\documentclass[10pt,a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{amsmath,hyperref,graphicx,booktabs,float}

% Paragraph indentation
\setlength{\parindent}{0pt}
\setlength{\parskip}{1ex plus 0.5ex minus 0.2ex}

\title{Image processing 3: Finding Waldo}
\author{Sander van Veen \& Tadde\"us Kroes \\ 6167969 \& 6054129}

\begin{document}

\maketitle

\section{Histogram intersection}

\subsection{Store a color histogram}

In order to store a color histogram in memory, a 3D array is mostly used. Thus,
one dimension for each base color (for the RGB color model, that is red, green
and blue).

\subsection{Color models}

There are two main categories in color models: absolute and non-absolute.
An absolute color model is unambiguous, which means that external factors do no
influence the color representation. CIEXYZ and sRGB are two examples of an absolute
color model. An example of non-absolute color model is RGB.

% Redo the previous exercise but this time using another color model. You can
% choose a color model of your own liking. Surf the web to find a model and
% describe your model in your report. What are the conversion formula’s and why
% was this color model invented?

It is useless to convert an absolute color model to a non-absolute color model,
since it will only lose information without any gain. On the other hand,
conversion between two non-absolute color models can be useful. We chose to
convert the RGB color model the HSV color model.

% Does your model improve the results compared with the RGB model?

In most cases, the HSV color model gives a better result. When comparing
picture 8 with picture 1, 2 and 3, the histogram intersections of the RGB color
model are resp. 52\%, 66\% and 66\%. With the HSV color model, these
intersections are resp. 28\%, 61\% and 45\%.

\begin{figure}[H]
\center
\includegraphics[height=3cm]{../database/8.jpg}
\caption{Picture 8 from the given database.}
\end{figure}

If we visually compare picture 8 with the pictures 1, 2 and 3, we can clearly
see that picture 2 shows more similarities (similar yellow color, similar size
and ratio of the object) than the other two pictures.

\begin{figure}[H]
\center
\includegraphics[height=3cm]{1-2-3.jpg}
\caption{From left to right picture 1, 2 and 3 from the given database.}
\end{figure}

However, in the RGB model, there is no clear distinction in the histogram
intersections (52\%, 66\% and 66\%), while the HSV model gives picture 1 and 3
a much lower intersection percentage (picture 2 has 61\%,  picture 1 has 28\%
and picture 3 has 45\%). 

So, using the HSV color model does improve the results.

\subsection{Histogram intersection tables}

\begin{table}[H]
\hspace{-1.5in}
\begin{tabular}{r|rrrrrrrrrrrrrrrrrrrr}
\toprule

   &   1 &   2 &   3 &   4 &   5 &   6 &   7 &   8 &   9 &  10 &  11 &  12 &  13 &  14 &  15 &  16 &  17 &  18 &  19 &  20 \\ 

\midrule

1  & 100 &  57 &  56 &  41 &  40 &  45 &  28 &  52 &  51 &  43 &  45 &  54 &  35 &  54 &  59 &  36 &  51 &  47 &  38 &  44 \\
2  &  57 & 100 &  60 &  56 &  47 &  53 &  36 &  66 &  59 &  39 &  41 &  63 &  39 &  65 &  60 &  51 &  59 &  61 &  36 &  54 \\
3  &  56 &  60 & 100 &  46 &  49 &  63 &  24 &  66 &  60 &  58 &  40 &  75 &  48 &  52 &  57 &  35 &  53 &  54 &  45 &  48 \\
4  &  41 &  56 &  46 & 100 &  47 &  47 &  35 &  50 &  48 &  36 &  39 &  49 &  39 &  42 &  48 &  34 &  40 &  63 &  38 &  35 \\
5  &  40 &  47 &  49 &  47 & 100 &  50 &  27 &  53 &  46 &  40 &  40 &  48 &  38 &  45 &  41 &  34 &  38 &  50 &  38 &  36 \\
6  &  45 &  53 &  63 &  47 &  50 & 100 &  25 &  60 &  53 &  43 &  38 &  59 &  49 &  46 &  47 &  35 &  45 &  55 &  46 &  42 \\
7  &  28 &  36 &  24 &  35 &  27 &  25 & 100 &  29 &  28 &  14 &  28 &  25 &  14 &  28 &  28 &  31 &  28 &  35 &  24 &  28 \\
8  &  52 &  66 &  66 &  50 &  53 &  60 &  29 & 100 &  51 &  44 &  40 &  67 &  44 &  57 &  54 &  43 &  52 &  57 &  44 &  45 \\
9  &  51 &  59 &  60 &  48 &  46 &  53 &  28 &  51 & 100 &  45 &  41 &  56 &  41 &  51 &  49 &  39 &  53 &  50 &  37 &  48 \\
10 &  43 &  39 &  58 &  36 &  40 &  43 &  14 &  44 &  45 & 100 &  29 &  48 &  52 &  35 &  39 &  20 &  35 &  40 &  53 &  27 \\
11 &  45 &  41 &  40 &  39 &  40 &  38 &  28 &  40 &  41 &  29 & 100 &  39 &  25 &  40 &  40 &  31 &  39 &  40 &  26 &  43 \\
12 &  54 &  63 &  75 &  49 &  48 &  59 &  25 &  67 &  56 &  48 &  39 & 100 &  41 &  57 &  62 &  41 &  55 &  56 &  36 &  51 \\
13 &  35 &  39 &  48 &  39 &  38 &  49 &  14 &  44 &  41 &  52 &  25 &  41 & 100 &  31 &  35 &  20 &  36 &  43 &  65 &  25 \\
14 &  54 &  65 &  52 &  42 &  45 &  46 &  28 &  57 &  51 &  35 &  40 &  57 &  31 & 100 &  55 &  51 &  57 &  48 &  31 &  54 \\
15 &  59 &  60 &  57 &  48 &  41 &  47 &  28 &  54 &  49 &  39 &  40 &  62 &  35 &  55 & 100 &  39 &  49 &  52 &  34 &  45 \\
16 &  36 &  51 &  35 &  34 &  34 &  35 &  31 &  43 &  39 &  20 &  31 &  41 &  20 &  51 &  39 & 100 &  57 &  41 &  19 &  45 \\
17 &  51 &  59 &  53 &  40 &  38 &  45 &  28 &  52 &  53 &  35 &  39 &  55 &  36 &  57 &  49 &  57 & 100 &  47 &  32 &  57 \\
18 &  47 &  61 &  54 &  63 &  50 &  55 &  35 &  57 &  50 &  40 &  40 &  56 &  43 &  48 &  52 &  41 &  47 & 100 &  40 &  40 \\
19 &  38 &  36 &  45 &  38 &  38 &  46 &  24 &  44 &  37 &  53 &  26 &  36 &  65 &  31 &  34 &  19 &  32 &  40 & 100 &  25 \\
20 &  44 &  54 &  48 &  35 &  36 &  42 &  28 &  45 &  48 &  27 &  43 &  51 &  25 &  54 &  45 &  45 &  57 &  40 &  25 & 100 \\

\bottomrule
\end{tabular}
\caption{Histogram intersection of 20 pictures with the RGB color model.}
\end{table}

\begin{table}[H]
\hspace{-1.5in}
\begin{tabular}{r|rrrrrrrrrrrrrrrrrrrr}
\toprule

   &   1 &   2 &   3 &   4 &   5 &   6 &   7 &   8 &   9 &  10 &  11 &  12 &  13 &  14 &  15 &  16 &  17 &  18 &  19 &  20 \\ 

\midrule

1  & 100 &  36 &  50 &  22 &  25 &  35 &  20 &  28 &  38 &  29 &  32 &  37 &  17 &  38 &  47 &  28 &  39 &  34 &  16 &  30 \\
2  &  36 & 100 &  48 &  49 &  43 &  48 &  34 &  61 &  52 &  12 &  37 &  55 &  12 &  51 &  50 &  47 &  52 &  57 &  11 &  49 \\
3  &  50 &  48 & 100 &  30 &  33 &  52 &  18 &  45 &  49 &  37 &  32 &  58 &  22 &  38 &  48 &  30 &  51 &  42 &  19 &  45 \\
4  &  22 &  49 &  30 & 100 &  41 &  34 &  33 &  40 &  36 &   5 &  32 &  36 &   7 &  30 &  37 &  26 &  27 &  54 &  10 &  28 \\
5  &  25 &  43 &  33 &  41 & 100 &  42 &  25 &  43 &  38 &   4 &  36 &  39 &   4 &  37 &  35 &  31 &  30 &  40 &   5 &  34 \\
6  &  35 &  48 &  52 &  34 &  42 & 100 &  22 &  48 &  42 &  13 &  34 &  53 &  12 &  42 &  43 &  33 &  41 &  46 &  10 &  40 \\
7  &  20 &  34 &  18 &  33 &  25 &  22 & 100 &  25 &  22 &   4 &  25 &  22 &   4 &  25 &  24 &  27 &  21 &  32 &  15 &  24 \\
8  &  28 &  61 &  45 &  40 &  43 &  48 &  25 & 100 &  41 &   4 &  33 &  56 &   5 &  41 &  40 &  36 &  44 &  47 &   6 &  43 \\
9  &  38 &  52 &  49 &  36 &  38 &  42 &  22 &  41 & 100 &  17 &  35 &  46 &  12 &  44 &  40 &  35 &  48 &  35 &   7 &  43 \\
10 &  29 &  12 &  37 &   5 &   4 &  13 &   4 &   4 &  17 & 100 &   6 &  17 &  45 &  14 &  20 &   5 &  18 &  11 &  42 &   7 \\
11 &  32 &  37 &  32 &  32 &  36 &  34 &  25 &  33 &  35 &   6 & 100 &  35 &   4 &  33 &  34 &  27 &  31 &  35 &   5 &  37 \\
12 &  37 &  55 &  58 &  36 &  39 &  53 &  22 &  56 &  46 &  17 &  35 & 100 &  12 &  48 &  55 &  37 &  51 &  46 &   9 &  48 \\
13 &  17 &  12 &  22 &   7 &   4 &  12 &   4 &   5 &  12 &  45 &   4 &  12 & 100 &  12 &  12 &   4 &  16 &  12 &  58 &   5 \\
14 &  38 &  51 &  38 &  30 &  37 &  42 &  25 &  41 &  44 &  14 &  33 &  48 &  12 & 100 &  46 &  45 &  44 &  40 &  13 &  40 \\
15 &  47 &  50 &  48 &  37 &  35 &  43 &  24 &  40 &  40 &  20 &  34 &  55 &  12 &  46 & 100 &  35 &  44 &  47 &  14 &  38 \\
16 &  28 &  47 &  30 &  26 &  31 &  33 &  27 &  36 &  35 &   5 &  27 &  37 &   4 &  45 &  35 & 100 &  49 &  35 &   5 &  41 \\
17 &  39 &  52 &  51 &  27 &  30 &  41 &  21 &  44 &  48 &  18 &  31 &  51 &  16 &  44 &  44 &  49 & 100 &  39 &  13 &  52 \\
18 &  34 &  57 &  42 &  54 &  40 &  46 &  32 &  47 &  35 &  11 &  35 &  46 &  12 &  40 &  47 &  35 &  39 & 100 &  14 &  36 \\
19 &  16 &  11 &  19 &  10 &   5 &  10 &  15 &   6 &   7 &  42 &   5 &   9 &  58 &  13 &  14 &   5 &  13 &  14 & 100 &   5 \\
20 &  30 &  49 &  45 &  28 &  34 &  40 &  24 &  43 &  43 &   7 &  37 &  48 &   5 &  40 &  38 &  41 &  52 &  36 &   5 & 100 \\

\bottomrule
\end{tabular}
\caption{Histogram intersection of 20 pictures with the HSV color model.}
\end{table}

\section{Histogram Backprojection}

\subsection{Finding Waldo}

The assignment is to find Waldo (\emph{waldo.tiff}) in a large image containing
Waldo and many other characters (\emph{waldo\_env.tiff}) using Histogram
Backprojection.

The idea of Histogram Backprojection is explained in the paper by Swain and
Ballard, so we will not explain it here. The algorithm as described in the
paper is as follows:

Given histograms $M$ (model, Waldo) and $I$ (environment), create the back
projection $b$ in the following steps:

\begin{enumerate}
    \item for each histogram bin $j$ do $R_j := frac{M_j}{I_j}$
    \item for each $x, y$ do $b_{x,y} := min(R_{h(c_{x,y})}, 1)$
    \item $b := D^r * b$
    \item $(x_t, y_t) := loc(max_{x,y}, b_{x,y})$
\end{enumerate}

However, the assignment tells us to only implement steps 1-3.

\subsection{Mask}

The algorithm is implemented in \emph{back\_projection.py}. First, a mask is
created to ignore the white background in the \emph{waldo.tiff}. This is needed
because the white color is not part of Waldo himself. In fact, Waldo in the
\emph{waldo\_env.tiff} has a yellowish background behind him. The usage of a
mask is simple: if the mask value of a pixel is \texttt{False}, the pixel's
color is discarded in the creation of the color histogram. The mask for Waldo
is created by discarding all pixels with RGB color (255, 255, 255), which has
the following result:

\begin{figure}[H]
    \label{fig:mask}
    \includegraphics{mask.png}
    \caption{The mask used to ignore the white background in \emph{waldo.tiff}.}
\end{figure}

\subsection{Basic algorithm}

The algorithm is implemented in the function \texttt{hbp}: First, $M$ and $I$
are created using \texttt{colHist}. $R$ is then created by division (step 1).
Afterwards, the back projection is created using the loop in step 2.  Finally,
a convolution function is used to soften the back projection. To create the
convolution, we have used the function \texttt{scipy.ndimage.correlate} with a
circular weight mask.

The following result is generated with 64 bins in each color dimension and a
convolution radius of 15 pixels:

\begin{figure}[H]
    \hspace{-4cm}
    \includegraphics[width=20cm]{found_waldo.png}
    \caption{\emph{found\_waldo.tiff}: Back projection of Waldo in the larger
    image, the red spot is Waldo's location.}
\end{figure}

This result is created in roughly 27 seconds on a laptop from 2009, with an
Intel dual core and 4 GB RAM.

\subsection{Experimental cluster detection using K-means}

In addition to the assignment, we thought it would be interesting to be able to
draw a rectangle around possible occurences of Waldo in the image. Therefore,
we created the function \texttt{find\_peaks}, which returns a number of
possible locations averages.  To do this, we eliminate all pixels in the back
projection which are below a certain threshold (which is given as e number from
0 to 1, and then multiplied by the maximum value in the projection). This
leaves us with a number of pixels clusters, of which we can calculate the
centers using K-means
algorithm\footnote{\url{http://en.wikipedia.org/wiki/K-means\_clustering}}
(function \texttt{scipy.cluster.vq.kmeans}). To find the initial estimators, we
find all pixels that are further away from all other estimators than the
diagonal of the model image (\emph{waldo.png}). Around each center pixel, a
rectangle with the size of the model image is drawn over the larger image. With
32 bins in each color dimension, a threshold of 0.25 and a convolution radius
of 10 pixels, the result is as follows:

\begin{figure}[H]
    \hspace{-4cm}
    \includegraphics[width=20cm]{k-means.png}
    \caption{\emph{k-means.png}: Multiple possible locations using a low
    threshold and convolution radius.}
\end{figure}

\end{document}

\end{document}
