Merge branch 'master' of github.com:taddeus/licenseplates

.gitignore

@@ -1,4 +1,9 @@
 *.swp
+*~
 *.pyc
 *.pdf
-*~
+*.aux
+*.log
+*.synctex.gz
+*.toc
+*.out

docs/plan.tex

@@ -0,0 +1,128 @@
+\documentclass[a4paper]{article}
+
+\usepackage{hyperref}
+
+\title{Using local binary patterns to read license plates in photographs}
+\date{November 17th, 2011}
+
+% Paragraph indentation
+\setlength{\parindent}{0pt}
+\setlength{\parskip}{1ex plus 0.5ex minus 0.2ex}
+
+\begin{document}
+\maketitle
+
+\section*{Project members}
+Gijs van der Voort\\
+Richard Torenvliet\\
+Jayke Meijer\\
+Tadde\"us Kroes\\
+Fabi\'en Tesselaar
+
+\tableofcontents
+\setcounter{secnumdepth}{1}
+
+\section{Problem description}
+
+License plates are used for uniquely identifying motorized vehicles and are
+made to be read by humans from great distances and in all kinds of weather
+conditions.
+
+Reading license plates with a computer is much more difficult. Our dataset
+contains photographs of license plates from various angles and distances. This
+means that not only do we have to implement a method to read the actual
+characters, but also have to determine the location of the license plate and
+its transformation due to different angles.
+
+We will focus our research on reading the transformed characters on the
+license plate, of which we know where the letters are located. This is because
+Microsoft recently published a new and effective method to find the location of
+text in an image.
+
+In short our program must be able to do the following:
+
+\begin{enumerate}
+    \item Use perspective transformation to obtain an upfront view of license
+          plate.
+    \item Reduce noise where possible.
+    \item Extract each character using the location points in the info file.
+    \item Transform character to a normal form.
+    \item Create a local binary pattern histogram vector.
+    \item Match the found vector with a learning set.
+\end{enumerate}
+
+\section{Solution}
+
+Now that the problem is defined, the next step is stating a solution. This will
+come in a few steps as well.
+
+\subsection{Transformation}
+
+A simple perspective transformation will be sufficient to transform and resize
+the plate to a normalized format. The corner positions of license plates in the
+dataset are supplied together with the dataset.
+
+\subsection{Reducing noise}
+
+Small amounts of noise will probably be suppressed by usage of a Gaussian
+filter. A real problem occurs in very dirty license plates, where branches and
+dirt over a letter could radically change the local binary pattern. A question
+we can ask ourselves here, is whether we want to concentrate ourselves on these
+exceptional cases. By law, license plates have to be readable. Therefore, we
+will first direct our attention at getting a higher score in the 'regular' test
+set before addressing these cases. Considered the fact that the LBP algorithm
+divides a letter into a lot of cells, there is a good change that a great
+number of cells will still match the learning set, and thus still return the
+correct character as a best match. Therefore, we expect the algorithm to be
+very robust when dealing with noisy images.
+
+\subsection{Extracting a letter}
+
+Because we are already given the locations of the characters, we only need to
+transform those locations using the same perspective transformation used to
+create a front facing license plate. The next step is to transform the
+characters to a normalized manner. The size of the letter W is used as a
+standard to normalize the width of all the characters, because W is the widest
+character of the alphabet. We plan to also normalize the height of characters,
+the best manner for this is still to be determined.
+
+\begin{enumerate}
+    \item Crop the image in such a way that the character precisely fits the
+          image.
+    \item Scale the image to a standard height.
+    \item Extend the image on either the left or right side to a certain width.
+\end{enumerate}
+
+The resulting image will always have the same size, the character contained
+will always be of the same height, and the character will alway be positioned
+at either the left of right side of the image.
+
+\subsection{Local binary patterns}
+
+Once we have separate digits and characters, we intent to use Local Binary
+Patterns to determine what character or digit we are dealing with. Local Binary
+Patters are a way to classify a texture based on the distribution of edge
+directions in the image. Since letters on a license plate consist mainly of
+straight lines and simple curves, LBP should be suited to identify these.
+
+To our knowledge, LBP has yet not been used in this manner before. Therefore,
+it will be the first thing to implement, to see if it lives up to the
+expectations. When the proof of concept is there, it can be used in the final
+program.
+
+Important to note is that due to the normalization of characters before
+applying LBP. Therefore, no further normalization is needed on the histograms.
+
+Given the LBP of a character, a Support Vector Machine can be used to classify
+the character to a character in a learning set. The SVM uses
+
+\subsection{Matching the database}
+
+Given the LBP of a character, a Support Vector Machine can be used to classify
+the character to a character in a learning set. The SVM uses the collection of
+histograms of an image as a feature vector.  The SVM can be trained with a
+subsection of the given dataset called the ''Learning set''. Once trained, the
+entire classifier can be saved as a Pickle object\footnote{See
+\url{http://docs.python.org/library/pickle.html}} for later usage.
+
+\end{document}

src/GrayscaleImage.py

@@ -0,0 +1,59 @@
+from pylab import imshow, imread, show
+from scipy.misc import imresize
+
+class GrayscaleImage:
+
+    def __init__(self, image_path = None, data = None):
+        if image_path != None:
+            self.data = imread(image_path)
+            self.convert_to_grayscale()
+        elif data != None:
+            self.data = data
+    
+    def __iter__(self):
+        self.__i_x = -1
+        self.__i_y = 0
+        return self
+            
+    def next(self):
+        self.__i_x += 1
+        if self.__i_x  == self.width:
+            self.__i_x = 0
+            self.__i_y += 1
+        if self.__i_y == self.height:
+            raise StopIteration
+        
+        return  self.__i_y, self.__i_x, self[self.__i_y, self.__i_x]
+            
+    def __getitem__(self, position):
+        return self.data[position]
+        
+    def convert_to_grayscale(self):
+        self.data = self.data.sum(axis=2) / 3
+        
+    def crop(self, rectangle):
+        self.data = self.data[rectangle.y : rectangle.y + rectangle.height, 
+                              rectangle.x : rectangle.x + rectangle.width]
+                              
+    def show(self):
+        imshow(self.data, cmap="gray")
+        show()
+    
+    # size is of type float
+    def resize(self, size):
+        self.data = imresize(self.data, size)
+        
+    def get_shape(self):
+        return self.data.shape
+    shape = property(get_shape)
+    
+    def get_width(self):
+        return self.get_shape()[1]
+    width = property(get_width)
+        
+    def get_height(self):
+        return self.get_shape()[0]
+    height = property(get_height)
+        
+    def in_bounds(self, y, x):
+        return x >= 0 and x < self.width and y >= 0 and y < self.height

src/Histogram.py

@@ -0,0 +1,17 @@
+class Histogram:
+
+    def __init__(self, bins, min, max):
+        self.bins = [0] * bins
+        self.min = min
+        self.max = max
+        
+    def add(self, number):
+        bin_index = self.get_bin_index(number)
+        self.bins[bin_index] += 1
+        
+    def remove(self, number):
+        bin_index = self.get_bin_index(number)
+        self.bins[bin_index] -= 1
+        
+    def get_bin_index(self, number):
+        return (number - self.min) / ((self.max - self.min) / len(self.bins))

src/LBP.py

@@ -1,15 +1,7 @@
 #!/usr/bin/python
 import Image
 from numpy import array, zeros, byte
-from matplotlib.pyplot import imshow, show, axis
-
-CELL_SIZE = 16
-
-def domainIterator(image):
-    """Iterate over the pixels of an image."""
-    for y in xrange(image.shape[0]):
-        for x in xrange(image.shape[1]):
-            yield y, x
+from matplotlib.pyplot import imshow, subplot, show, axis
 
 # Divide the examined window to cells (e.g. 16x16 pixels for each cell).
 
@@ -28,26 +20,36 @@ def domainIterator(image):
 # Optionally normalize the histogram. Concatenate normalized histograms of all
 # cells. This gives the feature vector for the window.
 
-image = array(Image.open("../images/test.png").convert('L'))
+CELL_SIZE = 16
+
+def domain_iterator(shape):
+    """Iterate over the pixels of an image."""
+    for y in xrange(shape[0]):
+        for x in xrange(shape[1]):
+            yield y, x
+
+image = array(Image.open('../images/test.png').convert('L'))
 
 def in_image(y, x, F):
     """Check if given pixel coordinates are within the bounds of image F."""
     return 0 <= y < F.shape[0] and 0 <= x < F.shape[1]
 
-def compare(image):
+def features(image):
     """Compare each pixel to each of its eight neigheach pixel to each of its
     eight neighbours."""
     features = zeros(image.shape, dtype=byte)
 
     def cmp_pixels(y, x, p):
+        """Check if two pixels (y, x) and p are in the image, and if the value
+        at (y, x) is larger than the value at p."""
         return in_image(y, x, image) and image[y, x] > p
 
-    for y, x in domainIterator(features):
+    for y, x in domain_iterator(features.shape):
         p = image[y, x]
 
-        # Walk around the pixel in counter-clokwise order, shifting 1 but less
+        # Walk around the pixel in counter-clokwise order, shifting 1 bit less
         # at each neighbour starting at 7 in the top-left corner. This gives a
-        # 8-bitmap
+        # 8-bit feature number of a pixel
         features[y, x] = byte(cmp_pixels(y - 1, x - 1, p)) << 7 \
                          | byte(cmp_pixels(y - 1, x, p)) << 6 \
                          | byte(cmp_pixels(y - 1, x + 1, p)) << 5 \
@@ -59,7 +61,14 @@ def compare(image):
 
     return features
 
-#print compare(image)
+def feature_vectors(image):
+    """Create cell histograms of an image"""
+    F = features(image)
+
+V = feature_vectors(image)
+subplot(121)
 imshow(image, cmap='gray')
+subplot(122)
+imshow(V, cmap='gray')
 axis('off')
 show()

src/LetterCropper.py

@@ -0,0 +1,34 @@
+from copy import deepcopy
+from Rectangle import Rectangle
+
+class LetterCropper:
+
+    def __init__(self, image, threshold = 0.9):
+        self.source_image = image
+        self.threshold = threshold
+        
+    def get_cropped_letter(self):
+        self.determine_letter_bounds()
+        self.result_image = deepcopy(self.source_image)
+        self.result_image.crop(self.letter_bounds)
+        return self.result_image
+
+    def determine_letter_bounds(self):
+        min_x = self.source_image.width
+        max_x = 0
+        min_y = self.source_image.height
+        max_y = 0
+
+        for y, x, value in self.source_image:
+            if value < self.threshold:
+                if x < min_x: min_x = x
+                if y < min_y: min_y = y
+                if x > max_x: max_x = x
+                if y > max_y: max_y = y
+        
+        self.letter_bounds = Rectangle(
+            min_x, 
+            min_y, 
+            max_x - min_x ,
+            max_y - min_y
+        )

src/LetterCropperTest.py

@@ -0,0 +1,10 @@
+from LetterCropper import LetterCropper
+from GrayscaleImage import GrayscaleImage
+
+image = GrayscaleImage("../images/test.png")
+
+cropper = LetterCropper(image)
+
+cropped_letter = cropper.get_cropped_letter()
+
+cropped_letter.show()

src/LocalBinaryPatternizer.py

@@ -0,0 +1,53 @@
+from Histogram import Histogram
+from numpy import zeros, byte
+from math import ceil
+
+class LocalBinaryPatternizer:
+        
+    def __init__(self, image, cell_size=16):
+        self.cell_size = cell_size
+        self.image = image
+        self.setup_histograms()
+
+        
+    def setup_histograms(self):
+        cells_in_width = int(ceil(self.image.width / float(self.cell_size)))
+        cells_in_height = int(ceil(self.image.height / float(self.cell_size)))
+        self.features = []
+        for i in xrange(cells_in_height):
+            self.features.append([])
+            for j in xrange(cells_in_width):
+                self.features[i].append(Histogram(256,0,256))
+
+                
+    def create_features_vector(self):
+        ''' Walk around the pixels in clokwise order, shifting 1 bit less
+            at each neighbour starting at 7 in the top-left corner. This gives a
+            8-bit feature number of a pixel'''
+        for y, x, value in self.image:
+            
+            pattern = (self.is_pixel_darker(y - 1, x - 1, value) << 7) \
+                    | (self.is_pixel_darker(y - 1, x    , value) << 6) \
+                    | (self.is_pixel_darker(y - 1, x + 1, value) << 5) \
+                    | (self.is_pixel_darker(y    , x + 1, value) << 4) \
+                    | (self.is_pixel_darker(y + 1, x + 1, value) << 3) \
+                    | (self.is_pixel_darker(y + 1, x    , value) << 2) \
+                    | (self.is_pixel_darker(y + 1, x - 1, value) << 1) \
+                    | (self.is_pixel_darker(y    , x - 1, value) << 0)
+                    
+            cy, cx = self.get_cell_index(y, x)
+            self.features[cy][cx].add(pattern)
+
+        return self.get_features_as_array()
+    
+    
+    def is_pixel_darker(self, y, x, value):
+        return self.image.in_bounds(y, x) and self.image[y, x] > value
+        
+        
+    def get_cell_index(self, y, x):
+        return (y / self.cell_size, x / self.cell_size)
+        
+        
+    def get_features_as_array(self):
+        return [item for sublist in self.features for item in sublist]

src/LocalBinaryPatternizerTest.py

@@ -0,0 +1,12 @@
+from GrayscaleImage import GrayscaleImage
+from LocalBinaryPatternizer import LocalBinaryPatternizer
+from LetterCropper import LetterCropper
+from matplotlib.pyplot import imshow, subplot, show, axis, bar
+from numpy import arange
+
+image = GrayscaleImage("../images/test.png")
+
+lbp = LocalBinaryPatternizer(image)
+histograms = lbp.create_features_vector()
+
+print histograms

src/NormalizedImage.py

@@ -0,0 +1,17 @@
+from copy import deepcopy
+class NormalizedImage:
+    
+    DEFAULT_SIZE = 100.0
+    
+    def __init__(self, image, size=DEFAULT_SIZE):
+        self.source_image = image
+        self.size = size
+
+    def add_padding(self):
+        pass
+
+    # normalize img
+    def get_normalized_letter(self):
+        self.result_image = deepcopy(self.source_image)
+        self.result_image.resize(self.size / self.source_image.height)
+        return self.result_image

src/Rectangle.py

@@ -0,0 +1,7 @@
+class Rectangle:
+
+    def __init__(self, x, y, width, height):
+        self.x = x;
+        self.y = y;
+        self.width = width;
+        self.height = height;

src/combined_test.py

@@ -0,0 +1,51 @@
+from GrayscaleImage import GrayscaleImage
+from LocalBinaryPatternizer import LocalBinaryPatternizer
+from LetterCropper import LetterCropper
+from matplotlib.pyplot import imshow, subplot, show, axis
+from NormalizedImage import NormalizedImage
+
+# Comment added by Richard Torenvliet
+# Steps in this test files are
+# 1. crop image 
+# 2. resize to default hight (in future also to width)
+# 3. preform LBP
+# 4. construct feature vector
+# 5. plot
+
+# Image is now an instance of class GrayscaleImage
+# GrayscaleImage has functions like resize, crop etc.
+image = GrayscaleImage("../images/test.png")
+
+# Crops image; param threshold is optional: LetterCropper(image, threshold=0.9)
+# image: GrayscaleImage, threshold: float
+cropper = LetterCropper(image, 0.9)
+cropped_letter = cropper.get_cropped_letter()
+
+# Show difference in shape
+print cropped_letter.shape
+
+# Resizes image; param size is optional: NormalizedImage(image, size=DEFAULT)
+# image: GrayscaleImage, size: float
+norm = NormalizedImage(cropped_letter)
+resized = norm.get_normalized_letter()
+
+# Difference is noticable
+print resized.shape
+
+lbp = LocalBinaryPatternizer(resized)
+feature_vector = lbp.create_features_vector()
+feature_vector /= 255 # Prepare for displaying -> 0 - 255 -> 0 - 1
+        
+subplot(141)
+imshow(image.data, cmap='gray')
+
+subplot(142)
+imshow(cropped_letter.data, cmap='gray')
+
+subplot(143)
+imshow(resized.data, cmap='gray')
+subplot(144)
+imshow(feature_vector, cmap='gray')
+
+axis('off')
+show()

src/histogram_test.py

@@ -0,0 +1,25 @@
+from Histogram import Histogram
+
+his = Histogram(10, 10, 110)
+his.add(10)
+his.add(19)
+his.add(20)
+his.add(29)
+his.add(30)
+his.add(39)
+his.add(40)
+his.add(49)
+his.add(50)
+his.add(59)
+his.add(60)
+his.add(69)
+his.add(70)
+his.add(79)
+his.add(80)
+his.add(89)
+his.add(90)
+his.add(99)
+his.add(100)
+his.add(109)
+
+print his.bins