Vehicle Detection Project

The main objectives of the project are:

• Perform a Histogram of Oriented Gradients (HOG) feature extraction on a labeled training set of images and train a classifier Linear SVM classifier
• Apply a color transform and append binned color features, as well as histograms of color, to the HOG feature vector.
• Normalize the features and randomize a selection for training and testing.
• Implement a sliding-window technique and use your trained classifier to search for vehicles in images.
• Run the pipeline on a video stream (starting with the test_video.mp4 and later implementing on full project_video.mp4) and create a heat map of recurring detections frame by frame to reject outliers and follow detected vehicles.
• Estimate a bounding box for vehicles detected.

For a more robust detector, I downloaded the Udacity labelled dataset (annotated by Autti) and pre-prossed the data to have more labelled dataset for the vehicle and non-vehicle classes and converted them to png format. I also down-scaled the images in accordance with the other training dataset provided in the exercise.

def loaddata (csvfile):
    vehicle_class = ['car', 'truck']
    with open(csvfile) as inputFile:
        reader = csv.reader(inputFile, delimiter=' ')
        for line in reader:  
            img = cv2.imread('data/'+line[0])    
            x1 = line[1]
            y1 = line[2]
            x2 = line[3]
            y2 = line[4]
            cropped_img = img[int(y1):int(y2), int(x1):int(x2)]
            scaled_img = cv2.resize(cropped_img, (64, 64), interpolation=cv2.INTER_NEAREST)
            if (line[6] in vehicle_class):
		cv2.imwrite('train_data/vehicle/'+line[0].split(".")[0]+'.png', img, [int(cv2.IMWRITE_PNG_COMPRESSION), 4])
            else:
                cv2.imwrite('train_data/novehicle/'+line[0].split(".")[0]+'.png', img, [int(cv2.IMWRITE_PNG_COMPRESSION), 4])

So, my training data consisted of 21645 vehicle class data and 15488 non-vehicle class data set.

The code for this step is contained in Step 4: Histogram of Oriented Gradients (HOG) of the IPython notebook.

I started by reading in all the vehicle and non-vehicle images. Here is an example of one of each of the vehicle and non-vehicle classes:

I then explored different color spaces and different skimage.hog() parameters (orientations, pixels_per_cell, and cells_per_block). I grabbed random images from each of the two classes and displayed them to get a feel for what the skimage.hog() output looks like.

I explored the different colour spaces and chose YUV as this would give optimal gradients.

Here is an example using the HOG parameters of orientations=9, pixels_per_cell=(8, 8) and cells_per_block=(2, 2):

I tried various combinations of parameters and settled down with the following parameters.

colorspace = 'YUV'
orientations = 11
pix_per_cell = 16
cells_per_block=(2, 2)
hog_channel = 'ALL' 
spatial_size = (16, 16)
hist_bins = 16

With this configuration, I achieve an accuracy of 0.9828 and it requires 0.0014 seconds to predict 10 labels with classifier. I also serialized the classifier in a pickled file, so that it could be used later on. The code is available under Step 8: Training a classifier.

I trained a linear SVM using a combination of histogram count, spatial binning and HOG features. The helper function can be found under extract_features in Step 4.

First, I used the implemented a version of sliding window function to draw the boxes in half of the image. This was to verify where the search regions would be. The results can be seen below.

I considered only half of the image in Y-direction as being a part of the road. I divided the scale of the search window as follows (height = image.shape[0]//2, start = image.shape[0]//2):

• Initially towards the middle of the image, I used smaller scales, as near the horizon the sizes of the cars would be small. This search window continued as follows -

ystart = start
ystop = start + height//3 

• For the later search windows, I increased the scale and the search boundary as well, as the objects nearer could take up significant portion of the image shape and can occulude the horizon. So my last search window looked as -

ystart = start
ystop = start + height 

Ultimately I searched using YCrCb, 3-channel HOG features plus and histograms of color in the feature vector, which provided a nice result. Here are some example of the sliding window search with these features.

Here's a link to my video result

I recorded the positions of positive detections in each frame of the video. From the positive detections I created a heatmap and then thresholded that map to identify vehicle positions. I then used scipy.ndimage.measurements.label() to identify individual blobs in the heatmap. I then assumed each blob corresponded to a vehicle. I constructed bounding boxes to cover the area of each blob detected.

Here's an example result showing the heatmap from a series of frames of video, the result of scipy.ndimage.measurements.label() and the bounding boxes then overlaid on the last frame of video:

• In the pipeline, more augmentation of the training data can be done.
• Also, the distant cars are not detected by the pipeline.
• Moreover, oncoming cars cannot be tracked as well.
• Sometimes, in the pipeline, we are not able to track the car. This may be beacuse of the accuracy of the classifier.

To track more intelligently, a state based monitoring with Kalman filters can be applied. Also, it would be interesting to solve the problem with Neural network approach.