Object detection with OpenCV

An interesting usage of the classification algorithms is object detection in live images. In this tutorial we will demonstrate how to detect a simple object using the open source library OpenCV. After a short description of OpenCV, we will see the steps needed to produce a model file using the OpenCV Cascade Classifier. This step is also called training the model. The model file will be then used to detect an object in the live images taken from a USB web camera. The model creation and the object detection code were tested on a laptop running Ubuntu 14.04 LTS. Basic C/C++ programming knowledge is required for this tutorial. A USB web camera is also needed.

OpenCV Library

OpenCV (Open Source Computer Vision) is a open source library which contains function for real-time image processing. The library is written in C++ and distributed under BSD License. It can be run under Linux, Windows, OS X, iOS, Android and has wrappers for Python, Java, C# and Ruby. It can be used for: object detection, motion detection, augmented reality, facial recognition or gesture recognition.

For this tutorial you will need to install OpenCV onUbuntu: OpenCV install on Ubuntu.

Trainig the model

In this section we will create a XML model file, which will be later used to detect the objects. For no special reason I choose a brezel (engl. prezel) as detection subject. The model creation process involves gathering a large amount of images containing the subject – we will call these positive images. We will also need a large a large amount of images not containing at all the searched object- negative images. Using the two folders and some description files we will run the OpenCV Cascade Classifier algorithm in order to produce the model file.

These are actual steps:

Step 1: I started by taking 201 JPG images of the brezel using a mobile phone.

Step 2: I installed GIMP for Ubuntu then I added David’s Batch Processor to the installation using the command:

sudo apt-get install gimp-plugin-registry

Step 3:  I created a batch process to convert the original images to grayscale and to resize them to 60 x 80 pixels. These 201 images will be used in the next steps to produce additional training images

Step 4: I took the grayscale images and applied the following transformations:upside-down (+201 images), blur 3.0 (+201 images), brightness +0.5 (+201 images),  brightness -0.5 (+201 images), contrast +0.5 (+201 images), contrast -0.5 (+201 images). I ended having 1407 positive images.

Step 5: Created a folder named positives and moved all the images to this folder.

Step 6: Created the positives.txt file. This is a descriptor file for the positives folder. Each line in this file contains the path to a positive image, the number of objects (brezels) in this image, the coordinates of the bounding rectangle containing the object.

find ./positives -iname "*.jpg" > positives.txt
sed -i 's/.jpg/.jpg 1 0 0 60 80/' positives.txt

Step 7: Downloaded the negative image files from this tutorial. Moved all negative images in a folder called negatives.

Step 8: Created the negatives.txt file. This is a descriptor file for the negatives folder.

find ./negatives -iname "*.jpg" > negatives.txt

Step 9: Prepare a training dataset from 1200 positive images,  using the following command:

opencv_createsamples -info positives.txt -num 1200 -w 60 -h 80 -vec training.vec

Step 10: Train the model using this commands

mkdir data
opencv_traincascade -data data -vec training.vec -bg negatives.txt -numPos 1000 -numNeg 1000 -numStages 10 -nsplits 2 -w 60 -h 80 -featureType LBP -minhitrate 0.999 -maxfalsealarm 0.5

Depending on the hardware the training can last a few hours. On my computer I got the model after 3:41 hours:

Training until now has taken 0 days 3 hours 41 minutes 55 seconds.

The resulting model file can be found found in the data folder and is named cascade.xml. I took this file and renamed it to brezel.xml. As a reference you can have a look at my trained model available in this archive brezel.zip.

More on training the model can be found here: http://docs.opencv.org/doc/user_guide/ug_traincascade.html

Using the trained model to detect objects

In this section we will see how to create a simple C++ Project, add the needed libraries to the project, add the model and do the actual object detection. As development tool I used Eclipse Luna (4.4.0) with CDT (C/C++ Development Tools).

Following, are the steps needed to setup and run the project:

Step1: Start Eclipse and create a new project: File > New >  Other > C++ Project.

Step2: Name the project BrezelDetectionProject, the press  Finish.

Step3: Right click on the project and choose to create a new folder named src.

Step 4: Right click the src folder and create a new .cpp file, BrezelDetector.cpp.

Step 5: Add the following code to the .cpp file.

#include <highgui.h>
#include <iostream>
#include <stdio.h>
#include <cv.h>

using namespace std;
using namespace cv;
using namespace std;

int main() {

	cvNamedWindow("Brezel detecting camera", 1);
	// Capture images from any camera connected to the system
	CvCapture* capture = cvCaptureFromCAM(CV_CAP_ANY);

	// Load the trained model
	CascadeClassifier brezelDetector;
	brezelDetector.load("src/brezel.xml");

	if (brezelDetector.empty()) {
		printf("Empty model.");
		return 0;
	}

	char key;
	while (true) {

		// Get a frame from the camera
		Mat frame = cvQueryFrame(capture);

		std::vector<Rect> brezels;

		// Detect brezels
		brezelDetector.detectMultiScale(frame, brezels, 1.1, 30,
				0 | CV_HAAR_SCALE_IMAGE, Size(200, 320));

		for (int i = 0; i < (int) brezels.size(); i++) {
			Point pt1(brezels[i].x, brezels[i].y);
			Point pt2(brezels[i].x + brezels[i].width,
					brezels[i].y + brezels[i].width);

			// Draw a rectangle around the detected brezel
			rectangle(frame, pt1, pt2, Scalar(0, 0, 255), 2);
			putText(frame, "Brezel", pt1, FONT_HERSHEY_PLAIN, 1.0,
					Scalar(255, 0, 0), 2.0);

		}

		// Show the transformed frame
		imshow("Brezel detecting camera", frame);

		// Read keystrokes, exit after ESC pressed
		key = cvWaitKey(10);
		if (char(key) == 27) {
			break;
		}
	}

	return 0;
}

Step 6: The project does not know where to find the OpenCV libraries. To fix that go to Project > Properties. In the C/C++ Build section, go to Settings, in the right pane select Tool Settings. In the GCC C++ Compiler section go to Includes and add to Include paths (-l) the folder where OpenCV is installed. For me the the path is /usr/include/opencv.

Step 7: Go to GCC C++ Linker section,  Library search path (-L) and add here the path to the OpenCV libaries. For me the path is /usr/lib/x86_64-linux-gnu/.

Under in Libraries(-l) add the following libraries one by one: opencv_core, opencv_objdetect, opencv_highgui.

Step 8: Add your xml model file to the src folder. The final project structure should look like this:

Step 9: Attach an USB web camera and run the program. The program can be stopped using the ESC key.

You can see results in the following short video:

Conclusion

We saw how we can use OpenCV to train a model for object detection and how to use this model to detect objects in live images. This is just a small use case of this computer vision library. Generally speaking, the computer vision has a wide application field and can handle areas like

• Controlling processes – industrial robots;
• Navigation – autonomous vehicle or mobile robots;
• Detecting events – for visual surveillance or people counting;
• Organizing information – indexing databases of images and image sequences;
• Modelling objects or environments – medical image analysis or topographical modelling;
• Interaction – input to a device for computer-human interaction
• Automatic inspection – in manufacturing applications.

Interested to find out the latest in the computer vision field? You may be interested also in the following article: https://technobium.com/computer-vision-next-generation/

References

http://docs.opencv.org/modules/objdetect/doc/cascade_classification.html

http://docs.opencv.org/doc/user_guide/ug_traincascade.html

http://docs.opencv.org/doc/tutorials/introduction/linux_eclipse/linux_eclipse.html

http://note.sonots.com/SciSoftware/haartraining.html