Realtime Object Detection & Tracking

In this project I designed an algorithm for detection and tracking hexagonal nuts on a conveyor belt. Python language with OpenCV image processing library were used to implement the algorithm. Template matching technique was used. 

Let me explain the algorithm. First, we have to include following libraries.

import cv2 as cv
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

Then I implemented following function in order to do image preprocessing. First, image thresholding with OTSU is used to get a binary image, then morphological closing operation was done to remove small holes that may be appeared in the image. After that connected component analysis is carried out in order to extract the contours of hexagonal nuts in the image.

def preprocess(im):
    """ Thresholding, closing, and connected component analysis lumped
    """ 

    th, img = cv.threshold(im,0,255,cv.THRESH_BINARY_INV+cv.THRESH_OTSU)
    kernel = np.ones((3,3), dtype = 'uint8')
    closing = cv.morphologyEx(img, cv.MORPH_CLOSE, kernel)
    retval, labels, stats, centroids = cv.connectedComponentsWithStats(closing)

    return retval, labels, stats, centroids 

The following function finds whether a detected nut is a new one or not. For this, two frames are used. The absolute difference between two nuts is measured and compared with a threshold value, to achieve this. The threshold value needs to be found experimentally by considering frame rate, and speed of the conveyor belt.

def is_new(a, b, delta, i):
    
    difference = np.absolute(a - b)
    return (difference[:,i] > delta[i]).all()

The following function is used to get the index of a nut in the previous frame, if that particular nut is not a new one. This is required for tracking the nuts.

def prev_index(a, b, delta, i):
    # Returns the index of the apppearance of the object in the previous frame.
    
    index = -1

    difference = np.absolute(a-b)
    if is_new(a, b, delta, i): return index
    return np.where(difference[:,i]&lt=delta[i])[0]

The following function is used to preprocess the template image. It's similar to preprocessing of images done before.

def preprocess_template(template_img):
    th_t, img_t = cv.threshold(template_img,0,255,cv.THRESH_BINARY_INV+cv.THRESH_OTSU)

    kernel = np.ones((3,3), dtype = 'uint8')
    closing_t = cv.morphologyEx(img_t, cv.MORPH_CLOSE, kernel)

    contours_t, hierarchy_t = cv.findContours(closing_t, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE)
    return contours_t

The main program is shown below.

cap = cv.VideoCapture(r'conveyor_with_rotation.mp4')
template_img = cv.imread(r'template.png', cv.IMREAD_GRAYSCALE)

contours_t = preprocess_template(template_img)

object_prev_frame = [[0, 0, 0, 0]]
total_nuts = 0
f = 0                               # Number of frames

size = []                           # Size of a frame image
Frames = []                         # Contour plots of Frames

while cap.isOpened(): 
    f += 1
    ret, frame = cap.read()
    
    if not ret:
        print("Can't receive frame (stream end?). Exiting ...")
        break

    frame_gray = cv.cvtColor(frame, cv.COLOR_BGR2GRAY)           # Converting frames to grayscale
    print_offset = 0

    # 1. Object Detection

    retval, labels, stats, centroids = preprocess(frame_gray)
    belt = ((labels >= 1)*255).astype(np.uint8)
    contours, hierarchy = cv.findContours(belt, cv.RETR_EXTERNAL, cv.CHAIN_APPROX_SIMPLE)

    object_current_frame = []
    count = 0
    for cont in contours:
        if (cv.matchShapes(contours_t[0], cont, cv.CONTOURS_MATCH_I1, 0.0) < 0.001):
            count += 1
            M  = cv.moments(cont)
            cx, cy = int(M['m10']/M['m00']), int(M['m01']/M['m00'])
            ca = cv.contourArea(cont)
            object_current_frame.append(np.array([cx, cy, ca, count]))

    # 2. Object Tracking

    im_contours_belt = np.zeros((belt.shape[0],belt.shape[1],3), np.uint8)
    size = im_contours_belt.shape
    conts = cv.drawContours(im_contours_belt, contours, -1, (0,255,0), 2).astype('uint8')

    delta = np.array([15])
    i = np.array([0])
    for nut in object_current_frame:
        if (is_new(object_prev_frame, nut, delta, i)):
            total_nuts += 1
            nut[-1] = total_nuts
        else:
            nut[-1] = object_prev_frame[int(prev_index(object_prev_frame, nut, delta, i))][-1]

        cv.putText(conts,str(int(nut[-1])), (int(nut[0]-20),int(nut[1])+10), cv.FONT_HERSHEY_PLAIN, 4 ,(255,0,255), 3, cv.LINE_AA)
        cv.putText(conts, 'Object {} : {}, {}, {}'.format(nut[-1], nut[0], nut[1], nut[2]), (50, 650 + print_offset*40),cv.FONT_HERSHEY_PLAIN, 2, (255,0,255), 2, cv.LINE_AA)
    
        print_offset += 1
    
    object_prev_frame = object_current_frame

    cv.putText(conts, ('Frame: ' + str(f)), (50,80), cv.FONT_HERSHEY_PLAIN, 2, (255,0,255), 2, cv.LINE_AA)

    Frames.append(conts)
    
    cv.imshow("Contours", conts)
    
    if cv.waitKey(1) == ord('q'):  
        break

cap.release() 
cv.destroyAllWindows() 

You can find the code and required files from this link.

The algorithm was tested using a video clip of a conveyor belt as follows.