Accurate Eye Center Location through Invariant Isocentric Patterns

Below, is a Python implementation of the paper Accurate Eye Center Location through Invariant Isocentric Patterns. Most of the comments in the code are copied directly from the paper.

Download: https://gist.github.com/dimitrs/d667ccb433d5f2baa8df

Read More

Mixing OpenCV and SciKit-image

I saw a Mathematica post that described how to detect and flatten a label on a jar. My goal here is to do something similar in Python. I am familiar with OpenCV-Python which is what I have always used for my computer vision projects, but it occurred to me that there is no reason why I should only use OpenCV-Python. I could use both OpenCV-Python and SciKit-image at the same time. After all, they are both based on Numpy. For this project I start with OpenCV-Python and then switch to SciKit-image for the last step.

jar.png

#Import both skimage and cv
from skimage import transform as tf
from skimage import io
import cv2

import numpy as np
from scipy import optimize
import matplotlib.pyplot as plt

# Could use either skimage or cv to read the image
# img = io.imread('jar.png')   
img = cv2.imread('jar.png')
gray_image = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
ret, thresh = cv2.threshold(gray_image,127,255,cv2.THRESH_BINARY)
edges = cv2.Canny(thresh ,100, 200)

# Find largest contour (should be the label)
contours,hierarchy = cv2.findContours(edges, 0, 1)
areas = [cv2.contourArea(c) for c in contours]
max_index = np.argmax(areas)
cnt=contours[max_index]

# Create a mask of the label
mask = np.zeros(img.shape,np.uint8)
cv2.drawContours(mask, [cnt],0,255,-1)

Mask of the label

# Find the 4 borders
scale = 1
delta = 0
ddepth = cv2.CV_8U
borderType=cv2.BORDER_DEFAULT
left=cv2.Sobel(mask,ddepth,1,0,ksize=1,scale=1,delta=0,borderType)
right=cv2.Sobel(mask,ddepth,1,0,ksize=1,scale=-1,delta=0, borderType)
top=cv2.Sobel(mask,ddepth,0,1,ksize=1,scale=1,delta=0,borderType)
bottom=cv2.Sobel(mask,ddepth,0,1,ksize=1,scale=-1,delta=0,borderType)

Left & Right borders after Sobel

# Remove noise from borders
kernel = np.ones((2,2),np.uint8)
left_border = cv2.erode(left,kernel,iterations = 1)
right_border = cv2.erode(right,kernel,iterations = 1)
top_border = cv2.erode(top,kernel,iterations = 1)
bottom_border = cv2.erode(bottom,kernel,iterations = 1)

Left & Right borders after calling erode

# Find coeficients c1,c2, ... ,c7,c8 by minimizing the error function. 
# Points on the left border should be mapped to (0,anything).
# Points on the right border should be mapped to (108,anything)
# Points on the top border should be mapped to (anything,0)
# Points on the bottom border should be mapped to (anything,70)
# Equations 1 and 2: 
#    c1 + c2*x + c3*y + c4*x*y, c5 + c6*y + c7*x + c8*x^2

sumOfSquares_y = '+'.join(["(c[0]+c[1]*%s+c[2]*%s+c[3]*%s*%s)**2" %
    (x,y,x,y) for y,x,z in np.transpose(np.nonzero(left_border)) ])
sumOfSquares_y += " + "
sumOfSquares_y += \
    '+'.join(["(-108+c[0]+c[1]*%s+c[2]*%s+c[3]*%s*%s)**2" % \
    (x,y,x,y) for y,x,z in np.transpose(np.nonzero(right_border)) ])
res_y = optimize.minimize(lambda c: eval(sumOfSquares_y),(0,0,0,0),method='SLSQP')

sumOfSquares_x = \
    '+'.join(["(-70+c[0]+c[1]*%s+c[2]*%s+c[3]*%s*%s)**2" % \
    (y,x,x,x) for y,x,z in np.transpose(np.nonzero(bottom_border))])
sumOfSquares_x += " + "
sumOfSquares_x += \
    '+'.join( [ "(c[0]+c[1]*%s+c[2]*%s+c[3]*%s*%s)**2" % \
    (y,x,x,x) for y,x,z in np.transpose(np.nonzero(top_border)) ] )
res_x = optimize.minimize(lambda c: eval(sumOfSquares_x),(0,0,0,0), method='SLSQP')

# Map the image using equatinos 1 and 2 (coeficients c1...c8 in res_x and res_y)
def map_x(res, cord):
    m = res[0]+res[1]*cord[1]+res[2]*cord[0]+res[3]*cord[1]*cord[0]
    return m
def map_y(res, cord):
    m = res[0]+res[1]*cord[0]+res[2]*cord[1]+res[3]*cord[1]*cord[1]
    return m

flattened = np.zeros(img.shape, img.dtype) 
for y,x,z in np.transpose(np.nonzero(mask)):
    new_y = map_y(res_x.x,[y,x]) 
    new_x = map_x(res_y.x,[y,x])
    flattened[float(new_y)][float(new_x)] = img[y][x]
# Crop the image 
flattened = flattened[0:70, 0:105]

Flattened Image

There is fair amount of distortion in the flattened image. Alternatively, use PiecewiseAffineTransform from SciKit-image:

# Use  skimage to transform the image
leftmost = tuple(cnt[cnt[:,:,0].argmin()][0])
rightmost = tuple(cnt[cnt[:,:,0].argmax()][0])
topmost = tuple(cnt[cnt[:,:,1].argmin()][0])
bottommost = tuple(cnt[cnt[:,:,1].argmax()][0])

dst = list()
src = list()
for y,x,z in np.transpose(np.nonzero(top_border)):
    dst.append([x,y])
    src.append([x,topmost[1]])
for y,x,z in np.transpose(np.nonzero(bottom_border)):
    dst.append([x,y])
    src.append([x,bottommost[1]])
for y,x,z in np.transpose(np.nonzero(left_border)):
    dst.append([x,y])
    src.append([leftmost[0],y])
for y,x,z in np.transpose(np.nonzero(right_border)):
    dst.append([x,y])
    src.append([rightmost[0],y])
src = np.array(src)
dst = np.array(dst)

tform3 = tf.PiecewiseAffineTransform()
tform3.estimate(src, dst)
warped = tf.warp(img, tform3, order=2)
warped = warped[85:170, 31:138]

Flattened label usgin skimage

Read More

How-to Install OpenCV on an Android VM

Installing an Android emulator on Ubuntu is actually quite easy. Below, are the steps I took to get OpenCV 2.4.5 working on a Android emulator installed on my Ubuntu 12.04 machine. Most guides I have read assume that you are going to be using Eclipse. I don't use Eclipse.

1) Install the JDK and ant. 

:~$ sudo apt-get install openjdk-6-jdk 
:~$ sudo apt-get install ant 
:~$ export JAVA_HOME=/usr/lib/jvm/java-6-openjdk 
:~$ export PATH=$PATH:/usr/lib/jvm/java-6-openjdk/bin 

Sanity check: type “javac” in your terminal to confirm JDK is available. 

2) Install the Android SDK. Download android-sdk_r22.3-linux.tgz and unpack it: 

:~$ cd android-sdk-linux 
:~$ tools/android update sdk –no-ui 
:~$ export PATH=$PATH:~android-sdk-linux/tools :~android-sdk-linux/platform-tools 

3) Install the Android emulator from Genymotion and add a virtual device. Once installed, use its UI to “Add” a virtual device: 

:~$ cd genymotion 
:~$ ./genymotion 

In order to enable the drag & drop of “apk” files from the host to the Android VM, enter the path to the android-sdk-linux directory in the Genymotion settings, on the ADB tab. 

Start the VM. 

4) Install OpenCV on the Android VM. 

Download OpenCV 2.4.5  and unpack it. 

Drag & drop OpenCV-2.4.5-android-sdk/apk/OpenCV_2.4.8_Manager_2.16_x86.apk into the VM. 

Sanity check: In the Android emulator click the “OpenCV Manager” to confirm that you see no error messages. 

5) Test an OpenCV sample application. Compile the 15-puzzle application, for example: 

:~$ cd OpenCV-2.4.8-android-sdk/samples/15-puzzle 
:~$ android update project --path . 
:~$ ant debug 
:~$ cd bin 

Sanity check: Drag & drop OpenCV-2.4.8-android-sdk/samples/15-puzzle/bin/Puzzle15Activity-debug.apk into the emulator. Confirm that it works. 

6) Test an OpenCV native C++ application. Download and unpack android-ndk-r9c-linux-x86_64.tar from http://developer.android.com/tools/sdk/ndk/index.html 

:~$ export PATH=$PATH:~android-ndk-r9c/ 
:~$ cd OpenCV-2.4.8-android-sdk/samples/native-activity 

Modify the ABI in Android.mk: APP_ABI := all 

:~$ vi jni/Android.mk 

Build it:

:~$ android update project --path . 
:~$ ndk-build -B 
:~$ ant debug 
:~$ cd bin

Read More

Encode and Decode Video from Memory

Download: https://github.com/dimitrs/video_coding

OpenCV provides a very simple api to connect to a camera and show the images in a window. It is also very simple to encode those images and store them on disk. However, the simple api provides no means by which to encode an image without storing it on disk and even worse, no means to decode an image stored in memory instead of on disk. Assuming I want to capture an image from a camera and send it over a socket to a receiver, one might want to first compress the image before sending it. A simple test I did indicated that an image can be compressed by a factor of 10 using the mpeg-4 codec. This is really important for streaming applications.

On Linux and Windows, OpenCV’s underlying implementation uses the ffmpeg library to encode/decode frames to and from disk. I found the implementation in opencv\modules\highgui\src\cap_ffmpeg_impl.hpp. In this post, I provide a small test project with a modified version of cap_ffmpeg_impl.hpp. This version allows you to encode/decode images to and from a buffer instead of disk. Warning: The code is a proof of concept and not production ready. You can find the project in github.

Below, you can see how to encode/decode video:

Initializing capture from a camera:

CvCapture* capture = cvCaptureFromCAM(CV_CAP_ANY);

Capturing a frame:

cvGrabFrame(capture); // capture a frame
IplImage* img = cvRetrieveFrame(capture); // retrieve the captured frame

Initialize a video encoder:

int fps = 10;
int width = 320;
int height = 320;
CvVideoWriter_FFMPEG writer;
writer.init();
writer.open("x.mpg", CV_FOURCC('D', 'I', 'V', 'X'), fps, width, height, true);

Encoding the previously captured image:

int wStep = img->widthStep;
int w =  img->width;
int h = img->height;
int c = img->nChannels;
writer.writeFrame((const uchar*)img->imageData, wStep, w, h, c, img->origin);

Initialize a video decoder:

CvCapture_FFMPEG cap;
cap.open(width, height);

Decoding the previously encoded image:

uint8_t* buffer = &writer.outbuf[0];
int length = writer.out_size;
cap.grabFrame(buffer, length);
IplImage* img2 = cap.retrieveFrame();

Show image in a window:

if (img2)
{
    cvShowImage("My Window", img2 );
    cvWaitKey();
}

Read More