Industrial camera system earns PhD student top honours in national research competition

Siddhant Ahuja never misses an episode of Dragon’s Den, so he thought he knew what to expect when he entered—and won first place in—a contest modeled after the popular television program.

“They didn’t really grill us as hard as I thought they might,” said Ahuja, a PhD student in electrical and computer engineering who was in Ottawa Tuesday to take part in TestDRIVE, an AUTO21 graduate student competition held during the Canadian Manufacturers and Exporters’ annual summit.

Read more on this here.

Developing innovative automotive technologies gives University of Windsor student a financial boost

A smart camera technology that can help increase a factory’s efficiency helped a University of Windsor student win a national automotive research competition. Siddhant Ahuja, a PhD student, won the AUTO21 TestDRIVE competition, receiving a $10,000 scholarship for his work on indoor real-time location tracking systems.

Wireless communication monitoring systems have been ineffective in tracking items on an assembly line due to the harsh shop conditions that can disturb radio frequencies. The visual sensor network technology presented by Ahuja can trace products through an assembly line and aid in automated inspection and identification, increasing the flexibility and productivity of automotive factories. The network uses spatially distributed smart cameras and web-enabled software.

Read more on this here.

PhD student hoping to wow judges in technology contest

Armed with nothing more than two cameras and a tiny circuit board, a UWindsor doctoral student will travel to Ottawa today to demonstrate how he can make car manufacturing more efficient and compete in a contest for as much as $10,000 in scholarship money.

Siddhant Ahuja, a student in electrical and computer engineering, will compete tomorrow against six other students in the TestDRIVE contest, an AUTO21 graduate student competition Tuesday at the Chateau Laurier hotel. The competition, structured like the television show Dragon’s Den, is held during the Canadian Manufacturers and Exporters’ annual event, Roadmap to Recoverysummit.

Read the full article here

Check event conflicts in Google calendar using PHP Zend Framework

In order to insert an event in to the Google calendar, i wanted to first check the calendar to make sure that there are no conflicts with existing events.

I wanted to be able to search either primary or secondary calendars. So i decided to write the code below to retrieve all events falling within the date-time range, then detect the number of events found. If even one event is found, then there is a conflict.

Remember that All-Day events within the date-time range also appear as conflicts.

<?php

/**
 * @author Siddhant
 * @copyright 2009
 */

$dateTimeStart="2009-12-05T16:00:00-05:00";
$dateTimeEnd="2009-12-05T18:00:01-05:00";

//Connect to google calendar
	//If you do not have access to the PHP.INI file on the remote server, you should use the following statement to set the path information for Zend framework
	$clientLibraryPath = 'libraries';
	$oldPath = set_include_path(get_include_path() . PATH_SEPARATOR . $clientLibraryPath);
	// load ZEND classes
	require_once 'Zend/Loader.php';
	Zend_Loader::loadClass('Zend_Gdata');
	Zend_Loader::loadClass('Zend_Gdata_ClientLogin');
	Zend_Loader::loadClass('Zend_Gdata_Calendar');
	Zend_Loader::loadClass('Zend_Http_Client');
	// connect to calendar service
	$gcal = Zend_Gdata_Calendar::AUTH_SERVICE_NAME;
	$user = "test@gmail.com"; //Insert your google username
	$pass = "testPassword"; //Insert your Google password
	$client = Zend_Gdata_ClientLogin::getHttpClient($user, $pass, $gcal);
	$gcal = new Zend_Gdata_Calendar($client);
//Search the calendar database to see if there is/are existing event(s) in the date/time range
	//generate a query to retrive events
	$query = $gcal->newEventQuery();
    //This sets the default calendar id
	//$query->setUser('default');
	//This sets the calendar to secondary or non-default calendar
	$query->setUser('secondaryCalendarID@group.calendar.google.com'); //The secondary id can be found from the link which looks like: the https://www.google.com/calendar/feeds/secondaryCalendarID@group.calendar.google.com/private/full
    $query->setVisibility('private');
    $query->setProjection('basic');
    //Order the events found by start time in ascending order
    $query->setOrderby('starttime');
    //Set date range
	$query->setStartMin($dateTimeStart);
	$query->setStartMax($dateTimeEnd);
	// Retrieve the event list from the calendar server
	//Remember that all-day events will show up while detecting conflicts
    try {
      $feed = $gcal->getCalendarEventFeed($query);
    } catch (Zend_Gdata_App_Exception $e) {
      echo "Error: " . $e->getResponse();
    }
    //If even one event is found in the date-time range, then there is a conflict.
	if($feed->totalResults!='0')
	{
		echo($feed->totalResults." Conflicts Found <br/>");
	 	echo("<ol>");
	    foreach ($feed as $event)
		{
	      echo "<li>\n";
	      echo "<h2>" . stripslashes($event->title) . "</h2>\n";
	      echo stripslashes($event->summary) . " <br/>\n";
	      $id = substr($event->id, strrpos($event->id, '/')+1);
	      echo "</li>\n";
	    }
	    echo "</ul>";
	    echo("</ol>");
	}
	else
	{
		echo("No Conflicts");
	}
?>

Image Segmentation Dataset

Here is a dataset that i prepared for testing image segmentation algorithms. All images are synthetic of size 128×128 (i.e. 16384 pixels in total) and grayscale.

Trin (4 classes)

A (5 classes)

B (3 classes)

Jigsaw (5 classes)

Snowflake (5 classes)

Sun (4 classes)

To download the dataset, please Click Here (13 KB, *.rar format)

How to select a proper Disparity Value for Stereo-Vision

I have been asked many times how to determine the proper disparity range? If you are taking the images (left, right and ground truth disparity maps) from Middlebury Stereo’s website, then the answer is pretty easy. The minimum disparity value is 0. The maximum disparity value is the maximum value of the pixel in the ground truth map, divided by the scale factor. If you have a stereo vision set-up taking real-world imagery, then you have to do a bit of math to calculate the values.

Here is one of the equations:

In the above equation,

r is the range of the object that you are trying to determine,
b is the baseline of the two cameras, i.e. the distance between the centers of the cameras,
f is the focal length of the image sensor,
x is the pixel size of the image sensor,
N is the maximum disparity value.

For example, lets say that the image sensor has the pixel size of 17um, focal length of 2.8mm, baseline of 28mm, and maximum disparity value is from 5-35. Range vs. disparity values can be plotted in a graphical form as below:

As can be seen above, for the characteristics of the stereo vision system selected, the maximum range of the object that you are trying to detect really depends on the range of disparity values.  For disparity values between 5-35, the detectable range of objects is roughly 15-100 cm.

However, there is another trade-off that people forget about, primarily between the uncertainty in detecting objects at a particular range and the actual range itself. Here is the equation which relates both these variables:

In the above equation, there are two new variables: Δr and ΔN.

Δr is the uncertainty in detecting the object at a certain range,
ΔN is the change in disparity value.

Taking the previous example, lets say that the image sensor has the pixel size of 17um, focal length of 2.8mm, baseline of 28mm, maximum disparity values between 5-35, and the range of 15-100cm, the uncertainty in detecting object at the desired range vs. range values can be plotted in a graphical form as below:

From the above, and for the characteristics of the stereo vision system selected, for a range of 50cm, the uncertainty in the range is 5cm, i.e. if an object has been detected at 50cm, the object could be anywhere from 47.5cm to 52.5cm with a delta of 5cm.

Thus you can see that there are some trade-offs that you should consider while developing a stereo-vision system, and selecting a proper disparity value.

For more information, please refer to:

[1] Khaleghi B., Ahuja S., Wu Q.M.J., “An improved real-time miniaturized embedded stereo vision system (MESVS-II),” IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008 (CVPRW’08), pp.1-8, 23-28 June 2008.
[2] Khaleghi B., Ahuja S., Wu Q.M.J., “A New Miniaturized Embedded Stereo-Vision System (MESVS-I)”, Canadian Conference on Computer and Robot Vision, 2008 (CRV ‘08), pp.26-33, 28-30 May 2008

Adding Vignetting Effect to Images

When the lens size is smaller than the dimensions of the actual image sensor, or inadequately covering it, it leads to an effect whereby the center of the image is clear, and a fading/darkening of the image borders is seen. This effect is commonly referred to as the Vignetting Effect. In graphics, it is used to de-emphasize the distracting background, and shed light on the foreground objects.

Example

Scale factor at image border=1.0	Scale factor at image border=0.8	Scale factor at image border=0.6
Scale factor at image border=0.4	Scale factor at image border=0.2	Scale factor at image border=0.1

MATLAB Code

% *************************************************************************
% Title: Function-Introduce Vignetting effect to the image
% Author: Siddhant Ahuja
% Created: September 2008
% Copyright Siddhant Ahuja, 2008
% Inputs: Input Image (var: inputImage), Scale Change level (var:
% scaleLevel) Valid values for scale change should go from 0.1 to 1
% Outputs: Vignetting effect added image (var: vignettImg, Time taken (var: timeTaken)
% Example Usage of Function: [vignettImg, timeTaken]=funcVignettingEffect('Baby1LeftColor.png', 0.5);
% *************************************************************************
function [vignettImg, timeTaken]=funcVignettingEffect(inputImage,scaleLevel)
% Read the input image
try
    % Read an image using imread function
    inputImage=imread(inputImage);
    % grab the number of rows, columns, and channels
    [nr, nc, nChannels]=size(inputImage);
     % Grab the image information (metadata) of input image using the function imfinfo
    inputImageInfo=imfinfo(inputImage);
    % Determine if input left image is already in grayscale or color
    if(getfield(inputImageInfo,'ColorType')=='truecolor')
        colored=1;
    else if(getfield(inputImageInfo,'ColorType')=='grayscale')
        colored=0;
        else
        error('The Color Type of Left Image is not acceptable. Acceptable color types are truecolor or grayscale.');
        end
    end
catch
    % if it is not an image but a variable
    inputImage=inputImage;
    % grab the number of channels
    [nr, nc, nChannels]=size(inputImage);
    if(nChannels)>1
        colored=1;
    else
        colored=0;
    end
end
if scaleLevel <= 0
    error('Scale value must be > 0');
end
vignettImg=inputImage;
tic; % Initialize the timer to calculate the time consumed.
imgCntX = nc/2;
imgCntY = nr/2;
maxDistance = sqrt (imgCntY^2 + imgCntX^2);
if(colored==1)
    for (i=1:nr)
        for (j=1:nc)
           dis = sqrt (abs(i-imgCntY)^2 + abs(j-imgCntX)^2);          
           %% reduce brighness of pixel based on distance from the image center
           vignettImg(i,j,1) = vignettImg(i,j,1)* (1 - (1-scaleLevel)*(dis/maxDistance) );
           vignettImg(i,j,2) = vignettImg(i,j,2)* (1 - (1-scaleLevel)*(dis/maxDistance) );
           vignettImg(i,j,3) = vignettImg(i,j,3)* (1 - (1-scaleLevel)*(dis/maxDistance) );
       end
    end
else
%gray
     for (i=1:nr)
        for (j=1:nc)
           dis = sqrt (abs(i-imgCntY)^2 + abs(j-imgCntX)^2);          
           %% reduce brighness of pixel based on distance from the image center
           vignettImg(i,j) = vignettImg(i,j)* (1 - (1-scaleLevel)*(dis/maxDistance) );
       end
    end
end
% Stop the timer to calculate the time consumed.
timeTaken=toc;

Left-Right Consistency (LRC) Check

Left-Right Consistency (LRC) check is performed to get rid of the half-occluded (objects scene in one image, and not in other) pixels in the final disparity map. This is computed by taking the computed disparity value in one image, and re-projecting it in the other image. If the difference in the values is less than a given threshold, then the pixels are half-occluded.

Example

Aloe-Left Image	Aloe-Right Image
Aloe-Ground Truth Left to Right Disparity Map	Aloe-Ground Truth Right to Left Disparity Map
Aloe-Left to Right SAD Disparity Map Window Size: 15×15 Disparity Range: 0-70	Aloe-Right to Left SAD Disparity Map Window Size: 15×15 Disparity Range: 0-70
Aloe-Ground Truth Occlusion Map	Aloe-LRC Map (threshold: 2)

MATLAB Code

% *************************************************************************
% Title: Function-Left/Right Consistency Check
% Author: Siddhant Ahuja
% Created: May 2008
% Copyright Siddhant Ahuja, 2008
% Inputs: Left Image (var: leftImage), Right Image (var: rightImage),
% Window Size (var: windowSize), Minimum Disparity (dispMin), Maximum
% Disparity (dispMax), Threshold for the check (var: thresh) typically 2.0
% Outputs: Disparity Map (var: dispMap), Time taken (var: timeTaken)
% Example Usage of Function: [dispMapLRC, timeTaken]=funcLRCCheck('TsukubaLeft.jpg', 'TsukubaRight.jpg', 9, 0, 16,2);
% *************************************************************************
function [dispMapLRC, timeTaken]=funcLRCCheck(leftImage, rightImage, windowSize, dispMin, dispMax, thresh)
% Initiate the Timer to calculate the time consumed.
tic;
% Perform SAD Correlation based matching (Right to Left)
[dispMapR2L, timeTakenR2L]=funcSADR2L(leftImage, rightImage, windowSize, dispMin, dispMax);
% Perform SAD Correlation based matching (Left to Right)
[dispMapL2R, timeTakenL2R]=funcSADL2R(leftImage, rightImage, windowSize,dispMin , dispMax);
% Prepare matrix for subtraction and scale it for comparison
dispMapL2R=-dispMapL2R;
% Find the size (columns and rows) of the L2R Disparity map and assign the rows to
% variable nrLRCCheck, and columns to variable ncLRCCheck
[nrLRCCheck,ncLRCCheck] = size(dispMapL2R);
% Create an image of size nrLRCCheck and ncLRCCheck, fill it with zeros and assign
% it to variable dispMapLRC
dispMapLRC=zeros(nrLRCCheck,ncLRCCheck);
% Find out how many rows and columns are to the left/right/up/down of the
% central pixel based on the window size
win=(windowSize-1)/2;
for(i=1:1:nrLRCCheck)
    for(j=1:1:ncLRCCheck)
        xl=j;
        xr=xl+dispMapL2R(i,xl);
        if (xr>ncLRCCheck||xr<1)
            dispMapLRC(i,j) = 0; %% occluded pixel
        else            
            xlp=xr+dispMapR2L(i,xr);
            if (abs(xl-xlp)<thresh)
                dispMapLRC(i,j) = -dispMapL2R(i,j);  %% non-occluded pixel            
            else
                dispMapLRC(i,j) = 0; %% occluded pixel                        
            end
        end
    end
end
% Terminate the Timer to calculate the time consumed.
timeTaken=toc;

Compute Variance Map of an Image

Variance map of an image is calculated by taking a square window of a set size around a center pixel, and calculating the variance of the values of the pixels.

The variance within the window can be calculated from the following equation:

Example

Monopoly-Left Image

Monopoly-Left Image-Variance Map (Pixels marked as black are low-texture regions)

MATLAB Code

% *************************************************************************
% Title: Function-Compute Variance map of the image
% Author: Siddhant Ahuja
% Created: May 2008
% Copyright Siddhant Ahuja, 2008
% Inputs: Input Image (var: inputImage), Window Size (var: windowSize),
% Threshold (var: thresh) Typical value is 140
% Outputs: Variance Map (var: varianceImg) , Time taken (var: timeTaken)
% Example Usage of Function: [varianceImg, timeTaken]=funcVarianceMap('MonopolyLeftColor.png', 5, 9);
% *************************************************************************
function [varianceImg, timeTaken]=funcVarianceMap(inputImage, windowSize, thresh)
try 
    % Grab the image information (metadata) of input image using the function imfinfo
    inputImageInfo=imfinfo(inputImage);
    if(getfield(inputImageInfo,'ColorType')=='truecolor')
    % Read an image using imread function, convert from RGB color space to
    % grayscale using rgb2gray function and assign it to variable inputImage
        inputImage=rgb2gray(imread(inputImage));
        % Convert the image from uint8 to double
        inputImage=double(inputImage);
    else if(getfield(inputImageInfo,'ColorType')=='grayscale')
    % If the image is already in grayscale, then just read it.        
            inputImage=imread(inputImage);
            % Convert the image from uint8 to double
            inputImage=double(inputImage);
        else
            error('The Color Type of Left Image is not acceptable. Acceptable color types are truecolor or grayscale.');
        end
    end
catch
    % if it is not an image but a variable
    inputImage=inputImage;
end
% Find the size (columns and rows) of the input image and assign the rows to
% variable nr, and columns to variable nc
[nr,nc] = size(inputImage);
% Check the size of window to see if it is an odd number.
if (mod(windowSize,2)==0)
    error('The window size must be an odd number.');
end
% Create an image of size nr and nc, fill it with zeros and assign
% it to variable meanImg
meanImg=zeros(nr, nc);
% Create an image of size nr and nc, fill it with zeros and assign
% it to variable varianceImg
varianceImg=zeros(nr, nc);
% Find out how many rows and columns are to the left/right/up/down of the
% central pixel based on the window size
win=(windowSize-1)/2;
tic; % Initialize the timer to calculate the time consumed.
% Compute a map of mean values
for(i=1+win:1:nr-win)
    for(j=1+win:1:nc-win)
        sum=0.0;
        for(a=-win:1:win)
            for(b=-win:1:win)
                sum=sum+inputImage(i+a,j+b);
            end
        end
        meanImg(i,j)=sum/(windowSize*windowSize);
    end
end
% Compute a map of variance values
for(i=1+win:1:nr-win)
    for(j=1+win:1:nc-win)
        sum=0.0;
        for(a=-win:1:win)
            for(b=-win:1:win)
                sum=sum+((inputImage(i+a,j+b)-meanImg(i,j))^2);
            end
        end         
        var=sum/((windowSize*windowSize)-1);
        % Apply threshold to produce a binarized variance map
        if (var > thresh)
            varianceImg(i,j) = 255;
        else
            varianceImg(i,j) = 0;
        end
    end
end
% Stop the timer to calculate the time consumed.
timeTaken=toc;

Finding Low-texture or textureless regions in images

Stereo vision algorithms typically compute erroneous results in regions where there is a little or no texture in the scene. They are defined in [1] as regions where the squared horizontal intensity gradient averaged over a square window of a given size is below a given threshold.

Example

Bowling1-Left Image

Bowling1-Left Image-Textureless Map (Pixels marked as white are low-texture regions)

MATLAB Code

% *************************************************************************
% Title: Function-Find Textureless regions of an image
% Notes: Textureless regions are defined as regions where the squared horizontal
% intensity gradient averaged over a square window of a given size 
% (windowSize) is below a given threshold (thresh);
% Author: Siddhant Ahuja
% Created: May 2008
% Copyright Siddhant Ahuja, 2008
% Inputs: Input Image (var: inputImage), Window Size (var: windowSize),
% Threshold (var: thresh) Typical value is 4
% Outputs: Textureless Map (var: texturelessImg) , Time taken (var: timeTaken)
% Example Usage of Function: [texturelessImg, timeTaken]=funcTexturelessRegions('imL.png', 9, 4);
% *************************************************************************
function [texturelessImg, timeTaken]=funcTexturelessRegions(inputImage, windowSize, thresh)
% Read the input image
try
    % Read an image using imread function
    inputImage=imread(inputImage);
    % grab the number of rows, columns, and channels
    [nr, nc, nChannels]=size(inputImage);
     % Grab the image information (metadata) of input image using the function imfinfo
    inputImageInfo=imfinfo(inputImage);
    % Determine if input left image is already in grayscale or color
    if(getfield(inputImageInfo,'ColorType')=='truecolor')
        colored=1;
    else if(getfield(inputImageInfo,'ColorType')=='grayscale')
        colored=0;
        else
        error('The Color Type of Left Image is not acceptable. Acceptable color types are truecolor or grayscale.');
        end
    end
catch
    % if it is not an image but a variable
    % grab the number of channels
    [nr, nc, nChannels]=size(inputImage);
    if(nChannels)>1
        colored=1;
    else
        colored=0;
    end
end
% Check the size of window to see if it is an odd number.
if (mod(windowSize,2)==0)
    error('The window size must be an odd number.');
end
% Create an image of size nr and nc, fill it with zeros and assign
% it to variable texturelessImg
texturelessImg=zeros(nr, nc);
% Create an image of size nr and nc, fill it with zeros and assign
% it to variable sqGradImg
sqGradImg=zeros(nr,nc);
% Find out how many rows and columns are to the left/right/up/down of the
% central pixel based on the window size
win=(windowSize-1)/2;
inputImage=double(inputImage);
tic; % Initialize the timer to calculate the time consumed.
% Produce Squared Horizontal Gradient image sqGradImg
for (i=1:1:nr)
    for (j=1:1:nc-1)
        sum = 0.0;        
        for (k=1:1:nChannels)
            diff =  inputImage(i,j,k) - inputImage(i,j+1,k);
            sum = sum + (diff*diff); 
        end
        sum = sum / nChannels;
        sqGradImg(i,j+1) = sum;
        if (j==1)
            sqGradImg(i,j) = sum;
        end
        if (sum > sqGradImg(i,j))
            sqGradImg(i,j) = sum;
        end
    end
end
% Compute average within predefined box window of size windowSize x
% windowSize
for (i=1+win:nr-win)
    for (j=1+win:nc-win)       
        % go over the square window
        sum = 0.0;
        avg = 0.0;
        for (a=-win:1:win)
            for (b=-win:1:win)                
                sum = sum + sqGradImg(i+a,j+b);
            end
        end           
        % Compute the average
        avg = sum / (windowSize*windowSize);
        % Apply threshold
        if (avg < (thresh*thresh))
            texturelessImg(i,j) = 255; % mark detected textureless pixel as white
        end
    end
end
% Stop the timer to calculate the time consumed.
timeTaken=toc;

References

[1] D. Scharstein and R. Szeliski, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” International Journal of Computer Vision, vol. 47(1/2/3), pp. 7-42, Apr. 2002.