Human-Robot Interaction in Unstructured Environments Progress Report October 28, 2013 Kevin DeMarco

Gesture Recognition
Last week I began researching state-of-the-art methods for gesture recognition without a depth sensor. There are many examples of individuals using a single web camera to detect a users hand and extracting some simple pose information. However, most of the examples involve an individual with their hand in front of a background that contrasts well with the hand. The common methods within OpenCV for hand pose recognition involve forming a convex hull around the hand, where the vertices of the hull imply the positions of the nger tips. While most of the examples end at this phase of the detection, the next step in gesture recognition involves detecting hand and nger movements in space. In this phase of detection, Hidden Markov Models (HMM) or Support Vector Machines (SVM) can be used for training the gesture models. I found an open source project, called the Gesture Recognition Toolkit (GRT), that uses SVMs for gesture recognition, which I plan on testing ( http://www.nickgillian.com/software/grt ). Most of the gesture recognition methods involve two steps: segmentation of the hand from the environment and pose or gesture recognition using machine learning. It seems that the hand segmentation aspect with a single non-depth camera is the most dicult aspect, since once the hand and nger poses are known, the machine learning algorithms can easily match the gestures to pre-trained models. I am interested in researching knowledge-based methods of hand segmentation since the human eye and brain are so adept at detecting hands in optical images. We should be able to develop a computer program that can extract the hands pose in arbitrary lighting and background conditions.

OpenCV Workbench
When I was working with OpenCV during the rst phase of my sonar-based diver detection research, I found that I spend a considerable amount of time writing custom C++ code to trim videos down to size and capturing new test videos. Thus, I decided to write my own OpenCV Workbench GUI application, as shown in Figure 1, to assist in development. The OpenCV Workbench facilitates jumping to and examining specic frames in the video sequence through the use of the horizontal frame slider and a number box that lets the user enter the exact frame to examine. Also, the Workbench has the capability to change the playback framerate. Not only can the OpenCV Workbench read all of the commonly used video formats (e.g., AVI, WMV, MP4, MPG, and MOV) and a standard web camera, but it can also read the sonar les from a BlueView 2D Imaging Sonar. This allows the user to use a single interface when prototyping computer vision algorithms. The OpenCV Workbench also has the capability of exporting newly recorded and edited video streams to new les on disk. The export video dialog window is shown in Figure 2. The Qt GUI framework provides a robust framework that enables error checking on user forms before invalid data entry can cause the program to crash. The next major feature that I will be implementing will be a visual interface for enabling and disabling computer vision algorithms (e.g., threshold, edge detection, face detection, background subtraction, etc.) and a GUI for arranging the order in which the algorithms execute.

TODO
Test out the Gesture Recognition Toolkit (GRT) with OpenCV Implement interface to enable and disable computer vision algorithms Research the use of knowledge-based computer vision for improved tracking of hand posture.

Figure 1: The OpenCV Workbench GUI

Figure 2: The OpenCV Workbench GUI: Export Video Frames