Human-machine interaction by tracking hand movements

Abstract

The vision-based hand tracking and gesture recognition is an extremely challenging problem due to the intricate nature of hand gestures this is a reason that available computer vision algorithms are computationally complex. In this research work a new methodology for 3D human hand gestures detection and recognition is proposed, which can be used for natural and intuitive human-computer interaction and other robotic systems. The proposed method based on Fourier’s descriptors, neural networks and morphology approaches to solve the problem of human hand tracking and gesture recognition of 3D objects from a single silhouette image. There are many constrains and challenges are there for the recognition of an object, like size change, translation, rotation around the three axes, partial occlusion, low intensity of light as well as the deformation of the shape. In this research work we used invariant Fourier’s descriptors and back propagation neural networks techniques for 3D objects recognitions from their 2D silhouette pattern to solve above mentioned challenges. The proposed approach used Fourier’s descriptors coefficients and back propagation neural network with different numbers of hidden layers to build the optimal classifier of 3D pattern from a single silhouette image. Besides that, another method is proposed using image processing and morphology technique in conjunction with various mathematical formulas to calculate hand position and orientation. The recognised objects are exposed to different intensities of light, are partially occluded, with size change, translation, rotation about all the axes and we used also deformed shapes. This new proposed method was applied and tested on the simulated Manipulated Robotic System (UniMAP Robot Manipulator Simulation System) that allows this robotic system to act as an intelligent system to track a human hand in 3D space and estimate its orientation and position in real time with the goal of ultimately using the algorithm with a robotic spherical wrist system. During experiment, there was no need for continuous camera calibration, and it required only once at the beginning for the registration of the hand and using proposed technique large number of hand movements and orientations are correctly identify. Experimental result shows that proposed method is a robust technique, unlike other approaches that use costly leaning functions or generalization methods. The high performance was achieved during experiments because of the accurate hand movement identification and the low computational load that results in a fast processing time. The proposed method could therefore be used with different types of teleoperated robotic manipulators or in other human-computer interaction applications in which a fast processing time was important.