A study of a vision-based lip movement analysis for hearing-impaired person

Abstract

The study of vision based lip movement is the interpretation of human lip movement while speaking. Deaf and hard of hearing people often have a problem being able to understand what being talking in the conversation. Sign language may be useful for them to communicate but not everyone may understand the sign language. To make the communication more interesting and does not have any obstacle to talk to anybody, the lip reading is the best way to communicate. Deaf and hard hearing person can talk like the normal person, however, they do not hear the word spoken by themselves. On the other hand, there is no sound feedback system for them. This is a reason they could not speak with the correct pronunciations. To teach them pronounce the word correctly, fluently and to understand others in the conversation, the lip reading systems that could train them pronounce the word correctly needs to be developed. Previous researcher proposed many methods to recognize the spoken word based on the lip movement. One of the method is attach the colour marker onto the lip surface and a vision sensor is used to track the moving marker. Other researchers extract the lip region by using Active contour Model (ACM) or a snake method to draw the keypoint around lip edges. In this studies, a system to track and recognize the spoken word based on lip movements is proposed. A camera is used as a vision sensor and an image processing technique is employed to extract and track the lip. The lip horizontal and vertical distances of the lip are used to measure the ellipse while the systems track the lip movements. The trajectories of the ellipse are resampled to 10 point called the features point. The word database contains 10 spoken word frequently speak at the hospital are designed based on the distribution of the 10 feature points. Two types of word database have been designed which are the individual word database and the universal word database. The individual word database is defined as the distributed feature points of each subject by speaks the words with repetition. Meanwhile, the universal word database is defined as the distributed feature point of all word regardless who is the subjects. The recognition experiments are conducted and the system recognizes the unknown words with the recognition rate 92.47% accuracy by using the individual database. Meanwhile, the system recognizes the unknown words with the recognition rate 90.39% accuracy by using the universal database.