SIGN LANGUAGE GESTURE RECOGNITION FROM VIDEO SEQUENCE USING TENSORFLOW.JS

Abstract

Sign language recognition is a critical problem in computer vision and machine learning, with the potential to improve communication and accessibility for deaf and hard-of-hearing individuals. In this research paper, we propose a novel approach for sign language recognition from the live feed using Tensorflow.js, a JavaScript library for machine learning in the browser. Our approach involves the use of convolutional neural networks to extract features from the video sequence, and a pre-trained model known as MobileNetV2 to classify images correctly. We have also introduced the autocorrect feature in order to make real-time detection faster. Our final model runs at 15FPS and detects finger spellings with an accuracy of 90%. Our model takes into account both the temporal and spatial features of the video sequence. Our results suggest that our approach has the potential to be a powerful and effective tool for sign language recognition, with the advantage of being a Next.js application which runs entirely in the browser and creating a PWA version of the app we make it accessible to a wider audience. Overall, our research represents a significant step towards improving the accessibility and inclusivity of sign language recognition.