Inspired by Jeremy Blum‘s innovation, the Sudo Glove, I set out to create a device that could use the same technology (flex sensors, accelerometer and Arduino) while accomplishing a completely different task. Translate sign language into text and speech.
My first prototype was powered by a wired connection and transmitted the gestures to a PC via a USB cable. Once the proof of concept was working, the USB cable was removed and replaced with a Bluetooth module, and LiPo battery and charger. Also, the glove was sewn in such a way to be aesthetically indistinguishable from an everyday glove. The final product felt very comfortable to wear, looked good, and was functional; however, it was still transmitting its gestures in raw text. Thus, an Android smartphone application had to be created. My app connects wirelessly to the glove, and displays and “speaks” the English translations to the sign language gestures being signed. This device can be used to learn sign language and helps bridge the communication barrier with which a mute or deaf person is faced, when communicating with someone unfamiliar with the language.
Below is a video I made when the translator was in its early stages of design.
Main Electrical Components:
Prototype 2 – Exposed View (Custom PCB, Rechargeable LiPo, Compact Design)
I made many modifications to the Sign Language Translator. When creating my first prototype I focused on making the innovation translate simple gestures, while in the later designs I focused on the user interface, design and feel of the device.
Prototype one had five flex sensors and a microcontroller called the Arduino Uno. The innovation was attached to the computer via USB. The main purpose for this prototype was to test and see if I could program simple hand gestures to prove the concept.
Once the concept of translating hand gestures into actions was proven by prototype one I went on to add more features and parts in prototype two. The parts included an accelerometer, a Bluetooth Module and a new glove that was more flexible and comfortable. I also started to experiment with code for an Android application. At the time I was using Google Eclipse to create an android app but it proved to unsuccessful because of the amount of code it would take just to connect a device via Bluetooth.
Prototype three proved to be a huge success because I was finally able to get all the parts to work together as one device. In this prototype I replaced the Arduino Uno with a smaller microcontroller called the Arduino LilyPad. The Lilypad played a big part in decreeing the overall size of the innovation. On the coding side, I continued to create an android application but this time I used MIT App Inventor.
Prototype One vs Two:
I designed and programmed an Android app in Eclipse. The application connects to the Sign Language Translator via bluetooth and provides the user with a neat UI. Below are some shots of V2 of the app.
The most difficult part about this project was distinguishing between the different sign language gestures.
For example: U and V both have the same flex sensor and accelerometer reading. In order to distinguish the two I needed to add a piece of conductive fabric on the sides of each finger. So when the two fingers were touching it could feed a +5v into one of the digital i/o pins on the Arduino.
It took a lot of patience and attention to coding to make the translator as reliable and accurate as possible.
In March of 2013 I competed at the Niagara Regional Science and Engineering Fair and won “Best Project in Fair” along with a healthy collection of other awards.
In May of 2013 I moved on to compete at the Canada Wide Science Fair in Lethbridge, Alberta and won a Silver Medal.
I was a finalist for the 2014 Weston Youth Innovation Award!
Currently, I am working on a similar innovation. Instead of using a flex sensor to detect the bend in the fingers, I am measuring muscle activity by detecting its electric potential. This is referred to as electromyography (EMG).
The device would resemble a thick rubber wristband. I made a quick mockup below.
Inside, the band would house EMG sensors, a Bluetooth module (to connect wirelessly with my app), a microcontroller, battery, and other smaller components.
Here are some challenges I will face: Making the device distinguish which fingers are bending (EMG sensors aren’t too reliable when it comes to sensing single nerves and muscles in one’s wrist), keeping the cost to a minimum, making it small and portable. Stay tuned to this post for updates!
Sign Language Translator In The News
2014-8-28 Freetronics | An Arduino-powered Sign Language Translator
2014-7-4 Tech Bang | Translation can be synchronized sound deaf sign language sign language friend quit
2014-4-7 Jeremy Blum | Engineers use Technology to Make Lives Better
2014-3-31 Ontario Science Centre| 2014 Weston Youth Innovation Award
Here is the basic code you can use if you would like to make your own Sign Language Translator. I would recommend adding additional features to make your project unique. I would love to see what you make! Send me your project and I’ll link to it.
Code < Click to download