Loading

Sign Language Translator Glove, Made with Arduino, Inspired by Jeremy Blum

My app connects wirelessly to the glove, and displays and “speaks” the English translations to the sign language gestures being signed. This device can be used to learn sign language and helps bridge the communication barrier with which a mute or deaf person is faced, when communicating with someone unfamiliar with the language.

Roman Kozak

The Abstract

Inspired by Jeremy Blum‘s  innovation, the Sudo Glove, I set out to create a device that could use the same technology (flex sensors, accelerometer and Arduino) while accomplishing a completely different task. Translate sign language into text and speech.

My first prototype was powered by a wired connection and transmitted the gestures to a PC via a USB cable. Once the proof of concept was working, the USB cable was removed and replaced with a Bluetooth module, and LiPo battery and charger. Also, the glove was sewn in such a way to be aesthetically indistinguishable from an everyday glove. The final product felt very comfortable to wear, looked good, and was functional; however, it was still transmitting its gestures in raw text. Thus, an Android smartphone application had to be created. My app connects wirelessly to the glove, and displays and “speaks” the English translations to the sign language gestures being signed. This device can be used to learn sign language and helps bridge the communication barrier with which a mute or deaf person is faced, when communicating with someone unfamiliar with the language.

Below is a video I made when the translator was in its early stages of design.

The Hardware

Main Electrical Components:

5 x Flex Sensors, 1 x Accelerometer, 1 x Arduino LilyPad, 1 x Bluetooth Module, 1 x Rechargeable Lithium Battery

Prototype 1

Sign Language Translator Arduino Roman Kozak

Prototype 2 – Exposed View (Custom PCB, Rechargeable LiPo, Compact Design)

Roman Kozak v2 Glove diagram

I made many modifications to the Sign Language Translator. When creating my first prototype I focused on making the innovation translate simple gestures, while in the later designs I focused on the user interface, design and feel of the device.

Prototype one had five flex sensors and a microcontroller called the Arduino Uno. The innovation was attached to the computer via USB. The main purpose for this prototype was to test and see if I could program simple hand gestures to prove the concept.

Once the concept of translating hand gestures into actions was proven by prototype one I went on to add more features and parts in prototype two. The parts included an accelerometer, a Bluetooth Module and a new glove that was more flexible and comfortable. I also started to experiment with code for an Android application. At the time I was using Google Eclipse to create an android app but it proved to unsuccessful because of the amount of code it would take just to connect a device via Bluetooth.

Prototype three proved to be a huge success because I was finally able to get all the parts to work together as one device. In this prototype I replaced the Arduino Uno with a smaller microcontroller called the Arduino LilyPad. The Lilypad played a big part in decreeing the overall size of the innovation. On the coding side, I continued to create an android application but this time I used MIT App Inventor.

Prototype One vs Two:

Roman Kozak

The App

I designed and programmed an Android app in Eclipse. The application connects to the Sign Language Translator via bluetooth and provides the user with a neat UI. Below are some shots of V2 of the app.

 

The Gestures

The most difficult part about this project was distinguishing between the different sign language gestures.

For example: U and V both have the same flex sensor and accelerometer reading. In order to distinguish the two I needed to add a piece of conductive fabric on the sides of each finger. So when the two fingers were touching it could feed a +5v into one of the digital i/o pins on the Arduino.

It took a lot of patience and attention to coding to make the translator as reliable and accurate as possible.

asl-american-sign-language

The Outcome

In March of 2013 I competed at the Niagara Regional Science and Engineering Fair and won “Best Project in Fair” along with a healthy collection of other awards.

Roman Kozak Science Fair 2013

In May of 2013 I moved on to compete at the Canada Wide Science Fair in Lethbridge, Alberta and won a Silver Medal.

Roman Kozak CWSF 2013

Roman Kozak CWSEF Silver

I was a finalist for the 2014 Weston Youth Innovation Award!

wyiabanner_2014_enter_en

Next Steps

Currently, I am working on a similar innovation. Instead of using a flex sensor to detect the bend in the fingers, I am measuring muscle activity by detecting its electric potential. This is referred to as electromyography (EMG).
The device would resemble a thick rubber wristband. I made a quick mockup below.
Inside, the band would house EMG sensors, a Bluetooth module (to connect wirelessly with my app), a microcontroller, battery, and other smaller components.
Here are some challenges I will face: Making the device distinguish which fingers are bending (EMG sensors aren’t too reliable when it comes to sensing single nerves and muscles in one’s wrist), keeping the cost to a minimum, making it small and portable. Stay tuned to this post for updates!

EMG Sciecne

Sign Language Translator In The News

2014-8-28 Freetronics | An Arduino-powered Sign Language Translator

2014-7-4 Tech Bang | Translation can be synchronized sound deaf sign language sign language friend quit

2014-4-7 Jeremy Blum | Engineers use Technology to Make Lives Better

2014-3-31 Ontario Science Centre| 2014 Weston Youth Innovation Award

Files

Here is the basic code you can use if you would like to make your own Sign Language Translator. I would recommend adding additional features to make your project unique. I would love to see what you make! Send me your project and I’ll link to it.

Code < Click to download

 

  • Outstanding articles and style of posting. It looks like I’ll come back on this website in the future and
    discover what exactly else you’ve gotten in store 🙂 !!!

    Now i’m likely to check out if my hubby and I will be able to track down nearly anything involving Accelerometer!!

  • Pingback: Engineers use Technology to Make Lives Better | JeremyBlum.com()

  • Hello Adrian,This is my firs Arduino and I also want to make exactly the same thing as you did but i have stulmbed across a serios of problems because some of the code that I am using in my arduino is writen in romanian and I don’t know exactly what to change in my code.Could you please help me ? The thing is that my arduino has Temperature, Humidity, Pressure and light sensor and there is a lot of code I must go through.Kind Regards,Flaviu Vlaicu

  • Pingback: Opiron Electronics Un guante basado en Arduino que traduce los gestos de la mano()

  • Pingback: Android Language Translation Apps | Israel Foreign Affairs()

  • do you have a twitter that i can follow

  • Safayet Ahmed

    Hi! I liked your project very much. I am also working on similar type of project. Can you suggest me where I can get the code and circuit diagram. Thank you in advance.

  • Adit Sanghvi

    I’m making a similar glove for a science fair(which also includes text to speech), and im using an arduino leonardo instead of a lilypad. I would like to know to which pin of the arduino did you attach your accelerometer to and how many accelerometers have you used? Thanks in advance!

  • Pingback: Engineers use Technology to Make Lives Better - Arduino collector blog()

  • Ian Lozano

    We are working on a similar project at our college. A quick question. How did you distinguish between the letters R, U and V? They all have the same flex and accelerometer sensor values.

    • You will need to add a hall effect sensor on the index and middle finger. Contact sensors signify which fingers are touching each other but they do not give any information of the relative position between fingers except for whether they are together or apart.

      I’d recommend you checkout this project by some EE students at cornell: http://people.ece.cornell.edu/land/courses/ece4760/FinalProjects/s2012/sl787_rak248_sw525_fl229/sl787_rak248_sw525_fl229/ They created sign language glove and included, in much detail all of the challenges they encountered.

      Good Luck! -r

      • Deepak

        Hey I have a small doubt,,why do u used minimum and maximum value????
        Can’t we just use the analog to digital mapped value using map function..
        Thnx….

  • Pingback: Top 5 Arduino / Raspberry Pi Sensors | Roman Kozak()

  • Pingback: Top 5 Arduino and Raspberry Pi Sensors | Roman Kozak()

  • Mahesh j Shetty

    This the best project. And it is very useful for dumb people’s. Can you please upload the circuit soon?

  • Mahesh j Shetty

    please email me the circuit..!

  • Mattia Granito

    hello! I love your project! I need your help. I want made a glove to move the mouse, press left, right and other 2-3 button on the keyboard. How i can make that? I can use your project but how i can move mouse? I want to make this glove because my grandfather can use his hands to use pc. He can use only one hand and i’d like to help him. Thanks in advance!

  • Conrad

    Sign language club was featured today on Fox news TV (USA). I commented about your project on:

    http://www.washington.edu/news/2016/04/12/uw-undergraduate-team-wins-10000-lemelson-mit-student-prize-for-gloves-that-translate-sign-language/

    • Conrad, I did see that yesterday and was thinking how familiar it looked 😉 Thank you for commenting and sharing!

  • Caroline

    Roman, we are trying to develop something similar. Do you think you can send us the circuit diagram? carolineruella@gmail.com

  • Minh Đức

    Share project for me.Help email : bui.17.11.1994@gmai.com

  • urooja memon

    please share your project ….how can i make it?
    here is my email:
    noorafshanmemon@gmail.com

  • shilpa more

    can u plz provide us circuit diagram of sign language translator into text and speech….. plz..plz..plz..

  • Irn-irn Gaizen

    hello your project is very awesome,,,i would like to ask if its ok to use other types of accelerometer like mma7631? pls can i ask for its circuit diagram?tnx!!!

  • Kamil

    please provide circuit diagram.