The Evolving World of Translation Technology for the Deaf and Hard of Hearing
There are 430 million people, or 5%, with disabling hearing loss in the world (World Health Organization: WHO, 2024), with 13 million, or 13%, in the United States (Quick Statistics About Hearing, Balance, & Dizziness, 2024). Since most of society is hearing, it can be easy to forget that there are some without the same accessibility as others. In the United States, for instance, it was only in the last 30 years that movie theaters were required by the Department of Justice (DOJ) to offer a form of accessible aid to all patrons (Questions and Answers: DOJ's Final Rule Under Title III Requiring Movie Theaters to Provide Closed Captioning and Audio Description, n.d.). The Federal Communications Commission (FCC) requires television broadcasts to be captioned (Closed Captioning on Television, 2021); however, this is only a fraction of media available. Streaming services like Netflix and sites with user-generated content like YouTube do not require videos to be captioned (Closed Captioning of Internet Video Programming, 2021).
Barriers exist beyond media, like using the telephone. Customer service often uses telephone systems, requiring the user to have access to video relay service (VRS), which "enables persons with hearing disabilities who use American Sign Language to communicate with voice telephone users through video equipment" (Video Relay Services, 2022). In many situations, such as medical appointments, interpreters can be used, which requires the availability of a trained professional. Technology and AI advancements have made a more equal and accessible world, filling the gaps when human interpreters are unavailable. Within the decade, there has been exponential growth in using AI to improve closed captioning. Researchers are studying alternative ways to interpret and store sign language (SL), including transmitting speech into signing avatars. Other forms of technology exist, such as AR glasses and syncing Amazon's Fire TV to hearing aids.
Machine learning, a subcategory of AI, is behind most of our automated technology, including Netflix recommendations and chatbots. It also enables computers to develop intelligence and continue improving without explicit programming. As AI expands, so too do our methods and ease of communication. Developments have been made with voice isolation, speech recognition, and closed captioning.
In 2018, Google developed an AI application that can separate one voice from a crowd (Cimpanu, 2018), allowing for an easier time focusing in a busy setting. In 2024, the University of Washington developed a system called "Target Speech Hearing" (TSH) which combines noise-canceling headphones and AI (Veluri et al.). The wearer hits a button on the side of the headphones and looks at the person they want to focus on, and the microphone detects the specific sound waves from the target to enhance and cancel out other noises (Milne, 2024). This technology can improve the communication experience for the Deaf community, particularly in noisy or crowded environments. While TSH is unavailable commercially, the developers have shared its code publicly, allowing others to build their own (Stake, 2024).
AI improvements can also be found within automatic speech recognition (ASR), the capability for a computer to process voice into text, which is used in everyday devices like Amazon Alexa. For individuals who need hearing or help to speak, interacting with voice technology is difficult (at best). Using the same concept as Amazon Alexa responding to the human voice, technology developments have used AI to create speech recognition software for individuals with "non-standard speech" due to accents, older adults, or speech disabilities (VoiceITT - Inclusive Voice AI, n.d.).
ASR is also used to generate video captions. Moreover, while closed captioning technology was first implemented in the early 1970s (Closed Captioning for the Hearing Impaired: How It Originated | NIST, 2016), automated captions are still frequently inaccurate; often enough, the term "craptions" has been introduced to the lexicon, combining the words "crap" and "captions" (Berube, 2023). Traditional methods can also include manual captioning, which is costly and time-consuming. Implementing AI in the captioning process increases speed and accuracy (The Enterprise World, 2023). AI also improves contextual understanding, ensuring the intended meaning is conveyed. For the DHH, AI integration can assist with capturing audio cues, leading to a more immersive experience.
While improvements to captions and speech are crucial for a more accessible world, it's important to remember that not everyone knows a written language. Often, a deaf person's first language is sign language (SL), which is communicated through hands, body movements, and facial expressions. There are 70 million SL users and more than 300 sign languages worldwide. Some SL users only know one language: their natural sign language from their region. A common misconception is assuming that if a person knows ASL (American Sign Language), they also know English (Florida, 2023). Each language is distinctly different in its grammar, syntax, and vocabulary. However, advancements in machine learning have led to AI signing avatars that can turn speech into SL in real-time, breaking down language barriers and allowing a DHH person to consume content in their first language.
Businesses like Signapse, a UK-based SL translation technology, have built their company on this advancement (Signapse AI | Sign Language Technology, n.d.). Signapse translates written text into BSL (British Sign Language) and ASL, which can be displayed on signage in airports to improve travel. An Israeli startup, CODA, can take speech and interpret it into ASL in real time.
CODA co-founder Shani Bibi said in an interview with NoCamel:
"It's beyond just the alphabet. We can take a sentence and change its essence just with our voice, right? Like I can say 'I want to go home' very calmly or very intensely. It's the same thing in sign language. Our approach is to make the AI understand this in creating the language."
Another area in which AI improves accessibility is cochlear implants (CI), an electronic device that provides a sense of sound by directly stimulating the auditory nerve (Cochlear Implants, 2024). Cochlear Limited, the leading company manufacturing cochlear implants, and Otoconsult, a medical software company (Talibov, n.d.), developed AI-FOX (Fitting to Outcomes eXpert), which uses AI fitting (adjusting) assistance to improve adjusting the CI processor for the recipient of cochlear implants (Cochlear Acquires Exclusive Licence to Artificial Intelligence Assistant to Support Better, Outcomes-based Cochlear Implant Fitting, 2017). A research study compared manual and AI-FOX to newly implanted patients. In each case, AI-FOX was equivalent to or superior to manual fitting. Patients were given the option to continue using either method. The majority of patients decided to use AI-FOX. Those who chose manual fitting were reported due to their comfort (Wathour et al., 2022).
One primary form of accessibility to visual elements is adding closed captions. As mentioned, captions have been present for roughly 50 years, but only recently has the translation shifted from solely manual translations. Along with a shortage of SL interpreters, the same is true for stenocaptioners, who are qualified individuals for translating broadcast television (National Association of Broadcasters, 2014). AI has aided in the workload with broadcast television and other forms of media.
The company XRAI Glass has taken that concept to the next level. XRAI Glass uses an app to translate conversations in real-time. They are worn in XREAL AR smart glasses and generate real-time subtitles of conversations around the wearer. Wearing the glasses allows the user to see subtitles displayed before them. It also saves transcriptions of conversations into the person's smartphone app (Hills, 2022). Mitchell Feldman, co-founder, and Chief Marketing Officer for XRAI, described the product as "Alexa for your eyes." (Captioned Smart Glasses Let Deaf People See, Rewind Conversations, 2022). The glasses have drawbacks, such as background noise, which may compromise accuracy. They may be uncomfortable to wear, especially with wearers of cochlear implants or hearing aids. The glasses must be connected by cable to a smartphone for the captions to operate. There is also the cost; after purchasing AR glasses, there can be a monthly cost for the phone app if premium services are desired. Amazon has also been innovating ways to appeal to the DHH. Their Fire TV can pair with specific models of Cochlear Nucleus, Baha implants, and Starkey hearing aids, allowing for an improved viewing experience (Amazon Fire TV, 2022).
Researchers need to be forever mindful and continually consider the Deaf community's acceptance of new interpreting technologies. According to researchers Shahin and Ismail, acceptance of automated SLMT systems depends on the user's background and comfort with technology. Also, the quality of the sign is a concern for those who mainly rely on sign language, where authenticity, dialect, and linguistic style may be a lower priority to non-daily sign users. To ensure SLMTs are designed to meet the needs of all users, Shahin and Ismail proposed using a participatory design approach to have DHH involved with the development process.
Having DHH people involved in the development process is ideal. Even with the most revolutionary technology and best intentions, hearing people will only partially grasp the needs and desires of their target consumers. For example, as appealing as Amazon's Fire TV connecting with hearing aids, the consumer must be a top priority. One user, Elliott Innes, commented on Medium's Amazon Fire TV article: "Very cool. Can I listen in my hearing aids while my family still listens through TV speakers? If not, this then creates some major issues with having to further isolate oneself" (Amazon Fire TV, 2023). According to the hearing aid manufacturer Starkey, only one device can play audio at a time.
Another consideration is comfortability with technology in general. Even though the DHH were early adopters of the Internet, e-mail, and texting (NAD, n.d.), some are more hesitant to use technology (as with their hearing counterparts), especially with older adults. This is seen in the abovementioned research regarding manual compared to AI-FOX fitting of Cochlear Implants (Wathour, 2022). Within the two groups of participants, 63% from the AI-FOX fitting group and 100% from the manual group decided to continue with the AI-FOX fitting after the study concluded. Comfortability was cited for those who opted for manual fittings going forward. Even with the equal to superior success of AI-FOX, some still decided to keep with traditional adjusting. For a hearing person, this may seem ridiculous. However, this further proves the necessity of including DHH in all research areas, concepts, and trials.
Advancements in AI and the development of accessibility tools have significantly advanced the quality of life for DHH individuals. It also addresses historical barriers in media, communication, and daily interactions. Even with government policies, gaps remain in online content, and there is a need for more human translators and interpreters. AI offers a promising future with speed, accuracy, and cost. More importantly, advancements are increasing inclusively. Inclusivity also involves DHH individuals as products are tested and technology is developed. There is not only one solution. Not everyone can read or understand written English. Not every Deaf person knows sign language. However, together, we can make communication between only, regardless of language or abilities, possible.
Fun fact / future tech…?
In January 2024, Marvel's miniseries "Echo" debuted. Maya Lopez is an Indigenous superhero, Echo, who is also deaf. Lopez is given a contact lens in the show to communicate with her adopted father, Fisk, who wears an earpiece. The lens translates Fisk's spoken words into signs via animated arms. Maya's signs are translated and fed through Fisk's earpiece. This system allows them to communicate without an interpreter. While that tech level has yet to be available, several companies are researching intelligent contact lenses.
References:
Abdallah, M. S., Samaan, G. H., Wadie, A. R., Makhmudov, F., & Cho, Y. I. (2022). Light-weight deep learning techniques with advanced processing for real-time hand gesture recognition. Sensors, 23(1), 2. https://www.mdpi.com/1424-8220/23/1/2
Amazon Fire TV. (2023, April 28). Hearing aid pairing on Fire TV - Amazon Fire TV. Medium. https://amazonfiretv.blog/fire-tv-launches-hearing-aid-pairing-423d56625683
Berube, C. (2023, May 12). Craptions - 99% invisible. 99% Invisible. https://99percentinvisible.org/episode/craptions/
Captioned smart glasses let deaf people see, rewind conversations. (2022, November 18). CBS News. https://www.cbsnews.com/miami/news/captioned-smart-glasses-let-deaf-people-see-rewind-conversations/
Cimpanu, C. (2018, April 23). Google develops AI that can separate voices in a crowd. BleepingComputer. https://www.bleepingcomputer.com/news/technology/google-develops-ai-that-can-separate-voices-in-a-crowd/
Closed Captioning for the Hearing Impaired: How it Originated | NIST. (2016, September 21). NIST. https://www.nist.gov/pml/time-and-frequency-division/time-services/closed-captioning-hearing-impaired-how-it-originated
Closed captioning of internet video programming. (2021, January 27). Federal Communications Commission. https://www.fcc.gov/consumers/guides/captioning-internet-video-programming
Closed captioning on television. (2021, January 27). Federal Communications Commission. https://www.fcc.gov/consumers/guides/closed-captioning-television
Cochlear acquires exclusive licence to Artificial Intelligence assistant to support better, outcomes-based cochlear implant fitting. (2017, May 26). Cochlear. https://www.cochlear.com/es/es/corporate/media/media-releases/2017/2017-05-26-cochlear-acquires-exclusive-licence-to-artificial-intelligence-assistant
Cochlear implants. (2024, June 13). NIDCD. https://www.nidcd.nih.gov/health/cochlear-implants#:~:text=A%20cochlear%20implant%20is%20a,directly%20stimulate%20the%20auditory%20nerve.
Contreras, D. (2023, May 5). How AI can benefit the deaf and hard of hearing community. InnoCaption. https://www.innocaption.com/recentnews/ai-helps-deaf-hard-of-hearing-community
Farooq, U., Rahim, M.S.M., Sabir, N. et al. Advances in machine translation for sign language: approaches, limitations, and challenges. Neural Comput & Applic 33, 14357–14399 (2021). https://doi.org/10.1007/s00521-021-06079-3
Find the best hearing aids for you | Starkey. (n.d.). https://www.starkey.com/
Florida, D. R. (2023, September 20). Debunking 7 Misconceptions about Sign Language, Deaf Culture, and Deafness. https://disabilityrightsflorida.org/blog/entry/7_misconceptions_ASL_Deaf_Culture#:~:text=Misconception%20%232:%20People%20who%20know%20American%20Sign,that%20differ%20from%20written%20and%20spoken%20ones.
Kim, E. (2023, August 13). AI avatar turns speech into sign language in real time. NoCamels. https://nocamels.com/2023/08/ai-avatar-turns-speech-into-sign-language-in-real-time/
Hill, S. (2022, December 21). XRAI Glass wants to subtitle life for the deaf and hard of hearing. WIRED. https://www.wired.com/story/xrai-glass-caption-ar-glasses-first-look/
Law, M. (2022, September 5). AI tech could transform how deaf people experience media. AI Magazine. https://aimagazine.com/articles/ai-tech-could-transform-how-deaf-people-experience-media
Meisfjord, T. (2024, January 14). Is the ASL technology used in ‘Echo’ real? We Got This Covered. https://wegotthiscovered.com/tv/is-the-asl-technology-used-in-echo-real/
Milne, S. (2024, May 23). AI headphones let wearer listen to a single person in a crowd, by looking at them just once. UW News. https://www.washington.edu/news/2024/05/23/ai-headphones-noise-cancelling-target-speech-hearing/
NAD - Technology. (n.d.). https://www.nad.org/resources/technology/
National Association of Broadcasters. (2014). In the Matter of Closed Captioning of Video Programming: Telecommunications for the Deaf and Hard of Hearing, Inc. Petition for Rulemaking. In nab.org (No. PRM11CG). Federal Communications Commission. https://www.nab.org/documents/filings/ClosedCaptioningQualityFNComments070914.pdf
Questions and Answers: DOJ’s final rule under Title III requiring movie theaters to provide closed captioning and audio description. (n.d.). https://archive.ada.gov/regs2016/movie_captioning_qa.html
Quick statistics about hearing, balance, & dizziness. (2024, September 20). NIDCD. https://www.nidcd.nih.gov/health/statistics/quick-statistics-hearing
Samaan, G. H., Wadie, A. R., Attia, A. K., Asaad, A. M., Kamel, A. E., Slim, S. O., ... & Cho, Y. I. (2022). Mediapipe’s landmarks with rnn for dynamic sign language recognition. Electronics, 11(19), 3228. https://www.mdpi.com/2079-9292/11/19/3228
Shahin, N., Ismail, L. From rule-based models to deep learning transformers architectures for natural language processing and sign language translation systems: survey, taxonomy and performance evaluation. Artif Intell Rev 57, 271 (2024). https://doi.org/10.1007/s10462-024-10895-z
Signapse AI | Sign Language Technology. (n.d.). https://www.signapse.ai/
Sonare, B., Padgal, A., Gaikwad, Y., & Patil, A. (2021, May). Video-based sign language translation system using machine learning. In 2021 2nd International Conference for Emerging Technology (INCET) (pp. 1-4). https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9456176
Stake, G. (2024, November 24). AI Headphones: Isolate Voices in a Crowd with Cutting-Edge Technology. Medium. https://medium.com/@gastricstake/ai-headphones-isolate-voices-in-a-crowd-with-cutting-edge-technology-891d8425bb73
Strobel, G., Schoormann, T., Banh, L., & Möller, F. (2023). Artificial Intelligence for Sign Language Translation – A Design Science Research study. Communications of the Association for Information Systems, 53(1), 42–64. https://doi.org/10.17705/1cais.05303
Talibov, Q. (n.d.). Home. Otoconsult. https://otoconsult.com/
The contact lens that could turn you into a camera. (n.d.). theatlantic.com. https://www.theatlantic.com/sponsored/vmware-2017/contact-lens/1634/#:~:text=Patents%20have%20already%20been%20filed,between%20involuntary%20and%20intentional%20blinking.)
The Editors of Encyclopaedia Britannica. (2024, November 15). Sign language | Definition, History, & Facts. Encyclopedia Britannica. https://www.britannica.com/topic/sign-language
The Enterprise World. (2023, November 22). Closed captions and subtitles: How AI helps improve quality? | The Enterprise World. The Enterprise World. https://theenterpriseworld.com/closed-captions-and-subtitles/
Veluri, B., Itani, M., Chen, T., Yoshioka, T., & Gollakota, S. (2024). Look once to hear: target speech hearing with noisy examples. Association for Computer Machinery. https://doi.org/10.1145/3613904.3642057
VoiceITT - Inclusive Voice AI. (n.d.). https://www.voiceitt.com/
Video relay services. (2022, July 18). Federal Communications Commission. https://www.fcc.gov/consumers/guides/video-relay-services
Wathour, J., Govaerts, P. J., Derue, L., Vanderbemden, S., Huaux, H., Lacroix, E., & Deggouj, N. (2022). Prospective comparison between manual and Computer-Assisted (FOX) cochlear implant fitting in newly implanted patients. Ear And Hearing, 44(3), 494–505. https://doi.org/10.1097/aud.0000000000001314
World Health Organization: WHO. (2024, February 2). Deafness and hearing loss. https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss
Zhou, Z., Chen, K., Li, X., Zhang, S., Wu, Y., Zhou, Y., Meng, K., Sun, C., He, Q., Fan, W., Fan, E., Lin, Z., Tan, X., Deng, W., Yang, J., & Chen, J. (2020). Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays. Nature Electronics, 3(9), 571–578. https://doi.org/10.1038/s41928-020-0428-6
Completed NM Lecture-Posts are due Tuesday, Nov. 26th, but if you've read this far and you need a few extra days, I will accept submissions until Friday, Nov. 29th (before 11:59pm) without penalty. Just copy and paste this notice at the end of your list of References.