How AI Is Helping Deaf Gamers Communicate

In 2022, AB-InBev's leading beer in Peru, Pilsen Callao, partnered with Fahrenheit DDB to create ‘E-nterpreters’. This AI-powered technology allows deaf gamers to add a ‘gamer sign language’ bot to their Discord applications [a popular VoIP programme used by gamers], allowing the hearing impared players to understand their teammates’ communication.

This year, the DDB team and Pilsen Callao took ‘E-nterpeters’ a step further. The innovative technology can now turn deaf players' facial gestures into audio messages. By allowing them to easily talk and interact with other players, players can benefit from the essential communication cycle in-game, creating inclusion and improving their gaming experiences. ‘E-nterpreters’ season one allowed deaf gamers to understand; now, season two helps them to be understood.

LBB’s Ben Conway caught up with Sergio Franco, ECD at FahrenheitDDB Peru -an agency part of DDB Latina - to discuss this new evolution of the technology and the responses from deaf gamers so far.

LBB> The last time we spoke, E-nterpreters was a Discord bot that turned audio chat into 'gamer sign language’ to allow deaf gamers to understand their teammates’ communication. Firstly, what was the response to that technology like? Do you have any stats on usage or feedback from deaf gamers?

Sergio> The launch of ‘E-nterpreters 1’ had a significant impact on the local gaming community, with over 22,000 active users before the release of ‘E-nterpreters 2’. This AI helped various users better integrate into online matches, enabling some of them to become part of professional ‘DOTA’ and ‘League of Legends’ teams [two popular ‘multiplayer online battle arena’ (MOBA) games]. A deaf pro player of DOTA, mentioned that this achievement would not have been possible without this development.

However, from the first day we launched, we received various feedback from the deaf gaming community. They asked us to improve the bot's usability, make the platform more user-friendly, and integrate new functionalities for spoken responses from their side. And that's what we did. With the release of ‘E-nterpreters’ season two, we incorporated several solutions to facilitate proper communication between deaf and hearing gamers.

LBB> The ‘E-nterpreters’ technology can now turn deaf gamers' facial gestures into audio messages, allowing them to communicate with their teammates.
How did this development come about?

Sergio> From the beginning of the project, we knew that we would need to implement a plan of continuous improvements, due to the type of artificial intelligence we developed. That's why assessing its performance and needs during the first three months was essential. It was during that time that we identified the necessary improvements. In June 2022, we began evaluating how to enhance our AI by transforming it into a platform with integrated resources and ensuring that they are heard - the most frequent feedback we received - finding a way to give a voice to deaf individuals within the game.

We began the study of facial recognition integration in approximately June 2022. We built a library of gamer gestures that could be recognised in real-time by utilising various research papers and facial capture technologies. Let's remember that deaf individuals take more time to gesture than hearing individuals, so we created a predictive model that anticipated the user's movements and what they wanted to communicate to their teammates. To achieve this, we received feedback and conducted testing at various stages of the project with over 30 deaf gamers involved in the development process. To achieve a high standard of accuracy, our primary reference was Google's ‘Deepmind’, an advanced facial recognition AI with an accuracy of 55% at that time.

LBB> How can people use this new tech? 

Sergio> Anyone with a webcam can use this development; it is free and available to everybody. The text-to-translation bot and the text-to-voice bot are also available as open-source AI within Discord here.

‘E-nterpreters’ season two is now a platform that brings various solutions for deaf gamers to listen and communicate with their teammates or new friends within Discord. Users must download the official software from the Pilsen Callao website to utilise the tool and have a complete experience with facial recognition and sign language translation bots. Today, we have an improved UX platform that guides users from start to finish on how to properly integrate ‘E-nterpreters’ with Discord.

LBB> What were some of the main challenges when developing this product? How did you overcome them?

Sergio> This type of development presents many challenges and as an innovation, we are aware of the desired outcome and the results we aim to achieve. However, things sometimes go differently than planned and different setbacks arise. For example, reducing the software's required power consumption for universal usage on any computer was a challenge. An AI prediction system consumes significant resources, so we had to find solutions to ensure the product's scalability. This led us to reconsider the AI architecture more than 15 times, running models in parallel and addressing challenges such as predicting micro-movements without producing false positive word predictions. This was one of the significant challenges because determining whether it was an ‘error’ or an advancement could only be done once the product was completed and undergoing final testing, potentially leading us to reconsider everything.

Our biggest challenge was interpreting gestures in real-time. Deaf individuals take longer to provide a verbal response, and when combined with internet connection delays, it resulted in a response time of over 10 seconds. This was a significant challenge because we had to utilise almost all the computer's resources to achieve faster processing. Therefore, we decided to develop a predictive model that, by pressing the ‘T’ key (‘talk’), it would open a predetermined time window for the user to speak. In milliseconds, the AI could predict what the user would say based on micro-movement knowledge.

LBB> What have been the responses to this innovation so far? And what have been the most fulfilling moments for you on this project?

Sergio> One of the initial responses we received from the first deaf gamer who tested the ‘E-nterpreters’ update was as follows: "I feel like you finally did it! You're changing the way we can communicate, and it's not just a promise. I'm using it, and I feel heard." The platform is now being used by thousands of deaf gamers seeking to break down the communication barriers that virtuality has imposed on this community. The downloads have already begun, and the response from each user has been very positive.