How Fahrenheit DDB and Pilsen Callao Used AI to Help Deaf Gamers

In the real world, communication solutions for the hearing impaired are a relatively common occurrence. However, in the video game industry, there are very few experiences that take into account the necessities and difficulties of deaf gamers. While they may be able to talk via text chat, these 300 million deaf gamers around the world are left out of the verbal communication that most modern online games use as the predominant method to talk.

In response to this, last year, beer brand Pilsen Callao teamed up with DDB Latina’s Peruvian agencies, Fahrenheit DDB and FTW DDB Peru, to develop ‘The E-nterpreters’ - AI bots that were programmed to translate in-game conversations into gamer sign language in real-time.

The project took nearly a year of development to create an algorithm capable of identifying the voice of every player, dividing it by channels and integrating it into a visual response. To discuss how the project was brought to life, LBB’s Ben Conway caught up with Fahrenheit DDB Peru’s ECD, Sergio Franco Tosso.

In 2022, Pilsen Callao - a beer brand in AB InBev’s portfolio - discovered a significant blind spot in the online gaming world: the fact that people with hearing impairments can’t hear or reply to other players’ verbal speech which serves as the main communication in online gaming. “That’s why we developed ‘The E-nterpreters’ in 2022,” says Sergio, “AI bots trained to translate real-time conversations into gamer sign language. Now we want to allow them to reply to the discussions by giving them a voice.”

Working closely with deaf communities and several deaf gamer groups, the DDB team identified deaf gamers’ needs and difficulties through a series of interviews. To name a few issues, they discovered that the gamers suffered from embarrassment and bullying, as well as the expected comprehension difficulties.

Explaining that Pilsen Callao is known in Peru as “the beer that supports equal friendship… without racial, gender, or physical differences”, the ECD says that for over 10 years, the brand has worked steadily to promote ideas where friends have a good time in an equal environment. 

He adds, “Deaf gamers find it hard to socialise and are constantly victims of bullying because of their hearing and speaking difficulties. Now, it’s time to continue the E-nterpreters' work and make online gaming a friendlier environment.”

The idea for a gamer sign language bot came to fruition when the creatives recognised that text chat communication is simply too slow and obtrusive in the “brutally fast” pace of online gaming. “We needed an agile tech that helps deaf gamers introduce themselves into conversations at the game’s speed.”

Working for 10 months with the social tech company Cirsys, they coded and trained an AI that combines the power of Google’s speech-to-text with Discord’s voice channel selector to convert spoken words into sign language. To make the translations precise and relevant to gamers, the team worked with deaf gamers to create a universal language that integrates the native gaming vocabulary. Hosted on Discord, the world’s largest platform for interacting among gamers, the E-nterpreters feature can link to any video game, running automatically when Discord is open and translating every conversation in real-time.

“The acceptance of the deaf gamer community was tremendous,” says Sergio. “It helped them integrate into a world where they couldn’t socialise with players that could hear. It’s become a helpful tool for meeting new friends and strengthening the bond between gamers. E-nterpreters also helped a deaf gamer to reach Immortal Level [the highest rank] in ‘Dota 2’ and is also helping other deaf gamers to become pro gamers.”

After overcoming several challenges during development - combining voice recognition by channel with voice identification, and reducing the translation time to around two seconds - the E-nterpreters feature is helping deaf gamers to ‘hear’ what their teammates are saying for the first time. But the next step, Sergio says, will be to help deaf gamers “complete the circle of communication” and respond without text chat.

“We are working on E-nterpreters season two, which seeks to make it possible for deaf people to respond using their voice through facial recognition, within Discord, and no longer rely on written responses.”