Problem Solved: The Virtual Human Protecting Japan's Deaf Community from Natural Disasters

---

"This project began when NHK Enterprises and NHK Global Media sought to revolutionise accessibility by creating a JSL concierge. This plan was part of the company’s continual investment in technology and commitment to better serve the Deaf community, with the goal to create an application that could receive text input and output a sign language video in near real time. While KIKI first appeared on the scene to provide important public safety information during emergencies, other initial engagements include sign language interpretation for museum exhibit guides, public announcements and more."

 

The Problem

  

"Japan is one of the most earthquake prone countries in the world. That’s why it has one of the most advanced early warning systems in the world designed to provide life-saving information in multiple languages – except sign language. Despite technological advancements and a sizable population of over 300,000 Deaf individuals in the country, there is a persistent lack of representation and accessibility for sign language users. This inadequacy is especially pronounced during emergencies, where critical information dissemination often excludes JSL users, leaving them vulnerable to safety risks. Despite Japan's technological advancements, including in disaster preparedness, scalable solutions to ensure inclusivity and accessibility for the Deaf community – particularly during emergencies – had yet to be effectively implemented."

 

 

"NHK Enterprises Inc. and NHK Global Media Services began by building the first-ever repository for sign language motion in partnership with the Tsukuba University of Technology, tapping the talent of our global partner Media.Monks. This involved the capture of sign language forms. Once captured, these forms were introduced to a first-of-its-kind translation engine. This engine considers emotion, pausing, rhythm and the unique syntax of sign language to enable a direct translation from plain text to lifelike movement.

 

"The ideation and development of KIKI involved a meticulous two-year process, collaborating with sign language experts and Tsukuba University of Technology to ensure linguistic authenticity and technical precision. 

"In spoken language, things like tone of voice greatly influence the way people interpret what’s said. Japanese Sign Language similarly relies on interpretation of motion, conveying a sense of tone that other methods of communication like subtitles fail to emulate. This informed the approach to ensuring KIKI could make the same expressive movements as a live human, able to implicitly convey, for example, seriousness and urgency in an emergency, or a feeling of fun and comfort in other scenarios. In designing a virtual interpreter,the team sought to ensure KIKI could provide vivid expression on par with a live human.

  

"NHK Enterprises Inc. and NHK Global Media Services have been trying to realise an inclusive society for a while. Thus far, creating a solution for the text-to-sign pipeline has been difficult due to its costly and labour-intensive nature. Working alongside Media.Monks in a collaborative effort with sign language experts and universities, the team explored innovative solutions to overcome these historic challenges. Recognising the complexity of the task, we drew inspiration from a collective commitment to inclusivity and prioritised meticulous planning to ensure a comprehensive approach. This stems from a shared desire to create meaningful change and address the barriers faced by the Deaf community, driving us to pursue creativity, innovation, and breakthrough thinking."

 

Prototype & Design

  

"The idea behind KIKI was to develop the world's first text-to-sign translation engine, revolutionising communication for the Deaf community. Drawing upon advanced technology and collaboration with sign language experts, KIKI seamlessly translates plain text into lifelike Japanese Sign Language (JSL) in real time. Through meticulous strategic planning, we made sure that the approach incorporated diverse perspectives and expertise. This included rigorous testing and experimentation to refine the solution and ensure its effectiveness in real-world scenarios. 

"With that, the idea was to create a dynamic virtual avatar capable of accurately conveying the nuances of sign language, including facial expressions and emotional cues. This innovative approach ensures rapid and effective communication during emergencies and beyond. By combining cutting-edge technology with linguistic expertise, KIKI represents a breakthrough in accessibility, empowering the Deaf community with vital information and fostering inclusivity on a global scale.

 

"In building the text-to-sign engine that powers KIKI, we had to navigate a handful of technical challenges, and worked closely with our internal team and Media.Monks. A speech-to-text (STT) module converts spoken language into text, though STT can sometimes turn up errors in its transcription. We used sentence analysis to correct any transcription errors, including a feature that allows for manual intervention for flagged words.

"Additionally, sentiment analysis was conducted to assess the emotional context and nuances of the speech, which aids in selecting appropriate facial expressions and movements for KIKI—pulling from a comprehensive database of over 8,000 motion capture animations, each corresponding to specific signed words. Using NHK Science and Technology Research Lab’s technology, these animations are integrated to reflect actions and emotions conveyed in the sentence, such as gifting or directional movement, by blending different animations for arm movements and body postures. Finally, the animation execution component generates a real-time sequence of animations that accurately represents the signed equivalent of the original spoken text.

"Together, we built the world’s first text-to-sign engine. I’m incredibly proud of our collaboration and the result, which applied existing technology to create innovative solutions, putting established tech to use in novel ways. Most importantly, we used technology to design an experience for a more accessible and inclusive future. 

  

"KIKI stands at the forefront of accessibility innovation – the world's first text-to-sign engine. This groundbreaking technology has brought the conversation of sign language inclusivity to the forefront of our minds in a way that was never thought possible before.

"We are already working on future developments for KIKI, including the use of AI and machine learning to make the experience more efficient, lifelike and scalable. For example, we’re excited about the opportunity to expand KIKI to interpret new languages like British and American Sign Language, and perhaps even a future where KIKI can be used to educate more people seeking to learn sign language. 

"KIKI has successfully been integrated as the first-ever virtual ambassador at the 2025 Deaflympics in Tokyo, an important step in our ultimate goal to make the world a more inclusive and accessible place for the Deaf community.

"NHK Enterprises and NHK Global Media’s integration of KIKI into Japan's emergency broadcast system ensures vital communication for over 300,000 Deaf individuals during crises. Since launch, KIKI has provided life-saving data at every emergency broadcast, most notably during the 7.6 magnitude earthquake on January 1st, 2024. KIKI continues to demonstrate the need, ease and use of real-time sign services, and the repository of over 8,000 sign language forms, captured in collaboration with the Tsukuba University of Technology, has the potential to expand even further – meaning KIKI and other virtual interpreters can continue to become even more communicative with the Deaf community."