Google Video Calls Can Use Sign Language

Google illustrations (doc. Antara)

JAKARTA - Recent research from Google now allows sign language to be changed in language in video calls. This will certainly make it easier for people with disabilities.

Google developed a real-time sign language detection engine that can tell when someone started with sign language and when they finished (not just detect sound).

The system is possible with very little latency, according to the TechCrunch report, quoted Monday. Latency has a big impact on sign language detection so that videos are not delayed or quality degraded. So, Google makes a system that is lightweight and reliable.

The system first runs a video model called PoseNet, which estimates body and limb position.

This simplified visual information (basically with lines) is sent to a system that is trained in pose data from videos of people using German Sign Language, and compares the live image to the resulting line views of the movement.

This simple process already results in 80 percent accuracy in predicting whether someone is using sign language or not, and with some additional optimizations it gets up to 91.5 percent accuracy.

Compared to how the detection of "active speech" on most video calls can only tell if someone is talking, can't even distinguish a cough, the numbers are pretty good.

Google video translates sign language (Google doc)

To work without adding the "someone is using sign language" signal to the call, the system uses a clever trick, by using a virtual audio source to produce a 20kHz tone, which is outside the range of human hearing, but noticed by the computer audio system.

This signal is generated whenever the person uses sign language, making the speech detection algorithm "think" that they are speaking aloud. Currently, the system is still in the demo stage.