Google’s Live Captions could be revolutionary (one day)

Image: raymond wong / mashable

Google went out of its way during I/O to emphasize the ways it’s trying to make its products more accessible. One project that could have a huge impact: Live Captions, a new feature coming to Android Q.

The feature automatically surfaces captions to any video or audio content on your phone in real time, regardless of whether you’re online or off. 

That would make just about all video and audio content — including video calls, audio messages, and podcasts — accessible to people who are deaf or hard of hearing. It would also benefit everyone who occasionally watches videos with the sound turned off. That this can happen instantly on your phone without a data connection is all the more impressive — provided Google can pull it off.

Google hasn’t given it a launch date, though it says the feature will be available as part of Android Q. But I got the chance to try it out and I walked away mostly impressed, though it’s clear there are some issues that need to be addressed.

When the feature rolls out, the idea is for Live Captions to be available throughout your phone regardless of what app you’re using, so you can call them up any time there’s audio playing. Whether you’re watching a video or listening to a podcast, all you do is press the volume down key to bring up the Live Captions button.

Image: raymond wong / mashable

One of the most impressive aspects of the technology is that captions begin appearing almost immediately, even when you’re offline. (Like many other new features Google announced at I/O, Live Captions does all of its processing on the device itself, so no connection is needed.) My entire demo was on a phone that was in airplane mode and not connected to WiFi. 

As for the actual captioning, well, it’s clearly still very much a work in progress. When my colleague Raymond Wong and I tried it out with videos of us speaking, we got mixed results.

When we were in a quiet room, and kept the phone’s microphone close, the results were damn near flawless. Though the captioning system struggled with a word here or there (“hashtag” seemed to throw it off), its accuracy was otherwise impressive. 

But, move the phone a few inches away from the speaker, and things were much less consistent. It misidentified a number of words even though the audio in the video was clear. These issues were even more pronounced when we tried out Live Captions in a room with a lot of background noise. In one instance, the captioning system generated significantly different text for the same video within the span of a few seconds.  

A Google spokesperson reiterated that it’s still early for Live Captions and that the team behind it is still making improvements. That’s true of any pre-release software, but it will be especially important for Google to address the issues we experienced. 

Not all audio is perfectly crisp and clear. Some people mumble or have accents. We share video from noisy places or with many people speaking. While it’s understandable these issues would be more difficult to address, it’s important they are fixed, as it’s more representative of the way people actually use their phones. If Live Captions can’t work reliably in these scenarios, it will be much less useful and risks becoming an afterthought.

Still, if Google can make it reliable, Live Captions would make a dramatic difference for the people who need it most. Let’s hope they get it right.