YouTube Adds Automatic Sound Effect Captions

youtube-adds-automatic-sound-effect-captions photo 1

YouTube now has a sound effect captioning system that can automatically identify music, sound, and laughter in videos.

It's a small but significant addition to the accessibility features of the internet's go-to video streaming site, a Google subsidiary. YouTube has offered automatic captions for dialogue tracks based on Google's voice recognition since 2009. But captioning sound effects is a much harder task, according to Google engineers, and one that's only feasible with the advancements in machine learning that have taken place over the past couple of years.

The problem is not so much a computer's ability to detect and classify things (products like Google Photos already have detection capabilities for images, for instance), but instead the lack of a significant database of sound effects to use when training the neural network that would identify them.

"While labeled ambient sound information is difficult to come by, we were able to generate a large enough dataset for training using weakly labeled data," Google engineer Sourish Chaudhuri wrote in a blog post. The team decided to focus on music, sound, and laughter first since they add meaningful context to a video's dialogue for people who are deaf or hard of hearing.


  • How to Download YouTube VideosHow to Download YouTube Videos

After processing thousands of hours of videos, YouTube now has a trained artificial intelligence algorithm for sound effects, which you can check out in videos like this clip from America's Got Talent (click the CC button to activate captions). The work is not yet done, however, according to Google engineer Noah Wang.

"Future challenges might include adding other common sound classes like ringing, barking and knocking, which present particular problems—for example, with ringing we need to be able to decipher if this is an alarm clock, a door or a phone," he wrote.

Recommended stories

Take Control of Your Smartphone’s Automatic Photo Uploads

Modern smartphones and cloud photo services want to automatically upload every single photo you take to the cloud. This ensures all those photos you take are safely backed up somewhere, but it isn’t ideal for every single photo. Unfortunately, companies like Apple and Google haven’t gotten that

Relax YouTube Adds Automatic Sound Effect Captions stories

More stories