My Co-Founder and I were kicking around the idea of a search engine that would let a person find phrases in a block of audio. We were looking for something that could peer into interviews, podcasts, video lecturesâââthings like that. And if it was done right, you would be able to search through many seasons of a certain TV show and find all the crucial moments like, âYouâre fired!â.
We thought, âThis has to exist, right?â. Surprisingly, no. There wasn’t a company out there that really provided the functionality. Certainly not in a way that was useful to us, at least. So we started hacking together a Google-based transcription to see if we can get a barebones prototype going. In a couple days it was runningâââsearch for something, and most of the time you got it. Huge pat on the back, right?
Speech recognition is hard.
Reality hit us when we noticed a problem. Sometimes the phrase was definitely spokenâyou could hear it plain as day in the audio streamâbut the search missed it. It turns out this is due to the inaccuracy of automatic speech transcription software.
We went on a quest to get our hands on some top quality speech recognition bad-assery. What we were met with was another dose of reality; speech recognition is hard. More evidence emerges when you dig into the current audio research scene and notice that this topic is still a very active topic.
The big tech companies (Google, Microsoft, Apple, etc.) put forth large efforts to get this sort of thing right. Even after that, you generally only get 90% word accuracy. Thatâs on very clean, well recorded speech. With input sources containing conversational speech of questionable qualityâsay, YouTube videosâthe word error rate get pretty bad (more than half is wrong sometimes!).
How Deepgram Works
Want to get more value out of your call center data, build the next game-changing voice feature, or save a lot of money on speech transcription? Learn why Deepgram is the platform to get you there.
Can audio search work well?
This got us wondering, âcan we improve the audio search situation?â. We landed on something we think is pretty goodâ search based on how a phrase sounds, not on the precise spelling in text. We were sure this would provide better results but we werenât sure just how much better it would be.
We dug into research to see if this technique had been tried in a production form. We turned up quite a few papersâmost were not totally relevantâbut a Google academic paper on searching through political speeches from 2008 was striking. âWhat was their method?â, you might wonder. They used just regular old text transcription with no additional incorporation of the way the audio actually sounded. Bummer, right?
Use the way words sound
What we were stumbling across was what speech researchers call keyword search. There is an existing method for doing this called acoustic keyword spotting, but that requires reprocessing the data every time for each and every searchâââthatâs totally impractical. So, yeah, applying this idea is a fairly difficult problem. We didnât really know just how hard at the time, but we know now (eight months of coding our first search engine and starting a company along the way helps beat that into you).
Our API allows you to upload audio and have the server process that audio into a giant searchable lattice. With a lattice like this, you can fuzzily go through the entire audio file for your search phrase in a fraction of a second. There is a huge improvement using this method when compared to the text-based approachâââsearch recall goes from a tepid 45% to a grin-inducing 90%+. Now we have our secret sauce.
A year and a half after writing this post, NVIDIA’s Jensen Huang demonstrated the power of our search on stage at GTC China.
