THIS IS HUGE Going multi-lingual becomes very accessible and scalable with @OpenAI whisper’s translate use-case With this, any content can be converted from base language to English 99 base languages across Indian, Russian, Roman, African, Asian, and more are covered ?

Details on data model

The model is trained on a total of 680k hours of audio and text This covers 3 levels of the dataset

– 17% Multilingual Speech Recognition

– 18% Translation data

– 65% English Speech Recognition

The dataset includes 125,000 hours of X→en translation data

The current approach is to do transcription -> translation is basically a form of Chinese whisper Acc to their paper – multilingual and multitask learning allows the translation directly from acoustic features Conceptually OpenAI whisper would beat Chinese whisper anyway

The impact this will have on any form of content is MASSIVE – Creators can now create content in their native language and get the same in English in the form of subs – This will make the content available to the larger world and they can truly go local to global

Huge Impact on Audio & Video Content

Every piece of audio and video content will become index-able which allows search, SEO, better recommendations and more ? For the first time in history, the content world will truly go beyond language and make every native content go global ?

To make this even more ACCESSIBLE we think there needs to be a product layer to this where any creator who is not well equipped to access APIs or even run colab We at @dubverse_ai are in beta for our dubbing use-case. We have some very very exciting news for all creators ?

We will solve the product layer for this Whisper by allowing features like – Easy Youtube Import allowing any video to be imported – A fully functional Editor where any incorrect word can be changed – Subtitle Burn and Export – Even Dub this using a synthetic voice

Release dropping soon on this. Starting with a google colab for this and then allowing access to our product for any video to be transcribed and making English subs available.

? Write to us at [email protected] to get your hands on the new releases ?

?UPDATE ?: We launched Whisper Colab

So here is the google colab demo where anyone can convert any non-English Video to English subs using

Try Now: OpenAI whisper Colab

Install Subtitles for Youtube Chrome Extension to apply custom Subtitles

Watch a demo on how to use this Colab

TLDR; Follow Varshul & Dubverse on Twitter for more OpenAI Whisper + Dubverse updates.