Submitted by Batuhan_Y t3_y2cpjc in MachineLearning
All you have to do is input a YouTube video link and get a video with subtitles (alongside with .txt, .vtt, .srt files).
Whisper can translate 98 different languages to English. If you want to give it a try;
Link of the app: https://huggingface.co/spaces/BatuhanYilmaz/Auto-Subtitled-Video-Generator
​
The video is age restricted, I assume pytube can't reach that video, so there is no input to the model.
Can you try with a different video?
Great news! Different error this time:
https://www.youtube.com/watch?v=PlUvLBRwLbw
RuntimeError: The size of tensor a (316) must match the size of tensor b (3) at non-singleton dimension 3Traceback:File "/home/user/.local/lib/python3.8/site-packages/streamlit/scriptrunner/script_runner.py", line 554, in _run_scriptexec(code, module.__dict__)File "/home/user/app/app.py", line 258, in <module>main()File "/home/user/app/app.py", line 138, in mainresults = inference(link, loaded_model, task)File "/home/user/.local/lib/python3.8/site-packages/streamlit/legacy_caching/caching.py", line 573, in wrapped_funcreturn get_or_create_cached_value()File "/home/user/.local/lib/python3.8/site-packages/streamlit/legacy_caching/caching.py", line 557, in get_or_create_cached_valuereturn_value = func(*args, **kwargs)File "/home/user/app/app.py", line 81, in inferenceresults = loaded_model.transcribe(path, **options)File "/home/user/.local/lib/python3.8/site-packages/whisper/transcribe.py", line 181, in transcriberesult: DecodingResult = decode_with_fallback(segment)File "/home/user/.local/lib/python3.8/site-packages/whisper/transcribe.py", line 117, in decode_with_fallbackdecode_result = model.decode(segment, options)File "/home/user/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_contextreturn func(*args, **kwargs)File "/home/user/.local/lib/python3.8/site-packages/whisper/decoding.py", line 701, in decoderesult = DecodingTask(model, options).run(mel)File "/home/user/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_contextreturn func(*args, **kwargs)File "/home/user/.local/lib/python3.8/site-packages/whisper/decoding.py", line 633, in runtokens, sum_logprobs, no_speech_probs = self._main_loop(audio_features, tokens)File "/home/user/.local/lib/python3.8/site-packages/whisper/decoding.py", line 588, in _main_looplogits = self.inference.logits(tokens, audio_features)File "/home/user/.local/lib/python3.8/site-packages/whisper/decoding.py", line 145, in logitsreturn self.model.decoder(tokens, audio_features, kv_cache=self.kv_cache)File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_implreturn forward_call(*input, **kwargs)File "/home/user/.local/lib/python3.8/site-packages/whisper/model.py", line 189, in forwardx = block(x, xa, mask=self.mask, kv_cache=kv_cache)File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_implreturn forward_call(*input, **kwargs)File "/home/user/.local/lib/python3.8/site-packages/whisper/model.py", line 124, in forwardx = x + self.attn(self.attn_ln(x), mask=mask, kv_cache=kv_cache)File "/home/user/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_implreturn forward_call(*input, **kwargs)File "/home/user/.local/lib/python3.8/site-packages/whisper/model.py", line 85, in forwardwv = self.qkv_attention(q, k, v, mask)File "/home/user/.local/lib/python3.8/site-packages/whisper/model.py", line 97, in qkv_attentionqk = qk + mask[:n_ctx, :n_ctx]
Ran into same error now. Changing the model size worked somehow. I'm not sure what the problem is.
I tried it on my localhost many times. I think it might be related to huggingface spaces.
>Ran into same error now. Changing the model size worked somehow. I'm not sure what the problem is.
I tried it on my localhost many times. I think it might be related to huggingface spaces.
Interesting. What model size did you end up using? Did it work eventually on the huggingface spaces?
Yes it worked on spaces. I've tried tiny, base and small models. Tried with 3 to 5 min. long videos.
Definitely needs more development for robustness but otherwise a great start! I just got it working on a 1:30 video.
You're right. Also needs more processing power. I'm still working on the app, trying to improve.
Thank you for your interest.
This would be really useful as a Reddit bot to translate Reddit videos, like the Ukranian ones. The bot could probably rehost a subtitled video on its own subreddit and leave a link in the comments to the original, along with the transcribed text.
That's a nice idea. I've seen a twitter bot doing that. I don't know how to create a reddit or twitter bot atm. I have free time so I can work on it.
If anyone wants to collaborate, I'd be more than happy.
I am not familiar with huggingface spaces, but do you intend to share source on github or colab?
Where does using streamlit come into hugging space.. like how do you integrate both?
You need to build the interface and select the streamlit option when creating a huggingface space. It takes care of the rest.
Sure. Here’s the link of the repo:
https://github.com/BatuhanYilmaz26/Auto-Subtitled-Video-Generator
Did you manage to solve this error? I'm getting it too. My video is unlisted and lasts 10:41 minutes.
I tried switching to the large model but it takes forever, it's still running.
HuggingFace gives 16GB RAM and 8 CPU cores for the spaces, when the app exceeds 16GB it crashes. In order to avoid that kind of situation I need deploy this model to one of the cloud providers with multiple GPU's. I can't afford that kind of heavy processing model at the moment. But I rebuild the hosted space so it works until it exceeds 16GB again :D
Small model works fine most of the time you can try using it.
I put it up on HuggingFace spaces for demonstration purposes.
Okay, thanks! I'd run it locally but it looks like it would be a bit much for my computer.
You can try on hf spaces, its up and running now. If you face an error keep clicking on Transcribe/Translate button (it worked for me on hf spaces :D, no errors on localhost).
Okay, thanks! Very useful app btw. I'd be nice if I could somehow replace the autogenerated YouTube subtitles by those. They're much better.
Do you mean it would be nice to edit the transcript and re-generate the video with that? If so, I actually tried to implement it 2 days ago, but couldn't make it work, I'll be working on that.
Hello,
Can I use this on a non-YouTube link (for example, a video uploaded on my own web server)?
Yes you can but you need to edit the code in order to do that.
Thanks. Would be great if the app has an option to add other links
It varies from site to site. I’m thinking of adding a file uploader.
i have tryed this video with the largest and the normal model, the transcrip feature has mistakes in all the lines, for the record, bouth people on the video are perfect neutral spanish people
https://www.youtube.com/watch?v=yiP2N7MIvAw
No but I can add a page where you can upload a video and a transcript as .srt file and generate a video with subtitles.
can Whisper AI do that, take our own transcription and sync it to audio as srt or vtt?
Not whisper but ffmpeg can do that.
cheecheepong t1_is32h30 wrote
Running into this issue:
https://imgur.com/a/sbhX9Xg
I tried running this video on it: https://www.youtube.com/watch?v=Mwt35SEeR9w