whisper base and an 11M param BERT i distilled from a 2B model, both running in the browser to caption videos
Reported by r/LocalLLaMA1 source ↓
A creator says their video-captioning tool runs Whisper Base and an 11-million-parameter BERT-mini tagger in the browser, using WebGPU or WASM depending on the device.
Key points
- The browser tool uses Whisper Base for transcription and BERT-mini to format captions
- The tagger has about 11 million parameters and was distilled using output from LFM3-Omni-2B
- Noisy audio can be sent to a cloud option; the tagger currently supports English only
Details
The tool's creator described it in a post on r/LocalLLaMA as a way to caption Blender tutorial videos without uploading every render to a captioning service. Transcription uses a timestamped Whisper Base model through Transformers.js. The creator says real GPU WebGPU adapters run it in FP32, while other devices use WASM with an 8-bit decoder; software WebGPU adapters also use WASM because they were slower.
A BERT-mini model with about 11 million parameters and eight labels per token handles word emphasis, line breaks, punctuation and capitalization. It runs in 8-bit mode on WASM and is about 12 MB. The creator says it replaced hand-written rules after outperforming them on held-out data, and that its training data was generated with LFM3-Omni-2B on a personal computer.
The creator says Whisper Base is less accurate than larger models, so the tool offers a cloud option for noisy audio. The tagger is English-only.
Written from coverage as of