PolyWhisper is our answer to a simple question: why does Whisper hallucinate so badly on Indian languages? The answer led us to build a dual LoRA expert system that reduces ASR hallucination by 20× while maintaining Whisper's zero-shot versatility.
The Problem with Whisper on Indic Languages
OpenAI's Whisper is remarkable for English and major European languages. But on Hindi, Tamil, Telugu, Bengali, and Marathi, it suffers from severe hallucination — generating confident but completely wrong transcriptions, especially on code-switched Hinglish speech.
Our Approach: Dual LoRA Experts
Instead of full fine-tuning (which would destroy Whisper's multilingual generality), we freeze the Whisper-Small encoder and attach language-specific LoRA expert adapters. A lightweight 33K-parameter router dynamically selects the appropriate expert based on the input audio segment.
Results
PolyWhisper achieves 20× hallucination reduction on Indic language benchmarks while maintaining competitive WER (Word Error Rate) against fully fine-tuned baselines. The model is ONNX-quantised for edge deployment and available on HuggingFace under Apache 2.0.