StyleTTS2 Hindi + Hinglish: Monica

A bilingual Hindi, English and code-mixed Hinglish voice ("Monica") built on StyleTTS2 by superU AI for low-latency conversational voice agents in India.

It reads Devanagari Hindi, English, and sentences that switch between the two mid-sentence, such as "आपका order कल deliver हो जाएगा", with no separate language-identification model.

This is the model described in our paper "Extending StyleTTS2 for Hindi and Bilingual Speech Synthesis: A Word-Level G2P Framework" (superU AI, April 2026).

English-only voice: see superU-AI/style-tts-2-english-anika.

Results

Evaluated on 150 held-out sentences (50 English, 50 Hindi, 50 Hinglish), single NVIDIA A100. Naturalness is predicted MOS from UTMOS (utmos22_strong), not a human listening study. WER uses Whisper large-v3 for English and Hindi, and Shunya Labs zero-stt-hinglish for Hinglish, since monolingual ASR mistranscribes code-switched speech.

Condition System UTMOS ↑ WER (%) ↓
English This model 3.785 2.66
English StyleTTS2 LibriTTS (unmodified) 4.446 0.68
Hindi This model 3.003 11.77
Hindi Indic Parler-TTS 3.116 31.28
Hinglish This model 3.328 10.32
Hinglish Svara-TTS 4.087 13.23

Latency: mean real-time factor of 0.045 on an A100 (unmodified English StyleTTS2: 0.0385), so the bilingual pipeline adds about 1.17x overhead and stays far below real time.

The trade-off: larger autoregressive and Hindi-only models score higher on predicted naturalness, while this model is markedly more intelligible on Hindi and Hinglish and much faster, which is what matters for a voice agent that has to be understood on the first try.

Ablation: word-level vs sentence-level G2P

Same checkpoint and weights; only the text front-end changes.

Condition Sentence-level WER (%) Word-level WER (%)
English 2.91 2.66
Hindi 14.38 11.77
Hinglish 28.91 10.32

Routing a whole mixed sentence through one phonemizer destroys the English words in it. Phonemizing per word is what makes Hinglish work.

How it works

StyleTTS2 is English-only out of the box. Four changes extend it:

  1. Word-level G2P routing. The text front-end tokenizes into words and phonemizes each one separately. A word containing any Devanagari character (U+0900–U+097F) goes to aksharamukha for Hindi IPA; anything else goes to eng_to_ipa. The script check is deterministic and zero-latency, so no language-ID model is needed.
  2. Custom pronunciation dictionary. A case-insensitive lookup in front of G2P overrides pronunciations for proper nouns, loanwords, brand names and symbols such as ₹.
  3. Expanded phoneme vocabulary. Hindi IPA symbols (retroflex ʈ ɖ ɳ ɽ, aspiration ʰ, nasalization and length marks, and Hindi vowels) are appended after the original English symbols, so the pretrained English embeddings keep their indices. Without them, Hindi phonemes map to the padding token and are silently erased.
  4. Multilingual PL-BERT (papercup-ai/multilingual-pl-bert) replaces the English-only PL-BERT, for coherent prosody and duration on Hindi input.

The model then sees an ordinary flat phoneme sequence; the acoustic architecture is unchanged.

Model details

Architecture StyleTTS2 (multispeaker config)
Decoder HiFi-GAN
Base checkpoint StyleTTS2 LibriTTS 2nd-stage
PL-BERT Multilingual PL-BERT
SLM discriminator microsoft/wavlm-base-plus
Sample rate 24 kHz
Style dim 128

Training

Data: 24 hours of synthetic speech: 10 h Hindi (Devanagari text, covering retroflex-heavy words and conjunct consonants), 10 h English (to prevent catastrophic forgetting) and 4 h code-mixed Hinglish, all at 24 kHz, split 80/20 train/validation.

Parameter Value
GPU 1x NVIDIA A100 80GB, about 2 days
Epochs 90 (2nd stage only; stopped on validation-loss convergence)
Batch size 2
Max frame length 300
Learning rate 1e-4 (PL-BERT 1e-5, acoustic modules 1e-5)
Style diffusion starts epoch 10
Joint / SLM adversarial training starts epoch 15
Final validation loss ~0.31 (duration 0.41)

No component was frozen; all modules were fine-tuned jointly.

Files

Hindi-MONIKA/
  config_ft_hindi.yml      # fine-tuning config, needed for inference
  epoch_2nd_000XX.pth      # checkpoints every 5 epochs; epoch_2nd_00089.pth is the final one
  train.log                # full training log

Usage

The weights load with the official StyleTTS2 inference code, using config_ft_hindi.yml and epoch_2nd_00089.pth, with Utils/PLBERT_Multilingual/ populated from papercup-ai/multilingual-pl-bert.

git clone https://github.com/yl4579/StyleTTS2.git
git xet install   # weights are stored with Git Xet: https://hf.co/docs/hub/git-xet
git clone https://huggingface.co/superU-AI/Style-tts-hindi-monica

Important: Hindi and Hinglish input must go through the word-level text front-end and the expanded symbol set described above. The stock StyleTTS2 TextCleaner will drop Hindi phonemes. The front-end code is not included in this repository yet.

Limitations

  • Synthetic training data. All training audio was generated, not recorded by native speakers, and this is the main ceiling on naturalness, especially for Hindi.
  • Romanized Hindi is read as English. "chai" or "roti" in Latin script get English phonemes and a slight English accent. Write Hindi in Devanagari.
  • Automated evaluation only. UTMOS and ASR-based WER are proxies; no human MOS study yet.
  • English quality trade-off. English UTMOS drops from 4.446 to 3.785 versus the unmodified model, though English WER stays low at 2.66%.

Citation

@techreport{superu2026styletts2hindi,
  title       = {Extending StyleTTS2 for Hindi and Bilingual Speech Synthesis: A Word-Level G2P Framework},
  author      = {{superU AI}},
  institution = {superU AI},
  year        = {2026},
  month       = {April}
}

Built on StyleTTS2 (Li et al., NeurIPS 2023).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support