k2-fsa/OmniVoice

High-Quality Voice Cloning TTS for 600+ Languages

Summary

OmniVoice is a Python project for zero-shot text-to-speech with support for more than 600 languages. It can clone a voice from a short reference recording, design a voice from attributes, or choose one automatically, with Python, command-line, web-demo, and training/evaluation workflows.

What it is useful for

OmniVoice fits multilingual speech work where collecting speaker- or language-specific training data is impractical. For voice cloning, an engineer supplies target text, a reference WAV, and its transcription; generation returns a list of NumPy waveform arrays at 24 kHz. A cloned-voice prompt can be saved and reused, avoiding repeated reference loading and automatic transcription in later sessions.

The project also supports voice design through instructions such as female, low pitch, british accent. Documented attributes cover gender, age, pitch, whisper style, English accent, and Chinese dialect. Inline [laughter] and [sigh] tags add non-verbal sounds, while pinyin and uppercase CMU-style pronunciation tokens can correct selected words. The shared API exposes diffusion steps, speaking speed, and fixed duration. For larger jobs, omnivoice-infer-batch reads JSONL and distributes inference across multiple GPUs.

The README calls out meaningful limitations. Voice cloning is the most stable mode; voice design is trained on Chinese and English only and can become unstable for low-resource languages or edge cases. Cross-lingual cloning keeps the reference language’s accent. References should be 3–10 seconds long because longer clips can slow inference and degrade quality. The project also prohibits unauthorized cloning, impersonation, fraud, and other illegal or unethical uses.

How engineers can use it

Start with a fresh virtual environment and install the PyTorch build appropriate for the host. The README’s NVIDIA CUDA 12.8 example is:

pip install torch==2.8.0+cu128 torchaudio==2.8.0+cu128 --extra-index-url https://download.pytorch.org/whl/cu128
pip install omnivoice

It also documents Apple Silicon, Intel Arc XPU, source, editable, and uv installation paths. To inspect the model without writing application code, launch the local UI:

omnivoice-demo --ip 0.0.0.0 --port 8001

For an application, load the model and call the common generation API:

from omnivoice import OmniVoice
import torch

model = OmniVoice.from_pretrained(
    "k2-fsa/OmniVoice", device_map="cuda:0", dtype=torch.float16
)
audio = model.generate(
    text="This is a test for text to speech.",
    ref_audio="ref.wav",
    ref_text="Transcription of the reference audio.",
)

The README also provides omnivoice-infer for single scripted outputs and omnivoice-infer-batch for JSONL workloads. If Hugging Face model downloads fail, it recommends setting HF_ENDPOINT to the mirror documented in the README.