LLaMA-Omni2: China’s Breakthrough in Real-Time Speech-Enabled Large Language Models

Introducing LLaMA-Omni2: A Modular Speech Language Model

Researchers at the Institute of Computing Technology, Chinese Academy of Sciences, have unveiled LLaMA-Omni2, a novel family of speech-capable large language models (SpeechLMs) now accessible on Hugging Face. This innovative modular framework integrates speech perception, synthesis, and language understanding into a unified, end-to-end pipeline, enabling real-time spoken dialogue with minimal latency while maintaining modular interpretability and reduced training costs.

Architecture Overview

LLaMA-Omni2 models range from 0.5 billion to 14 billion parameters and are built upon the Qwen2.5-Instruct series. The architecture includes several key components:

• Speech Encoder: Uses Whisper-large-v3 to convert speech input into token-level acoustic representations.
• Speech Adapter: Employs a downsampling layer and feed-forward network to align encoder outputs with the language model’s input space.
• Core LLM: The Qwen2.5 models act as the main reasoning engine.
• Streaming TTS Decoder: An autoregressive Transformer generates speech tokens, which are then converted into mel spectrograms using a causal flow matching model inspired by CosyVoice2.

A gating mechanism merges LLM hidden states with textual embeddings before speech synthesis, enhancing the contextual accuracy of generated audio.

Streaming Generation and Latency Optimization

The model implements a read-write scheduling strategy, producing speech tokens in tandem with textual output. For every R tokens generated by the LLM, W speech tokens are synthesized, allowing synchronized acoustic and textual generation. Empirical results show that setting R = 3 and W = 10 strikes an optimal balance, achieving approximately 583 ms latency, a low ASR word error rate (3.26), and high perceptual quality (UTMOS 4.19).

Efficient Training with Limited Data

Training leverages a relatively small dataset of 200,000 multi-turn speech-to-speech dialogue samples synthesized from instruction-following text datasets like Alpaca and UltraChat, using various input voices and a consistent output voice via FishSpeech and CosyVoice2.

Training proceeds in two phases:

• Stage I: Independent optimization of speech-to-text and text-to-speech modules.
• Stage II: Fine-tuning the entire speech-to-speech generation pathway, including gating and autoregressive decoding.

Benchmark Performance

LLaMA-Omni2 outperforms previous models such as GLM-4-Voice and its predecessor LLaMA-Omni in spoken question answering and speech instruction tasks, with performance scaling alongside model size. The 14B parameter variant notably surpasses all baselines despite using significantly less training data.

Component Impact and Insights

• Gate Fusion Module: Crucial for maintaining alignment between textual and contextual signals; its removal degrades performance.
• TTS Pretraining: Initializing from Qwen2.5 and fine-tuning in streaming mode yields superior results.
• Read/Write Ratios: Adjustments affect latency and quality trade-offs.

Multi-turn dialogue data proves more effective than single-turn samples, with performance stabilizing around 200K samples.

LLaMA-Omni2 sets a new standard for real-time, low-latency spoken interaction with large language models, demonstrating that high-quality speech capabilities are achievable without massive speech corpus pretraining. This paves the way for practical, real-time speech applications integrating advanced language understanding and synthesis.

For more details, check out the paper, model on Hugging Face, and GitHub repository.