Argmin AIArgmin AI
About usOptimize your agent
RLHFBasic RAGAgentic RAGQuantizationSafety BenchmarksLLM-as-a-JudgeChain-of-ThoughtMulti-Agent SystemsFunction CallingEfficient InferenceHallucination Detection
LLM ArchitecturesAudio-Language ModelsMultimodal BenchmarksDataset Construction

Step-Audio: production-ready unified speech-text model with dual-codebook audio tokens, synthetic TTS data engine, and real-time tool-calls

February 17, 202510 min

Overview

Production Readiness

0.8

Novelty Score

0.75

Cost Impact Score

0.8

Citation Count

0

Authors

Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, Hongyu Zhou, Jianjian Sun, Brian Li, Chengting Feng, Changyi Wan, Hanpeng Hu, Jianchang Wu, Jiangjie Zhen, Ranchen Ming, Song Yuan, Xuelin Zhang, Yu Zhou, Bingxin Li, Buyun Ma, Hongyuan Wang, Kang An, Wei Ji, Wen Li, Xuan Wen, Xiangwen Kong, Yuankai Ma, Yuanwei Liang, Yun Mou, Bahtiyar Ahmidi, Bin Wang, Bo Li, Changxin Miao, Chen Xu, Chenrun Wang, Dapeng Shi, Deshan Sun, Dingyuan Hu, Dula Sai, Enle Liu, Guanzhe Huang, Gulin Yan, Heng Wang, Haonan Jia, Haoyang Zhang, Jiahao Gong, Junjing Guo, Jiashuai Liu, Jiahong Liu, Jie Feng, Jie Wu, Jiaoren Wu, Jie Yang, Jinguo Wang, Jingyang Zhang, Junzhe Lin, Kaixiang Li, Lei Xia, Li Zhou, Liang Zhao, Longlong Gu, Mei Chen, Menglin Wu, Ming Li, Mingxiao Li, Mingliang Li, Mingyao Liang, Na Wang, Nie Hao, Qiling Wu, Qinyuan Tan, Ran Sun, Shuai Shuai, Shaoliang Pang, Shiliang Yang, Shuli Gao, Shanshan Yuan, Siqi Liu, Shihong Deng, Shilei Jiang, Sitong Liu, Tiancheng Cao, Tianyu Wang, Wenjin Deng, Wuxun Xie, Weipeng Ming, Wenqing He, Wen Sun, Xin Han, Xin Huang, Xiaomin Deng, Xiaojia Liu, Xin Wu, Xu Zhao, Yanan Wei, Yanbo Yu, Yang Cao, Yangguang Li, Yangzhen Ma, Yanming Xu, Yaoyu Wang, Yaqiang Shi, Yilei Wang, Yizhuang Zhou, Yinmin Zhong, Yang Zhang, Yaoben Wei, Yu Luo, Yuanwei Lu, Yuhe Yin, Yuchu Luo, Yuanhao Ding, Yuting Yan, Yaqi Dai, Yuxiang Yang, Zhe Xie, Zheng Ge, Zheng Sun, Zhewei Huang, Zhichao Chang, Zhisheng Guan, Zidong Yang, Zili Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu

Links

Abstract / PDF

Why It Matters For Business

Step-Audio cuts voice data costs with a synthetic-data TTS engine and delivers a production-ready speech agent that supports real-time tool calls and fine-grained voice control—useful for voice assistants, contact centers, and localization pipelines.

Summary TLDR

Step-Audio is an open-source, production-oriented speech+text system that unifies recognition, understanding and synthesis in one pipeline. It uses a dual-codebook audio tokenizer (linguistic + semantic tokens), a 130B multi-modal LLM, a distilled 3B TTS model trained on synthetic audio, and runtime engineering (speculative streaming, async tool calls). On internal and public tests it improves instruction-following and ASR/TTS metrics versus open-source baselines, while introducing a synthetic-data TTS engine to reduce data collection costs.

Problem Statement

Open-source speech systems either separate understanding and generation (ASR→LLM→TTS), or attempt end-to-end designs that struggle with controllability, emotion, dialects and tool integration. High-quality multi-style voice data is costly. The field lacks a deployable, controllable open framework that combines robust speech understanding, flexible generation, and real-time tool calling.

Main Contribution

A unified 130B-parameter multi-modal model (Step-Audio) that performs ASR, semantics, dialogue, voice cloning, audio editing and TTS; Step-Audio-Chat released.

A generative data engine and distillation pipeline that synthesizes large-scale TTS data and yields a lightweight Step-Audio-TTS-3B model.

A dual-codebook speech tokenizer (linguistic + semantic tokens) with 2:3 interleaving to balance intelligibility and acoustic quality.

Runtime innovations: speculative response generation (40% committed rate, ~500ms latency reduction) and asynchronous tool calling for real-time voice interactions.

A new benchmark StepEval-Audio-360 covering 9 dimensions (language, emotion, reasoning, instruction following, role-play, singing/RAP, safety) and mixed LLM/ human evaluation.

Key Findings

Dual-codebook tokenization reduces ASR CER on tested ASR sets.

NumbersCER improved from 25.5% → 18.4% (3B ASR ablation)

Step-Audio pretrain achieves top open-source ASR among discrete-token models.

NumbersAverage CER 4.64% (Step-Audio Pretrain) vs 4.32% best hidden-feature model listed

Real-time speculative generation reduced latency and commits a useful fraction of predictions.

Numbers~40% speculative commits; ~500 ms latency saved per response

Instruction-following and chat accuracy improved over open-source baselines on several benchmarks.

NumbersAverage improvement ~9.3 points on open-source benchmarks (claimed)

Reward model and RLHF required explicit mitigation for a 'deaf hacking' reward bias.

NumbersReward model pairwise accuracy 70.51%; mitigation data construction described

Results

ASR average CER (discrete-token models)

ValueStep-Audio Pretrain 4.64%

BaselineWhisper Large-v3 7.28% (listed)

ASR CER (dual-codebook ablation)

Value25.5% → 18.4%

BaselineSingle-codebook semantic

TTS resynthesis (SEED test)

ValueStep-Audio-TTS-3B CER 1.31% (test-zh), WER 2.31% (test-en)

BaselineCosyVoice 3.63% CER, 4.29% WER

Voice chat overall score (GPT-4o eval)

ValueStep-Audio-Chat chat score 4.11 / 5

BaselineGLM4-Voice 3.49 / 5

Who Should Care

Product ManagerMl EngineerCtoFounderEngineering Lead

What To Try In 7 Days

Run the Step-Audio-Chat demo to compare instruction following on your domain-specific prompts.

Distill or fine-tune the released 3B TTS model on a small target-speaker seed to test rapid voice cloning.

Integrate speculative response generation in your voice app and measure perceived latency and extra compute.

Agent Features

Memory

  • Short-term context managed as ASR transcripts (text)
  • Historical audio can be used but text is primary compact storage

Planning

  • Speculative response pre-generation to reduce latency
  • Context manager preserves dialogue history as text

Tool Use

  • Asynchronous tool calling for external API queries
  • Text-threaded tool invocation decoupled from audio rendering

Frameworks

  • PPO RLHF for AQTA
  • Reward model with Bradley-Terry loss

Is Agentic

true

Architectures

  • AQTA (audio input, text output) + TTS pipeline
  • dual-codebook token stream (linguistic + semantic)

Collaboration

  • Role-playing support for multi-turn persona tasks
  • Controller coordinates audio/text subsystems

Optimization Features

Token Efficiency

  • Text-to-audio token compression ratio ~1:14 used for history
  • Interleaving 2:3 linguistic:semantic tokens

Infra Optimization

  • Thousands of H800 GPUs; reported 35% MFU
  • Custom GPU kernels and communication overlap

Model Optimization

  • Dual-codebook tokenization to balance semantic and acoustic fidelity
  • 3B speech decoder separate from 130B LLM

System Optimization

  • StarWeaver RPC-based disaggregated data preprocessing
  • Disaggregated model placement to reduce pipeline bubbles

Training Optimization

  • Audio continual pretraining on Step-1 backbone
  • Stagewise pretrain/posttrain schedule mixing audio/text/image ratios

Inference Optimization

  • Speculative response generation (saves ~500ms per response)
  • Streaming audio tokenizer with fixed-duration segmentation

Reproducibility

Code Urls

Data Urls

Code Available

Data Available

Open Source Status

  • partial

Risks & Boundaries

Limitations

  • Large compute and data requirements; trained on thousands of H800 GPUs.
  • Reliance on synthetic TTS data risks distributional gaps for niche voices.
  • Reward-model bias ('deaf hacking') required manual negative-example mitigation.
  • Some evaluations depend on LLM judges (GPT-4o) and internal human setups, which can bias results.

When Not To Use

  • When strict on-device low-latency constraints prevent use of large remote models.
  • If you require provably curated real human recordings for legal/compliance reasons.
  • If you cannot afford the compute and infrastructure for a 130B backbone.

Failure Modes

  • Reward hacking where the model learns shortcut responses like 'I didn't hear' without improving understanding.
  • Speculative generation waste: extra compute and occasional incorrect committed replies.
  • Vocoder degradation if acoustic tokens are discarded (single-codebook failures).

Core Entities

Models

  • Step-Audio (130B)
  • Step-Audio-Chat
  • Step-Audio-TTS-3B
  • Step-Audio-TTS
  • Step-1 (130B backbone)
  • Step-2 (text LLM used for rewriting)

Metrics

  • CER (Character Error Rate)
  • WER (Word Error Rate)
  • MOS (Mean Opinion Score)
  • Instruction Following (IF)
  • SS (Speaker Similarity score)
  • Factuality
  • Relevance
  • Chat Score

Datasets

  • StepEval-Audio-360
  • SEED TTS
  • Aishell-1
  • Aishell-2
  • Wenetspeech
  • LibriSpeech
  • Llama Question (audio version)
  • Web Questions (audio version)
  • TriviaQA (subset used)
  • ComplexBench (audio version)
  • HSK-6 (listening)

Benchmarks

  • StepEval-Audio-360
  • Llama Question
  • Web Questions
  • TriviaQA
  • ComplexBench
  • HSK-6
  • SEED TTS tests

Context Entities

Models

  • Whisper Large-v3
  • Qwen2-Audio
  • Moshi
  • GLM-4-voice
  • MinMo
  • LUCY
  • CosyVoice / CosyVoice2
  • FireRedTTS
  • MaskGCT

Datasets

  • Public corpora (web text, books, code, images) used in pretraining