Apple confirms: speech diffusion models scale like autoregressive ones

Apple ML Research has proven that neural networks for speech scale as predictably as text models thanks to a new pJSD metric, though speech systems still lag behind text systems in quality.

Why speech models lag behind text models

Language models working directly with sound without intermediate text representation — so-called speech-only SLM — currently significantly lag behind text and multimodal systems. This seems paradoxical: voice interfaces are in demand, audio data is abundant, yet progress is slow.

The dominant approach — discrete autoregressive SLM (AR SLM) — first converts continuous audio signals into discrete tokens, similar to how text models work with words. The problem is that speech is fundamentally different from written language: discretization inevitably loses acoustic details carrying semantic meaning. To compensate for these losses, AR models require enormous computational resources and huge training datasets — making them expensive and difficult to scale.

How continuous diffusion circumvents limitations

Continuous diffusion (CD) offers a radically different approach: work directly with continuous audio signal representations, completely bypassing discretization. This eliminates information loss at the input and potentially makes training more efficient.

However, a key question remained: do CD models obey scaling laws? These very laws have become the foundation of modern large language models — predicting how quality grows as parameters increase and data volume expands. Without predictable scaling, investing major resources in an architecture is risky: there's no guarantee that investments will pay off.

Apple investigated both types of metrics. Key findings:

• CD SLM demonstrate clear scaling laws for validation loss
• The pJSD metric (linguistic quality) decreases predictably as computation increases
• An optimal ratio between number of tokens and number of parameters was identified — a speech analog of Chinchilla's law for text LLMs
• The scaling behavior of CD models closely corresponds to that of AR models

Why we need the new pJSD metric

Standard metrics — primarily loss — measure how accurately the model reproduces the training distribution, but poorly reflect whether speech is substantive from a linguistic perspective. Apple introduced phoneme Jensen-Shannon divergence (pJSD).

The metric calculates the divergence between the distribution of phonemes in speech generated by the model and real human speech. The closer the value to zero, the more "human-like" the speech is in a linguistic sense. Critically, pJSD is applicable to both architectures — AR and CD — allowing fair comparison of fundamentally different approaches on a single scale.

"To quantitatively assess the linguistic quality of speech language models, we introduce the pJSD metric —

Jensen-Shannon divergence at the phoneme level," states the Apple ML Research paper.

What this means

Apple's research removes a key uncertainty in speech AI: continuous diffusion scales predictably and is comparable to the autoregressive approach. This gives researchers and companies justification for long-term investment in CD architecture. Next-generation speech AI systems will be able to work without intermediate text — potentially making voice interaction more natural and computationally efficient.

What is a speech neural network?

This is a language model working directly with sound without intermediate text representation — a speech-only language model (speech-only SLM).