Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesis

Introduction

Modern large-scale text-to-speech (TTS) models are typically auto-regressive, generating speech representations with an LLM backbone, optionally paired with an audio refinement module. Regardless of architecture, the two minimal conditioning signals are the text to synthesize and the target voice. Given its conditional generative nature, TTS benefits significantly from Classifier-Free Guidance (CFG), which steers generation toward the conditioning by interpolating between conditional and unconditional predictions.

Reliable CFG requires the model to be robust to unconditional generation. In compiled AR inference, this is commonly achieved by replacing dropped conditions with a fixed null vector (usually zeros) rather than dynamic attention masking, which triggers recompilations. But a single fixed null has two limitations in multi-conditioning TTS: it cannot distinguish orthogonal signals (speaker identity vs. linguistic content), and it may lie outside the training distribution, causing instabilities or numerical issues.

We propose replacing the fixed vector with a set of learnable null embeddings, one per conditioning signal. They adapt to the model's distribution, converge to a stable unconditional baseline, and share gradients across conditioning axes: when text is replaced by its null, that null still receives gradients from the speaker-only conditional state, and vice versa.

Method

Model

Our TTS model has two components: (1) an autoregressive GPT 0.6B backbone with lightweight diffusion heads, following the next-token diffusion paradigm; and (2) a causal transformer-based VAE that encodes speech into 64-dim latents $z$ and decodes them back to 48 kHz audio. The backbone $g_\theta$ is conditioned on speaker latents $s$ (a reference mel encoded via a Perceiver encoder) and BPE-compressed text tokens $t$. At each AR step, the last hidden state $h_i$ conditions a diffusion head $\epsilon_\theta$ that predicts the VAE latent $z_i$ through iterative denoising.

Classifier-Free Guidance

We adopt a conditional-baseline CFG formulation in the diffusion heads, steering the prediction away from an unconditional reference:

where the conditional and unconditional hidden states are

and $w \geq 0$ controls the steering strength.

Learnable Null Embedding

Instead of dropping all conditions to a fixed null $\varnothing$, we introduce a learnable null per modality: $\bar{s}$ for the speaker and $\bar{t}$ for the text, giving

During training, each condition is replaced by its learnable null with fixed probability, letting the model learn a stable, in-domain unconditional baseline instead of an arbitrary predefined vector.

Independent Attribute CFG

Because each modality controls a distinct speech attribute, guidance can be decoupled. We define modality-specific unconditional states and extend the formulation with independent weights:

where $w_s \geq 0$ and $w_t \geq 0$ independently control speaker and text guidance.

Experiments & Results

We train two identical models: one using a fixed zero vector for dropped conditions, one using learnable null embeddings for the same steps on the same seeded data, dropping each condition with probability 0.1. Evaluation uses 65 unseen expressive speakers × 2 sentences = 130 samples, benchmarked against open-source SOTA systems.

Objective Metrics

We evaluate on our own set rather than an open benchmark such as Seed-TTS-eval. Public benchmarks tend to have mediocre reference audio and stylistically flat, monotonous speech, and they carry a real risk of overlapping with the training data of the systems being compared. Our speaker references are proprietary and unpublished, so no system here can have trained on them. An extensive description of the evaluation suite is left to future work.

The learnable null wins speaker similarity (SECS, PRO, PMR) and stability (CER 0.9), with higher pitch variation at matched intelligibility. It stays robust for $w \geq 1.0$ where the fixed-zero baseline degrades sharply. Absolute PQ/UTMOS favor the fixed baseline, but quality-correlation to the reference favors learnable nulls, reflecting greater faithfulness rather than lower quality.

Attribute-Control Analysis

Decoupling the guidance exposes interpretable knobs. Text guidance ($w_t$): raising it lowers CER but reduces pitch variation — a stability ↔ expressiveness trade-off, and also raises speech rate. Speaker guidance ($w_s$): drives SECS, PRO, and PMR up, improving reference faithfulness — a similarity ↔ quality trade-off. Empirically, a high $w_s = 1.2$ with a moderate $w_t = 0.4$ gives faithful, expressive speech with controlled rate.

Subjective CMOS

Both learnable variants are preferred over the fixed-zero baseline on naturalness and similarity. Coupled $w$ wins naturalness; tuned ($w_s, w_t$) wins similarity.

Ablation Samples

Hear the progression across the configurations reported above for the same model and training, only the unconditional representation and guidance differ. The last two columns (learnable null) are the proposed learnable null embedding.

Audio Samples

Our proposed model against open-source state-of-the-art TTS. See the objective comparison in Table 1. For sharing speech samples, we use the 4 available open speech references from the Expresso dataset.

No audio wired up yet. Add .mp3 files under samples/ and populate the SAMPLES manifest at the bottom of this file.

Citation

@misc{vecino2026learnableclassifierfreeguidancenull,
      title={Learnable Classifier-Free Guidance Null Embeddings for Enhanced Controllable Speech Synthesis},
      author={Biel Tura Vecino and Yoach Lacombe and Julian Weber and Zbigniew Łatka and Haitong Zhang and Logan Hart and Eren Gölge},
      year={2026},
      eprint={2609.25411},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2609.25411},
}