Inside Frankenstein's lab: a full-duplex model at home • Nick Wall

A live session, unedited24 s · me, and the model on one GPU

me
Should I wear a sweater tomorrow? It's 65 degrees outside.
model
You should probably skip the sweater since 65—stops
me
Sorry, I misread. It's 61.talks over it
model
61 is even cooler, so a light sweater is a smart call. It will keep you cozy without being too heavy.

That’s a 12-billion-parameter full-duplex model I trained at Maple. It’s built out of parts from off-the-shelf models, sewn together with LoRA adapters and some stubbornness, on a minuscule GPU budget. LoRA, low-rank adaptation, trains small low-rank updates to a model’s weight matrices while the original weights stay frozen, so fine-tuning a 12B model fits on a single GPU (Hu et al., 2021). Getting it to generate speech sent me down a rabbit hole: a clean voice, the base model’s knowledge and natural conversation kept pulling against each other, and fixing one often broke another.

It’s a cheap testbed I built as part of our full-duplex work, and in this post, I’ll show how we used it to experiment with full-duplex models cheaply, which let us iterate quickly on RL methods, data recipes and architecture changes before committing to costlier, more polished ones. Data is where most of a model’s behavior comes from, and a model I could retrain in under an hour made that loop fast enough to learn things before scaling up. Instead of training a model from scratch, I took a text LLM that already hears audio, taught it to run on an 80 ms clock and decide every frame whether to listen or speak, and gave it a separate pretrained speech model as its voice.

I’ll get into how it works, why my first design, with Gemma making its own audio, gave way to the split, and why the split is a good fit for experimenting and points toward a more coherent end-to-end design. Then how it learned to converse, how reinforcement learning taught it when to talk, and finally what it still gets wrong and where it goes next. I’m not going to cover delegation here, which is how we actually build voice agents at Maple. That one deserves its own post.

The small budget was kind of the point. Most full-duplex work so far has come from labs with far more compute than most people have, which makes the whole direction look out of reach for small teams and independent researchers. I want this post, and the code and weights released with it, to be a resource for anyone who wants to explore new directions in voice and new paradigms for how people and AI interact, without worrying about immense compute requirements, or about the daunting task of building a complete end-to-end system with nowhere obvious to start.

Why full duplex

First, why I care about full duplex at all.

Why voice is still a cascade

Almost every voice product you’ve talked to is a cascade: speech-to-text, then an LLM, then text-to-speech. It’s so pervasive that it’s almost taken for granted, and there’s a good reason for that. Each piece is improving at a ridiculous pace on its own, and anyone can stitch together the best STT, the best LLM and the best TTS of the month. That convenience is also the problem. When every part gets better for free, nobody has to rethink the whole, and so for a long time nobody did.

The cost of that is turn-taking. A cascade needs some way to decide when you’ve finished talking, so it bolts on voice activity detection (VAD) and waits for silence. It can’t overlap with you, it can’t say “mhm” while you’re mid-thought, and every pause forces a re-evaluation of whether you’re thinking or done. Semantic turn detection, models that read the words or the audio to judge whether you’ve finished, has gotten a lot better. OpenAI’s Realtime API has a semantic VAD mode, and LiveKit’s turn detector and Pipecat’s Smart Turn are open. They still only decide at a pause, though, and a wrong guess either cuts you off or adds dead air. They’re also still a gate in front of the model: it only gets to talk when the detector says so, so it can’t backchannel, start early on an obvious answer, or hear that you’ve changed your mind. It’s not very bitter-lesson-pilled either, and it’s a task even people get wrong at times, so a small model bolted on in front stands no chance. That’s a limit of the architecture, and of turn-based LLMs in general (I’ll write more about that another time). It also blocks the products I actually want to build: an assistant that sees what you see, hears what you hear, and speaks up when it has something useful to say, not just when it’s spoken to.

Cascade

Each stage waits for the one before it, and nothing starts until the caller is silent

callerCan we move it……to Friday?Wait, Saturday

pipelineVADSTTLLMTTS

modeldead airtalks over the caller

Full duplex

One model hears every 80 ms frame and decides each frame whether to speak

callerCan we move it……to Friday?Wait, Saturday

modelmhmFriday works…Saturday too

Figure 1: The same exchange through a cascade and through a full-duplex model. The caller pauses mid-sentence, then changes their mind. The cascade can't start until the caller goes quiet, then talks over the correction. The full-duplex model backchannels during the pause, answers as the caller finishes and stops when interrupted.

Full duplex removes that limit (Figure 1): one model that listens and speaks at the same time, and decides moment to moment whether to talk. I think it’s the next big step in voice interaction.

From cascades to full duplex

“Voice model” covers a few different designs, and the names get used loosely, so it’s worth being precise about what exactly we’re talking about here.

	Hears the audio itself	Decides when to talk	Listens while it talks	Voice comes from
Cascade	No, only a transcript	A turn detector	No	A separate TTS model, from text
Speech to speech, turn-based	Yes	A turn detector	No	The model itself
Full duplex	Yes	The model, continuously	Yes	The model itself
The model in this post	Yes	The model, continuously	Yes	A separate speech model, from the model’s text

A cascade is the pipeline above: speech-to-text, an LLM, text-to-speech, with a turn detector deciding when you’ve finished. The LLM only ever sees a transcript.

Speech-to-speech models take audio in and produce audio out, so they hear how something was said and can answer in kind. Most of them are still turn-based. OpenAI’s Realtime API and Gemini Live put a turn detector in front of the model, which answers when the detector decides you’re done and gets cut off when it hears you start again. That recovers what the transcript throws away, but the turn-taking is the cascade’s.

Full-duplex models drop the rigidity of sequential turns. They hear every frame, including while they’re talking, and decide every frame whether to speak; Moshi, PersonaPlex and Nemotron VoiceChat work this way. It’s the bitter lesson applied to conversation: instead of hand-building turn-taking out of silence thresholds and turn detectors, you let the model learn it from data. Moshi and PersonaPlex are also speech-native: the audio comes out of the same model that’s following the conversation, so the voice’s tone and emphasis can carry what the model understood.

The model in this post is full duplex in its timing but not in its voice. Gemma hears the audio directly and makes every decision to listen or speak itself, with no turn detector, but it writes its reply as text, and a separate speech model says it. That’s a thinker and a talker, the split Qwen’s Omni models are named for, although their talker also gets the thinker’s audio and visual features (Qwen2.5-Omni’s read its hidden states too), where mine only gets the text.11 Qwen team, Qwen3-Omni Technical Report, 2025. Qwen3-Omni’s talker “no longer consumes the Thinker’s high-level text representations.” I started out aiming for speech-native and ended up here. For what this post is about, the duplex behavior, speech-native doesn’t add much, and it adds a lot of complexity. A separate voice can also be swapped for any other speech model, and with how many great TTS models have come out recently, that’s the more practical choice, at least for now. The hand-written rules for when to pass Gemma’s text to the talker are exactly the kind of hand-engineering the bitter lesson says will lose, which is part of why I think of this design as a step toward end-to-end rather than the destination.

The open gap

Nobody has had much reason to push on open full duplex, so the open infrastructure and research for it (and for speech-to-speech in general) is thin, and the closed work is too expensive to copy. There are some good open models, Kyutai’s Moshi and NVIDIA’s PersonaPlex (built on Moshi), MiniCPM-o, the Qwen Omni line, and several others. What’s missing is everything around them: recipes for adapting your own LLM, open data that looks like real conversation, evaluation that doesn’t fall apart on audio, and honest write-ups of what goes wrong. Tool calling is thin too: few open voice models support it at all, and the ones that do, do it poorly. Even the closed frontier struggles with it and has only recently made real strides.

It’s part of a broader gap. Very few frontier-scale models understand arbitrary audio at all. On the closed side it’s essentially Gemini. On the open side, Inkling from Thinking Machines is the largest open model with native audio input, Meta’s Muse Spark Open just joined it, and beyond that you’re into smaller models and specialists like Qwen-Omni and Step-Audio.22 Thinking Machines, Welcome Inkling, July 2026.

That gap is what a cheap, open testbed is for, so here’s how this one works.

How the model works

The model I’ve put together has two parts, and for the rest of this post I’ll call them the thinker and the talker. Gemma 4 12B is the thinker: it hears a continuous stream of your audio, writes the reply as text, and decides every 80 ms whether to listen or speak. CSM-1B is the talker: it takes the words Gemma writes and turns them into speech. Let’s walk through how the pieces fit together.

Running on a clock

Like Moshi, this model runs on a clock: every 80 milliseconds it takes in a slice of your audio and decides whether to keep listening, keep speaking, or start speaking. Because it never stops hearing you, there’s no turn boundary for anyone to detect. Overlap, interruptions and backchannels (the “mhm” a listener says to show they’re following) get handled the same way as everything else, one frame at a time (Figure 2).

my audio Gemma writes a token SIL EPS talker audio

Youshouldprobablyskipthesweatersince65degreesisquitemild.Alightjacketoralongsleeveshirtwillkeepyoucomfortablewithoutmakingyouoverheat.61isevencooler,soalightsweaterisasmartcall.Itwillkeepyoucozywithoutbeingtooheavy.

0 s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14 s15 s16 s17 s18 s19 s20 s21 s22 s23 s

you cut in · stops

Press play to follow along.

wrote

said

0.0 / 23.5 s

Figure 2: One conversation, 80 ms at a time. The session from the top of the post, with the final model, live and with no noise gate. Each column is one frame: what the model heard from me, what Gemma wrote in the assistant's slot (a word, SIL to keep listening, or EPS to keep speaking), and what the talker played. Gemma starts writing 0.4 s after I finish, and the talker's voice follows about 1.2 s later. When I talk over it, its audio stops within two frames, and it answers the corrected question. Press play, or drag along the overview to jump.

The pretrained parts

Training a full-duplex model from scratch is a lab-scale project. We’re on a budget, so I started from pretrained pieces, and I picked them less for raw quality than for what they make easy: fast experiments and flexibility.

Gemma 4 12B, for language and hearing. There are larger models with better audio understanding, and I could have taken any LLM and trained a small projector to map a pretrained audio encoder’s outputs into it. Gemma is more interesting than either: it hears audio natively, and it does it without a speech encoder at all. Each 40 ms of audio goes through a linear projection straight into the model. That’s interesting for two reasons. It’s a much simpler starting point, and it means hearing and language already share one embedding space, which fit a hunch I had that full-duplex conversation is best learned as a joint audio and text task. Thinking Machines’ recently released Inkling is encoder-free for audio too, a choice they tie to the design of their interaction models.22 Thinking Machines, Welcome Inkling, July 2026. The case for dropping the encoder is newer than you’d expect, but it’s leaning one way. For images, a large study found no inherent advantage to bolting on a pretrained encoder.33 Shukor et al., Scaling Laws for Native Multimodal Models, ICCV 2025. Across 457 trained models, early fusion with no image encoder matched or beat late fusion with a pretrained one, and was cheaper to train. For speech, the first careful comparison found encoder-free models close the gap on transcription as data grows and do better on how things are said, though they still trail on knowledge-heavy spoken questions.44 Fan et al., LLM can Read Spectrogram: Encoder-free Speech-Language Modeling, 2026. Encoder-free trailed on transcription by about 11% relative, narrowing to about 4% with 10× the data; it was well ahead on emotion and speaker tasks and well behind on spoken MMLU. Shi et al., Teaching LLMs to Hear Who Spoke What, 2026, add that encoders trained for transcription tend to discard acoustic detail. For full duplex there’s a practical reason too: most audio encoders look ahead at audio that hasn’t arrived yet, so they have to be retrained to stream before a full-duplex model can use them.55 Chang et al., D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones, 2026, which first distills Qwen3-Omni’s audio encoder into a streaming one. Gemma projects each 40 ms slice on its own, so there’s nothing to retrain. I’m fairly confident encoder-free is where this goes. It’s the bitter lesson again: one model learning from the raw signal, rather than a front end trained for someone else’s objective. Somehow, we’ve landed in a world where “audio” is niche and has come to mean “voice.” Supporting audio, and working toward omni models, means hearing the world far more expressively than an encoder trained to transcribe speech can. Everything beyond the transcript gets thrown away: a laugh, music, a door closing, the room itself, and how something was said, not just what was said. Strapping one of those encoders onto an LLM and calling it audio input shuts off access to so much of the world.

Mimi, the audio codec. To generate audio, the model needs something discrete to predict, and a pretrained neural codec is the standard way to get it: it compresses audio into tokens and decodes them back. I picked Mimi, the codec Kyutai built for Moshi, mostly for its ecosystem. It’s one of the most studied codecs, and Moshi, CSM and Kyutai’s newer speech models all use it, so parts built around it tend to fit together. Its codes are discrete, so they can be predicted just like text tokens and sit in the same sequence as text, and at 12.5 frames a second they keep the audio sequence short.

CSM-1B, a pretrained speech decoder. Sesame’s CSM is a speech model trained on roughly a million hours of audio. It isn’t the best TTS model, or even the best at its size, but it produces Mimi codes, so it slots straight into this setup, and I’ve appreciated the care the Sesame team has put into designing voice interactions. I first borrowed only its decoder, and ended up using the whole model as the voice.

The layout

The layout design is what makes our approach cheap. Every 80 ms frame becomes three positions in one token sequence: two for the user’s audio (one per 40 ms) and one for the assistant. The assistant’s slot holds a text token or a control token: SIL while it listens, EPS while it’s still speaking with nothing new to say, and words when it has them. Gemma’s existing language-model head writes all of them (Figure 3).

Interleaved (this model)

Each frame adds three positions to one token sequence: two 40 ms user slots, then the assistant slot Gemma's own head writes: a word, SIL to listen or EPS to keep speaking.

∿∿SIL∿∿SIL∿∿SIL∿∿SIL∿·SIL··It's··EPS··EPS··Paris··EPS··EPS··EPS··SIL··SIL

1.12 s42 positions

Parallel streams (as in Moshi)

Each frame is one step. The streams advance together and are summed into one input. Flexible, but further from what a pretrained chat model has seen.

user

∿∿∿∿∿·········

text

·····It's··Paris·····

audio

·····♪♪♪♪♪♪♪··

1.12 s14 steps

Figure 3: Two ways to lay out a duplex conversation. The user asks a question and the assistant answers, one 80 ms frame at a time. Interleaving spends three sequence positions per frame where parallel streams spend one, so an hour of conversation is about 135,000 positions. In exchange, duplex stays ordinary next-token prediction, which a chat model already knows how to do.

Parallel streams, as Moshi uses, are the other common choice: each step sums the user’s audio, the model’s text and the model’s audio into one input, and extra heads predict each stream. Interleaving keeps the input a plain token sequence, which is what a pretrained LLM was trained on, and that has two advantages. The transfer is small: the model already knows how to continue a chat, so learning to continue a chat chopped into 80 ms slots takes very few steps, and the LM head it already has writes the control tokens. And standard tooling works unchanged: chat templates, sequence packing in training, KV caching, and the batching in ordinary LLM serving engines. Parallel streams batch fine too, but they need their own model code and serving path. It does make the sequence longer: three positions per frame instead of one.

Emitting text in the assistant’s slot, rather than speech, is a deliberate simplification. A true speech-to-speech model would output audio there. Text is what lets me stitch pretrained parts together and learn quickly from a chat model’s priors, and the slot can carry other channels too: a thinking channel, or one for handing work to other agents.

That slot has a cost: throughput. The assistant gets one text token per frame, so it can write at most 12.5 tokens a second. A spoken reply needs roughly three or four tokens a second, so speech fits easily. Everything else the model writes shares the same budget: a reply meant to be read, a tool call’s arguments, a thinking channel. All of them share that cap, several times slower than the same Gemma decoding plain text. As built, the model is only useful speech to speech. Giving the assistant several slots per frame, or letting it write a variable-length burst of text between frames, would lift the cap at the cost of a longer sequence and a less regular layout.

The 80 ms frame timing itself wasn’t a choice. Mimi’s frame rate and Gemma’s audio input both land there, and it’s common in the literature, so I went with it. Personally, I don’t think it’s the right size: at 80 ms, every frame is a hard real-time deadline, and the faster reactions it allows don’t buy much in practice. NVIDIA’s Nemotron VoiceChat also runs on 80 ms frames, and on Full-Duplex-Bench, a public benchmark for turn-taking,66 Lin et al., Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities, 2025. This post uses v1.0. Published systems’ numbers are the benchmark’s; this model’s are timed from the talker’s audio rather than with a speech recognizer. it takes about half a second to yield when you interrupt it.77 NVIDIA, NemotronLabs VoiceChat 11B, 2026. Full-Duplex-Bench 1.0, user interruption. People typically leave gaps of around 200 ms between turns,88 Stivers et al., Universals and cultural variation in turn-taking in conversation, PNAS, 2009. so reacting within a frame isn’t what makes a conversation feel natural. A larger chunk, like the 200 ms micro-turns in Thinking Machines’ interaction models, is worth investigating.99 Thinking Machines Lab, Interaction Models: A Scalable Approach to Human-AI Collaboration, May 2026.

Thinker and talker

Every frame, Gemma decides whether to listen, keep talking or yield, and it writes the reply a little ahead of the speech. CSM speaks what it writes, in chunks (Figure 4).

Figure 4: The final design: a thinker and a talker. Every 80 ms frame, Gemma hears the user's audio and writes one token in the assistant's slot: a word, SIL to listen, or EPS to keep speaking. Only its LoRA adapters train. The playout gathers the words into chunks, and CSM-1B, unchanged, speaks each one through Mimi's decoder. Every frame, Gemma is also told whether the talker is still playing, so it can tell an interruption from a new turn, and a SIL while it plays empties the queue.

Most of the engineering is in the playout, the code that hands Gemma’s text to CSM in chunks and plays what comes back. The first chunk waits for a sentence end or a clause past 8 words, because CSM speaks each chunk as its own utterance, and shorter chunks put pauses in the wrong places. That wait costs about a second before the first audio, which the reinforcement learning section comes back to. After that it speaks every complete sentence, each chunk conditioned on the recent audio so the voice stays continuous, cut early at a clause if the talker is about to run dry. When Gemma yields, the playout throws away anything not yet spoken, so the voice stops mid-word. This heuristic came from trying a few designs and keeping what sounded best in practice. With short chunks, one sentence could become half a dozen separate generations, and each restart broke the rhythm. Speaking each chunk as a continuation of the reply so far kept the rhythm but made CSM drop words, so longer chunks won, at the cost of a later first word. A talker built to speak streaming text wouldn’t need any of this.

Writing ahead is a requirement of CSM, which needs words before it can speak them, not of duplex models in general. A model trained end to end for streaming could get by with less lead, especially with larger chunks. At 80 ms, a frame is a fraction of a syllable, so the latest frame carries almost no local context, and the model leans on lookahead and long history instead. At around 200 ms a chunk holds roughly a syllable to a word, which is enough to decide prosody, the rhythm, stress and intonation of speech, locally.

I chose CSM over a talker that already streams text (more on those in a bit) because it produces Mimi codes, which my first design also needed, so it was already in the stack, and it sounded great from the first time I heard it.

Initially I tried having Gemma generate the audio itself, with a small head that turns its hidden state into Mimi’s codes.

Giving the voice its own model

Having Gemma make its own audio worked better than I expected, but two things went wrong. The voice kept glitching, with quick hiccups in the middle of sounds, like audio buffering. And the model got noticeably dumber: it started getting facts wrong that it used to know. They had different causes, but moving the voice out of Gemma fixed both, and it turned out to be a better setup for experimenting, too.

Generating audio means predicting Mimi’s codes. Each 80 ms frame is a stack of eight codes, each from its own codebook, a learned vocabulary of 2,048 entries, and each code encodes what the codes before it left out. The first, codebook 0, is trained to match a self-supervised speech model’s representation, so it carries most of what was said and when. The later codebooks add the detail: timbre, breath, the room (Figure 5).

Recordingbefore Mimi

1 codebookcodebook 0 alone

2 codebooks0 and 1

4 codebooks0 to 3

8 codebookswhat my model predicts

32 codebooksall of Mimi; what CSM-1B renders

Figure 5: One recording decoded from more and more of Mimi's codebooks. Codebook 0 carries most of what was said and when; the later codebooks add how it sounds. Recording from LibriTTS-R (CC BY 4.0), reading Willa Cather's Alexander's Bridge.

In the first design (Figure 6), LoRA adapters train Gemma itself. A depth head, a small transformer initialized from CSM-1B’s decoder that works within one frame, reads Gemma’s hidden state at the assistant’s slot and predicts that frame’s eight codes one codebook at a time. The codes it just produced are fed back in, so the model can hear itself (Figure 7).

recall keptshare of base Gemma's answers

better →

55%

voicecodebooks 1–3 loss, lower is better

better →

2.27 nats

Audio gradient into Gemma

Figure 6: One frame of the single-model design. The hidden state at the assistant's slot feeds both the language-model head, which writes the next word or control token, and the depth head, which writes the frame's eight Mimi codes. The dashed lines are the two losses' gradients flowing back into that one hidden state, and the audio loss was many times the text loss. With the audio gradient flowing into Gemma, the voice is cleaner but recall drops. Stop it and recall comes back, but the voice falls apart. Switch between the settings, or click a dot, to compare.

The sum joins frame t+1's input

Figure 7: Inside one frame of the single-model design. Gemma's hidden state at the assistant's slot conditions the depth head, which writes the frame's eight Mimi codes one at a time, each conditioned on the codes before it. The codes go to Mimi to become 80 ms of audio, and through the feedback tables, one embedding table per codebook. Their sum is added to the assistant slot's input on the next frame, so the model hears what it just said. The code values are illustrative.

Learning to speak cost the model recall

Fine-tuning a model toward a new objective always risks breaking what it already knew, and generating speech is a big one for a model that has only ever written text. I saw it early: in the same live conversation that impressed me, it confidently got facts wrong that base Gemma reliably knew. Ask base Gemma for the tallest mountain in Europe and it gets it right. After speech training it often says Everest, in a complete, confident sentence. Nothing about the reply sounds broken, and that’s worse in voice, where the listener can’t skim or double-check the answer the way they would with text. I wanted to know how widespread that was, so I built a facts test: short questions from Natural Questions, a set of real search queries with reference answers, read aloud by a synthetic voice and by recorded human speakers. The model hears each one live and answers. It tests simple recall, not reasoning. The score is retention: of the questions base Gemma gets right, the share the trained model still gets right.

Every checkpoint trained to speak lost about half of those answers, and none of the usual knobs (data, run length, layout, learning rate) moved it (Figure 8). Base Gemma itself only gets about a quarter of these right, because my scoring is strict: one reference answer, which has to appear in the first few words of the reply.1010 The Gemma 4 report doesn’t publish Natural Questions scores. Its predecessor, Gemma 3 12B, scores 31% counting any accepted answer: Gemma Team, Gemma 3 Technical Report, 2025 (5-shot, pretrained model). So the losses are on questions it found easy, and the model is losing things it could already do.

correct, and base was too base was correct, lost newly correct wrong

Base Gemma

125 of 500 correct

Speech-trained

kept 55% · +30 new

Audio gradient stopped

kept 73% · +38 new

Hover or tap a square to see the question and that model's answer.

Figure 8: Outcomes on 500 short factual questions. Each square is one question, in the same position in every panel, with base Gemma's correct answers first. Speech training loses many of base Gemma's answers; stopping the audio gradient recovers most of them, and some answers change in both directions. Hover or tap a square to read the question and that model's answer.

This looks like catastrophic forgetting, where training on a new task degrades what a model could already do. But with LoRA the base weights are frozen, so the knowledge is still there: switch the adapters off and you have base Gemma back. Something the adapters learned is getting in the way. Scaling them down brings answers back smoothly, while letting base Gemma read the question and switching the adapters on only for the answer barely helps. So the trouble is in how the model answers, and the prime suspect is the audio loss, which trains the same hidden state the answer comes from.

Where the gradient goes

To see why the audio loss does this, it helps to look at where each loss’s gradient goes. Here’s the objective for one frame:

\mathcal{L} = \underbrace{\mathrm{CE}_{\text{text}}}_{\mathcal{L}_{\text{text}}} + \underbrace{2\,\mathrm{CE}_0 + \textstyle\sum_{k=1}^{7} \mathrm{CE}_k}_{\mathcal{L}_{\text{audio}}}

\begin{aligned} \mathcal{L} &= \underbrace{\mathrm{CE}_{\text{text}}}_{\mathcal{L}_{\text{text}}} \\ &\quad + \underbrace{2\,\mathrm{CE}_0 + \textstyle\sum_{k=1}^{7} \mathrm{CE}_k}_{\mathcal{L}_{\text{audio}}} \end{aligned}

The text term is one token, usually an easy SIL or EPS. The audio term sums all eight codebooks, with codebook 0 counted twice, and it ends up many times larger than the text term. Both reach the same hidden state $h$ at the assistant’s slot, and the scale on the audio term, $\lambda$ , is the setting you can switch in Figure 6:

\frac{\partial \mathcal{L}}{\partial h} = \frac{\partial \mathcal{L}_{\text{text}}}{\partial h} + \lambda\,\frac{\partial \mathcal{L}_{\text{audio}}}{\partial h}, \qquad \lambda \in \{1,\ 0.3,\ 0\}

\begin{gathered} \frac{\partial \mathcal{L}}{\partial h} = \frac{\partial \mathcal{L}_{\text{text}}}{\partial h} + \lambda\,\frac{\partial \mathcal{L}_{\text{audio}}}{\partial h} \\[0.4em] \lambda \in \{1,\ 0.3,\ 0\} \end{gathered}

Every update is also clipped to a fixed size:

g \leftarrow g \cdot \min\left(1,\ \frac{c}{\lVert g \rVert}\right), \qquad c = 1

\begin{gathered} g \leftarrow g \cdot \min\left(1,\ \frac{c}{\lVert g \rVert}\right) \\[0.4em] c = 1 \end{gathered}

The gradient ran over that limit on every step, so every update was clipped to the same size. The much larger audio term took up most of it.1111 A loss ratio isn’t a gradient ratio, and I didn’t log the two gradient terms separately. By the end of training the audio loss was roughly twenty times the text loss.

With $\lambda$ set to zero, the depth head still learns from audio but none of that learning flows back into Gemma, and the model was nearly as sharp as base Gemma again. But without that gradient the voice fell apart and hearing became fragile.

Recall can’t tell you whether the model is listening, because a model can answer from the words it half caught or from the conversation so far. So I measure hearing directly, with the hearing gap. I score the same reply twice, teacher-forced: once with the user’s real audio, and once with the audio from a different conversation swapped in. A model that’s listening finds its reply far more likely with the real audio, so the gap is the difference in loss on the reply’s words. A nat is a unit of information, like a bit but counted with the natural log, and cross-entropy computed with the natural log, PyTorch’s default, comes out in nats. A loss of 0.69 nats is a coin flip, and 2.3 nats is about one in ten. Differences read as ratios, so a gap of 1 nat means one prediction is about 2.7 times more likely than the other. Near zero means it isn’t using what it hears (Figure 9).

Without the audio gradient, whether the model kept hearing depended on where training started. Started from a checkpoint I’d first trained on conversations where the user never spoke, it lost its hearing in a single step and never got it back, while the same recipe from base Gemma heard fine (Figure 9).

audio gradient into Gemma0.87no gradient, after a silent-user stage0.00no gradient, from base Gemma1.71

Figure 9: The hearing gap over training. How much worse the model predicts its reply when given another conversation's audio instead of the user's; zero means it ignores what the user said. Without the audio gradient, a model first trained with a silent user learned to hear for a few hundred steps, lost it in a single step and never recovered. Started from base Gemma, the same recipe without the gradient heard fine.

Passing back only part of the audio gradient landed in between. Under Adam that’s the same as reweighting the two losses, and each setting traded voice for recall without getting both (Figure 10). Within one model, I couldn’t find a way out of that trade at this budget.

Hover a point for what happened.

Figure 10: Recall against voice quality for each way of sharing the audio gradient. Right means more of base Gemma's correct answers kept; up means lower acoustic loss. No variant reaches the top right with intelligible words: the two hollow points come closest, but kept recall by garbling the words, which neither axis shows. Hover a point for details.

I think the pressure is inherent, but its size isn’t. Base Gemma answers about as well from audio as from text, so here the cost comes from generating speech, and I’ve put my reasons for thinking so, and how it compares with other speech models, in the appendix.

The glitches come from codebooks 1 to 7

The second problem was the voice. To find where the glitches came from, I ran a listening test that swapped codebooks between my model and a reference recording (Figure 11).

Reference0 and 1–7 from the reference

CSM-1BSesame's model, same words

Our codebook 00 ours · 1–7 reference · clean

Our codebooks 1–70 reference · 1–7 ours · glitches

All oursall eight ours, true history · glitches

Running freeits own history · glitches

Figure 11: A listening test that swaps codebooks between my model and a reference. Every clip says the same words, “What is the biggest dog breed in the world?”, and all but the last are decoded from the reference's history. With my model's codebook 0 the voice stays clean; with its codebooks 1 to 7 the glitches appear, even given the true history.

My model’s codebook 0, the words and timing, sounded fine dropped into the reference. Its codebooks 1 to 7 brought the glitches with them, even when every frame was given the true history, which rules out drift from the model running on its own output. The codec isn’t the problem either, since the reference and CSM-1B both sound clean through Mimi.

So the depth head gets the words right and misses the fine acoustic detail on top, which your ear catches immediately. Either the head isn’t given enough to work with (one hidden state trained mostly to predict text) or it hasn’t learned from enough audio (a hundred hours of one voice, against CSM’s million). Starting the head from CSM’s decoder helped more than anything else I tried, but it didn’t make the glitches go away, and both real fixes cost far more data and compute than I had.

Splitting the thinker from the talker

The recall cost came from the audio’s gradient flowing into Gemma, and the glitches from a speech decoder that needed far more training than I could give it, so a pretrained speech model doing the talking removes both. The depth head’s glitches could land anywhere in a reply, while CSM’s flaws show up only where chunks join and are much milder, which is a problem you can engineer around (Figure 12). With no audio gradient into Gemma, recall comes back: it performs better than any single-model checkpoint.

Design	Who makes the audio	Voice	Recall	Hearing
One model, audio gradient into Gemma	Gemma’s depth head	glitches throughout	about half of base Gemma’s answers kept	yes
One model, audio gradient stopped	Gemma’s depth head	falls apart	most answers kept	fragile
Thinker and talker	CSM-1B, from Gemma’s text	clean, with audible joins	best of the three	yes

Depth headevery frame

CSM talkerin chunks

Who speaks

Figure 12: The same reply from the depth head and from the CSM-1B talker. The depth head writes eight codes every frame, and the glitches can land in any of them. The talker speaks Gemma's text in chunks, a little behind the writing, and its defects sit where chunks join. Timings in the drawing are schematic; both recordings are real.

A better place to experiment from

Splitting the model also made the rest of this post cheap to do. Gemma never has to predict audio in training, only text and control tokens, and the reinforcement learning later doesn’t run the talker at all. That means the duplex behavior, when to listen, speak or stop, can be worked on without touching how the voice sounds.

It doesn’t close the door on end to end, either. The next steps are to let the talker read Gemma’s hidden state, and maybe the user’s audio, and then train the two together. I’ll get into that at the end of the post.

So now we have a functioning model, but it still can’t really hold a conversation.

Teaching it to converse

A model that can talk still has to learn to converse: to answer the way people speak rather than the way chatbots write, to stop when it’s interrupted, and to tell a pause from the end of a turn. Data and decoding got me the first two, and the third took reinforcement learning.

The data is the model

Almost all the data and RL environments that exist assume a chat-assistant UX, with one complete user message, a hard turn boundary, markdown and lists, long thorough answers, and a lot of hedging. Speech doesn’t work like that, and the differences go well past formatting.

In real conversation, people overlap, backchannel (“mhm”, “right”), fix sentences halfway through and leave things unsaid (Figure 13).1212 Carletta et al., The AMI Meeting Corpus, 2005. Audio and annotations used under CC BY 4.0; Figure 13 uses meeting ES2002d, 864.0–886.8 s. They build sentences differently too, putting the answer first in short clauses, saying numbers aloud and using words people rarely write. And speech carries a whole channel that text drops, in prosody, emphasis, hesitation and laughter.

As it happened

Word timings from the corpus, grouped into utterances

holding the turn backchannel abandoned mid-sentence both talking

Aof stylized or in different concept that thatYeahSure okayYeah alrightOkay sureWhat about button shape Square buttonsOkay

BI think that's something that's very hard to catch so you you restrict the number of people who wanna try something The the look and the colour is something which is cool but I think that there's also that factor of if it's too unfamiliarthen um because when you put it on the shelfYeah button shape might be a good idea to change

Press play to follow along.

0.0 / 22.8 s

As chat data

The same exchange the way a chat transcript would record it, written for this figure

Some devices stylize the standard grid, or use a different concept.

I think that's very hard to catch on, so you restrict who wants to try it. The look and the colour are cool, but there's also the risk of it being too unfamiliar.

What about button shape? Square buttons?

Button shape might be a good idea to change.

Figure 13: Twenty-three seconds of a real conversation, and the same exchange as chat data. From a design meeting in the AMI Meeting Corpus (ES2002d, CC BY 4.0). A backchannels through B's turn without taking it, B abandons a sentence when A cuts in with a new idea, and B picks the idea up. As chat data, the same exchange becomes four tidy messages, and all of that disappears. Press play to follow along.

Run chat data through a TTS voice and you get written language read aloud, which sounds like speech but has none of its structure. I suspect that’s a big part of why real human voices remain this model’s weak spot. The gap shows up everywhere I measured it. The model answers recorded human questions far less often when they’re spoken than when they’re typed, and, as the next section shows, it at first yielded to synthetic interruptions far more reliably than to human ones.

The clearest sign of the chat style coming through was dodging. The multi-turn conversations I trained on came from the helpful part of hh-rlhf, Anthropic’s public dataset of assistant conversations, and the model picked up their habit of hedging. Rewriting every assistant reply in them as spoken conversation (answer first, no filler, short sentences, no dodging) made it dodge far less often (Figure 14). My first rewrite cost some recall. A second pass, which wrote numbers as digits for the voice to read aloud and dropped the filler, fixed that. So rewriting replies for speech changed which questions it could answer as well as how it sounded.

hedges or dodges filler turns the question back the answer, first

userDo you think you can explain to me how gravity works?

Written (hh-rlhf)51 words

I think you’re looking for an answer beyond what I can say. I can understand physics in a very high level, but I think there’s a large gap in my knowledge that I can’t bridge without a lot of more advanced understanding. Is there something specific that you’re trying to understand?

Rewritten for speech39 words

Gravity is the pull between anything with mass, it is what keeps your feet on the ground and Earth going around the sun. Do you want me to stick to the everyday effects you can actually see and feel?

Example

Figure 14: The same first reply, written for reading and rewritten for listening. Each pair is a real training session: the original reply from hh-rlhf, and the version a teacher model rewrote for speech, which is what the talker trained on. The rewrite answers in its first sentence, drops the hedging and filler, and writes numbers as digits. A question back survives only where the person's next turn answers it. Switch between the examples to compare.

Beyond the wording, the timing has to look like speech too, and interruptions are where that showed up first.

Yielding to interruptions

A useful voice model has to stop talking when it’s interrupted, but only when stopping makes sense. A traditional voice pipeline yields aggressively; any sound from the user cuts the reply off, whether it’s a correction or a “mhm”. A full-duplex model hears the interruption itself, so it can learn to handle it the way people do, stopping for a correction and carrying on through a backchannel. So far this model only does the first half. It stops reliably whenever someone talks over it, including when they just say “okay”.

Training examples where people interrupt. In ordinary chat data, the user only speaks once the assistant has finished. Trained on it, the first thinker–talker model had never heard anyone talk over it, and it didn’t stop (Figure 15). The fix was interruption examples, which are existing training conversations cut so the user’s next turn starts in the middle of a reply. One hazard of training on whole conversations is that Adam’s default beta2 is fragile. Beta2 sets how much history Adam’s running estimate of gradient size keeps. At the default 0.999, hearing collapsed after a single gradient spike. How much they overlap matters, because with a fixed overlap the model learned to stop on exactly one frame, and if it missed that frame it kept talking. Spreading the overlap over a wide range made stopping the right answer on every frame of the interruption. It’s a small case of a problem that comes back later in a bigger form: a per-frame loss rewards stopping on particular frames, when the behavior I want is defined over the whole interruption.

What it was trained on: strict turns

The user only speaks after the reply has ended

userWeather in London?New York?

modelLondon's answerNew York's answer

What happened live

I asked about New York twice while it answered London. It kept going

meWeather in London?What about New York?…and again

talkerLondon's answerpauses mid-answerstill London

Figure 15: Why the first version didn't stop. It had only ever been trained on strict turns, where the user speaks after the reply ends, so it had never heard anyone talk over it. In its first live test I asked about the weather in London, then asked about New York City twice while it was answering. It kept talking about London. Gemma did write a New York answer after the second ask, London's word for word with the city swapped, rain and umbrella included, but it sat in the queue behind London's and was never spoken. The pause mid-answer is CSM stalling at a parenthesis, a separate bug.

Stopping on a threshold. Sampled, the decision to stop is a fresh draw on every frame, so a model that’s fairly sure it should stop can still talk through a whole interruption (Figure 16). Stopping as soon as the probability crosses a threshold, while still choosing words greedily, makes it stop reliably.

me model speaking model talking over me P(STOP) above 0.1

Figure 16: One live interruption, replayed frame by frame. I start talking while the model is mid-answer. In every 80 ms frame the model gives a probability of writing STOP, which hands the floor back to me (the bars; this session used an earlier layout with its own STOP token; the current one stops by writing SIL). Sampling STOP from that probability, it had a 89% chance of stopping at some point while I talked, never drew it, and kept talking over me. Writing STOP as soon as the probability crosses 0.1 stops it 0.40 s after I start.

Knowing whether its voice is still playing. The thinker/talker split added a failure of its own. Gemma writes much faster than CSM speaks, so it usually finishes writing a reply while the talker is still saying it. To Gemma, the person talking at that point looked like a new turn arriving at a quiet moment, not an interruption.

The fix is to tell Gemma on every frame whether the talker is still playing, and to train it to keep writing EPS while it is. Then a SIL during playback means stop.

That also closed a gap our synthetic data had hidden. The model had yielded to synthetic interruptions but often not to recorded human ones. On my interruption test, where synthetic and recorded voices cut in mid-reply, it now yields to both within a frame or two. The training examples put the stop right at the interruption, so a frame or two shows how well the model learned them. Whether that timing feels natural to people is a separate question.

Data and decoding got the model to stop when it’s interrupted. Telling a pause from the end of a turn was a different problem, and more data alone wasn’t going to fix it.

Reinforcement learning for interactivity

Supervised training got the content right and the timing close, but the rest of the timing is hard to teach that way. Good timing comes down to a handful of decisions, like whether to start talking at this pause or at this turn end. Each is one frame’s choice, and at 80 ms a frame, whether it was the right one only shows over the next few seconds of the conversation. A per-frame loss can’t see that far, and it scores the decision frame alongside the hundreds of frames around it where the answer is obvious, so the model learns the average timing of its training examples rather than the decision.

Whether a conversation went well is a property of the whole exchange, and that’s what reinforcement learning scores. It rewards a whole stretch of conversation for how it went, so the credit can find the few decisions that mattered. It also gets around the data problem. Two-channel recordings of real conversations exist, but they aren’t labeled for these decisions or shaped like someone talking to an assistant, and relabeling them was more effort than I wanted to put in for this project. A reward only needs to know where the pauses and turn ends are, which is far cheaper than a correct target for every frame.

Training only the decision to speak, with a simple reward and a few dollars of compute a run, taught the model to wait through pauses, once three things about the setup were right.

A failure the benchmark couldn’t see

Full-Duplex-Bench scores a model from the audio it plays, and scored that way this model never talks over a speaker’s pause. But the talker only starts once Gemma has written a full opening clause, and Gemma stops itself the moment the person carries on, so a false start never reaches the audio. Looking at what Gemma writes before the playout tells a different story. It starts writing an answer in a lot of real mid-sentence pauses (Figures 17 and 18). The wait that hides those starts is also most of why the model is the slowest on the benchmark to answer at the end of a turn. So the playout was hiding false starts, and we were paying for it in latency. If Gemma rarely started in a pause, the first chunk could be shorter, and that’s the decision I wanted reinforcement learning to fix.

Hover or tap a point for what it means.

Figure 17: Full-Duplex-Bench before and after reinforcement learning, as the benchmark hears the model and as Gemma writes. Left is fewer starts in a speaker's real mid-turn pauses; down is a faster answer when a turn ends. The gap between each filled point and its hollow one is what the playout hides. Before reinforcement learning, Gemma starts in nearly a quarter of the pauses. After, it starts in about one in eight, but answers later, and as heard it's the slowest system here. The other systems are the benchmark's published numbers. Hover or tap a point.

It happens live too. In Figure 18, from a conversation with me before reinforcement learning, Gemma writes half an answer, stops when I keep talking, and nobody hears a word of it.

my audio Gemma writes a token SIL EPS talker audio

MountEverestisthetallestinAsia,standingat290AconcaguaisthetallestinSouthAmerica,at22,837feetinArgentina.

0 s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14 s15 s16 s17 s

written, never said

Press play to follow along.

wrote

said

0.0 / 17.0 s

Figure 18: A start the playout hid. From a live session with the checkpoint before reinforcement learning (Figure 17's lighter points). I ask about Asia and go quiet; 0.9 s later Gemma starts writing about Everest. A second after that I change the question, and Gemma stops on the next frame. The talker says none of it, because its first chunk was still waiting for a clause past 8 words, and stopping empties it. Then Gemma answers the new question. This start came at the end of my question, so it wasn't wrong, but in Full-Duplex-Bench's real pauses the same thing happens mid-sentence, and the benchmark, which only hears the audio, never counts it. Press play, or drag along the overview.

Training only the decision to speak

My approach is guided by two recent papers, Kyutai’s interactivity post-training for Moshi1313 Ohashi, Zeghidour, Défossez and Kharitonov, Post-training speech models for better interactivity, Kyutai, June 2026. and Qualcomm’s HiPLEX,1414 Jung et al., HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models, Qualcomm AI Research and KAIST, 2026. with changes that make it cheap. HiPLEX came out right as I was starting on this, and a lot of it matched where my head already was, especially splitting the decision to speak from the choice of words and caring about which frames get the credit. I think there’s a bigger thread here. Most reinforcement learning for full duplex so far scores the model’s output as one stream with one reward, and a lot of the fixes, theirs and mine, come down to where the credit goes. Each training example is a short clip from a real two-person recording, Meta’s Seamless Interaction,1515 Meta, Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset, 2025. The dataset is CC BY-NC 4.0. and there are two kinds. A pause clip ends inside a speaker’s mid-sentence pause, and its reward is −1 if Gemma writes any word before the speaker resumes, and 0 if it waits. A turn clip ends where the speaker finishes, and its penalty grows with how long Gemma takes to start, with the full penalty if it starts before they’re done. The model hears each clip on its live 80 ms timeline, and for each clip I sample several continuations (rollouts). The update is GRPO (group relative policy optimization): each rollout’s reward is compared with the average of its group, rollouts that did better than their siblings are made more likely, and a KL penalty keeps the model close to where it started.

Three choices keep a run to a few dollars and about an hour. The first is training only the decision to speak. Each frame, Gemma writes SIL, EPS or a word, so “write a word, any word” is a separate decision from which word. Live, the decision to start is sampled (stopping uses the threshold from earlier) while the words are chosen greedily, so rollouts work the same way, and only that decision is ever explored or trained. The loss never depends on which word Gemma picks. The second is reading the timing straight from Gemma’s tokens. Both papers time replies by running speech recognition on the decoded audio, but the talker says exactly the words Gemma wrote, so the tokens give the same timing, and training never has to run the talker at all. The third is training only the LoRA adapters, on one GPU, where both papers train the whole model on many more.1616 Kyutai trained on 32 H100s and HiPLEX on 16. Runs this cheap are what made it possible to iterate and experiment quickly!

There’s no reward for content, either. Kyutai found that training on timing alone made the replies worse, so it adds a reward from an LLM judge, and HiPLEX routes that judge’s reward only to the choice of words. Both need it because their timing update reaches the words. Here it mostly doesn’t. The adapters are shared, so an update to the timing decision can still nudge which word comes out on top, but the KL penalty keeps that small. The risk I watched instead is indirect. A model that starts sooner has heard less of the question when it answers, so I ran the facts test on every checkpoint I kept, and recall held on all of them (the evaluation section covers the test’s blind spots).

What made it work

Every early run learned to speak sooner, in pauses too (Figure 19). Three things fixed that, and each is likely to matter for reinforcement learning on any timing decision.

Hover or tap a run for what changed.

Figure 19: Every run answered sooner and started in more pauses, until one trained on pauses alone. Each row is one run, from step 0 to a held-out check; left is better in both columns. The first four, from the checkpoint that only answers in silence, learned to speak in room tone, which is both a faster answer and a false start. The last two hear room tone throughout: with turn clips in the mix it still learned to speak sooner, and with pause clips alone it learned to wait, at the cost of answering later. Hover or tap a run for what changed.

Credit has to land on the decisions that differ. Two easy mistakes pushed every early run toward speaking.

Two ways the credit went wrong

Averaging over each rollout’s own frames. That gives every rollout the same total weight, however many frames it was charged on, so on the frames the rollouts shared, one rare false start could outweigh all the quiet ones. Averaging over every frame in the step weighs each frame the same. It’s the length bias Dr. GRPO describes.1717 Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective, 2025.
Charging each reward only to the frames near its event, as HiPLEX does. Done per rollout, a rollout that started early was charged from the start of the clip, and the ones that waited only from the end of the turn. So the listening every rollout did while the person was still talking was only ever pushed toward speaking. Charging every frame of every rollout lets that shared listening cancel out within the group, so the update only moves the decisions where the rollouts actually differed.

The clips have to reflect the environment the model actually works in. A real room is never silent. There’s always a low hum of background noise, called room tone. Every turn the supervised model trained on ended in exact digital silence, so it learned to take its turn only on silence, and in a real room it barely answered. Live, I worked around this with a noise gate on the microphone, which turns any quiet stretch longer than a short hold into exact silence. It works, but every answer has to wait out the hold, and a hand-tuned filter deciding what the model gets to hear is the opposite of bitter-lesson thinking. The model should learn to hear the room itself, which is what the rest of this fix does. Once it trained on room tone, it answered in a real room without the gate. It also started talking in most mid-sentence pauses, since a quiet room now sounded like the end of a turn. Reinforcement learning on clips heard over the same room tone taught it to wait through them.

The reward you care about has to drive the update. GRPO only learns from a group whose rollouts score differently. A turn clip’s reward, graded by delay, nearly always differs within a group. A pause clip’s pass-or-fail reward often doesn’t, because all the rollouts start or none do. With both kinds of clip in every step, the turn clips drove almost all of the learning, and the model kept learning to answer sooner. Training on pause clips alone fixed that.

The result

Trained on pause clips alone, from the checkpoint that hears room tone, the model learned to wait. On Full-Duplex-Bench it starts in far fewer real pauses than before, and almost none of them reach the audio (Figure 17). Retention held too. Live, with no noise gate, it didn’t cut into any of the pauses I left mid-question, where the checkpoint it started from cut into several (Figure 20), though that’s only a handful of pauses.

Before reinforcement learning

The checkpoint that answers in room tone, which reinforcement learning started from

my audio Gemma writes a token SIL EPS talker audio

MountEverestisthetallestmountainintheAconcaguaisthetallestmountaininSouthAmerica,reaching22,891feet.It'slocatedinArgentinaandisamassivepeak.

0 s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14 s15 s16 s

written, never said

Press play to follow along.

wrote

said

0.0 / 16.2 s

After

Trained on pause clips alone for 20 steps

MountAconcaguaisthetallestmountaininSouthAmerica.Itstandsat22,837feetintheAndes.ItisamassivepeakinArgentina.Itistrulyimpressive.

0 s1 s2 s3 s4 s5 s6 s7 s8 s9 s10 s11 s12 s13 s14 s15 s16 s17 s18 s19 s20 s

Press play to follow along.

wrote

said

0.0 / 20.8 s

Figure 20: The same question, with the same pause, before and after reinforcement learning. Both live, with no noise gate. Before, Gemma starts writing "Mount Everest is the tallest mountain in the" while I pause, stops when I say "South America", and the talker never says it; it did the same on its other take. After, it waits through the pause. Across these sessions, the checkpoint before started in 6 of my 19 mid-question pauses and the one after in none of 13. The cost shows too: the after answer comes later. Play each to compare.

It overcorrected, though. With nothing but the KL penalty pushing it to answer promptly, it now takes noticeably longer to answer at the end of a turn and occasionally misses one, and it got slightly worse on the benchmark’s synthetic pauses. That’s where the model stands, so the rest of the post covers how far to trust these numbers, what it still gets wrong, and where it goes next.

Evaluation and limitations

Everything above rests on a handful of internal evals. Before getting to what the model still gets wrong, here’s what those evals can and can’t catch.

What the evals can and can’t catch

Five measurements carry this post: the facts test, the hearing gap, the interruption tests, Full-Duplex-Bench and my ear. The hearing gap is the most bespoke, and the most direct, since it asks whether the audio changes what the model predicts. That lets it catch a model that has stopped listening even when its answers still look fine. The interruption tests and Full-Duplex-Bench measure timing from what the model does, so they say nothing about whether its reply was any good. For the voice, my ear was the final test; automatic quality scores like UTMOS, a model that predicts how listeners would rate a clip, didn’t track what I heard.

For the facts test, a text-only LLM judge marks each answer right or wrong against the reference answers, and separately whether the model misheard the question, so a hearing problem doesn’t count as forgetting. The same questions typed, and base Gemma’s own answers, give the comparison, and together they answer whether the model still knows what base Gemma knows.

It’s a weaker guard than it looks, though, for two reasons. First, scoring the same checkpoint twice gives the same answer, but training the same recipe twice doesn’t. A lot of the questions are borderline for the model, and which way those fall shifts from one training run to the next. The run that scored best on one question set came last on a fresh one. So I only treat big, consistent movement as real; small differences between checkpoints are noise.

Second, it can’t see the risk reinforcement learning actually adds. The questions are short and asked in one go, so there’s no mid-question pause to start in early. A model that learned to jump in would score just as well here and still cut people off before they finish asking. The pause and turn-end measures cover that, but nothing yet checks the answers a model gives after starting early.

The bigger gap is that speech evaluation doesn’t scale yet. The judges I used see transcripts, so they’re good at what was said, but timing, overlap, prosody and voice quality never reach them, and my ear can’t listen to everything. Audio-native LLM judges don’t fix it. In my experience, Gemini-class models listening to the audio miss timing and overlap, are insensitive to artifacts, and aren’t calibrated. Public duplex benchmarks lean on exactly these judges, so they inherit the weakness. Part of the reason is that very few frontier models have real audio understanding to build judges from.

What the model still gets wrong

These are in rough order of how much they matter.

Real human speech. Everything was trained on synthetic voices, and the model answers far fewer recorded human questions spoken than typed. It hears rare words as common ones. A quiet mic goes unheard, and its own voice echoing back through the speakers breaks its speech into a repeated syllable, so live use needs echo cancellation or headphones.

A voice that can’t hear. The thinker/talker split costs the voice its expression. CSM picks prosody from Gemma’s text and its own recent audio, and nothing else; Gemma hears your tone, but there’s no way to pass it on. In practice that means:

It can’t match you. If you’re upset, quiet or in a hurry, the voice sounds the same as it always does.
It can’t say a word the way you said it. A name you just spoke gets pronounced from its spelling.
It can’t adjust to the room. People speak up in a noisy room; the talker never knows the room is noisy.
It can’t stop like a person. When you interrupt, the audio is cut, rather than trailing off the way a speaker who hears you would.
It can’t make sounds that aren’t words: no laugh, no “hm” that carries meaning.

The chunk joins are also audible if you listen for them, and there’s only one voice.

No room for text or tool calls. Gemma gets one token per frame, which is plenty to keep up with speech. It isn’t enough for text meant to be shown on screen, like a link or a code snippet, or for a long tool call, which would have to squeeze into the gaps between spoken words.

Serving. I spent little time on kernels and serving, so staying inside the 80 ms budget is less reliable than it could be (Figure 21). Gemma’s attention shapes are awkward on Blackwell: current FlashAttention kernels there don’t support the sliding-window layers’ head size, and the global layers’ heads are wider still. Fighting that wasn’t worth it for a limited-compute experiment, at least so far.

Figure 21: Time per frame on the live server, one B200. Gemma and the talker run on separate threads, so each has its own 80 ms. Bars run to the 95th percentile; the dot marks the median and the tick the slowest frame. Gemma's step holds at about 31 ms. The talker is idle on most frames, but the frame that starts a new chunk can take almost all of the budget: the slowest took 79 ms.

Finally, every number in this post comes from my own runs, without the ablations or review a paper would get. It was also a solo project. I built it to give us the confidence to invest further in this direction, then came back to it to dig deeper for this post. So treat the findings as strong hints.

What’s next

There’s a lot left to do, in speech, in interaction and timing, and in training, data and evaluation.

Speech

The most interesting open problem I ran into is streaming speech generation: speaking text you haven’t fully seen, coherently with audio you’ve already played. The talker never has the whole input. Text arrives a clause at a time and can be cut off by a yield (Figure 22). Each chunk has to stay coherent with audio already committed, acoustically (timbre, loudness, room) and in delivery, where intonation depends on words that haven’t arrived yet, like a question at the end of a sentence.

In chunks (live)

The talker starts once the first sentence is written, then speaks each sentence as it completes

Gemmawrites the reply, a token a frame

talkerThe great dane, originated from Germany.It is one of the two largest dog breeds…

Whole

CSM gets the full sentence at once, so it can't start until Gemma has finished writing

Gemmawrites the reply, a token a frame

talkerdead airThe great dane, originated from Germany. It is one…

The same reply, rendered both ways

Listen for the pause and the rhythm resetting where the sentences join

In chunksas live

Wholefull text first

Figure 22: Speaking text that hasn't finished arriving. Gemma writes the reply about a token a frame. In chunks, the talker waits for a sentence end or a clause past 8 words; here the first sentence ends after six words, so it starts at 0.64 s. Each later chunk is a whole sentence, spoken conditioned on the audio before it but never on the words after it, which is where the join comes from. Rendered whole, CSM sees the full reply and sounds continuous, but it can't start until the writing is done. The lane timings are schematic; both recordings are real CSM renders of the same reply. Play both to compare.

My workaround, clause-level chunks conditioned on recent audio, works, but the joins are still audible. Streaming-text TTS does exist, and it’s worth trying in place of CSM: Kyutai’s delayed streams modeling can start speaking from text as it streams in.1818 Kyutai, Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling, 2025. Interruption-aware streaming TTS is starting to appear, like NVIDIA’s VoiceChat-TTS, which follows an LLM’s text stream and stops on an interruption token.1919 Casanova et al., VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents, NVIDIA, 2026. What I haven’t seen is one that follows a thinker that can change its mind mid-sentence. I have a few experiments running on this now.

The bigger step is a talker that hears. Right now CSM only gets Gemma’s words, which is where most of the limitations above come from. The plan is to let CSM read Gemma’s hidden state, and maybe the user’s audio, so the voice can follow the conversation, and then fine-tune thinker and talker together. Promptable voices, set from an in-context example, are the other obvious gap on the voice side. Those steps are also the path back toward end to end. Each step moves more of the voice into what the model learns, and each one is a small change to a setup that’s already cheap to experiment with. Once the two are trained jointly, folding the talker back into Gemma is the last step, and the hard part of that step is the voice-versus-recall tradeoff from earlier.

This is where encoder-free hearing should pay off. Gemma hears raw audio in the same space it writes text, with no encoder in front that was trained to keep only the transcript, so its hidden state should carry tone, pace and the room along with the words. A talker that reads it would get all of that without anything trained to bridge two representations. It also shortens the last step, since a model that already hears in the same space it writes is part of the way to one that speaks there too, which is why the ablations further down include symmetric input and output representations.

Two changes would free up frame budget. One-step generation, like Kyutai’s Pocket TTS, which samples each frame’s continuous latent in a single pass using drifting, a recent one-step training objective,2020 Deng et al., Generative Modeling via Drifting, 2026. Kyutai recently retrained Pocket TTS with it. would produce a frame’s audio at once instead of one codebook at a time. Larger chunks, as in the layout section, would give every step far more slack.

Interaction and timing

Timing comes first, picking up where the reinforcement learning leaves off. The model now waits through pauses but answers later than we’d like, so the next thing to try is putting the turn clips back, weighted so pauses and turn ends each drive half of the learning, to win the delay back. Now that Gemma rarely starts in a pause, the playout’s first chunk can get shorter too, and most of the remaining delay at a turn end goes with it. Interruptions need their own training clips, since the recordings held too few. And backchannels need training examples where the model speaks briefly over the person before any reward can learn where they go; right now even an “okay” from the person makes it stop.

The assistant’s slot can carry a thinking channel alongside the reply. A conversational model spends most of its time listening, and that time can go to reasoning or to starting work before you’ve finished talking (Figure 23). It needs more than the one text token per frame the layout allows, though, or the thinking crowds out the reply.

Turn based

Reasoning and the lookup start only after the caller finishes

callerMy order from Tuesday never arrived and…

thinkingreasoning

agentorder lookup

modeldead airIt shipped…

Thinking while listening

The model reasons and starts the lookup while the caller is still talking

callerMy order from Tuesday never arrived and…

thinkingreasoning while listening

agentorder lookup

modelmhmIt shipped Wednesday, it's out today

Figure 23: Waiting for the turn against working while listening. A conversational model spends most of its time listening. Spending that time on reasoning and delegated work means the answer is ready when the caller stops talking.

Delegation is the other half. In my last post, Intelligence is not instant, I argued that the model talking to you shouldn’t also be the agent doing the work. The work has to come off the spoken path, and the conversation has to keep moving while it happens. In July, OpenAI shipped GPT-Live-1, a full-duplex model that hands harder reasoning and tool calls to a backend agent while the conversation continues.2121 OpenAI, GPT-Live 1, July 2026. It’s the same architecture we run in production at Maple, and it’s good to see a frontier lab land there too. I’m surprised more of the industry hasn’t gone this way, since to me it’s the most natural approach. I plan to write a follow-up on how we build that behavior and the relevant details.

Further out are proactivity and vision, an assistant that sees what you see and speaks up unprompted.

Training, data and evaluation

Moving the voice back into one model brings the voice-versus-recall tradeoff back with it, so the first open question is whether that tradeoff shrinks with scale. There’s a whole range of fixes worth trying first, from routing the gradient more carefully to large interleaved pretraining, and I’ve collected them in the appendix. I’d also like proper ablations of encoder-free versus encoder-plus-projector hearing, and of symmetric input and output representations, to see how much they’d buy back toward speech-native.

The data is still missing two things. One is robustness to real acoustics, from training with the model’s own voice leaking into the user’s channel and with real mic and room conditions. The other is a data engine for spoken interaction, one that produces dialogue with realistic timing, overlap and spoken phrasing.

Evaluation needs judges built for audio, and specifically decision models. A decision model answers one narrow question about an input, like “did the assistant cut the speaker off here?”, with a typed, calibrated probability instead of free text. TypeSafe’s Jev is an example for text; they call the category System One models. They should give calibrated probabilities, hear everything in the clip, including non-speech sounds, paralinguistics, overlap and the room, and be cheap enough to run on every rollout, both to evaluate and as RL rewards. Text-side RL is already moving this way. Applied Compute runs a cheap judge over sampled traces at every training step to catch reward hacking, calibrated to a fixed false-positive rate.2222 Evan Ellis and Aniruddh Sriram, How reward hacking detection works in Ari, Applied Compute, October 2026. I want that for speech RL, run on the audio itself.

Wrapping up

Take a text LLM that already hears audio, lay the conversation out so duplex is ordinary next-token prediction, and train LoRA adapters on synthetic data, and you get a full-duplex model cheap enough to iterate on quickly. Having Gemma make its own audio glitched and made it noticeably dumber, so the voice became a separate model, which also made everything after it cheaper to try. Most of the conversational behavior came from cooking quality data, and the timing dynamics came from reinforcement learning.

The split is a step on the way to end to end, and the next one is a talker that hears the conversation. A lot else is still open. I haven’t gotten the model to be patient and quick at the same time, and there’s still no good way to evaluate audio as audio. Next time, I want to get into what the thinker can do with all that listening time, like handing off work while the conversation keeps going.

Try it

The code is on GitHub and the weights are on Hugging Face.

The weights are released under CC BY-NC 4.0, for non-commercial use, and the code under Apache 2.0. The model card lists what the weights build on and their attributions.

Appendix

Why producing speech might erode recall

A few reasons producing speech might erode recall:

Audio carries far more information than text. Eight codebooks of mostly non-linguistic detail against one text token, so the audio dominates the gradient and the shared weights spend their capacity on it.
The answer and the audio share one hidden state. The depth head reads Gemma’s final hidden state at exactly the positions where Gemma answers, so those positions have to encode acoustic detail too.
The fine-tuning is narrow. A hundred hours of one synthetic voice would cause some forgetting under any new objective, though stopping the gradient shows most of this cost is the audio specifically.

What I keep coming back to is that base Gemma answers about as well from audio as from text, so here the cost comes from generating speech. Other speech models show a similar intelligence drop from their text backbones,2323 Qwen team, Qwen3-Omni-Flash, on the intelligence drop from casual spoken input. Moshi against Helium for one,2424 Défossez et al., Moshi: a speech-text foundation model for real-time dialogue, 2024. Moshi answers spoken questions well below Helium, its text backbone, which the authors partly put down to its conversational fine-tuning. but those comparisons change hearing and speaking at once, so they can’t say which one costs the recall.

Shrinking the voice-versus-recall tradeoff

The fixes that might shrink it fall into five families:

Route the gradient more carefully. I tried passing back part of the audio gradient, and haven’t tried gradient surgery, which projects out the parts of one objective’s gradient that conflict with another’s.
Give the acoustics their own capacity, with an acoustic branch, modality-specific adapters, or a head that reads intermediate layers.
Make the backbone predict less acoustic detail, with semantic tokens or a continuous head it conditions on.
Protect what’s there, with a KL term to base Gemma or replayed text data.
Scale, with large interleaved pretraining, which is probably how the big labs pay this cost down and exactly what we can’t do here.

Beyond those, repeating the investigation on a bigger backbone, or another model family, would help say whether it’s a scale problem or a structural one. On the optimizer side, per-objective optimizer states, as in AdaTask, would normalize audio and text separately before mixing them. The worry with leaving Adam is that Gemma was pretrained with it, but Qu et al. find LoRA largely mitigates optimizer mismatch, especially at low rank.2525 Qu, Huang and Horvath, Can Muon Fine-tune Adam-Pretrained Models?, ICML 2026.

Acknowledgements

Thanks to my colleague James Rosen, who listened to a lot of crazy ideas and half-formed brainstorming along the way.

If you want to talk about any of this, DM me on X.

Citation

Please cite this work as:

Nick Wall, “Inside Frankenstein’s lab: a full-duplex model at home”, walln.dev, October 2026.

@misc{wall2026fullduplex,
  author = {Nick Wall},
  title = {Inside Frankenstein's lab: a full-duplex model at home},
  year = {2026},
  month = {oct},
  howpublished = {\url{https://walln.dev/blog/full-duplex-at-home/}},
  note = {Blog post}
}