Most speech-to-text you've used works like a stream: you talk, words appear under you, and it feels like the model is listening continuously. That feeling is a construction. The on-device engine we run to keep transcription private has no streaming mode. None. And the way we're building live captions on top of it is the interesting part — so this is a walkthrough of the hard part, honestly labelled: it's in progress, not a shipped feature.

The problem: a model that only knows how to answer once

The engine is a small, quantized Whisper model running through whisper.cpp — a lean C++ port of a well-known open speech model, chosen precisely because it runs entirely on your machine with nothing phoning home. The catch is in its shape. Its decode call takes a fixed window of audio, transcribes it, and returns. That call is stateless: the internal working memory it builds up — the attention cache a transformer accumulates as it reads — is reset on every invocation. There is no "keep listening" entry point. There is no handle you hold across calls. Each call is a fresh, sealed question: here are N seconds of sound, what were the words? — and then it forgets you were ever there.

So the naive approach — "feed it audio continuously and read the captions off the other end" — is not a thing that exists. The API you'd want was never written, because the model wasn't built to work that way. It's a batch tool. We wanted a live one.

The move is to stop waiting for a streaming API and build the streaming ourselves, around a batch core we call over and over.

The trick: own the window, overlap the decodes, keep the tail hot

The component that does this is a small Rust piece that holds three things: the loaded model, a ring buffer, and a resampler. Together they turn a one-shot decoder into something that produces running text.

Resample first. Microphones on these machines typically capture at 48kHz. Whisper wants 16kHz. So the first thing that happens to incoming audio is a high-quality 3:1 downsample — a proper sinc resampler, not a crude every-third-sample throwaway, because feeding the model dirty audio costs you accuracy you can't get back.

Then a sliding ring buffer. The resampled audio flows into a fixed-size ring buffer — 30 seconds of it, 480,000 samples at 16kHz — a circular chunk of memory that always holds the most recent slice of sound and quietly overwrites the oldest as new audio arrives. It never grows. It's a moving window onto "the recent past," and it's the surface we decode from.

Then overlapping decodes on a heartbeat. Roughly once a second, we take a 7-second window out of that buffer and run the batch decoder on it. Seven seconds for a caption that updates every one second means each decode overlaps its neighbors heavily — the same words get transcribed several times across successive windows. That overlap is not waste; it's the whole mechanism. It's how we paper over the model's amnesia. Because the decoder has no memory of the last call, the audio itself has to carry the context forward, and the only way to do that is to let windows overlap so each new decode re-hears the recent past.

Then split the window: hot tail, cold body. Here's the part that makes it read as live rather than jumpy. Of that 7-second window, we treat the last 2 seconds as a context tail — provisional, still changing, likely to be revised as more audio arrives to disambiguate it. The 5 seconds before it are "hot" and settling. Text that has scrolled out past that trailing 2-second tail is considered stable: we emit it as final and it won't move again. Text still inside the tail stays provisional — shown, but subject to revision on the next pass. It's the same instinct as a person catching the tail end of a half-heard sentence and mentally rewriting it a beat later once the next words land.

Then de-duplicate at the seams. Because windows overlap, the same word gets decoded in consecutive passes, and a dumb concatenation would stutter — "the the quick quick brown." A language-aware layer above the decoder detects and removes those cross-window duplicates, so the stitched output reads as one clean running transcript instead of a pile of overlapping fragments.

flowchart TD
  A[Microphone audio · 48kHz] --> B[Sinc resampler · 3 to 1 downsample to 16kHz]
  B --> C[(Sliding ring buffer · 30s / 480,000 samples · oldest overwritten)]
  C -->|about once per second| D[Take a 7-second window · 5s hot + 2s context tail]
  D --> E[Batch decoder · stateless · KV cache reset every call]
  E --> F[Stitch overlapping decode with previous · drop duplicate words]
  F --> G{Is this text outside the trailing 2 seconds?}
  G -- yes --> H[Emit as FINAL · settled, won't change]
  G -- no --> I[Keep PROVISIONAL · revise on next pass]

The invariant we won't trade away: the audio is gone the moment it's used

There's a rule threaded through this that a cloud transcription service structurally cannot make. When raw audio samples cross the boundary into the speech component, that component copies them on entry and never holds onto the caller's buffer past the return. The samples it works from are its own; the originals aren't retained. Audio doesn't accumulate somewhere waiting to be mishandled — it's consumed and released as it flows through.

This matters because the sliding-window design is, by nature, holding a rolling buffer of your recent speech in memory. That's unavoidable — you can't decode sound you've already discarded. So the discipline is to make that window the only place raw audio lives, keep it bounded to those 30 seconds, and guarantee nothing leaks out the sides into some long-lived copy. A remote service, by contrast, has to receive your audio over the wire and hold it long enough to process on hardware you don't control. The privacy property here isn't a promise in a policy document; it's a shape in the code — bounded buffer in, transcript out, raw bytes gone. That's the Honesty axiom applied to sound.

One more tell that this path is meant to be non-optional: the release build literally refuses to compile unless the speech feature is switched on. Voice isn't a bolt-on that might or might not be present — the binary won't exist without it.

What's real today, and what isn't

Now the honest part, because the honest part is the point.

This live-transcription path is in progress, not finished, not celebrated. The broader voice-and-video effort it belongs to had its overall celebration retracted earlier this year, and it's back in implementation. The one rung that is owner-accepted is narrow: audio actually arriving at the tap — the microphone-to-buffer plumbing genuinely working end to end. The transcription mechanism this article describes is wired up and lives on the main line, but it has not been rung-celebrated as a finished, user-facing feature. Recent work around it has been plumbing and gap-closing — for instance, making sure the small model file ships to the right place so the engine can find it — not the epic's completion.

Two more limits worth stating plainly. On Apple platforms, the live path uses a different speech engine entirely, tuned for that hardware, rather than the whisper.cpp path described here. And on Android, this Rust whisper path isn't present yet at all. So "we do live on-device captions" is not a flat, everywhere claim — it's a mechanism that is real on one path, being built out, with the edges honestly ragged.

What's genuinely settled is the idea, and it's a good one: you don't need a model with a streaming API to build a live caption stream. You need to own the window yourself — resample clean, keep a bounded rolling buffer, decode overlapping slices on a heartbeat, hold the tail provisional and let the body settle, stitch and de-duplicate the seams — and trade a little latency and some recompute for the feeling of the words keeping up with the voice. It's a batch model wearing a real-time coat we tailored by hand. We're still hemming it. But the clever part — the part that makes a stateless, one-shot decoder feel like it's listening — is real, and it's on the main line today.

For the wider reasoning behind keeping the voice on your own machine in the first place, see giving the system a voice that stays home.

Written by AI agents from real project logs; owned and edited by Mujo.

Written by AI agents from real project logs; owned and edited by Mujo.