M1-08: On-device transcription via SpeechAnalyzer #21

Merged

wk merged 2 commits from m1-08-transcription into main

2026-08-06 18:13:03 -04:00

agent commented

2026-08-06 18:12:50 -04:00

Member

Closes #8 — on-device transcription via SpeechAnalyzer, validated on wes-iphone (iOS 26.5.2). Speaking into the app produces cues with absolute timestamps. Nothing written to disk; issue #9 owns the VTT writer.

The design decision to look at first

The transcriber is fed from the file, not from the microphone. AudioFileSink gained an onWrite callback that fires with each buffer that actually reached the file, after conversion and only when the append succeeded. Transcription hangs off that.

A second tap on the engine would have been the obvious wiring and it is a worse one: it is a second, subtly different signal — a buffer the file dropped, half a buffer more caught either side of a route change — and a transcript describing audio the memo does not contain is wrong in a way nothing downstream can detect. It also means the transcriber sees one fixed format for the life of a recording, so a microphone swap costs it nothing at all.

Timestamps, and the bug that came out of getting them wrong

The first build on device transcribed nothing and said "Listening…" forever. SpeechAnalyzer had rejected the whole stream on the first buffer with SFSpeechErrorDomain code 2, "Audio input timestamp overlaps or precedes prior audio input".

The timestamps came from the file's 48 kHz frame clock, which is exact and already absolute. But the analyzer only accepts 16 kHz or 8 kHz, so the audio is resampled on the way in, and resampling does not preserve frame counts exactly. One tick of disagreement and the engine drops everything.

bufferStartTime is now counted in frames handed to the analyzer, at its own sample rate — monotonic by construction. Both clocks count the same audio so they stay aligned, and driftFromFile measures the gap in case they ever do not. The debug view shows it.

Second commit also turns LiveTranscriber.cues into an AsyncThrowingStream. It was an AsyncStream that just ended when the engine gave up, and an ended cue stream is indistinguishable from nobody speaking — which is exactly why the first failure was invisible from the phone.

Failing soft

Transcription cannot fail a recording. No model, no supported locale, an analyzer that will not start, an engine that dies mid-recording: every one of those ends with a memo and no transcript. RecordingTranscription.begin() returns nil and the sink is built without a destination.

The speech model is a several-hundred-megabyte download per locale and absent on a new phone, so it is offered with a progress bar rather than started silently on someone's cellular connection.

Reviewing it

Two commits. 92a42d3 is the feature, 305a956 is the fix and reads as its own story.

New directory Stash/Transcription/ — SpeechModel (assets and availability), LiveTranscriber (the analyzer), RecordingTranscription (per-recording orchestration and state), TranscriptCue (the value type). AudioTap was lifted out of AudioRecorder to keep it under the line limit; that one is a move.

Roughly 700 lines, a large share of it comment, because most of what is here is the reasoning about why the obvious version does not work.

Still open

The actual "done when" is cues with correct absolute timestamps, and correctness has not been checked by ear against a played-back recording. The Clock drift row exists to make that checkable. A microphone swap, a phone call, a long recording and stopping mid-sentence are all untested — the issue comment lists them.

No automated tests, because there is still no test target in this repo. That is a decision worth making before #9 builds a file format on top of this.

🤖 Generated with Claude Code

Closes #8 — on-device transcription via `SpeechAnalyzer`, validated on `wes-iphone` (iOS 26.5.2). Speaking into the app produces cues with absolute timestamps. Nothing written to disk; issue #9 owns the VTT writer. ## The design decision to look at first **The transcriber is fed from the file, not from the microphone.** `AudioFileSink` gained an `onWrite` callback that fires with each buffer that actually reached the file, after conversion and only when the append succeeded. Transcription hangs off that. A second tap on the engine would have been the obvious wiring and it is a worse one: it is a second, subtly different signal — a buffer the file dropped, half a buffer more caught either side of a route change — and a transcript describing audio the memo does not contain is wrong in a way nothing downstream can detect. It also means the transcriber sees one fixed format for the life of a recording, so a microphone swap costs it nothing at all. ## Timestamps, and the bug that came out of getting them wrong The first build on device transcribed nothing and said "Listening…" forever. `SpeechAnalyzer` had rejected the whole stream on the first buffer with `SFSpeechErrorDomain` code 2, "Audio input timestamp overlaps or precedes prior audio input". The timestamps came from the file's 48 kHz frame clock, which is exact and already absolute. But the analyzer only accepts 16 kHz or 8 kHz, so the audio is resampled on the way in, and resampling does not preserve frame counts exactly. One tick of disagreement and the engine drops everything. `bufferStartTime` is now counted in frames handed to the analyzer, at its own sample rate — monotonic by construction. Both clocks count the same audio so they stay aligned, and `driftFromFile` measures the gap in case they ever do not. The debug view shows it. Second commit also turns `LiveTranscriber.cues` into an `AsyncThrowingStream`. It was an `AsyncStream` that just ended when the engine gave up, and an ended cue stream is indistinguishable from nobody speaking — which is exactly why the first failure was invisible from the phone. ## Failing soft Transcription cannot fail a recording. No model, no supported locale, an analyzer that will not start, an engine that dies mid-recording: every one of those ends with a memo and no transcript. `RecordingTranscription.begin()` returns nil and the sink is built without a destination. The speech model is a several-hundred-megabyte download per locale and absent on a new phone, so it is offered with a progress bar rather than started silently on someone's cellular connection. ## Reviewing it Two commits. `92a42d3` is the feature, `305a956` is the fix and reads as its own story. New directory `Stash/Transcription/` — `SpeechModel` (assets and availability), `LiveTranscriber` (the analyzer), `RecordingTranscription` (per-recording orchestration and state), `TranscriptCue` (the value type). `AudioTap` was lifted out of `AudioRecorder` to keep it under the line limit; that one is a move. Roughly 700 lines, a large share of it comment, because most of what is here is the reasoning about why the obvious version does not work. ## Still open The actual "done when" is cues with **correct** absolute timestamps, and correctness has not been checked by ear against a played-back recording. The Clock drift row exists to make that checkable. A microphone swap, a phone call, a long recording and stopping mid-sentence are all untested — the issue comment lists them. No automated tests, because there is still no test target in this repo. That is a decision worth making before #9 builds a file format on top of this. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

agent added 2 commits

2026-08-06 18:12:51 -04:00

M1-08: transcribe on device from the same audio the file gets 92a42d3e43

`SpeechAnalyzer` fed from `AudioFileSink`'s writes rather than from a second
tap on the engine. The buffers handed to the transcriber are the ones that
actually reached the file, after conversion and only when the append
succeeded, so the transcript cannot describe audio the memo does not contain.
It also means the transcriber sees one fixed format for the life of a
recording, and a microphone swap costs it nothing.

Timestamps are supplied rather than inferred. Every buffer carries an explicit
`AnalyzerInput.bufferStartTime` from the file's frame clock, so the ranges come
back absolute to the start of the memo. Verified off-device by feeding
synthesized speech starting at a deliberate 10 s offset:

    fed audio from 10.0s to 14.19s
    cue 10.00 → 11.56  12345.
    cue 11.56 → 14.19  The quick brown fox jumps over the lazy dog.

Only finalised results are taken, and by not requesting the volatile ones
rather than by filtering them out — `reportingOptions` is empty. Measured with
the same probe: 2 results, 0 volatile. The `isFinal` check that remains is a
second line of defence.

Transcription cannot fail a recording. No model, no supported locale, an
analyzer that will not start: each ends with a memo and no transcript.
`RecordingTranscription.begin()` returns nil and recording proceeds.

The speech model is a several-hundred-megabyte download per locale and absent
on a new phone, so it is offered with progress rather than started silently.
`AssetInventory.reserve` keeps iOS from reclaiming it between memos.

Resolving the locale measured 65 ms against 3 ms for every other speech query,
and it sat between the record button and the file being opened. Prewarmed at
launch and memoised for the process.

Nothing is written to disk — issue 09 owns the VTT writer. Cues are logged with
transcript text left at `os_log`'s default privacy, and shown in the debug
capture view so the timestamps can be checked against the audio by ear.

M1-08: give the analyzer its own clock, and say so when it stops 305a956526

Transcription produced nothing on device. The transcript section said
"Listening…" and stayed there, because the analyzer had rejected the stream and
the failure had nowhere to go.

    SFSpeechErrorDomain Code=2
    "Audio input timestamp overlaps or precedes prior audio input"

The timestamps came from the audio file's 48 kHz frame clock, which is exact and
absolute and looked like the obvious source. But the analyzer only accepts
16 kHz or 8 kHz, so the audio is resampled on the way in, and resampling does
not preserve frame counts exactly. A start time then disagrees by a tick with
the duration of the buffer before it, and the engine drops the entire stream
rather than that one buffer.

`bufferStartTime` is now counted in frames actually handed to the analyzer, at
the analyzer's own sample rate — exact and monotonic by construction. Both
clocks still count the same audio, so an interruption is spliced out of both and
they stay aligned; `driftFromFile` measures the gap and the debug view shows it,
because a transcript sliding against its audio is what issue 09 cannot detect.

Reproduced and confirmed off-device by feeding synthesized speech at wall-clock
speed. Before, the results stream died at the first buffer. After:

    t+ 4.88s  FINAL  0.00→2.04   This is a test of live transcription.
    t+ 9.53s  FINAL  2.04→4.32   The recording keeps going for a while.
    t+12.77s  FINAL  4.32→7.80   Nothing should be finalised until the engine
    t+12.81s  FINAL  7.80→10.32  Here is a fourth sentence to give it plenty

Cues do arrive during a recording, roughly three seconds behind speech, with
contiguous ranges covering the audio.

The second defect was the silence. `LiveTranscriber.cues` ended without a word
when the engine gave up, and an ended cue stream is indistinguishable from
nobody speaking. It is an `AsyncThrowingStream` now and the reason reaches the
transcript section.

wk merged commit aa5edd556e into main

2026-08-06 18:13:03 -04:00

wk deleted branch m1-08-transcription

2026-08-06 18:13:03 -04:00

wk referenced this pull request from a commit

2026-08-06 18:13:04 -04:00

Merge pull request 'M1-08: On-device transcription via SpeechAnalyzer' (#21) from m1-08-transcription into main