M1-08: On-device transcription via SpeechAnalyzer #8

New issue

Closed

opened 2026-08-06 10:53:41 -04:00 by agent · 1 comment

agent commented

2026-08-06 10:53:41 -04:00

Member

Transcribe on-device, using Apple's speech engine, from the same audio tap that
feeds the recorder.

On-device is non-negotiable. Nothing about a Stash capture leaves the phone unencrypted,
ever — that constraint starts here and holds through M3, where the relay is blind by design.

Scope

SpeechAnalyzer fed from the same AVAudioEngine tap used by issue 05, so the audio and
the transcript are guaranteed to describe the same signal.
Emit finalized cues with timestamps absolute to the start of the recording, not
relative to any chunk.
Nothing written to disk yet — issue 09 owns the VTT writer. This issue ends at logged cues.

Two things to get right

Take the final transcription, not the live one. The live/partial hypotheses are lower
confidence and get revised. v2 has an open question about whether it kept the right one; do
not inherit the ambiguity.

Absolute timestamps from the start. Issue 09's whole design depends on it, and retrofitting
offsets later is the kind of thing that produces a transcript that is subtly wrong everywhere.

Not in scope

Custom vocabulary. On-device STT reliably mangles jargon — NIXL, CMX, Omnistation, DGX — and
an SFSpeechLanguageModel contextual-strings pass is the real fix. It is a known, wanted
enhancement and it is not M1.

Done when

Speaking into the app produces logged cues with correct absolute timestamps.

Milestone 1 of the Stash v3 rebuild, scoped 2026-08-06. M1 is a functioning iOS app on Wes's
phone with no sync at all — the entire goal is recording voice memos reliably and getting
them off the device by hand. Sync arrives in M3.

Working agreement for every issue in this repo: feature branch, tested before the PR opens
(on-device where the label says so), then Wes reviews the PR and we walk the code together.
Docs update in the same commit. A PR that takes more than 20 minutes to review is too big —
say so and split it.

Transcribe on-device, using Apple's speech engine, from the same audio tap that feeds the recorder. **On-device is non-negotiable.** Nothing about a Stash capture leaves the phone unencrypted, ever — that constraint starts here and holds through M3, where the relay is blind by design. ## Scope - `SpeechAnalyzer` fed from the same `AVAudioEngine` tap used by issue 05, so the audio and the transcript are guaranteed to describe the same signal. - Emit **finalized** cues with timestamps **absolute to the start of the recording**, not relative to any chunk. - Nothing written to disk yet — issue 09 owns the VTT writer. This issue ends at logged cues. ## Two things to get right **Take the final transcription, not the live one.** The live/partial hypotheses are lower confidence and get revised. v2 has an open question about whether it kept the right one; do not inherit the ambiguity. **Absolute timestamps from the start.** Issue 09's whole design depends on it, and retrofitting offsets later is the kind of thing that produces a transcript that is subtly wrong everywhere. ## Not in scope Custom vocabulary. On-device STT reliably mangles jargon — NIXL, CMX, Omnistation, DGX — and an `SFSpeechLanguageModel` contextual-strings pass is the real fix. It is a known, wanted enhancement and it is **not** M1. ## Done when Speaking into the app produces logged cues with correct absolute timestamps. --- *Milestone 1 of the Stash v3 rebuild, scoped 2026-08-06. M1 is a functioning iOS app on Wes's phone with **no sync at all** — the entire goal is recording voice memos reliably and getting them off the device by hand. Sync arrives in M3.* *Working agreement for every issue in this repo: feature branch, tested before the PR opens (on-device where the label says so), then Wes reviews the PR and we walk the code together. Docs update in the same commit. A PR that takes more than 20 minutes to review is too big — say so and split it.*

agent added the

labels

2026-08-06 10:53:41 -04:00

agent commented

2026-08-06 18:12:22 -04:00

Author

Member

Validated on wes-iphone, iOS 26.5.2. Speaking into the app produces cues.

The bug worth recording

The first build on device transcribed nothing. The transcript section said "Listening…" and stayed there. The analyzer had rejected the stream on the first buffer:

SFSpeechErrorDomain Code=2
"Audio input timestamp overlaps or precedes prior audio input"

Timestamps were taken from the audio file's 48 kHz frame clock — exact, already absolute, and the obvious source given what this issue asks for. But SpeechTranscriber.availableCompatibleAudioFormats is only 16 kHz and 8 kHz Int16, so the audio is resampled on the way in, and resampling does not preserve frame counts exactly. One tick of disagreement between a declared start time and the previous buffer's real duration, and the engine drops the entire stream rather than the one buffer.

bufferStartTime is now counted in frames actually handed to the analyzer, at the analyzer's own sample rate — monotonic by construction. Both clocks count the same audio, so an interruption is spliced out of both and they stay aligned. driftFromFile measures the gap and the debug view shows it in milliseconds, red past 250 ms.

The second half of that failure was that none of it was visible. LiveTranscriber.cues was an AsyncStream that simply ended when the engine gave up, and an ended cue stream is indistinguishable from nobody speaking. It is an AsyncThrowingStream now and the reason reaches the screen.

Verified off-device

A throwaway probe on macOS running the real SpeechAnalyzer path with Stash's exact formats and conversions, on synthesized speech. Not committed.

Timestamps are absolute. Audio fed starting at a deliberate 10 s offset returned cues at 10.00→11.56 and 11.56→14.19, not 0.00→1.56. AnalyzerInput.bufferStartTime is the time base.

Cues arrive during a recording, roughly three seconds behind speech, with contiguous ranges:

t+ 4.88s  FINAL  0.00→2.04   This is a test of live transcription.
t+ 9.53s  FINAL  2.04→4.32   The recording keeps going for a while.
t+12.77s  FINAL  4.32→7.80   Nothing should be finalised until the engine decides.
t+12.81s  FINAL  7.80→10.32  Here is a fourth sentence to give it plenty to work with.

No volatile results are emitted at all. With reportingOptions empty: 2 results, 0 volatile, 2 final. "Take the final transcription, not the live one" is satisfied by not requesting the live ones rather than by filtering them out, which is the stronger guarantee. The isFinal check that remains is a second line of defence.

Timing. supportedLocale(equivalentTo:) measured 65 ms against 3 ms for every other speech query, and it sat between the record button and the file being opened. Prewarmed at launch and memoised for the process.

Not validated

Whether the timestamps are right against real audio. Cues appear with plausible times, but nobody has played a recording back and checked that a cue names the moment the words were said. That is the actual "done when" of this issue and it is the one thing still open. The Clock drift row is there to make it checkable.
A microphone swap mid-recording. The design says the transcriber cannot notice one — it is fed the file's fixed format — but that has not been exercised.
A phone call mid-recording, and whether the transcript splices the interruption out the way the audio does.
A long recording. Everything tested has been under a minute. Nothing here bounds the cue array or the analyzer's memory.
Stopping mid-sentence. finish() awaits finalizeAndFinishThroughEndOfInput specifically so the last few seconds are not dropped, and that path has not been checked against a recording that ends mid-word.
A locale other than the one on this phone, and the model-download prompt, which never appeared because the model was already installed from a previous Stash build.
Nothing is covered by an automated test. There is still no test target in this repo.

Out of scope, as filed

Custom vocabulary. On-device STT mangles jargon and SFCustomLanguageModelData is the fix; the API is present in the SDK and unused here.

Validated on `wes-iphone`, iOS 26.5.2. Speaking into the app produces cues. ## The bug worth recording The first build on device transcribed nothing. The transcript section said "Listening…" and stayed there. The analyzer had rejected the stream on the first buffer: ``` SFSpeechErrorDomain Code=2 "Audio input timestamp overlaps or precedes prior audio input" ``` Timestamps were taken from the audio file's 48 kHz frame clock — exact, already absolute, and the obvious source given what this issue asks for. But `SpeechTranscriber.availableCompatibleAudioFormats` is only 16 kHz and 8 kHz Int16, so the audio is resampled on the way in, and resampling does not preserve frame counts exactly. One tick of disagreement between a declared start time and the previous buffer's real duration, and the engine drops the **entire stream** rather than the one buffer. `bufferStartTime` is now counted in frames actually handed to the analyzer, at the analyzer's own sample rate — monotonic by construction. Both clocks count the same audio, so an interruption is spliced out of both and they stay aligned. `driftFromFile` measures the gap and the debug view shows it in milliseconds, red past 250 ms. The second half of that failure was that none of it was visible. `LiveTranscriber.cues` was an `AsyncStream` that simply ended when the engine gave up, and an ended cue stream is indistinguishable from nobody speaking. It is an `AsyncThrowingStream` now and the reason reaches the screen. ## Verified off-device A throwaway probe on macOS running the real `SpeechAnalyzer` path with Stash's exact formats and conversions, on synthesized speech. Not committed. **Timestamps are absolute.** Audio fed starting at a deliberate 10 s offset returned cues at 10.00→11.56 and 11.56→14.19, not 0.00→1.56. `AnalyzerInput.bufferStartTime` is the time base. **Cues arrive during a recording**, roughly three seconds behind speech, with contiguous ranges: ``` t+ 4.88s FINAL 0.00→2.04 This is a test of live transcription. t+ 9.53s FINAL 2.04→4.32 The recording keeps going for a while. t+12.77s FINAL 4.32→7.80 Nothing should be finalised until the engine decides. t+12.81s FINAL 7.80→10.32 Here is a fourth sentence to give it plenty to work with. ``` **No volatile results are emitted at all.** With `reportingOptions` empty: 2 results, 0 volatile, 2 final. "Take the final transcription, not the live one" is satisfied by not requesting the live ones rather than by filtering them out, which is the stronger guarantee. The `isFinal` check that remains is a second line of defence. **Timing.** `supportedLocale(equivalentTo:)` measured 65 ms against 3 ms for every other speech query, and it sat between the record button and the file being opened. Prewarmed at launch and memoised for the process. ## Not validated - **Whether the timestamps are right against real audio.** Cues appear with plausible times, but nobody has played a recording back and checked that a cue names the moment the words were said. That is the actual "done when" of this issue and it is the one thing still open. The Clock drift row is there to make it checkable. - **A microphone swap mid-recording.** The design says the transcriber cannot notice one — it is fed the file's fixed format — but that has not been exercised. - **A phone call mid-recording**, and whether the transcript splices the interruption out the way the audio does. - **A long recording.** Everything tested has been under a minute. Nothing here bounds the cue array or the analyzer's memory. - **Stopping mid-sentence.** `finish()` awaits `finalizeAndFinishThroughEndOfInput` specifically so the last few seconds are not dropped, and that path has not been checked against a recording that ends mid-word. - **A locale other than the one on this phone**, and the model-download prompt, which never appeared because the model was already installed from a previous Stash build. - **Nothing is covered by an automated test.** There is still no test target in this repo. ## Out of scope, as filed Custom vocabulary. On-device STT mangles jargon and `SFCustomLanguageModelData` is the fix; the API is present in the SDK and unused here.

agent referenced this issue from a pull request that will close it,

2026-08-06 18:12:50 -04:00

M1-08: On-device transcription via SpeechAnalyzer #21