M1-08: On-device transcription — choose the engine #8

New issue

Open

opened 2026-08-08 20:03:44 -04:00 by agent · 0 comments

agent commented

2026-08-08 20:03:44 -04:00

Member

Transcribe on the device, with no audio leaving the phone. This is the issue in the Android port
that does not translate, and it should be worked as a decision before it is worked as a feature.

Why iOS's answer does not port

iOS uses SpeechAnalyzer, which streams a live recording and emits finalised results for as long
as the recording runs. Forty minutes of walking produces forty parts and that is the design.

Android's platform SpeechRecognizer is documented as not intended for continuous recognition,
and it stops on roughly five seconds of silence. createOnDeviceSpeechRecognizer() (API 33+) forces
recognition on-device and fails cleanly if no local engine is available, which is the right privacy
posture — but it does not remove the silence timeout or the short-utterance design. A forty-minute
walk with pauses in it is precisely the case it is not built for.

Restarting a session on every timeout is the common workaround. It drops words at every seam, burns
battery, and would put a gap in the transcript exactly where somebody stopped to think — which is
often right before the sentence worth keeping.

The Pixel 6's Tensor chip does run Google's own on-device speech model, which is what the stock
Recorder app uses for its live transcription. That engine is not exposed to third-party apps. Being
told the phone "has a hardware transcription engine" is true and does not mean this app can reach
it — worth confirming rather than assuming, because if some route to it exists it is the best
answer available.

The candidates

whisper.cpp over minute-sized windows. Fully offline, no silence timeout, quality close to what
Apple's engine gives. It is batch rather than streaming — but the transcript design is already
one VTT part per minute, so running it over each completed minute of audio maps onto the existing
format more directly than iOS's streaming analyzer does. Costs: a model in the APK or downloaded on
first run (tens to a couple of hundred MB), CPU and battery during a walk, and a transcript that
lags the recording by up to a minute plus processing time.

Vosk. Genuinely streaming, light enough for constrained hardware, no timeout problem. Lower
accuracy than Whisper on long-form speech.

Platform SpeechRecognizer with session restart. Cheapest to build, and it has the seam problem
described above.

The recommendation is whisper.cpp, on the strength of it fitting the parts design rather than
fighting it. Wes's call at the planning checkpoint, and it should be an informed one — this
choice sets transcript quality for every Android memo and it is expensive to revisit.

Non-negotiable regardless of engine

No audio and no transcript ever leaves the device. The 8/6 scope is explicit: under no
circumstances, debug logging or not, is unencrypted audio or unencrypted speech-to-text available
to anyone but the user. An engine that falls back to a network service when the local model is
missing is disqualified — it has to fail visibly instead.

Suggested shape

Worth splitting after the decision: a spike that measures the chosen engine on the Pixel against a
real walk recording, then the implementation. If the spike says the answer is wrong, that is a
cheap finding rather than an expensive one.

Done when

An engine is chosen with the reasoning written down in stash-docs, and a forty-minute recording
on the Pixel produces a transcript with no gaps at the seams and nothing sent off the device.

Milestone 1 of Stash for Android, mirroring stash-ios M1. M1 is a functioning app on the test
device with no sync at all — record voice memos reliably and get them off the device by hand.
Sync arrives in M3.

Test device is a Pixel 6 (Tensor G1) borrowed from a friend, until something better turns up.
Stock Android, so no OEM battery-killer behaviour to fight — worth remembering that a phone from
another manufacturer may not be as forgiving about background work.

Translation of Stash/stash-ios#8. Where the platform forces a different answer this issue says so; where it
does not, the iOS issue is the fuller statement of intent and should be read alongside this one.

Working agreement for every issue in this repo: feature branch, tested before the PR opens
(on-device where the label says so), then Wes reviews the PR and we walk the code together. Docs
update in the same commit. A PR that takes more than 20 minutes to review is too big — say so and
split it.

Transcribe on the device, with no audio leaving the phone. **This is the issue in the Android port that does not translate, and it should be worked as a decision before it is worked as a feature.** ## Why iOS's answer does not port iOS uses `SpeechAnalyzer`, which streams a live recording and emits finalised results for as long as the recording runs. Forty minutes of walking produces forty parts and that is the design. **Android's platform `SpeechRecognizer` is documented as not intended for continuous recognition**, and it stops on roughly five seconds of silence. `createOnDeviceSpeechRecognizer()` (API 33+) forces recognition on-device and fails cleanly if no local engine is available, which is the right privacy posture — but it does not remove the silence timeout or the short-utterance design. A forty-minute walk with pauses in it is precisely the case it is not built for. Restarting a session on every timeout is the common workaround. It drops words at every seam, burns battery, and would put a gap in the transcript exactly where somebody stopped to think — which is often right before the sentence worth keeping. **The Pixel 6's Tensor chip does run Google's own on-device speech model**, which is what the stock Recorder app uses for its live transcription. That engine is not exposed to third-party apps. Being told the phone "has a hardware transcription engine" is true and does not mean this app can reach it — worth confirming rather than assuming, because if some route to it exists it is the best answer available. ## The candidates **whisper.cpp over minute-sized windows.** Fully offline, no silence timeout, quality close to what Apple's engine gives. It is batch rather than streaming — but the transcript design is *already* one VTT part per minute, so running it over each completed minute of audio maps onto the existing format more directly than iOS's streaming analyzer does. Costs: a model in the APK or downloaded on first run (tens to a couple of hundred MB), CPU and battery during a walk, and a transcript that lags the recording by up to a minute plus processing time. **Vosk.** Genuinely streaming, light enough for constrained hardware, no timeout problem. Lower accuracy than Whisper on long-form speech. **Platform `SpeechRecognizer` with session restart.** Cheapest to build, and it has the seam problem described above. The recommendation is whisper.cpp, on the strength of it fitting the parts design rather than fighting it. **Wes's call at the planning checkpoint**, and it should be an informed one — this choice sets transcript quality for every Android memo and it is expensive to revisit. ## Non-negotiable regardless of engine **No audio and no transcript ever leaves the device.** The 8/6 scope is explicit: under no circumstances, debug logging or not, is unencrypted audio or unencrypted speech-to-text available to anyone but the user. An engine that falls back to a network service when the local model is missing is disqualified — it has to fail visibly instead. ## Suggested shape Worth splitting after the decision: a spike that measures the chosen engine on the Pixel against a real walk recording, then the implementation. If the spike says the answer is wrong, that is a cheap finding rather than an expensive one. ## Done when An engine is chosen with the reasoning written down in `stash-docs`, and a forty-minute recording on the Pixel produces a transcript with no gaps at the seams and nothing sent off the device. --- *Milestone 1 of Stash for Android, mirroring `stash-ios` M1. M1 is a functioning app on the test device with **no sync at all** — record voice memos reliably and get them off the device by hand. Sync arrives in M3.* *Test device is a **Pixel 6** (Tensor G1) borrowed from a friend, until something better turns up. Stock Android, so no OEM battery-killer behaviour to fight — worth remembering that a phone from another manufacturer may not be as forgiving about background work.* *Translation of `Stash/stash-ios#8`. Where the platform forces a different answer this issue says so; where it does not, the iOS issue is the fuller statement of intent and should be read alongside this one.* *Working agreement for every issue in this repo: feature branch, tested before the PR opens (on-device where the label says so), then Wes reviews the PR and we walk the code together. Docs update in the same commit. A PR that takes more than 20 minutes to review is too big — say so and split it.*

agent added the

labels

2026-08-08 20:03:44 -04:00