Running a Health Voice: All Private on Your Mac

There is this promise that a doctor can just talk, and a note will write itself. It feels good & easy when you say it like that. Talk, note, done. But the patient's voice has to go somewhere, and most of these products send it to a cloud box you are supposed to trust. That part never sat right with me. You are in a room with a person. Their audio leaving that room is not a small detail.

So I picked up the "Voice AI Assistant for Healthcare" case study from ombharatiya's system design guide. I did not want another slide with boxes and arrows. I wanted to know, if I actually sit and build this, how much of it is real. Press record, talk, and at the end a signed note is sitting in a real FHIR server. That was the itch.

A few questions I had:

• Can the audio stay on my machine and still become a proper note?
• If the case study says real time, what does that even mean on a laptop?
• Who is allowed to invent a sentence here, and who is not?

The rule I gave myself: the voice never leaves. Transcription, who is speaking, medical terms, all of that has to happen locally. The only thing that can go out is a finished transcript, and even that comes back as a draft. A human has to look at it, change it, put a name on it. Otherwise I am just making another box to trust.

Ok, let us walk through it the way it actually runs, not the way the diagram looks.

You open the console. First you record about ten seconds of the nurse. That is just to make a voiceprint, so later I can tell who is who. Then the encounter starts. The browser takes the mic, squashes it down to 16kHz mono, and pushes it over a WebSocket to FastAPI.

Silence is the first problem. If I send every quiet frame into Whisper, Whisper starts making things up. So silero VAD cuts the stream into utterances. Only speech goes forward.

Then there are two Whisper models, and this is where I stopped pretending. The case study talks about real time as if the model just keeps up. On this Mac, large-v3-turbo needs around 2 seconds before it even starts, every single call, even for a short clip. There is no trick around that locally. Their numbers assume a datacenter GPU. Mine does not have one. So base.en talks first, fast and messy, just so the screen feels alive. The big model comes behind it and writes the line I actually keep. I did not like this at first. It felt like cheating. Then I realised the other option was to lie about latency, and I liked that even less.

Each finished line gets a speaker label from the voiceprint, and a biomedical NER pass for symptoms, meds, vitals, the usual. I also run three checks that are just code. Safety. Spoken self-correction. Later, did the note forget something that was said. I did not want a model doing those. If something is wrong I want to point at the exact phrase, not at a score.

When you hit stop, the whole speaker-labeled transcript goes to a SOAP endpoint. What is SOAP here? Basically the shape of a clinical note. Subjective, objective, assessment, plan. gpt-5.5 fills that shape from the transcript, with a strict JSON schema, and I tell it in the prompt: do not invent a fact. It comes back as a draft. Someone edits it. Types their name. Hits approve. Only then I build a FHIR bundle myself, in code, and POST it to a local HAPI server, and write one boring line in an audit file. FHIR is just the hospital's way of storing the record. I am not going to let a model be creative at the door of the medical record. That idea made me uncomfortable from day one.

A few moments that stayed with me.

I first put the SOAP call inside the WebSocket, right when the encounter ended. Felt tidy. Then I turned on pyannote to refine speakers. On CPU that can sit there for 30, 40, 90 seconds. The frontend gave up waiting. The note never arrived. No crash. Just an empty screen, and me staring at it thinking the model had failed, when it was my own wiring. I pulled SOAP out into POST /soap. Suddenly I could also regenerate a note from an old transcript without recording again. That bug annoyed me and then I was glad it happened.

The one that still makes me laugh is Whisper on silence. Near-quiet room, and it starts speaking YouTube. "Thanks for watching." "Please subscribe." "Consult a qualified healthcare professional." One afternoon it wrote "love love love" sixteen times. The NER, very sincerely, marked "love" as a lab value. Then the completeness check got angry that "love" was missing from the note. I sat there looking at this thing I had built, a clinical pipeline arguing about love. The fix was ugly and I like that it is ugly. Drop the high no-speech segments. Drop the one-word-repeated nonsense. Regex for the stock phrases. If the utterance is weak, do not even send it to NER. Real speech stays. Garbage does not get a second chance to become a vital.

Safety I refused to outsource. Regex. If it fires I can show you the words. A false alert is a glance. A miss is the actual failure. Red banner, no FHIR push, someone has to check a box. I do not want to argue with a model about whether a sentence was dangerous.

Even the SOAP call had a small stupid moment. I send temperature because that is what I have always sent. gpt-5.5 says no, reasoning models only want the default. So I catch that complaint and retry without it. Nothing deep. Just the kind of thing that ruins a demo at 2am if you assume the old API still behaves.

I keep coming back to the same feeling. Listening can be a model. Drafting can be a model. Deciding that this note is true enough to file cannot be. That last step needs a name on it.

That is Health Voice for me. Not a product pitch. A way to sit with that discomfort and see how far I could take it on one machine, without sending the room away to someone else.