Health Voice

Clinicians spend a huge chunk of every day typing instead of looking at the patient. Ambient scribes exist, but most of them are cloud boxes you talk into and trust. For a hospital that means patient audio and PHI leaving the building, which makes the compliance story hard and the trust story harder.

I picked up the "Voice AI Assistant for Healthcare" case study from ombharatiya's system design guide and wanted to know how much of it I could actually build for real, not as boxes-and-arrows on a slide. Not a mock with fake data and a happy-path demo. A thing where I press record, talk, and a signed note shows up in a real FHIR server at the end.

The catch I set for myself: keep the patient audio on the machine. Transcription, speaker ID, and medical term extraction all run locally. The only thing allowed to touch the network is the step that turns a finished transcript into a structured note, and even that gets reviewed and edited by a human before anything is filed.

HealthVoice is the full pipeline from the case study, running on a single Apple Silicon Mac. ▎ You open the console in the browser and record about ten seconds of the nurse's voice once. That builds a voiceprint. Then you start an encounter. The browser downsamples the mic to 16kHz mono PCM and streams it over a WebSocket to a FastAPI server. From there: ▎

• silero VAD splits the stream into utterances so I am not transcribing silence.
• A fast Whisper model (base.en) transcribes live partials so the screen feels alive. A slower, accurate model (large-v3-turbo) re-transcribes each finished utterance.
• Every finished utterance gets a speaker label by comparing its voiceprint to the enrolled nurse, plus a pass of biomedical NER that pulls out symptoms, meds, vitals, conditions, and so on.
• Three deterministic checks run on the same text: a safety scan (self-harm, abuse, mandatory reporting), a self-correction detector, and later a completeness check against the note.
• When you stop, the whole speaker-labeled transcript goes to a stateless SOAP endpoint. gpt-5.5 turns it into a structured note with a strict JSON schema. It is told, in the system prompt, to never invent a fact.
• The note shows up as an editable draft. The clinician fixes it, types their name, and hits approve. That builds a FHIR R4 transaction bundle (Patient, Encounter, MedicationStatement, Composition, plus a Provenance record naming the human and the AI), POSTs it to a local HAPI server, and writes a local audit line.

▎ The honest version of "real time": true sub-500ms streaming is not possible locally because the MLX Whisper encoder has a roughly 2 second fixed cost per call on this Mac. The case study's latency number quietly assumes datacenter GPUs. So I made it feel live with the draft model while the accurate model catches up, instead of pretending the hardware is something it isn't.

A few of the parts that were more annoying than they look.

▎ The 2 second wall. MLX Whisper large-v3-turbo pays a fixed encoder cost of around 2 seconds per call on this Mac regardless of clip length. There is no streaming around it locally. So the design is two models: base.en for fast partials that keep the screen moving, large-v3-turbo for the per-utterance final you actually trust. Ingestion is decoupled from transcription with a producer/consumer queue so the mic never waits on the model.

▎ Whisper hallucinates on silence. This was the demo-killer. On near-silent audio Whisper confidently emits things it learned from YouTube: "thanks for watching", "please subscribe", and the medical disclaimer "consult a qualified healthcare professional". One day it produced "love love love" sixteen times and the biomedical NER happily labeled "love" as a lab value. So the entity panel filled with garbage and the completeness check started complaining that "love" was missing from the note. The fix is three deterministic layers, no model involved: drop decoded segments whose no_speech_prob or compression_ratio is too high or that are just one token repeated or that match a known stock-phrase regex; a small denylist in the NER stage as a backstop; and skip NER entirely for any utterance below the confidence threshold so untrusted audio never reaches the note. Clinical speech is untouched.

▎ Safety is not the LLM's job. The self-harm / abuse / mandatory-reporting scan is curated regex, on purpose. An alert can always be explained by the exact phrase it matched, it can't be argued away, and it can't be hallucinated into or out of existence. It is tuned for recall: a false alert costs a glance, a missed one is a real failure. When it fires, the UI shows a red banner that blocks the FHIR push until someone checks a box.

▎ gpt-5.5 rejects custom temperature. The reasoning models only allow the default temperature, so the SOAP call sends my temperature and, if the API complains specifically about temperature, retries once without it. Small thing, but it's the kind of detail that breaks a demo if you assume the older Chat Completions contract.

▎ FHIR by hand. The bundle is built explicitly rather than by an LLM, because filing is the one place you really do not want a creative model. The note's assessment stays as narrative text the clinician wrote. I never fabricate a coded Condition resource just to look complete.

_HALLUCINATION_RE = re.compile(
    "|".join([
        r"thank(s| you)?(?: (?:so|very) much)?(?: for (?:watching|listening))",
        r"please (?:like|subscribe|comment)",
        r"don'?t forget to subscribe",
        r"substitute for (?:professional )?medical advice",
        r"qualified healthcare professional",
        r"for (?:educational|informational) purposes",
        r"consult (?:your|a) (?:doctor|physician|healthcare)",
    ]),
    re.IGNORECASE,
)

def _is_degenerate(text: str) -> bool:
    """True for an utterance that is just one token repeated (e.g. 'love love')."""
    words = _WORD_RE.findall(text.lower())
    return len(words) >= 2 and len(set(words)) == 1

@staticmethod
def _keep(segment: dict) -> bool:
    """False for a decoded segment that looks like a Whisper hallucination."""
    if segment.get("no_speech_prob", 0.0) > config.NO_SPEECH_THRESHOLD:
        return False
    if segment.get("compression_ratio", 0.0) > config.COMPRESSION_RATIO_THRESHOLD:
        return False
    text = (segment.get("text") or "").strip()
    if not text:
        return False
    if _is_degenerate(text) or _HALLUCINATION_RE.search(text):
        return False
    return True

def scan_safety(text: str) -> list[dict]:
    """Return safety alerts: [{category, label, severity, term, snippet}]."""
    if not text:
        return []
    alerts, seen = [], set()
    for cat in _SAFETY_CATEGORIES:
        for rx in cat["_compiled"]:
            m = rx.search(text)
            if not m:
                continue
            key = (cat["category"], m.group(0).lower())
            if key in seen:
                continue
            seen.add(key)
            alerts.append({
                "category": cat["category"],
                "label": cat["label"],
                "severity": cat["severity"],
                "term": m.group(0),
                "snippet": _snippet(text, m.start(), m.end()),
            })
    return alerts

Snippet 3 — label gpt-5.5 only allows the default temperature, language python:
try:
    resp = client.chat.completions.create(
        model=config.OPENAI_MODEL,
        temperature=config.SOAP_TEMPERATURE,
        messages=messages,
        response_format=response_format,
    )
except Exception as exc:
    # Reasoning models (gpt-5 / o-series) only allow the default temperature.
    if "temperature" not in str(exc).lower():
        raise
    resp = client.chat.completions.create(
        model=config.OPENAI_MODEL,
        messages=messages,
        response_format=response_format,
    )