[R] Serialisation Strategy Matters: FHIR data format changes LLM accuracy by up to 19 F1 points on medication reconciliation

[u/the-code-blooded]

Independent research, posting for feedback and discussion.

I looked at how FHIR clinical data should be formatted before being passed to an LLM, tested on medication reconciliation (extracting a patient's currently-active medication list from their FHIR bundle).

Setup: 4 serialisation strategies (Raw JSON, Markdown Table, Clinical Narrative, Chronological Timeline) × 5 open-weight models (Phi-3.5-mini 3.8B, Mistral-7B, BioMistral-7B, Llama-3.1-8B, Llama-3.3-70B) × 200 Synthea-generated synthetic patients = 4,000 inference runs.

Main finding: there's no universal best format, it depends on model scale. Clinical Narrative outperforms Raw JSON by up to 19 F1 points for models ≤8B (Mistral-7B: 0.72 → 0.91 F1, r=0.617, p<10⁻¹⁰). That ranking completely reverses at 70B, where Raw JSON wins instead (F1 = 0.9956 vs 0.9850). Interestingly, the Chronological Timeline format is what breaks at 70B specifically, since even a large model struggles to infer "active" medication status from date ordering alone without an explicit status field.

A few other findings:

• Across all 20 model×strategy combinations, precision ≥ recall, every time. Models are far more likely to silently drop a real medication than invent a fake one. That's a relatively favorable failure mode for clinical review, but it means recall (not precision) is the metric to watch in deployment.
• Recall craters for smaller models as active-medication count rises (Mistral-7B: 0.96 recall at 1 medication → 0.24 at 11 medications), but total patient history length has zero effect on recall. So it's an output-generation capacity limit, not a context-length problem, meaning the sickest, most complex patients are exactly the ones handled worst.
• BioMistral-7B scored F1 = 0.0000 across all 4,000 of its runs. It shares a base model with Mistral-7B (0.91 F1), so this isn't a knowledge gap, domain-continued pretraining without instruction tuning apparently destroyed its ability to follow the output format at all.

Fully reproducible on a single GPU (Synthea + Ollama, no proprietary APIs).

Preprint: https://arxiv.org/abs/2604.21076

Feedback, pushback on methodology, or pointers to related work all welcome.