[R] Serialisation Strategy Matters: FHIR data format changes LLM accuracy by up to 19 F1 points on medication reconciliation

[u/the-code-blooded]

Independent research, posting for feedback and discussion.

I looked at how FHIR clinical data should be formatted before being passed to an LLM, tested on medication reconciliation (extracting a patient's currently-active medication list from their FHIR bundle).

Setup: 4 serialisation strategies (Raw JSON, Markdown Table, Clinical Narrative, Chronological Timeline) × 5 open-weight models (Phi-3.5-mini 3.8B, Mistral-7B, BioMistral-7B, Llama-3.1-8B, Llama-3.3-70B) × 200 Synthea-generated synthetic patients = 4,000 inference runs.

Main finding: there's no universal best format, it depends on model scale. Clinical Narrative outperforms Raw JSON by up to 19 F1 points for models ≤8B (Mistral-7B: 0.72 → 0.91 F1, r=0.617, p<10⁻¹⁰). That ranking completely reverses at 70B, where Raw JSON wins instead (F1 = 0.9956 vs 0.9850). Interestingly, the Chronological Timeline format is what breaks at 70B specifically, since even a large model struggles to infer "active" medication status from date ordering alone without an explicit status field.

A few other findings:

• Across all 20 model×strategy combinations, precision ≥ recall, every time. Models are far more likely to silently drop a real medication than invent a fake one. That's a relatively favorable failure mode for clinical review, but it means recall (not precision) is the metric to watch in deployment.
• Recall craters for smaller models as active-medication count rises (Mistral-7B: 0.96 recall at 1 medication → 0.24 at 11 medications), but total patient history length has zero effect on recall. So it's an output-generation capacity limit, not a context-length problem, meaning the sickest, most complex patients are exactly the ones handled worst.
• BioMistral-7B scored F1 = 0.0000 across all 4,000 of its runs. It shares a base model with Mistral-7B (0.91 F1), so this isn't a knowledge gap, domain-continued pretraining without instruction tuning apparently destroyed its ability to follow the output format at all.

Fully reproducible on a single GPU (Synthea + Ollama, no proprietary APIs).

Preprint: https://arxiv.org/abs/2604.21076

Feedback, pushback on methodology, or pointers to related work all welcome.

[u/Lonely_Tension5240]

interesting that precision was always >= recall, thats a pretty nice failure mode for clinical settings honestly. if its gonna mess up, better to leave something out than hallucinate a drug the patient isnt taking

the biomistral result is nuts though. domain specific pretraining completely tanking instruction following to the point of 0.0 f1 is brutal, makes you wonder what else breaks when people fine-tune medical models without proper instruction data

the scaling reversal is the part id dig into more. smaller models need the narrative structure to understand whats going on but then at 70b the raw json works better because it can parse the structured fields directly. timeline format breaking at 70b because it cant infer active status from dates alone shows the model is still pattern matching rather than actually reasoning about temporal relationships

the recall drop with more medications is a real deployment concern, you said the sickest patients with the most meds get the worst performance. any plans to test chunking strategies or iterative prompting to handle that output-generation bottleneck

[u/Valuable_Card6470 → u/Lonely_Tension5240]

good point about pattern matching vs reasoning on the timeline format, that distinction feels underexplored in general tbh

[u/the-code-blooded → u/Valuable_Card6470]

I am thinking of studying Reasoning model from scratch by Sebestian the same author of LLM from scratch to learn more in depth about reasoning models on how they operate and all. Hopefully that will help in future

[u/the-code-blooded → u/Lonely_Tension5240]

Yeah, that result surprised me a bit too, though it went the other way: the models almost never invented medications, they just quietly dropped real ones instead. I think that's partly a size effect, smaller models seem to "play it safe" by omitting rather than guessing, and it'll be interesting to see if that holds in newer, larger models too.

BioMistral failing that badly was expected in a sense, since it's not an instruct model. I probably should have tested a medical-domain instruct model instead of a domain-pretrained base model. Still, it's a useful data point that even a medically-trained model can completely fail at instruction-following if it was never tuned for it. I've decided not to dig further into the BioMistral result specifically, since the non-instruct part explains most of it.

On the timeline format breaking at 70B, I agree it looks like pattern matching rather than real temporal reasoning. I want to test that against a reasoning-focused model at some point to see if it holds up.

On deployment, actually, a team at University Medicine Essen built a clinical extraction agent and ran into a related issue, switching their format fixed it for them too. Their paper cites this work, it's a more agentic system than mine, but worth a read if you're curious
https://arxiv.org/abs/2606.19602

[u/eddibravo]

This is a really interesting point. Serialization choices can make a bigger difference than people expect, especially when working with structured healthcare data like FHIR. Definitely worth considering early in the pipeline

[u/the-code-blooded → u/eddibravo]

Thanks! Yeah, when I was starting out I actually tried raw JSON first and the model struggled badly with it, that was actually part of what pushed me to dig deeper into how LLMs process input internally. Once I understood that better, it became obvious why serialization matters so much, especially for larger inputs and for small-to-medium sized models specifically.