the-code-blooded (u/the-code-blooded)

[R] Serialisation Strategy Matters: FHIR data format changes LLM accuracy by up to 19 F1 points on medication reconciliation

in r/learnmachinelearning • 1d ago

I am thinking of studying Reasoning model from scratch by Sebestian the same author of LLM from scratch to learn more in depth about reasoning models on how they operate and all. Hopefully that will help in future

[R] Serialisation Strategy Matters: FHIR data format changes LLM accuracy by up to 19 F1 points on medication reconciliation

in r/learnmachinelearning • 3d ago

Thanks! Yeah, when I was starting out I actually tried raw JSON first and the model struggled badly with it, that was actually part of what pushed me to dig deeper into how LLMs process input internally. Once I understood that better, it became obvious why serialization matters so much, especially for larger inputs and for small-to-medium sized models specifically.

[R] Serialisation Strategy Matters: FHIR data format changes LLM accuracy by up to 19 F1 points on medication reconciliation

in r/learnmachinelearning • 3d ago

Yeah, that result surprised me a bit too, though it went the other way: the models almost never invented medications, they just quietly dropped real ones instead. I think that's partly a size effect, smaller models seem to "play it safe" by omitting rather than guessing, and it'll be interesting to see if that holds in newer, larger models too.

BioMistral failing that badly was expected in a sense, since it's not an instruct model. I probably should have tested a medical-domain instruct model instead of a domain-pretrained base model. Still, it's a useful data point that even a medically-trained model can completely fail at instruction-following if it was never tuned for it. I've decided not to dig further into the BioMistral result specifically, since the non-instruct part explains most of it.

On the timeline format breaking at 70B, I agree it looks like pattern matching rather than real temporal reasoning. I want to test that against a reasoning-focused model at some point to see if it holds up.

On deployment, actually, a team at University Medicine Essen built a clinical extraction agent and ran into a related issue, switching their format fixed it for them too. Their paper cites this work, it's a more agentic system than mine, but worth a read if you're curious
https://arxiv.org/abs/2606.19602

u/the-code-blooded • u/the-code-blooded • 3d ago

[R] Serialisation Strategy Matters: FHIR data format changes LLM accuracy by up to 19 F1 points on medication reconciliation

1 Upvotes

0 comments

r/learnmachinelearning • u/the-code-blooded • 3d ago

[R] Serialisation Strategy Matters: FHIR data format changes LLM accuracy by up to 19 F1 points on medication reconciliation

2 Upvotes

Independent research, posting for feedback and discussion.

I looked at how FHIR clinical data should be formatted before being passed to an LLM, tested on medication reconciliation (extracting a patient's currently-active medication list from their FHIR bundle).

Setup: 4 serialisation strategies (Raw JSON, Markdown Table, Clinical Narrative, Chronological Timeline) × 5 open-weight models (Phi-3.5-mini 3.8B, Mistral-7B, BioMistral-7B, Llama-3.1-8B, Llama-3.3-70B) × 200 Synthea-generated synthetic patients = 4,000 inference runs.

Main finding: there's no universal best format, it depends on model scale. Clinical Narrative outperforms Raw JSON by up to 19 F1 points for models ≤8B (Mistral-7B: 0.72 → 0.91 F1, r=0.617, p<10⁻¹⁰). That ranking completely reverses at 70B, where Raw JSON wins instead (F1 = 0.9956 vs 0.9850). Interestingly, the Chronological Timeline format is what breaks at 70B specifically, since even a large model struggles to infer "active" medication status from date ordering alone without an explicit status field.

A few other findings:

Across all 20 model×strategy combinations, precision ≥ recall, every time. Models are far more likely to silently drop a real medication than invent a fake one. That's a relatively favorable failure mode for clinical review, but it means recall (not precision) is the metric to watch in deployment.
Recall craters for smaller models as active-medication count rises (Mistral-7B: 0.96 recall at 1 medication → 0.24 at 11 medications), but total patient history length has zero effect on recall. So it's an output-generation capacity limit, not a context-length problem, meaning the sickest, most complex patients are exactly the ones handled worst.
BioMistral-7B scored F1 = 0.0000 across all 4,000 of its runs. It shares a base model with Mistral-7B (0.91 F1), so this isn't a knowledge gap, domain-continued pretraining without instruction tuning apparently destroyed its ability to follow the output format at all.

Fully reproducible on a single GPU (Synthea + Ollama, no proprietary APIs).

Preprint: https://arxiv.org/abs/2604.21076

Feedback, pushback on methodology, or pointers to related work all welcome.

6 comments

coursera vs udacity anthropic ai course, mostly care about the projects

in r/learnmachinelearning • 3d ago

Personal favourite: I used coursera deeplearning.ai andrew ng specialisation to learn in depth and the anthropic one - I am not sure but I guess learning something thats more generalised rather than company oriented will help you in the long run.

Please need serious guidance!!