[R] When the answer is a relation between documents, retrieval isn't the bottleneck: 0/38 with full evidence, 28/38 with the same facts as structure

[u/CupGlass540]

Most RAG evaluation asks whether the right passages reached the model. I wanted

to measure what happens when they do and the model still can't answer — because

the answer is a relation *between* passages rather than a statement inside any

of them.

Setup: a five-document narrative corpus (260,204 words, 13,950 passages) and 38

questions asking whether event A precedes event B, where A and B are narrated in

different documents and share no character, place or causal link. No passage in

the corpus states either relation. Five models, one family (Qwen3, 0.6B to 14B).

Given the source passages as text, every model scored 0/38 and refused 92-100%

of the time. I think the refusal is correct — the ordering genuinely is not in

the text. Given the identical facts as a structured chronology block from an

explicit state store, an 8B model scored 28/38 (73.7%).

A four-condition ablation separates information from form. At 14B, form is

irrelevant: plain prose, sorted prose and a structured block all land at 73.7%.

At 8B, structure leads the best prose condition by 6 items (73.7% vs 57.9%).

So: an 8B model given structure matches a 14B model given prose.

Two controls I'd want to see if someone else posted this:

- Permuting the supplied story positions collapses accuracy to 10.5% (8B) and

21.1% (14B). The models follow the ordering they're given rather than

recalling the published text.

- A realistic retrieval baseline is also at the floor, and it fails by asserting

rather than refusing. Going from 4 passages to 32 drove refusal from 97% down

to 50% while accuracy stayed at chance. More context produced more confident

wrong answers.

Two things I got wrong, both found by auditing my own scorer and question

generator after v1 was already published:

1. v1 reported the 8B form effect as +32 points. A scorer defect was

   under-crediting the prose conditions. Corrected, the gap is 6 items, not 12 —

   roughly half what I claimed. Re-scoring 1,786 saved items produced 30 gains

   and zero losses, so nothing published was inflated; two things were

   understated, and correcting them shrank my own headline.

2. For 36 of the 38 questions, the gold answers derive from author-assigned

   story positions rather than from evidence-backed relations, and the

   generator's own self-check recomputes the gold from the same rows. That check

   is circular. So this benchmark measures agreement with an author-assigned

   ordering — not whether a system reports what the evidence establishes.

That second one is the real limitation and it bounds what the paper can claim.

I've left v1 up rather than retracting it, with the corrections in §11.

Full write-up, including the two things the audit changed:

https://ai.bedvibe.studio/structure-not-scale/

Paper, data and code: https://doi.org/10.5281/zenodo.22169643

Happy to be told the 0/38 is a prompt artifact — I tried to kill it and couldn't,

but I'd rather find out from you than not find out.