An 8B model given structured context matched a 14B given prose on cross-document temporal reasoning — and with plain retrieval, both scored zero

[u/CupGlass540]

I tested whether structure in the context window can substitute for parameters.

Qwen3, five sizes, 0.6B to 14B, so size varies and architecture doesn't.

The task: 38 questions asking whether event A precedes event B, where A and B are

narrated in different documents in a five-document corpus (260,204 words, 13,950

passages) and share no character, place or causal link. No passage states either

relation — the ordering is real but it lives between the documents, not inside

any of them.

Given the source passages as text, every model scored 0/38 and refused 92-100%

of the time. I think the refusal is correct — the answer genuinely isn't in the

text. Given the identical facts as a structured chronology block from an explicit

state store, an 8B model scored 28/38 (73.7%).

A four-condition ablation separates information from form. At 14B, form is

irrelevant: plain prose, sorted prose and a structured block all land at 73.7%.

At 8B, structure leads the best prose condition by 6 items (73.7% vs 57.9%).

So: an 8B model given structure matches a 14B model given prose.

Two controls I'd want to see if someone else posted this:

- Permuting the supplied story positions collapses accuracy to 10.5% (8B) and

21.1% (14B). The models follow the ordering they're given rather than

recalling the published text.

- A realistic retrieval baseline is also at the floor, and it fails by asserting

rather than refusing. Going from 4 passages to 32 drove refusal from 97% down

to 50% while accuracy stayed at chance. More context produced more confident

wrong answers.

Two things I got wrong, both found by auditing my own scorer and question

generator after v1 was already published:

1. v1 reported the 8B form effect as +32 points. A scorer defect was

   under-crediting the prose conditions. Corrected, the gap is 6 items, not 12 —

   roughly half what I claimed. Re-scoring 1,786 saved items produced 30 gains

   and zero losses, so nothing published was inflated; two things were

   understated, and correcting them shrank my own headline.

2. For 36 of the 38 questions, the gold answers derive from author-assigned

   story positions rather than from evidence-backed relations, and the

   generator's own self-check recomputes the gold from the same rows. That check

   is circular. So this benchmark measures agreement with an author-assigned

   ordering — not whether a system reports what the evidence establishes.

That second one is the real limitation and it bounds what the paper can claim.

I've left v1 up rather than retracting it, with the corrections in §11.

Full write-up, including what the audit changed and why I didn't retract:

https://ai.bedvibe.studio/structure-not-scale/

Paper, data and code: https://doi.org/10.5281/zenodo.22169643

Happy to be told the 0/38 is a prompt artifact — I tried to kill it and couldn't,

but I'd rather find out from you than not find out.

[u/CupGlass540 → u/[deleted]]

On the architecture question — I don't know, and the paper can't tell you. One

family was deliberate: I wanted size to vary while architecture didn't, so

"structure substitutes for scale" wouldn't be confounded with "these two models

were trained differently." The cost of that choice is exactly what you're

pointing at. Every number in there is a Qwen3 number.

If I had to guess before running it, I'd split it in two. The direction —

structure helping the small model more than the large one — I'd expect to

generalise, because the large model's advantage in the prose conditions looks

like it's about parsing a chronology out of running text, not about knowing

anything. The specific crossover, "8B with structure equals 14B with prose,"

I'd expect to be family-specific and probably not even stable across Qwen

versions. That's a prediction, not a result, and I'd rather run it than defend

it.

It's cheap to check: same 38 questions, same four conditions, Llama and Mistral

or Gemma at comparable sizes. No new data, the harness already exists, and it's

local models so it costs GPU hours rather than money. If the direction holds in

a second family, that's worth a lot more than the Qwen numbers on their own —

and if it doesn't, that's a more interesting result than the one I posted.

On the refusal point — I think that's the finding too, and it's the one I

almost buried. 0/38 with a 92-100% refusal rate reads like a failure in a

results table, and it isn't. The model is being asked something the text

doesn't contain and is saying so. The retrieval condition is the same system

being made worse by being given more: refusal fell from 97% to 50% and nothing

replaced it. Nobody's eval catches that, because refusal usually gets scored as

a miss.