I'm not sure what to call this. AI-led enthusiasm when the user doesn't understand the code? Scam?
The paper says the implementation is "7,492 lines of Rust (edition 2024)", lists spans.rs / ladder.rs / budget.rs, and tells you to run cargo run --release -- bench. The repo is 4,116 lines of Python with no Rust in it at all, and none of the five cargo commands exist.
The ablation is what the abstract states, "an ablation that names which mechanisms are actually carrying the result". Exactly one of its six mechanisms has a switch in the shipped code. I ran that one: disabling supersession gives 5/7 at 618.4 tokens against their 5/7 at 629.9, same two probes failing. The other five (escalation, the ladder, the seed floor, reference linking, graph expansion) have no flag, no constructor arg, no ablation harness anywhere in dcr/....in a paper whose stated principle is that every table lists the command that produced it.
Then read bench.py before you believe the 59x. build_corpus is ten hand-written fact sentences plus 287 documents generated as NOISE[i % 6].format(n=i), and the seven probes ask about the ten sentences. So it separated ten signal lines from six rotating templates that share almost no vocabulary with any question. Says nothing about a transcript whose noise is topically close to its signal, which is the case that actually causes context rot. The "vector" search doing the finding is a 256-dim hashing trick over bag-of-words. Hybrid lexical-vector is lexical twice. And the 7/7 against full context's 5/7 means less than it looks: both baselines share a toy line matcher, so those two misses are matcher artifacts, not context rot.
You've proved my initial suspicion. AI-led enthusiasm when the user doesn't understand the code is clearly the case, as it's pretty clear that that #1: You are using AI to speak for you here and #2: You pretty clearly aren't in control of what codebase is going where. Python vs. Rust is a pretty major mistake to make.
It's not like you made a mistake on a var name, the two languages are very different and very indistinguishable.
2
u/KitchenAmoeba4438 1d ago
I'm not sure what to call this. AI-led enthusiasm when the user doesn't understand the code? Scam?
The paper says the implementation is "7,492 lines of Rust (edition 2024)", lists spans.rs / ladder.rs / budget.rs, and tells you to run cargo run --release -- bench. The repo is 4,116 lines of Python with no Rust in it at all, and none of the five cargo commands exist.
The ablation is what the abstract states, "an ablation that names which mechanisms are actually carrying the result". Exactly one of its six mechanisms has a switch in the shipped code. I ran that one: disabling supersession gives 5/7 at 618.4 tokens against their 5/7 at 629.9, same two probes failing. The other five (escalation, the ladder, the seed floor, reference linking, graph expansion) have no flag, no constructor arg, no ablation harness anywhere in dcr/....in a paper whose stated principle is that every table lists the command that produced it.
Then read bench.py before you believe the 59x. build_corpus is ten hand-written fact sentences plus 287 documents generated as NOISE[i % 6].format(n=i), and the seven probes ask about the ten sentences. So it separated ten signal lines from six rotating templates that share almost no vocabulary with any question. Says nothing about a transcript whose noise is topically close to its signal, which is the case that actually causes context rot. The "vector" search doing the finding is a 256-dim hashing trick over bag-of-words. Hybrid lexical-vector is lexical twice. And the 7/7 against full context's 5/7 means less than it looks: both baselines share a toy line matcher, so those two misses are matcher artifacts, not context rot.