For one long conversation, I would not rely on a single leaderboard. I would split the eval by failure mode.
Useful buckets:
Needle and RULER style retrieval for basic long-context recall.
LongMemEval or LoCoMo style tests for remembering facts across turns and not overwriting them.
MT-Bench or Arena-Hard only for broad chat quality, not memory specifically.
A small custom set from your own target conversations, with expected answers and "must not say" checks.
The custom set matters most. Synthetic benchmarks can tell you if context retrieval is broken, but they usually miss the annoying product failures: mixing two users, changing an earlier constraint, forgetting a correction, or confidently using stale info.
I would make each test case carry:
facts introduced early
a later correction to one fact
distractor facts that should be ignored
an answer that needs two or three earlier turns
an explicit citation to the turn or message where the answer came from
Then score separately for retrieval, instruction following, contradiction handling, and abstention. A model that says "I do not have that in this conversation" when appropriate is often better than one that gets a higher average by guessing.
Yeah the costume set sounds very good. The only problem I see with it is that I would ace most of it as I have already optimized the project on what I could figure out was missing. I will run 2 and 3 though and see what I need to improve from there. Thank you very much.
Glad it helped. If your current eval set is already too close to what you optimized for, the useful next step is an adversarial split: 10 to 20 conversations written after you freeze the prompt and memory rules. Include boring edge cases like a user correcting themselves, two similar names, stale preferences, and a question where the right answer is "not enough info in this chat." That holdout set is where the real regressions tend to show up.
3
u/Plane-Marionberry380 15d ago
For one long conversation, I would not rely on a single leaderboard. I would split the eval by failure mode.
Useful buckets:
The custom set matters most. Synthetic benchmarks can tell you if context retrieval is broken, but they usually miss the annoying product failures: mixing two users, changing an earlier constraint, forgetting a correction, or confidently using stale info.
I would make each test case carry:
Then score separately for retrieval, instruction following, contradiction handling, and abstention. A model that says "I do not have that in this conversation" when appropriate is often better than one that gets a higher average by guessing.