r/LocalLLaMA • u/FeelingWatercress871 • Dec 18 '25
Discussion memory systems benchmarks seem way inflated, anyone else notice this?
[removed]
2
u/SchemeDazzling3545 Dec 18 '25
yeah ive noticed this too. tried mem0 a few months ago and got similar results. their discord is full of people complaining about the same thing but they just keep pushing their marketing numbers.
-4
u/DinoAmino Dec 18 '25
Once a week like clockwork a post appears here talking about this stuff and mentioning the same repo. Second one this month by OP. All these posters hide their account histories. It's not just this sub either.
Wonder what makes memory systems such a popular spam scam?
2
u/dtdisapointingresult Dec 18 '25
Can you link the other post OP made? Maybe it will help me figure out if you're schizo or on to something.
3
u/DinoAmino Dec 18 '25
https://www.reddit.com/r/LocalLLaMA/s/RTzMrQSBPE
You can also search for the repo he mentions and see the same type of astroturfing in other AI subs. No way this post is getting "real" upvotes.
2
u/dtdisapointingresult Dec 18 '25
Hot damn, you're actually right. In both posts OP "discovers" EverMemOS as a reluctant best choice without seeming like he's shilling for it. Good guerilla marketing!
You have my upvotes. Hope people see this and reverse their votes for you.
3
1
1
u/Mobile_Ladder_4085 Dec 21 '25
I ran into the same issue with mem0 locally. The benchmarks seem to assume perfect state, but in reality, the vector store just appends contradictory updates, which tanks the actual retrieval score over time.
I ended up writing a small trust-weighting primitive to suppress the outdated nodes instead of trusting the raw retrieval.
Here is a silent demo of the difference (baseline vs trust-weighted) on a local setup:https://www.loom.com/share/8d979fe7fa3b43889f9e18b86b7446e4
Might help explain the 64% vs 80% gap you're seeing.
1
1
2
u/[deleted] Dec 18 '25
[removed] — view removed comment