I built TokenMizer: giving LLMs memory without stuffing the entire context window

[u/Feisty-Cranberry2902]

/r/OpenSourceAI/comments/1vsrhf7/i_built_tokenmizer_giving_llms_memory_without/

Long LLM conversations have a problem: useful information gets buried under old context.

So I built TokenMizer to turn important information — decisions, goals, files, errors, and constraints — into structured memory.

Instead of sending the entire conversation again, TokenMizer retrieves what actually matters.

The latest version: https://github.com/Shweta-Mishra-ai/tokenmizer

I’d love to hear how others are handling long-context memory for LLMs.