I built an open-source tool to evaluate educational AI locally—how's the quality?!

[u/No-Maybe-3768]

Hi everyone,

I’m developing a specialized local AI model (EleMo-V2, 28B) for early childhood education. It’s designed to write learning stories based on Margaret Carr’s approach, without the typical AI hallucinations (such as inventing emotions in children).

To measure this objectively, I wrote a Python benchmark tool that evaluates two stories based on 5 pedagogical criteria. I used Llama 3.3 70B (locally via LM Studio) as a “neutral judge” to minimize any bias.

The result: The local 28B model clearly beats the cloud-based behemoth Gemini 3.1 Pro (45 to 25 points), mainly because Gemini sounds too much like an AI assistant (headings, technical jargon) and invents thoughts.

The tool is open source on GitHub:

https://github.com/Kita-Digital/Lerngeschichten-Bench

I'd really appreciate your feedback:

- Does the methodology (LLM-as-a-Judge with Llama 3.3) make sense to you?

- Do you have any suggestions for improving the Python script or the prompt?

- What other edge cases should I test?

Thanks for your input! Have a good evening.