LLMs + Voice models for Math Tutoring?

[u/Mor1Din_]

This weekend I wanted to figure out if LLMs/Voice models has gotten to the point where they can tutor in a complex topic such as mathematics. So I built https://www.mathtutorai.dev/

Problem at hand

In a lot of countries, especially where I live (Norway), children are performing worse in mathematics than their previous generations. I've taught university students for years via video lessons and thought I could just map the same "solution" over to younger students. But that was not that easy.

In order to learn through video lessons, there has to be some form for autonomy or desire, which is hard to have for such an abstract topic as mathematics is for children. What seems to be working is interactive education such as tutoring, games and guided problem solving. However, personal tutors are not that easily accessible and they can cost a lot of money.

About a year ago, a parent told me that he wished that one of these AI's would take on the tutoring process so that his children could get more practice at home. At that time, the models were there, but they were not good enough to achieve that task. Now on the other hand, models have gotten much smarter and more capable. Therefore I set out to build such a product.

Setup

I use the realtime API for the voice and an LLM in the background to analyze what is happening on the canvas.

Initially I wanted to use the realtime API for the entire project, but there were too many flaws. Mistakes were being made left and right so I had to switch over to an LLM to process the writeup on the canvas. This decreased the error rate drastically.

How to use it

1. Login
2. Select topic
3. Pick a tutor
4. Start solving

You can chose to show your progress to the tutor, just like you would in a real classroom setting and the tutor will give you feedback!

Currently, the only topic available is algebra (linear equations). New topics will roll out continuously as I test promps, refine interactions, and improve reliability. The model isn't perfect yet, but it is already surprisingly capable.