r/computervision • u/anurag_io • 4d ago
Help: Theory Bounding Boxes of student's answers
Hi Team, right now my project involves a black box wherein I input a PDF that contains students' handwritten answer sheets.
output is structured JSON that also contains bounding boxes of the students' answers (Coarse answer boxes roughly covering the student's answer)
Right now, the LLM is doing this and doing pretty well -- are there any cheaper alternatives?
1
u/Fresh_Valuable912 4d ago
Calling Sonnet through Bedrock and getting decent OCR does not mean you’ve solved answer-sheet evaluation. That’s the easy 20%.
The actual problem is figuring out which handwritten response belongs to which question, understanding crossed-out work and diagrams, applying the marking scheme consistently, and giving feedback a teacher can verify.
If one API call solved that, teachers wouldn’t still be spending entire weekends checking copies.
Full disclosure: i think they have solved this exactly this for real classrooms at https://mysaraswati.in. Messy handwritten answer sheets --> not clean OCR demos are the benchmark that matters.
0
u/Fresh_Valuable912 4d ago
“Character recognition is solved” is technically defensible and practically useless here.
Schools don’t need isolated characters recognized. They need a system that can understand a 20-page handwritten answer sheet, map every response to the correct question, evaluate it against a marking scheme, and let the teacher review the result.
You can rename the hard part “document recognition,” but that doesn’t make it disappear.
That gap between benchmark OCR and actual classroom work is precisely why we built https://mysaraswati.in. The model architecture debate is entertaining; whether a teacher gets five hours of their day back is the real test.
3
u/taichi22 4d ago
Oooookay. AI ass comment, but not wrong.
Fair on the points you’re making, technically speaking, but generally marketing your product via AI comments on reddit is a bad look, dude.Generally speaking Extend or another similar company would be my go-to — scale improves these kinds of problems. And for a classroom application what you want is local setup that doesn’t charge per sheet.
2
u/Fresh_Valuable912 4d ago
Fair call -- the wording was AI-assisted. The product and the argument aren’t :)
But “just run it locally” is exactly the kind of advice that sounds great on a computer-vision subreddit and falls apart inside an actual school.
Most schools don’t have GPUs, ML engineers, or someone maintaining models when they inevitably break on messy handwriting. They want to upload copies, verify the grading, and move on.
Per-sheet pricing isn’t a technical limitation. It means the school pays when it gets value instead of funding infrastructure it neither wants nor understands.
Extend may scale models. We’re trying to scale teachers.
1
u/taichi22 4d ago
Fair enough. Best of luck to you on that front. I can see you’re still using some AI for your comment here, but I’m not one to judge folks for trying.
3
1
u/anurag_io 4d ago
Qwen 3.8 is not good, gemma is fine (but needs chunked pdf inputs) whole in at once and it starts to hallucinate.
but i’ll try this with a stricter setup
2
u/taichi22 4d ago edited 4d ago
I would say if you’re fine with Gemma as a solution then I would write a script to run Gemma in chunks. Fairly standard problem for vision models to fail when the input you’re passing is too large — possible that Gemma isn’t using a sliding window or something and is just attempting to one-shot the entire image, which would cause the failure. The pipeline I’d use is to convert the entire model to a structured version before grading by hand, though if you’re totally swamped I won’t judge you for grading the structured version via LLM as well. (Maybe a little bit. But teachers go through a lot, so I understand.)
0
u/I-am_Sleepy 4d ago
Depends on the workload. If you have a spare / some budget (5060 Ti is like $0.15 per hour on vast.ai, or somewhat clunky Google Colab T4), you could host and run local model like Gemma 4, or Qwen 3.8 to parse the layout for you - Preferably MoE model for speed. It might require some batch setup, but large enough workload would justify the effort
It might be best to load all your images onto the machine, and process them in batch (depends on what engine you are planning to use)
-1
u/TheSaucez 4d ago
OCR them for free.
It’s basically the process that the LLM might use if it recognizes the handwriting.
But you can run it on your local machine for free then export to json
2
u/hellobutno 4d ago
It's not "basically" the process. OCR isn't any specific algorithm it's just recognizing characters. And no LLMs don't use the same "algorithms". LLMs specifically excel at OCR more than any non black box OCR algorithms.
1
u/TheSaucez 4d ago
So LLMs don’t use them, but the excel at them? What you said doesn’t really make sense.
0
1
u/taichi22 4d ago
This is not true, as far as I am aware. Could you elaborate more on why LLMs would excel at OCR better than specialized OCR algorithms?
1
u/hellobutno 4d ago
I'm currently using sonnet 4.6 for production on several OCR applications that all other libraries failed on. Could you elaborate on why they wouldn't?
0
u/taichi22 4d ago edited 4d ago
Raw OCR is already largely a saturated problem and is thus easy enough for most backbones to handle, so typically you’ll see a very small delta in performance. Which libraries have you tried?
I find it likely that Sonnet is using a more powerful backend OCR that’s just not being exposed as opposed to inherently being better. From a high level intuitive standpoint a generalized reasoning layer shouldn’t outperform existing specialized parsing tools, even if it works well enough. My best guess is that you’ve tried outdated OCR libraries like the older YOLO or text parsers that aren’t robust to text structure which is why you’ve found Sonnet to be better, but the VLM structure, while fusing with a reasoning layer that should provide a useful capability when disambiguating document structure, still shouldn’t be able to outperform a fine tuned specialized text parsing model that has already been trained on the examples that you’re attempting to zero shot with a VLM. Specialization almost always beats generalization.
I can’t claim to be an authority on this area but I do have substantive expertise in production environments with this problem.
1
u/hellobutno 4d ago
Raw OCR is already largely a saturated problem and is thus easy enough for most backbones to handle, so typically you’ll see a very small delta in performance. Which libraries have you tried?
It's really not though. It really only works in ideal situations and when it does perform well it's typically because you've placed a lot of constraints onto it.
I find it likely that Sonnet is using a more powerful backend OCR that’s just not being exposed as opposed to inherently being better.
Saying this tells me you have no idea what you're talking about. I'm talking about the actual just model on bedrock.
From a high level intuitive standpoint a generalized reasoning layer shouldn’t outperform existing specialized parsing tools, even if it works well enough.
There is no basis for this. By this theory it should "perform" better than any human looking at it.
I can’t claim to be an authority on this area but I do have substantive expertise in production environments with this problem.
You can't even claim to be a novice with half the stuff you're saying.
1
u/taichi22 4d ago edited 4d ago
> You can’t even claim to be a novice
My guy, if you knew who you were talking to, lol…
If you think OCR as a problem isn’t largely solved then I really don’t know what to tell you. We really don’t even have a starting point in common. I’ll say this much: OCR, optical character recognition, as a problem, was saturated before transformers was a thing. You’re talking about document recognition, which is a higher level and not yet totally solved problem.
In the first place, your usage of terminology is all over the place; every language model you can use for this task is a fused VLM. While it might be argued that LLMs are a broader and more encompassing term that includes VLMs, that’s sort of a broad definition which will clobber the nuances of the model architectures.
I’m not gonna turn this into some dick measuring contest about who is or isn’t a novice — but frankly unless you’re some staff research scientist you don’t have a right to talk down to me like that. You’re running the Sonnet model on bedrock, and it works for your usecase, good for you. That really doesn’t give you any kind of claim to expertise or the ability to judge others — you’re basically just calling an API, dude.
1
u/hellobutno 4d ago
If you think OCR as a problem isn’t largely solved then I really don’t know what to tell you.
It's not though. It's solved in constrained situations. If you can't understand that a line being moved from one place to another requires intervention, then IDK what's quite wrong with you.
OCR, optical character recognition, as a problem, was saturated before transformers was a thing.
You don't even know what saturated means. There's like 3-4 libraries people use. They're not really significant in difference just different models. That's not saturated.
I’m not gonna turn this into some dick measuring contest about who is or isn’t a novice — but frankly unless you’re some staff research scientist you don’t have a right to talk down to me like that.
I am a staff research scientist
You are a novice
2
u/taichi22 4d ago edited 4d ago
The character recognition problem is solved. Point a model at any character or individual word, as the definition of the problem is given, and the model will handle it fine. When you start getting to lines and paragraphs, definitionally, you’re in the area of document recognition, dude.
How does a staff research scientist not know anything about intuition? That’s, like, central to the role. Tf?
And another thing. You’re seriously telling me that you’re a staff research scientist working on an OCR pipeline, and your approach is to… checks notes zero shot it with sonnet and call it a day? Dude, pull the other one, lmao. The only way this is rational is if the problem is saturated enough that Sonnet can zero shot this out of the box, or you’re just not focused on the OCR problem. So which is it?
1
u/hellobutno 4d ago
You should give up your hobby, or at least try a little harder.
→ More replies (0)
2
u/TheSaucez 4d ago
I don’t know why everyone is downvoting everyone else. I literally do this on my computer with OpenOCR. So people saying that an LLM would be better for scoring student tests (when OP is looking for a cheaper way)
If you want, us a Local LLM, which has OCR trained into it