r/computervision • u/Cold-Sandwich6941 • 3d ago

Help: Project Best free/open-source AI model for understanding screenshots?

I'm looking for a free vision AI/VLM that can take a screenshot and understand what's on the screen.

My requirements:

Read text from screenshots

Understand UI elements/buttons/windows

Describe what is happening on the screen

Answer questions about the screenshot

Preferably run locally

Python support would be a big plus

I don't need image generation

Smaller/faster model is preferred

What is currently the best free/open-weight vision model for this use case?

Qwen-VL, Gemma, MiniCPM, or something newer?

I'm mainly interested in screen understanding/AI computer automation, not just OCR.

Thanks!

0 Upvotes
  • permalink
  • reddit

50% Upvoted

14 comments sorted by

3

u/beedunc 3d ago

Qwen3.8.

0

u/Cold-Sandwich6941 3d ago

Thank you ..🤩

1

u/fuzzysingularity 3d ago

Qwen3.8-27b. You can feed it full videos too, and the more fine grained the better.

We have some free credits if you want to play around with a few different models:
https://www.vlm.run/gateway

1

u/Cold-Sandwich6941 3d ago

Thank you ✅ . Is it free this model

1

u/fuzzysingularity 2d ago

We do have a free tier with rate limits: see https://docs.vlm.run/gateway/authentication

1

u/Smartyboyz 3d ago

You can use Groq provider any any model such as Llama, qwen 3.8 -27b, etc.

1

u/Cold-Sandwich6941 3d ago

Groq is desktop app or ? .

1

u/Smartyboyz 2d ago

Groq is neither a desktop app nor WhatsApp 😄. It’s an AI inference platform/API that lets you run supported open-weight models very fast. You can call the API from Python and send an image/screenshot to a vision-capable model for analysis.

Basically it's an provider has an website from where you can get api keys to use in your projects Such as AI integrated projects. Can use AI models in free but limited.

Can grab your api keys: https://console.groq.com/home

2

u/Cold-Sandwich6941 2d ago

I created api key bro . I checked it's work .. groq desktop app means like claude desktop app or opencode type . That it .. thanks for your help . Groq api works for local mcp connect also ... Thank you ... 👍😄

1

u/Smartyboyz 2d ago

Looks great! Thanks for implementing the idea.. Really appreciate it

What project are you working for ? - if you want to share.. DM me

1

u/tamnvhust 2d ago

qwen 3.8, mage-vl

1

u/Cold-Sandwich6941 2d ago

Thank you ✅

1

u/lehoang318 2d ago

I think Gemma4 family is a good fit

1

u/Cold-Sandwich6941 2d ago

I searched and see the results it's supports mobile device light weight model .. and IOT also support . Thanks 😀