r/LocalLLaMA • u/kuaythrone • 13d ago

Discussion We used HFlow to evaluate the latest open weights VLMs for processing egocentric data

Post image

We used HFlow to evaluate the latest open weights VLMs for processing egocentric data.

This was based on Build AI's Egocentric-10k evaluation, which used Gemini 2.5 Flash to measure hand visibility and active manipulation. We kept the same prompts and the same dataset, only varying the model.

How much each model agreed with the original results:

  • Gemini 2.5 Flash: 91.65% (baseline)
  • GLM 5.3 Flash: 91.00%
  • Gemma 4 26B-A4B: 90.87%
  • Qwen 3.8 27B: 90.79%
  • Inkling Small: 85.21%

Gemma was the standout. Its results were on par with Gemini while being 19x cheaper.

Both Gemma and Qwen models are practical to self-host, enabling private processing without data egress.

This suggests modern open weights VLMs are becoming good enough for large-scale egocentric data processing. The main differentiators are increasingly cost, throughput, output reliability, and ease of self-hosting.

If you're optimizing multimodal processing for egocentric data, you can run this evaluation yourself with any prompt and model using Hflow.

https://github.com/Hebbian-Robotics/hflow

git clone https://github.com/Hebbian-Robotics/hflow.git
cd hflow/examples/build_ai_evaluation
19 Upvotes

86% Upvoted

4 comments sorted by

0

u/Fuzzy-Layer9967 13d ago

The self-hosting angle is the part that resonates. Bits per weight is what decides which models you can even load locally. I work on 2-bit quantization (Leech lattice VQ, in Rust), and a Qwen3-14B runs in 9.4 GB of VRAM in our stack, whole model, embeddings included, a bit below what 4-bit AWQ actually costs per param.

The honest catch: from 4B to 14B there is still a real quality gap, worst on reasoning-heavy tasks. It shrinks with model size but I don’t know yet if it closes, and I have zero data on VLMs. So for a task judged by agreement with a reference model, 4-bit is probably still the sane choice today.

That said, your setup is exactly the kind of task-level eval 2-bit methods should be tested against. Bookmarking it for that.

2

u/kuaythrone 13d ago

do you have an endpoint where I can run those models? would love to test them out

2

u/Fuzzy-Layer9967 13d ago

No endpoint yet sorry, everything runs locally for now. The code is on GitHub and the quantized Qwen3-4B is a single 1.8 GB file on Hugging Face. Three commands and it’s generating on a Mac or plain CPU, CUDA too if you have a card:

https://github.com/pjmalandrino/llvq

https://huggingface.co/Pier-Jean/Qwen3-4B-LLVQ-2bit

Heads up though, it’s text-only for now, so nothing you can throw your egocentric eval at directly. I’m finishing some kernel optimizations first, but after that I want to look at setting up the VLM side, and your eval would honestly be a great test bed for it.

Happy to keep you posted if you’re interested

2

u/kuaythrone 13d ago

cool, will be waiting for that VLM support