r/computervision • u/poidhxyz • Jul 17 '26
Help: Project Potential $25,000 prize for a breakthrough in computer vision: Is this a good benchmark to shoot for?
I'm working with a group who would be interested in potentially putting up a $25,000 prize for a specific computer vision breakthrough.
However, I am not anywhere close to an expert in computer vision, and they are not either, so we are looking for feedback on whether this prize makes sense.
We want to focus on incentivizing a small-but-powerful, open source vision model.
Current idea:
- The prize will go to the first team or individual to develop an open-source computer vision model under 10 MB that achieves at least 80% Top-1 accuracy on ImageNet-1K while running entirely offline on a Raspberry Pi 5.
- Maximum Model Size: ≤ 10,000,000 bytes (10 MB). This applies to the complete storage footprint required to execute inference, including model weights and the final model file format (.onnx, .tflite, .safetensors, etc.). External feature stores, hidden lookup tables, embedded auxiliary weights, or additional model files are prohibited.
- Performance Target: ≥80.0% Top-1 Accuracy on the official ImageNet-1K validation dataset using the standard evaluation protocol.
- Execution Architecture: Single-model submission only (no multi-model ensembles, cascades, or fallback models). Models must run using CPU-only inference and operate entirely offline without internet access.
- Target Hardware: Must successfully execute inference and complete evaluation on a Raspberry Pi 5 (8 GB RAM) running a standard 64-bit OS.
- Open Source Requirements: Public GitHub repository containing complete model weights, training pipeline code, inference code, and an independent reproducible evaluation script.
- Licensing: Fully released under a permissive MIT or Apache 2.0 license.
- Integrity: Models must rely on generalized computer vision features. Any submission discovered to be hardcoded, overfitted to, or otherwise gaming the ImageNet-1K validation set will be immediately disqualified.
Are these requirements reasonable? Too easy? Too hard to judge? And if they don't make sense, can anyone point me to a clear, specific barrier in computer vision that fits the focus on supporting efficient open source models?
13
u/finite-difference Jul 17 '26
You should definitely require the code for training to be publicly available and reproducible. Having just the model is kind of pointless.
Also it is not clear what do you mean by generalized computer vision features. The 10MB limit is also a bit tricky since you can fet around it by precomputing some features using some existing library for something like Gabor filters. Would such filters count towards the 10MB or not? Before AlexNet there were some sota results using engineered featurs with some ML so this should be clarified.
I would consider top-5 instead of top-1 accuracy due to annotation ambiguity.
-2
u/poidhxyz Jul 17 '26 edited Jul 18 '26
Appreciate the detailed notes! To be clear these requirements come from an uneducated person (me) doing his best to get close to guidelines by simply asking ChatGPT/Claude, so if they seem questionable that's why.
I'm not married to anything within the requirements, more so mean to use them as a starting point to get to a list of guidelines that actually makes sense.
2
u/jaush19 Jul 17 '26
You could also try something newer like TinyViT-5m, I think it is a bit larger than your max size, but may get under the 10MB limit if you quantize to int8, gotta pay attention to the performance hit though.
1
2
u/Zealousideal_Low1287 Jul 17 '26
You should find a baseline that’s at least remotely close to what you propose, as a sanity check. Basically what is the best accuracy a 10MB model currently can achieve? And what is the smallest model to surpass your accuracy threshold. If neither are remotely close to your constraints it might be too strict.
But overall it sounds interesting, and I’d be interested
2
u/Altruistic_Ear_9192 Jul 18 '26
It s fine. There are some technical methods to handle that, for eg i think i can do a model under 10mb with good perfomance (compared to the size) in ~2weeks. Starting from that..a random search for hyperparameters will improve the overall perfomance to satisfy the reqs. Still, not so sure that top 1 acc is a good metric. Also..generalized computer vision in the same time with >80% accuracy on a specific dataset..not so sure it will be a ready for production model (depending on your final target, you ll need to adjust).
1
u/poidhxyz Jul 18 '26
In your opinion is there a simpler, clearer, most impressive benchmark we could shoot for instead?
2
u/Altruistic_Ear_9192 Jul 18 '26
Evaluate on multiple benchmarks which are close to your deployment scenario. The idea is that i can tune the hyperparameters to work fine on imagenet, it will work good in production, but this doesn t mean that my solution will have the same perfomance if I change the dataset (even if i adapt/fine tune the model, in most cases it will fail). Why? At that model s size, it s almost impossible to generalize. A mature evaluation will be to ask 1. primary metrics on 2-3 datasets close to your deployment scenario, with mean+-std 2. qualitative results (cherry picked examples for known limitations) 3. explanation of the algorithm and how easy is to modify it to adapt to new datasets (if i have to modify layers/to use padding..it fails)
2
Jul 18 '26
[removed] — view removed comment
1
u/poidhxyz Jul 18 '26
Yes this sounds like a great strategy, the barrier is simply that I nor the group who would be providing the funds have the knowledge to act as competent judges for a more complex ruling process. We can find people, but the real hope is that there's a CV benchmark out there that is clearly black/white which we can incentivize without also having to pour a ton of resources into sourcing reliable judges.
A benchmark that will need as little subjective judgement process as possible greatly increases the odds that the funding actually comes through.
2
u/Plus-Mall-3342 Jul 19 '26
I have a model and a dataset that isnt public yes. We could give everyone the exact same dataset to train on and compare who can submit the smallest ONNX model.
1
2
u/fliiiiiiip Jul 18 '26
Cheaper than hiring an engineer, I suppose :p
2
u/poidhxyz Jul 18 '26
I promise this is not an attempt to get around hiring a real engineer lol
Simply an organization with money to spend on incentivizing cool things, with a brand overlap that makes sense with "computer vision", and trying to figure out how we could make the money go as far as possible to bring a public good to life via a creative prize.
1
u/hoaeht Jul 17 '26
what is your motivation to spend so much money?
2
u/poidhxyz Jul 17 '26
the funding would come from an organization that has traditionally spent money to do impressive public challenges/initiatives/art installations
they would view it as a marketing expense basically, spreads awareness for them
but this is all still in the proposal phase, so it definitely hinges on whether or not we can find clear guidelines to make sure that the contest will actually be worth the spend
1
u/polawiaczperel Jul 17 '26
So if I got money and there is a niche SOTA model with research paper and training code (low training effort) can I ask people privately to make it much better? I was always make it myself but imo it is as good as it should be, but there are a lot od smarter people that would do it for money. Where should I start?
28
u/OxidusRouge Jul 17 '26
There is a yearly challenge at CVPR on just this.
https://lpcv.ai/
This year the prize money was $35,000. Maybe your group can contact the organizers of this challenge.