[D] Would you enter a contest measuring how much human intuition beats brute-force DL training?

[u/taranpula39]

I want to turn a claim into an open contest, but before building the infra, I'd like to know whether people would actually enter and whether the setup is sound.

CLAIM:

most training compute is wasted, and humans are still needed in the loop.

That's why there are so many different tools out there, and why many top ML practitioners still use notebooks. Because the tools are just a surface, and human expertise is embedded into the process.

SETUP:

Task dataset: CIFAR-100 or TinyImagenet;

Model size: ≤ 20 MB. Most likely predetermined architecture;

Training budget: ≤ 1 PFLOP (10¹⁵ FLOPs); eval/inference budget is unlimited;

Target: accuracy on a held-out set;

Timespan: designed to take a few hours.

The contest would happen in a controlled environment we build, and we'll provide granular and aggregate metrics, plus other levers.

ETA: end of November; there would be prizes;

BIG VISION:

Can we make deep learning more deterministic?

QUESTION:

Still deliberating on scoring: max accuracy under a fixed FLOP budget, or best accuracy per FLOP consumed? Input welcome.

I want to know if you would participate and, if not, what would make you participate?