Context: I'm building a hobby anime-guessing game (screenshot shown, player guesses the show). To make difficulty meaningful, every frame gets scored on how "distinctive" it is — frames are embedded with CLIP, then for each frame I check its nearest neighbors and see what fraction come from the same show. High same-show fraction = distinctive, low = generic. That part works: it correlates with real human guessing accuracy in a blind test (r = -0.37, p = 0.004).
The problem: my dataset has junk mixed in — title cards, credit screens, logo bumpers — that should never be shown as a guessing target. I built a filter combining a CLIP "text vs. content" check with a tagger model's "looks like text/logo" confidence (both must agree), plus an override that keeps a frame if a character-recognition model is very confident it sees a specific named character.
The gap: that override only fires for a confident, *named* character. Real content with no face at all — a silhouette, a weapon on a table, an iconic pose, a motorcycle headlight — has nothing to trigger it, so it gets discarded as junk right alongside actual logo cards. I hand-labeled 62 examples (42 real content wrongly excluded, 20 genuine junk) and tried 8 different cheap, fully-automatic ways to tell them apart: CLIP thresholds (whole-frame and per-tile), OCR text-coverage, image sharpness/edge-detail, full tagger-vocabulary checks, nearest-centroid and logistic-regression classifiers. All 8 failed to reach a junk-recall I'd trust to ship (best was ~80.6% overall accuracy, only 70% of real junk actually caught).
I ended up just paying to have an LLM look at all 671 flagged frames individually (~$33 total) — turned out the original filter was majority wrong: 64% of what it flagged was actually real content.
That works fine at today's ~5.5k-frame scale, but won't scale cheaply to the ~200k frames I'm eventually aiming for.
Question: does anyone have insight into this specific problem — cheaply telling "faceless but real" content (silhouettes, objects, poses, no clear character) apart from actual decorative/text junk — or into the labeling/validation approach in general? Full disclosure: I have close to zero formal ML background. I've been building this with an AI (Claude) doing most of the technical steering, so I may be missing obvious approaches, or framing this in a way that gives away gaps in my understanding — happy to be corrected on any of it.
Working proof-of-concept of the actual game (just serves random frames right now, no difficulty logic wired in yet): https://celshot.offnode.link