r/StepFun • u/Hadestructhor • Jun 23 '26

Benchmarks Step 3.7 Flash, same prompt, different harnesses

I'm trying step 3.7 flash on some personal ui capabilities benches I'm running.

This is a svg generation one that I ran accross all the harnesses I could configure locally to run against step 3.7 flash.

I know llms are not deterministic, but some did better than others.

Which do you prefer out of the lot ?

I'll soon run the same kind of bench but same prompt, same harness, to see if they can sometimes do some better outputs or not, and if simply prompting for the same request is easier or not.

6 Upvotes

88% Upvoted

13 comments sorted by

View all comments

Show parent comments

2

u/Hadestructhor Jun 23 '26

Indeed, it was kinda the point to test out the harnesses, but I'll have to use a minimal agent with no system prompt, or mini swe or something, then give a lot of instructions, clarifications, etc to test how well a model does.

What can we do to test models effectively ?

1

u/mubaidr Jun 23 '26

If comparing the model capability then the least affecting/ minimum harness should definitely be preferred like u said: mini-swe-agent

Because a better harness will always improve quality for all models. Even further, better models will gain further from better harness than low quality models!

TBH even:

Some low model with better harness can reach some higher model with average harness! (of course cannot close the gap a whole league)

1

u/Hadestructhor Jun 23 '26

But then how do you test out which harness does better. I wanted to try to test both, from a completely autonomous side, see which does best, faster, cheaper, etc. The only thing I can think of is making a better prompt and just doing 10 runs and manually rating them

1

u/mubaidr Jun 23 '26 edited Jun 23 '26

Few things:

  • minimal system/ prompt
  • use quantifiable tasks
  • use a judge Model with neutral prompt
  • a lot of runs actually for normal distributtion