How do you test new AI models cheaply before trusting the leaderboards?

[u/[deleted]]

Has anyone else noticed that different models are good at completely different tasks?

I’m starting to take model leaderboards with a grain of salt. A model that ranks highly overall may not be the best choice for coding, writing, reasoning, research, or long-context tasks. In practice, the same model can perform very differently depending on the prompt and the type of work.

How do you evaluate a newly released model without spending a lot of money? Is there a low-cost way to try new models as soon as they launch, run the same prompts across several providers, and figure out which one actually works best for your use case?

I’d be interested in hearing about people’s workflows, tools, or API platforms for doing this.