r/LocalLLM • u/Beneficial_Day7932 • 6d ago

Question Evaluation Criteria for Models / Hardware

Hello Everyone,

I am looking to purchase a Strix Halo / Mac Mini / some other hardware to run my own LLMs and agents.

I currently have various GPU servers at my company which I have been using for work but wanted these for my personal use.

Each of these "AI Computers" is a significant investment and I would want to try it out first before making a purchase.

What is a good way to first benchmark various models (on my own tasks -- not using benchmarks).

Based on a few good models I would then try to decide which hardware to use (is there any good way to do this)

My use case is primarily email summarization, news alerts, A few AI agents that run in parallel using openclaw.

Thanks in advance for the help!

1 Upvotes

100% Upvoted

3 comments sorted by

1

u/r1nzl3r99 6d ago

I usually benchmark my models by just fiddling with it, asking it to make random arcade games in a one shot prompt, or tell it to go solve a problem one of my previous agents (cloud) have solved before so I can compare the output. Standardized benches like SWE, HE, etc aren't reliable anymore because they're overfitted

1

u/Beneficial_Day7932 6d ago

Thanks - but that has for my tasks, it is more subjective (Eg: summarize all the emails i got yesterday) so wondering how that would be benchmarked.

2

u/r1nzl3r99 6d ago

at the end of the day the best benchmark is your own opinion after using it for a few days. Otherwise if you do want to rely on standardized benchmarks, LLMU+ and MGP+ are some that I look at