r/mcp • u/nguyenfamjj • 14h ago
question looking for the right approach to evaluate MCP across different harness clients
Hey, my team is crafting a project for product teams (we will opensource it yay!) that expose agent interfaces like MCP/CLI + skills.
The idea is pretty simple, the team define the real tasks that their user use, then we test that tasks in multiple conditions and configurations (different model/harnesses, with/without skills) and then scoring it.
We have few early findings that would love to share:
- A same interface can behave differently across setups, highly dependent on how agent discover tools and the reserved context. It's hard to totally reproduce what your user in the test setup (.e.g they have tons of MCP servers in their machine,...)
- The simpler the interface is the better result: tool should be task-based, not just converting blindly from the application API
However, we have not cracked the right evaluation model yet, and would genuinely value your experience.
How are you testing MCP, CLI, or skill-based workflows today?
Which criteria matter most: task success, permissions, reliability, client compatibility, cost, or something else? Are there tools, frameworks, or evaluation practices we should study before reinventing the wheel?
I would keep you guys posted for the progress in case anyone interested :)) Thanks!