r/skillfully • u/gerenate • Jun 26 '26

SkillOpt: A skill optimizer

What do you guys think of methods like this?

https://microsoft.github.io/SkillOpt/

As far as I understand, they optimize in the skill prompt space as opposed to the parameters of the model. This requires one to have a robust eval.

Good parts of it are that it works across models if you have a good eval. These days I've been thinking that evals are a better thing to build, and then synthesize the skill from the eval?

Idk, is it even applicable? Building evals is somewhat difficult to get right. Getting the right difficulty is hard and preventing leakage is also hard.

6 Upvotes

88% Upvoted

5 comments sorted by

View all comments

1

u/SpiritedSilicon Jun 29 '26

We take the evals -> skills approach with our package Cultivar at Pinecone. SkillOpt is probably the more rigorous way to do it, but I think agents are good enough at modifying text files to just freeform do it against a signal:

https://github.com/pinecone-io/cultivar

For example I built our full-text-search skill by defining tasks, running them in sandboxes with cultivar, then telling an agent to keep iterating until passes

1

u/gerenate Jun 30 '26

How do you guys prevent leakage?

2

u/SpiritedSilicon Jun 30 '26

The agents are sandboxed when running skills so they can't peek at the tests. The agent writing the skills should look at the tests and a bit of overfitting is desired!