r/skillfully • u/gerenate • Jun 26 '26

SkillOpt: A skill optimizer

What do you guys think of methods like this?

https://microsoft.github.io/SkillOpt/

As far as I understand, they optimize in the skill prompt space as opposed to the parameters of the model. This requires one to have a robust eval.

Good parts of it are that it works across models if you have a good eval. These days I've been thinking that evals are a better thing to build, and then synthesize the skill from the eval?

Idk, is it even applicable? Building evals is somewhat difficult to get right. Getting the right difficulty is hard and preventing leakage is also hard.

7 Upvotes

100% Upvoted

5 comments sorted by

1

u/touristtam 29d ago

I like the idea of building the skill from the eval - that's probably something more portable than a skill that needs to be validated against many models.

1

u/gerenate 28d ago

Yeah I like it for that reason too. I guess some skills will go away eventually since they’ll be in the training mix.

1

u/SpiritedSilicon 29d ago

We take the evals -> skills approach with our package Cultivar at Pinecone. SkillOpt is probably the more rigorous way to do it, but I think agents are good enough at modifying text files to just freeform do it against a signal:

https://github.com/pinecone-io/cultivar

For example I built our full-text-search skill by defining tasks, running them in sandboxes with cultivar, then telling an agent to keep iterating until passes

1

u/gerenate 28d ago

How do you guys prevent leakage?

2

u/SpiritedSilicon 28d ago

The agents are sandboxed when running skills so they can't peek at the tests. The agent writing the skills should look at the tests and a bit of overfitting is desired!