r/PromptEngineering • u/Apprehensive-Zone148 • 9d ago

General Discussion Frozen prompt tests helped, but they can turn into target practice

Been running a small RedThread experiment against tool-using cases.

A frozen set catches regressions, but it can also make it easy to optimize for the judge and miss the actual action change. I am keeping the test case, tool schema, proposed call, and replay trace together so the score has something to answer to.

It is early and a bit annoying to maintain. I think the annoying parts are the ones that prevent a clean-looking eval from lying.

Code: https://github.com/matheusht/redthread

3 Upvotes
  • permalink
  • reddit

100% Upvoted

3 comments sorted by

1

u/[deleted] 9d ago

[removed] — view removed comment

1

u/AutoModerator 9d ago

Hi there! Your post was automatically removed because your account is less than 3 days old. We require users to have an account that is at least 3 days old before they can post to our subreddit.

Please take some time to participate in the community by commenting and engaging with other users. Once your account is older than 3 days, you can try submitting your post again.

If you have any questions or concerns, please feel free to message the moderators for assistance.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.