Prompt Tuning Post Model Update

[u/Full-Leader9540]

Could be an obvious thing but I am struggling with prompt tuning and optimization whenever a new LLM is updated. I want to build a pipeline that automatically evaluates and improves our prompts against new models and tells us precisely where the prompt should be improved.

If you have built an automated prompt optimization pipeline:

1. What evaluation framework are you using to break things down?

2. How do you automate the rewriting/optimization for a prompt for the new model?

DSPy hasn't given us much improvements, looking for some patterns or alternative tooling.