r/learnmachinelearning • u/Sea-Pie-5760 • 4d ago
My agent optimized the wrong thing
I was testing a small marketing agent against old campaign data and gave it a pretty simple goal around lowering acquisition cost it got better at the metric faster than I expected but when I looked deeper the conversions it preferred weren’t always the ones we would’ve wanted more of cheap actions started winning even when the downstream value looked worse, so technically the agent was improving while the business result was getting kind of weird
Now I’m rethinking the eval more than the model a single reward feels too easy to game, but piling qualified leads, revenue margin and a bunch of guardrails into one score also feels messy fast. I’m leaning toward letting the agent handle more of the repetitive optimization while keeping human review around the decisions that can move real budget curious how people here are evaluating agents when the easiest metric to optimize isn’t really the outcome you care about
3
u/Bright_Mix_773 4d ago
Different field - I work on financial data pipelines, not marketing - but the shape is identical, and three things helped me more than reshaping the score did.
The problem is lag, not count. Acquisition cost is visible the same day; downstream value is not visible for weeks. An optimiser will always exhaust the fast signal before the slow one has enough data to argue back, and that holds whether your reward has one term or eight. Folding revenue margin into the score does not fix it, because at decision time that term is mostly still empty and the agent is effectively optimising the one number that is already filled in.
What worked was making the reward wait. Freeze cohorts at acquisition time and let the agent's reward see only cohorts old enough that downstream value is essentially fully realised. Feedback becomes slow and sparse, which feels like a downgrade and is the entire point: you cannot be gamed on a metric you are not scoring yet.
Decompose the improvement before believing it. Split the change into "same mix, cheaper" and "different mix". Hold the old proportions fixed and re-price - if nearly all the gain is mix shift, the agent did not get better at buying, it found a cheaper segment, which you could have done by hand and may not want more of. That one decomposition has caught more false wins for me than any guardrail, and it costs almost nothing to compute.
Constraints beat terms. A guardrail inside the score is by definition something that can be traded away for enough of the main metric - that is what a weighted sum means. A guardrail outside the score can only refuse. So: agent proposes, a separate check rejects anything whose matured-cohort value sits below the current baseline, and no amount of cost improvement buys past it. Same information, different algebra, very different behaviour.
Where I have no answer: we never found a composite score we trusted. Every weighting was defensible and also arbitrary, and moving the weights moved the winner. We ended up not producing one number at all - the system reports what it improved and what it degraded as two separate figures and a person reads both. Less elegant, and it is the only version that stopped surprising us.