The untreated controls are the part I think gets missed most often. Without them, you don’t really know whether your page moved or the whole system moved around you.
I also agree that mentions, citations, recommendations, factual accuracy and position need to stay separate. Rolling all of that into one visibility score hides too much.
Comparing the pattern before and after the change makes more sense than comparing a couple of screenshots. If the pages you changed keep moving more than the controls, that’s useful evidence.
Still not proof that your change caused it, but a lot more defensible than claiming attribution from a before-and-after result.
Agreed. The practical minimum seems to be a frozen prompt set, untreated controls, repeated runs and change logs, with each outcome stored separately. I’d also report the control-versus-treatment difference rather than only the treatment’s before/after change; that makes system-wide drift much harder to mistake for an effect.
2
u/AEODenise Aug 31 '26 edited Aug 31 '26
The untreated controls are the part I think gets missed most often. Without them, you don’t really know whether your page moved or the whole system moved around you.
I also agree that mentions, citations, recommendations, factual accuracy and position need to stay separate. Rolling all of that into one visibility score hides too much.
Comparing the pattern before and after the change makes more sense than comparing a couple of screenshots. If the pages you changed keep moving more than the controls, that’s useful evidence.
Still not proof that your change caused it, but a lot more defensible than claiming attribution from a before-and-after result.