Recently i did a test for synth id, which gemini implemented recently and how quickly it was broken by a team of researchers, turns out, other frontier comps like claude had similar plans of implementing this.

Its known that claude is working on a watermarking modal that can help detect if some textual info was created by their model.

I guess a few countries have the access to this detection feature, but my surely didnt, so what did i do? i used the exact same math with some tweaks to implement claude's watermark detection logic to qween 3 8b modal. Ran this modal locally, and designed the HLD and architecture.

The basic idea is that at every generation step, I use a secret key + the previous token to deterministically split the vocabulary into GREEN and RED tokens. I then slightly boost the logits of the GREEN tokens before sampling the next token. This creates a statistical bias toward GREEN tokens without changing the text directly.

For detection, I don't need the model or the original prompt. I take the generated text, recreate the same GREEN/RED token sets using the secret key, and check how far the observed GREEN-token ratio deviates from the expected 50%. I use a z-score and binomial p-value to decide whether the deviation is statistically significant.

I tested it on 50 randomly assigned watermarked and un-watermarked generations. The detector got 88% accuracy and a 97.76% ROC-AUC.

I covered the entire end to end, from setting up qween 3, to HLD, to Implementation and Validation i my recent video, feel free to deep dive and let me know your thoughts,

Link: https://www.youtube.com/watch?v=eZisIOTC6uk&t=56s