The Problem with AI Text Watermarks

Anthropic announced that for “Claude models launched on or after August 2, 2026”, “generated text will carry embedded watermarks”. Now, you might say, this is an EU requirement, and the EU is famously damagingly overregulated (What else could we expect from our European brethren besides more regulation? Innovation?). BUT, while this is only required in the EU, because Anthropic hates you, they are rolling this out globally. After scraping the world’s knowledge and using it without attribution, they now will not let you use it without attribution. How ethical!

Ok, ill-advised rant aside, this is a consequential decision with ramifications. At root, the issue is obviously that AI-generated text could carry real-world consequences if it’s masquerading as something authentically generated by a human (think of a deepfake video of a nuclear launch, though how this relates to pure text is less clear) and people are taking credit for AI’s work.

Right now, SynthID is already deployed by Google as a solution. The short summary is that it creates a hidden pattern in the text. Currently, LLMs choose next tokens probabilistically. In the sentence the cat ran ____. There might be a 90% chance that the next token is fast, a 5% chance that it is quick, and a 5% chance that it is well. SynthID adds an extra layer that uses a secret to sample multiple candidate tokens and select among them using keyed, context-dependent scoring functions, without changing the marginal output probabilities.

Think of a six-sided dice roll where we have the secret “Prefer Even Numbers” or “Prefer Odd Numbers”. If we roll two dice and get 2 and 5, using our secret, we may select 2. Now, in this instance, the odds of Even/Odd are 50/50, so the final output probability for any dice number remains 16.7%. However, if you know the secret key, you can see how the outputs align with the expectation of that secret:

Secret preference: O E O E E
Actual output:     5 2 3 4 6
Matches?          ✓ ✓ ✓ ✓ ✓

Without the watermarking secret, you would expect to see 50% of the outputs match the preference. With the secret, it is much higher, again without disturbing the aggregate distributions. So, the AI-generated watermark is not an actual watermark, but a correlation of output tokens to some pattern.

What are the problems here? A few things.

1. Performance depends on length (“longer texts contain more watermarking evidence, and so we have more statistical certainty when making a decision”).
2. Performance depends on model settings (“if the LLM distribution is very low entropy, meaning it almost always returns the exact same response to the given prompt, then Tournament sampling cannot choose tokens that score more highly under the g functions. In short, like other generative watermarks, Tournament sampling performs better when there is more entropy in the LLM distribution, and is less effective when there is less entropy.”)
3. A tradeoff remains between detectability and quality-distortion (“weaker levels of non-distortion can reduce text quality and diversity, whereas stronger levels of non-distortion can reduce detectability and increase computational complexity”).
4. Understanding the above constraints demonstrates that evading this mechanism is possible. Thus, by introducing it as a “watermark” it provides a false sense of certainty about knowing that the text is AI-generated.

Overall, this is a bad idea from Anthropic, which unfortunately has become the bad idea fairy. Bookmark it: This will push people toward Chinese models that don't watermark. Great job EU!!