OpenZeppelin says OpenAI’s EVMbench smart contract benchmark is tainted by data contamination

What EVMbench is trying to measure

OpenZeppelin's core claim: contamination undermines the signal

That overlap is a big deal in practice:

• If a model has seen the exact contract snippet before, it can "solve" the task by recall rather than reasoning.
• Even partial overlap (near-duplicates, mirrored repos, repeated patterns) can inflate scores because the model is effectively getting hints.
• Contamination tends to show up as unnaturally strong performance on specific subsets, and it can make model-to-model comparisons misleading.

OpenZeppelin also pointed to methodological flaws, and notably said it found at least four invalid high-severity vulnerability classifications in the benchmark. That second point is not just academic. If the label is wrong, then "correct" answers become ambiguous, and participants can be penalized for being accurate.

The net effect: contamination plus questionable labels can turn a benchmark into a vibes-based leaderboard.

The label problem: why "high severity" needs to be precise

Smart contract security is brutally sensitive to context. A pattern that is catastrophic in one contract can be benign in another due to:

• permissions and ownership boundaries
• upgradeability constraints
• assumptions about trusted callers
• invariants enforced elsewhere in the system

So when OpenZeppelin says at least four "high severity" items are invalid, it's a flashing warning light about ground truth. Benchmarks live and die on clean labeling.

A practical way to think about it: if the dataset says "this is an exploitable bug," but it isn't, then an AI system that calls it out is being trained to hallucinate exploits. On the flip side, a model that correctly says "not a vuln" might score worse, pushing teams toward noisier tooling.

That is how you end up with security products that spam false positives, drain engineering time, and still miss the real issues.

Why contamination is especially messy for EVM security tasks

Contamination is a known problem across AI evals, but smart contract benchmarks have a few extra traps:

Public repos are effectively pretraining fuel

Vulnerability writeups are widely indexed

When a contract gets exploited, the postmortems and exploit analyses spread everywhere. If EVMbench includes contracts tied to high-profile incidents, the "answer" might already be present in the training data, sometimes explicitly.

Near-duplicate patterns can be enough

Even when the exact file is not present, common templates and repeated anti-patterns can make it difficult to tell whether a model is reasoning or pattern-matching. That is not useless, but it is not the same claim as "the model can audit novel contracts."

Market structure angle: why this matters beyond academic benchmark drama

EVMbench landed in a moment where AI x crypto tooling is fighting for mindshare and budget. Benchmarks are the ammo used to justify:

• procurement decisions by protocols
• VC narratives around "AI auditors"
• dev teams trusting copilots for security-critical code
• new agent frameworks that promise continuous monitoring

It also impacts how quickly teams should delegate to AI. Plenty of devs already use LLMs to draft Solidity. Handing the same class of model a security sign-off based on a benchmark with leaks is a recipe for misplaced confidence.

What a cleaner benchmark would likely require

OpenZeppelin's findings put pressure on benchmark creators to adopt the same kind of rigor expected in security work. In practice, that usually means:

• Provenance tracking: clear sourcing for each sample, plus evidence it is not trivially indexed or duplicated.
• Deduplication and similarity checks: not just exact matches, but near-duplicate detection across repos and datasets.
• Reproducible splits: strict separation between train-like material and evaluation material, plus documentation.
• Label adjudication: multiple reviewers, clear severity rubric, and public dispute resolution when labels are challenged.
• Versioning: benchmarks evolve, so scores should be tied to dataset versions, not a floating target.

What to watch next

A few concrete signals will matter over the next weeks:

1. Will EVMbench maintainers publish a response detailing what was contaminated and how it will be fixed?
2. Will the dataset be patched and versioned so past scores are clearly separated from new scores?
3. Will third parties replicate OpenZeppelin's findings, especially on the mislabeled high-severity cases?
4. Will vendors keep advertising old scores without clarifying the dataset revision?

If fixes land quickly with transparent methodology, EVMbench can still become useful. If the response is hand-wavy, expect the benchmark to become another leaderboard that mainly measures who trained closest to the test set.

Takeaway: treat EVMbench scores as "soft data" until provenance is nailed down

Until EVMbench's dataset is demonstrably cleaned and the labeling disputes are resolved, teams should treat high scores as directional at best, then validate with old-school controls: human review, formal methods where feasible, differential fuzzing, and adversarial testing. The thesis that "AI auditors are getting good fast" only holds if the eval is clean, and OpenZeppelin just challenged that assumption with specifics, including training data leaks and multiple high-severity labels that appear wrong.