Olympix vs. AI
We run Olympix against frontier AI and publish every result
Each benchmark runs Olympix and a frontier AI model on protocols that professional reviewers have already audited, and counts how many of the published findings each one reaches. Every run ships with its corpus and results, so you can check the numbers and see how the gap moves from one run to the next.
Why it matters
More real findings, each one proven
More of the real issues. A model on its own reasons about where code could break, and Olympix executes the paths to find where it does.
Proof with every finding. Each one arrives with an exploit trace and a proof of concept you can run.
Less time triaging. Engineering hours go to fixing confirmed issues instead of dismissing plausible-sounding ones.
A number you can check. Every run publishes its results and ground truth sources.
Olympix vs. AI
Olympix BugPoCer and Claude Opus 5.5 ran on the same fifteen audited EVM protocols. BugPoCer found 94 of the 160 published high and medium severity findings, and Opus 5.5 found 53.
Olympix found 94 of the 160 issues professional reviewers published.
High and medium severity findings from published audit and contest reports across fifteen EVM protocols.
recall for BugPoCer across all fifteen protocols
recall for Opus 5.5 on the same corpus
recall gap across all fifteen protocols
recall gap on the Best 10 subset
Method
How every benchmark runs
The method stays the same from run to run, which keeps one result comparable to the next. As new frontier models ship, the AI side of the comparison moves to newer models, and every run states which model it used.
Start from a published review
Every protocol has already been through a public audit or audit competition with a published report. Ground truth is the high and medium severity findings in those reports, and none of them come from us.
Run both tools on the same code
Olympix and the AI model run on the same audited codebases, and every finding they report is checked against the published reviews.
Count independent matches only
A finding counts when a tool surfaces an issue the reviewers reported. Ground truth sources are published with every run so anyone can check the arithmetic.
Olympix and a frontier model
A model on its own vs. a model inside a system
Olympix uses frontier AI too. Inside Olympix, AI orchestrates a deterministic engine that executes the code until an invariant breaks, which is why each finding can arrive with a proof attached.
Beyond the benchmark
Tested against real exploits too
Benchmarks against published findings measure how much a tool reaches. In a separate study we took every 2025 EVM hack with a smart contract issue at its root cause, fifty incidents in total, and assessed each one against Olympix with no additional intervention.
Read the 2025 exploit analysis →2025 EVM hacks Olympix would have prevented at the time of the study, and updates since then cover the fiftieth
of losses in that scope preventable at the code level, based on the backtest
Archive
Every benchmark, on record
Each run is published with its corpus, ground truth sources and method, so the history shows how Olympix moves against frontier AI over time. Models and corpora can change between runs, so compare results with both in view.
Your codebase
See what Olympix finds on your own code
Send us a repo that's already been audited, and we'll run Olympix at the same commit and show you what comes back.