Olympix vs. AI

The corpus is the same fifteen EVM protocols we used for our August audit benchmark. Each one had already been through a public security review by an audit firm or a competitive audit, with a published report. Ground truth is the high and medium severity findings in those reports, and none of them were produced by us.

Ground truth counts were cleaned on 29 September 2026. Uniswap v4 shows what the cleanup does: its six original findings were copied from three earlier reviews that ship inside the repo, three of them were fixed before the commit under test, and the competition ruled the other three out of scope, so all six came out.

Both tools ran on the same fifteen codebases, and each finding they reported was checked against the published reviews.

We report two sets and two views. All 15 is the full corpus. Best 10 is a subset of ten protocols holding 118 of the findings. TP counts true positives, where a tool independently surfaced an issue the reviewers reported. TP+NFR also counts.

Most of the criticism of AI vulnerability discovery this year has been about what lands on the defender's desk. A piece in The Hacker News in April argued that a tool producing "high-confidence-sounding false positives at scale" adds to a security team's workload, since every spurious critical finding has to be triaged and dismissed. When AISLE re-ran its tests on code where the bug had already been patched, most of the models that found the bug also flagged the fix as vulnerable.

The record on Mythos itself is mixed. Epoch AI's review cites an independent test on curl, one of the most heavily reviewed codebases in open source, that turned up one low-severity vulnerability alongside four false positives, and it also concludes that Mythos's real edge is more likely a lower false positive rate than a jump in discovery. Either way, a finding that arrives as a description still needs someone to confirm it.

Olympix handles this inside the pipeline. BugPoCer's verification phase takes every candidate the detection phase flags, tries to reproduce it as a working exploit, writes a proof of concept for the ones that hold, and discards the ones it can't reproduce. A developer receives an exploit trace and a PoC they can run, so the finding arrives as a reproducible fact rather than a suspicion to triage.

That's what the TP tag in this benchmark means. BugPoCer's 94 true positives across all fifteen protocols are findings it tagged as proven, with a proof of concept and evidence attached. Findings that are lower impact, tied to a user-defined invariant, or without a compiling proof of concept yet are tagged Needs Further Review and counted only in the TP+NFR view. Findings BugPoCer concludes aren't real or aren't exploitable are tagged False Positive and kept out of both.