The Vault With 100% Branch Coverage: What Smart Contract Test Coverage Doesn't Measure

Picture a vault that's ready to ship. It follows ERC-4626, the team has written tests for every function, and forge coverage reports 100% across lines, statements, branches and functions. The audit came back with every finding resolved, and by every number on the dashboard the contract is done.

We're going to break it three times. The first break makes an off-by-one change to a single line of its code while every test stays green, and the other two don't touch the code at all. Each one follows an attack pattern that has already cost real protocols millions, and each one teaches a different way to measure what your tests actually protect. By the end, "100% branch coverage" should read the way it really works, as a count of code that ran, which is a much smaller claim than it sounds like.

Branch coverage is only one part of the picture, though. The coverage we care about is how well your code is protected against problems people already know about: bug classes we've seen before, attack vectors that have already been used and the invariants your protocol relies on. Each break below falls outside what branch coverage measures. Detection is the other half, finding the bugs nobody has named yet, and it comes up once we get to invariants.

Why a Green Coverage Report Feels Safer Than It Is

Coverage tools count which parts of a contract your tests execute. Foundry breaks that down into lines, statements, branches and functions, and Hardhat produces similar reports. A test earns coverage by running code, whether or not it asserts anything about what the code did.

That's always been true, but it matters a lot more now that so many test suites are written by AI.

Ask a coding assistant to get coverage up and it will, generating thousands of tests in an afternoon that call every function and assert whatever the code happened to return. The suite ends up confirming what the code does rather than what it should do, and when the same model wrote both the contract and the tests, they share the same blind spots.

Researchers have started putting numbers on this, so far for general-purpose software rather than smart contracts. In one 2025 study, LLM-written tests for Java benchmark code hit 100% line and branch coverage on a function while catching just 4% of the bugs injected into it, and on Meta's TestGenEval benchmark, GPT-4o's tests for real open-source Python repos averaged 35% coverage and caught under a fifth of injected bugs. The pattern predates AI, too, since a 2014 study of 31,000 test suites had already shown that coverage barely predicts how many bugs a suite catches. None of these studies measured Solidity, but the mechanism is the same one the vault runs into below, and AI just makes it much easier to pile up tests that look thorough.

Low coverage is still a useful warning, because code no test touches is code nobody checked. High coverage just means the tests visited the code, and the three breaks below show how much can hide behind that.

Break One: Off by One

ERC-4626 vaults round against the user, so a withdrawal always burns at least enough shares to cover the assets taken out. Our vault does it with one line, using a common round-up idiom:

shares = (assets * totalSupply() - 1) / totalAssets() + 1;

The team's tests deposit 1e18 at a 1:1 share price and withdraw 5e17, then check that the user received 5e17 tokens. Every division comes out exact, and since the function has no branches, any test that calls it leaves it fully covered.

Now take the + 1 off the end of that line. Without it, the line rounds down, so every withdrawal burns one wei of shares less than it should, in the user's favor. Every test still passes, because the tests check the tokens the user received and never check how many shares were burned. The suite ran the line, counted it as covered, and never checked which way it rounds.

That off-by-one change is called a mutant, and mutation testing is basically making lots of them on purpose. The tool flips comparisons, swaps operators, deletes require checks and nudges values by one in either direction (x + 1 becomes x, and x becomes x + 1), then reruns your tests against every version. Off-by-one mutants are especially handy on rounding code, since a single unit is exactly what rounding is there to protect. If your tests fail, the mutant is killed, and if they still pass, it survives and points at behavior nobody is checking. Your mutation score is the share of mutants your tests kill. It's the same measure the AI studies above used to catch their weak suites, and research on open-source Solidity projects found the same pattern, with suites that had near-complete statement coverage still catching a low share of mutants.

A rounding bug this small looks harmless, and this failure mode is older than Ethereum. In 1982 the Vancouver Stock Exchange launched an index at 1,000 points. It recalculated the index after every trade and truncated the result to three decimal places instead of rounding. Each calculation was off by a sliver, and twenty-two months of slivers later the index read 524.811 when it should have been 1,098.892. In a vault the same drift happens one withdrawal at a time, and an attacker can amplify it on purpose.

The fix is a single assertion on the shares burned, checked at an exact share price and at one that doesn't divide evenly. Without a mutation run, though, nobody would have known that assertion was missing.

Break Two: The Attacker Who Never Calls Deposit

The team adds that assertion and runs mutation testing again, and this time every mutant dies. The vault now has 100% branch coverage and a perfect mutation score. We're going to break it again without changing anything in its code.

Many vaults compute their exchange rate from their own token balance, and anyone can change that balance with a plain ERC-20 transfer that never touches the vault's functions. In the ERC-4626 inflation attack, the attacker makes a tiny first deposit and then transfers a large amount straight to the vault. The next depositor's shares round down to zero, and their deposit effectively becomes the attacker's. The team's tests call deposit, withdraw and redeem many times, and none of them ever sends tokens to the vault another way, because that isn't a vault function.

This is the same pattern as one of DeFi's better-known exploits. In April 2023, an attacker flash-loaned WBTC, deposited into an empty Hundred Finance market, donated WBTC directly to the contract to inflate its exchange rate, then used a rounding error on redemption to drain about $7.4M. The flaw had been in the Compound v2 code Hundred forked since launch, through multiple audits. Seven months later, Onyx Protocol lost around $2.1M to the same issue in a market deployed five days earlier with no liquidity. Across the whole industry, audits and losses overlap more than you'd expect. CoinGecko's 2026 State of Crypto Security report found that platforms that had completed an independent audit accounted for 88% of the $3.63B stolen across 245 incidents between January 2025 and July 2026, although many of those losses came from key compromises and infrastructure breaches that sit outside what an audit reviews.

Mutation testing can't see this one, because no line of code is wrong. The problem is a path, a sequence of ordinary calls that leads the contract into a state its tests never set up. Measuring it takes a third number, adversarial path coverage. You fix the bounds first: how many calls deep a sequence can go, which outside behavior gets modeled (direct token transfers, token callbacks, flash-loan-sized balances, first-depositor and empty-vault setups) and which attack strategies apply. Within those bounds, you map the paths the contract can reach, work out which are feasible, attack each one with the strategies that fit, and record which paths were tested and which weren't. A contract's full state space is effectively infinite, but a bounded set of paths is something you can enumerate and count, so you can report it as a number right next to the other two. The number means exactly what its bounds say. A tested path with no violation tells you none of the modeled attacks worked within those limits, which is a narrower claim than proving the path safe.

Break Three: Two Calls at Once

Say the vault's underlying asset is an ERC-777 token. ERC-777 tokens are ERC-20 compatible, but when tokens arrive they call a tokensReceived hook on the recipient, which hands control to the recipient's code in the middle of the transfer. Now suppose the vault's redeem function reads the caller's share balance into a local variable, sends the tokens, and only afterwards writes the new balance back from that cached copy:

uint256 bal = balanceOf[msg.sender];
require(bal >= shares);
uint256 assets = convertToAssets(shares);
asset.transfer(msg.sender, assets); // ERC-777 calls the receiver's tokensReceived hook here
balanceOf[msg.sender] = bal - shares;
totalSupply -= shares;

Trace it with the vault holding 10,000 tokens against 10,000 shares and an attacker contract holding 100 shares. The attacker calls redeem(100). The vault caches a balance of 100, passes the check and sends 100 tokens, which fires the attacker's hook. Inside the hook, the attacker calls redeem(100) again. Nothing has been written back yet, so the cached balance still reads 100, the check passes, and the vault sends another 99 tokens (slightly fewer, because the first 100 tokens already left while the share supply hadn't dropped yet). The inner call writes the attacker's balance to 0 and returns, and then the outer call finishes and writes 100 minus 100, which is 0 again. The attacker burned 100 shares and walked away with 199 tokens, and the remaining depositors cover the difference.

That's reentrancy, and every test that calls redeem once from an ordinary wallet will pass, because a wallet has no hook to call back with. In April 2020 the same mechanism cost Lendf.Me about $25M, when an attacker used imBTC's ERC-777 hook to re-enter the protocol before it updated its records.

Payments engineers know this failure as a race condition. In 2015 a security researcher sent the same $5 Starbucks gift card transfer twice at the same instant from two browser sessions, and the destination card was credited twice. The EVM runs transactions one at a time, so reentrancy is how a smart contract ends up with the same bug: an external call hands control back to the attacker in the middle of a function.

Paths like these are why adversarial testing works from a list of attack strategies, since a team's own tests follow the paths the team had in mind. Reentrant callbacks, direct transfers, empty-state and first-depositor setups, flash-loan-sized amounts and price manipulation are each a way of leaving the expected path. Some of the costliest failures in finance came from a path nobody tested anymore. In 2012, Knight Capital left a retired trading function on one of eight servers, and a repurposed flag triggered it. In 45 minutes the firm lost more than $460M, and the SEC found it lacked adequate controls for code deployment and testing. In a smart contract, a deprecated function or an old implementation behind a proxy stays callable until someone removes it.

Reentrant callbacks belong on that list, so with adversarial path coverage, a path like this one is either attacked and reported or shows up as untested.

The Property That Would Have Caught All Three

Each break violated something the team believed about their vault but never wrote down. A withdrawal should never burn fewer shares than the assets are worth. A deposit that would mint zero shares, or fewer than the minimum the depositor agreed to accept, should be rejected rather than quietly accepted. A single set of shares should only ever be redeemed once.

Those beliefs are invariants, and writing them down changes the problem. Once an invariant is explicit, the question becomes whether any reachable sequence of calls breaks it, and if one does, that sequence is a counterexample you can replay as a test. Foundry's invariant testing searches for one by calling your contracts in random sequences. Automated tools can go further, inferring the invariants a codebase relies on without stating them and then searching for a counterexample to each. Known attack strategies cover known vulnerability classes, and invariants are how you catch the ones nobody has named yet.

What "Tested" Should Mean

It helps to think about smart contract security as two questions. Coverage asks how well the code is protected against what's already known (known bug classes, known attack vectors and the invariants the protocol depends on) and where that protection has gaps. Detection asks what's wrong that nobody has named yet, the bugs no checklist or attack strategy describes. You need both, since a clean detection result only means much if the known stuff has been covered first, and perfect coverage of the knowns tells you nothing about the unknowns.

Branch coverage is only the first layer of coverage, and the vault shows why coverage itself has to be more than one number.

Once a team tracks all three, writes down its invariants and sets a threshold for each, "tested" starts meaning something a passing build can actually back up. It also gives auditors, partners and governance something concrete to look at, and an auditor who starts from that picture gets to spend their time on the parts nobody has measured yet. For financial institutions moving onchain, these numbers look a lot like the evidence their risk and compliance teams already ask for from software controls, and since a bad transaction onchain can't be reversed, that evidence has to exist before deployment.

Olympix is built around that definition. Detection and all three coverage numbers come from one engine and one pass over the whole codebase, during development, at PR merge and before release, measured against thresholds the team sets, including at the seams where independently audited scopes meet.

Start With Your Own Vault

You can run every step below on your own codebase. Each one repeats one of the vault's lessons on code you know, so you'll see where your coverage number is promising more than your tests actually check.

1. Run forge coverage and treat the result as a floor, never a finish line. That's the number on the vault's dashboard before anything broke.
2. Run a mutation pass on your most important contract, starting with the functions that move funds. Every surviving mutant is a test you're missing, exactly like Break One. If most of your tests were generated with AI, start there, since that's where coverage and mutation score tend to drift furthest apart.
3. Write down three invariants your protocol can't survive breaking, and turn them into invariant tests. These are the properties that would have caught all three of the vault's breaks.
4. Write one test for each way an attacker leaves the expected path: a direct transfer, a reentrant callback, an empty-state first deposit and a flash-loan-sized amount. These cover Breaks Two and Three, the paths an attacker takes that your existing tests never do.
5. Remove or lock any deprecated function or old implementation that's still callable. That's the Knight Capital lesson: code nobody tests anymore still runs if something calls it.

Doing this once by hand is the quickest way to see the gaps in your own suite. Keeping all three numbers current across a whole codebase on every pull request takes more than a one-off exercise, and that's where tooling comes in.

What the Vault Teaches

The vault ends this article where it started, with 100% branch coverage, and it was broken three times anyway. The coverage number was accurate the whole time, because it only ever measured which lines ran, and every break lived somewhere else: a rounding direction no test checked, a donation path no test took and a reentrant call no test made. Each gap had its own measurement, and together they turn "well tested" from a feeling into a number a team can set a bar for and hold every release to.

Audits and the tools teams already use still matter. What changes is what a release has to show before it ships, and for contracts that hold value, whether that's a DeFi protocol or an institution moving assets onchain, the bar should include detection plus every layer of coverage, from branch coverage to mutation score and adversarial path coverage, across the whole codebase.

Where Olympix Comes In

Olympix measures detection and every layer of coverage from one engine, Deductive Interception, in one pass over the whole codebase. It runs during development, at PR merge and before release, against thresholds your team sets, including where separately audited scopes meet. The same engine and thresholds run in your terminal, in CI, in your editor and in your coding agent, and in CI the checks show up as required status checks on the PR, so a failing one blocks the merge.

Static analysis runs a detector set written specifically for smart contracts on every function on every commit, and reports line-level findings with severity right in the PR and the IDE.

Mutation testing runs against your own test suite on every change, with 14 mutation operators that include off-by-one mutants like the one in Break One, and reports every surviving mutant by location along with a mutation score you can enforce.

BugScout covers the known side through Bounded Adversarial Verification and fuzzing. It enumerates feasible paths within set bounds, runs the known attack strategies that fit each one (reentrant callbacks included), returns a proof of concept for every violation, and reports which paths were tested and which weren't. A path with no violation means no modeled attack worked within those bounds, which is a narrower claim than a proof of safety.

BugPoCer covers detection through Counterexample-Guided Invariant Refutation. It infers the invariants a codebase rests on from the code and the documents around it, such as specs, design docs, NatSpec comments, threat models and past audits, then symbolically executes until one breaks, with AI proposing candidates and the deterministic engine deciding. Findings it can reproduce come back as an executable proof of concept with the trace and the minimum conditions to trigger them, and candidates it can't reproduce yet are labeled Needs Further Review instead of being reported as confirmed. The Deductive Interception whitepaper shows how BugScout, working inward from known signals, and BugPoCer, working outward from the system, meet on the same paths.

Every one of these numbers can be a release gate. Thresholds are set per tool and per repo in the dashboard, for example zero open high or medium BugPoCer findings, zero BugScout exploits and a mutation score of at least 90%. A repo that misses a threshold fails at HEAD, a required tool that didn't run on HEAD gets flagged instead of skipped, and the records can be exported for auditors, boards or regulators.

Olympix also publishes its benchmarks. In an August 2026 run on 15 EVM protocols that had already been through professional review, BugPoCer reproduced 128 of the 188 high and medium severity findings those audits published. The October 2026 benchmark re-scored the same 15 protocols against a cleaned set of 160 findings and ran Claude Opus 5.5 alongside it: BugPoCer found 94 confirmed findings (101 counting the ones still marked Needs Further Review) and Opus 5.5 found 53. The two runs use different denominators, so the percentages shouldn't be compared directly. Looking back at 2025, a retrospective backtest found the engine would have surfaced the underlying vulnerability in 49 of 50 in-scope EVM exploits, meaning every 2025 EVM exploit with a smart contract root cause, assessed against the code as it stood before each attack, about $247M in code-level losses. Key compromises, social engineering and infrastructure breaches were out of scope.

Circle, an Olympix design partner from day one, runs static analysis on every EVM contract, mutation testing in CI and BugPoCer ahead of external audits.

To see those numbers on your own code, book a technical walkthrough. We'll run the engine on a codebase you've already had audited and show you what it finds, how much of its reachable behavior has actually been checked, and which paths nobody had tested yet.