Olympix vs. Slither vs. Vertigo-rs: A Mutation Testing Comparison
Executive Summary
Mutation testing is a critical component of smart contract security, ensuring that vulnerabilities are detected before deployment. Olympix Mutation Testing outperforms its competitors—Slither and Vertigo-rs—across three key areas: mutant operator effectiveness, runtime & cost, and continuous development.
Mutant Operators: Precision Over Noise
Olympix focuses on high-impact vulnerabilities, selecting mutant operators based on real-world exploits to ensure maximum security relevance. Unlike Slither, which includes low-value mutations that generate unnecessary noise, Olympix’s operators are curated for accuracy and impact, with quarterly updates adding new security-relevant mutations.
Runtime & Cost: Speed and Scalability
While Slither and Vertigo-rs run locally and take 20+ hours per test cycle, heavily taxing a developer’s machine, Olympix offloads all compute to the cloud, running tests in minutes and allowing for unlimited runs at a fixed cost per year. This makes Olympix the most scalable, cost-effective solution, eliminating both infrastructure and opportunity costs.
Continuous Development: Future-Proof Security
Olympix is continuously improving, investing millions of dollars in research and development. Every major exploit is analyzed, leading to tool enhancements or new features. Olympix introduces quarterly updates for its mutation testing, automated unit testing, and static analyzer, and is set to release fuzz testing in 2025. In contrast, Slither’s development is limited as it was originally built as a lead generation tool for an audit firm, with no strong incentives for ongoing improvements. Vertigo-rs, maintained by RareSkills, sees only occasional updates, making it less reliable for evolving security needs.
For teams that prioritize fast, scalable, and cutting-edge security testing, Olympix is the clear leader. Its targeted mutant operators, rapid cloud-based execution, and ongoing investment in innovation make it the best choice for serious security teams looking to stay ahead of emerging threats.
Introduction to Mutation Testing
Mutation testing is a meta-testing technique that evaluates the effectiveness of a test suite by introducing small, controlled modifications (mutants) into a program’s source code. These mutants mimic common coding errors—such as logic mistakes, incorrect operator usage, or typos—to assess whether existing tests can detect them. The test suite is then executed against these mutants: if a test fails due to a mutation, the mutant is "killed," indicating that the test successfully caught the issue. If the test suite does not detect the mutation, the mutant "survives," highlighting potential weaknesses in the test coverage.
This approach is particularly crucial in Web3 and DeFi security, where immutable smart contracts, financial risk, and complex state management make thorough testing non-negotiable. Traditional unit tests and code coverage tools often fall short in identifying deep-seated vulnerabilities, as seen in real-world incidents like the CrowdStrike outage and the xz backdoor exploit. In contrast, mutation testing can expose untested code paths, improve test suite quality, and enhance overall smart contract security. Given the high stakes in DeFi, where millions of dollars are locked in contracts, mutation testing serves as an essential safeguard against catastrophic exploits before deployment.
This curated selection of mutant operators focuses on real-world vulnerabilities rather than generating unnecessary noise, ensuring high-impact security testing.
Mutant operators marked in purple are unique to Olympix.
1. Address Swap Mutations
Mutation Type: Swaps addresses in contract logic, such as sender/recipient.
Why It’s Dangerous: Can redirect funds or privileges to unintended addresses.
Example Hack: DAO Hack (2016) – Vulnerabilities in address handling allowed reentrancy attacks, draining $60M from the DAO.
2. Arithmetic Operator Mutations
Mutation Type: Alters arithmetic operations (+ to -, * to /).
Why It’s Dangerous: Can lead to incorrect calculations, overflows, or underflows.
Example Hack: SafeMoon Hack (2022) – Improper arithmetic checks led to an exploit allowing attackers to steal tokens.
3. Assignment Operator Replacement
Mutation Type: Changes assignment operators (= to += or -=).
Why It’s Dangerous: Can cause unintended state changes or incorrect fund allocations.
Example Hack: Beanstalk Flash Loan Attack (2022) – Misuse of assignments in governance logic allowed an attacker to drain $182M.
4. Comparison Operator Mutations
Mutation Type: Alters comparison logic (> to <, == to !=).
Why It’s Dangerous: Affects conditional logic, leading to bypassed security checks.
Example Hack: BNB Bridge Exploit (2022) – Improper comparison checks led to a $560M exploit.
5. Condition Negation Mutations
Mutation Type: Flips conditional statements (if (x) → if (!x)).
Why It’s Dangerous: Can bypass critical security checks.
Example Hack: Cream Finance Exploit (2021) – Poor condition handling led to an $18M attack.
Example Hack: Fei Protocol Exploit (2021) – Improper logic in conditional checks allowed attackers to withdraw locked funds.
9. Modifier Removal Mutations
Mutation Type: Removes function modifiers (e.g., onlyOwner).
Why It’s Dangerous: Allows unauthorized function execution.
Example Hack: Euler Finance Hack (2023) – Missing access control led to a $197M loss.
10. Remove delete Operator
Mutation Type: Removes delete statements that reset state variables.
Why It’s Dangerous: Leaves stale, exploitable data in storage.
Example Hack: Compound Finance Bug (2021) – A failure to reset variables allowed repeated withdrawals, draining $80M.
11. Remove emit Statement
Mutation Type: Removes event emissions for transactions.
Why It’s Dangerous: Reduces transparency and auditability.
Example Hack: Multiple Rug Pulls – Attackers remove emit to hide malicious transactions.
12. State Variable Initialization Changes
Mutation Type: Alters initial state values in contracts.
Why It’s Dangerous: Can leave contracts in vulnerable or unintended states.
Example Hack: Yam Finance Failure (2020) – Incorrect initialization led to governance failure.
13. Storage Location Mutations
Mutation Type: Alters whether variables are stored in memory, storage, or calldata.
Why It’s Dangerous: Can lead to unintended overwrites or incorrect reads.
Example Hack: Multichain Hack (2023) – Poor storage handling led to loss of $126M.
14. Ternary Conditional Mutations
Mutation Type: Alters ternary expressions (x ? y : z).
Why It’s Dangerous: Can lead to incorrect execution paths.
Example Hack: ThorChain Exploit (2021) – A flaw in condition handling led to a $7.6M attack.
Slither Unique Mutant Operators
Slither’s unique mutant operators do not target high-impact security vulnerabilities like reentrancy, logic errors, or access control flaws. No notable hacks have occurred due to these mutation types. This makes them less valuable for real-world vulnerability detection, as they introduce unnecessary noise rather than simulating critical exploits.
Why It’s Not Likely to Cause Vulnerabilities: Bitwise operations are rarely used in security-critical logic, making their impact on smart contract security minimal.
Notable Hack Example: None—No major exploits have been caused by bitwise operator misuse in Solidity contracts.
2. Comment Mutations
Mutation Type: Modifies or removes comments in the code.
Why It’s Not Likely to Cause Vulnerabilities: Comments do not affect contract execution; they are only meant for human readability.
Notable Hack Example: None—Comments have no execution impact and cannot introduce vulnerabilities.
3. Function Header Replacement
Mutation Type: Modifies function headers by changing visibility (public, private, internal, external).
Why It’s Not Likely to Cause Vulnerabilities: Most smart contract vulnerabilities arise from logic errors rather than visibility changes, which developers carefully control.
Notable Hack Example: None—While incorrect visibility settings can lead to issues, no major exploits have occurred from automated visibility mutations.
4. Integer Literal Mutations
Mutation Type: Alters fixed numeric values in the code.
Why It’s Not Likely to Cause Vulnerabilities: Contracts typically use predefined constants or external inputs, reducing the impact of arbitrary number changes.
Notable Hack Example: None—No critical exploits have stemmed from simple integer literal changes.
5. Misc Solidity-Based Mutants
Mutation Type: Various Solidity-specific modifications that do not fit into other categories.
Why It’s Not Likely to Cause Vulnerabilities: These are typically syntactic rather than logical changes and do not consistently introduce exploitable conditions.
Notable Hack Example: None—No general Solidity-based mutations have been linked to major exploits.
6. Remove if/while Statements
Mutation Type: Eliminates conditional (if) and looping (while) statements.
Why It’s Not Likely to Cause Vulnerabilities: Smart contracts often have minimal loop logic due to gas constraints, and removing conditionals usually results in compilation failures rather than vulnerabilities.
Notable Hack Example: None—No major exploits have resulted from removing control structures.
7. Revert Replacement
Mutation Type: Alters revert statements, potentially removing error handling.
Why It’s Not Likely to Cause Vulnerabilities: While missing revert statements can impact contract behavior, Solidity’s structured exception handling makes it unlikely to introduce exploitable vulnerabilities in well-tested code.
Notable Hack Example: None—No major exploits have been linked to revert statement mutations.
Vertigo-rs Unique Mutant Operators
Vertigo-rs does not have any unique mutant operators.
Olympix mutation testing is a fully managed, cloud-based solution, allowing developers to offload all compute-heavy processes to Olympix’s powerful servers. Unlike local mutation testing tools that slow down a developer’s machine, Olympix runs efficiently in the cloud, completing in minutes rather than hours. This enables engineers to run dozens of mutation test cycles per day, ensuring continuous security improvement without performance bottlenecks. With a fixed cost per year for unlimited runs, Olympix eliminates compute constraints, making it infinitely scalable while significantly reducing opportunity costs.
Slither
Slither’s mutation testing tool runs locally on a developer’s machine, leveraging Forge and solc, both of which are extremely compute-intensive. Running a full mutation testing cycle with Slither can take 20+ hours, heavily taxing CPU and memory resources, making it impractical to work on other tasks simultaneously. Since every test run competes for local machine resources, running multiple iterations is cumbersome and slows down development. The opportunity cost is high, as developers either have to wait for results or dedicate a separate high-performance machine to handle testing.
Vertigo-rs
Like Slither, Vertigo-rs is a local tool that runs directly on a developer’s machine, utilizing high CPU and memory to process mutations through Forge and solc. While Vertigo is optimized for efficiency, it still suffers from long runtimes (potentially over 24 hours for larger contracts) and high compute demands, making it impractical for frequent re-testing. Running multiple mutation test cycles locally is both time- and cost-prohibitive, as developers either need to invest in powerful hardware or accept long downtimes while tests execute, significantly reducing iteration speed during security testing phases.
Runtime per Mutant Comparison (Olympix, Slither, Vertigo-rs)
The number of mutants generated per test cycle depends on the contract size, but a reasonable estimate is 200 mutants for a medium contract.
Based on previous findings, a single run on Olympix takes 20+ minutes (~1,200 seconds) for a typical large contract, on Slither takes 20+ hours (~72,000 seconds), and on Vertigo-rs takes 25+ hours (~90,000 seconds).
At an average of 200 mutants generated for a typical large contract, Olympix’ time per mutant is 0.8 seconds, Slither’s is 48 seconds, and Vertigo-rs’ is 60 seconds.
Cost per Run Comparison (Olympix, Slither, Vertigo-rs)
Mutation testing should be run on every new commit during the testing phase of a development life cycle.
As such, the typical cost per run can be calculated based on the following estimations:
A team of 2 solidity developers
A typical large contract - generating approximately 1,500 mutants
A quarterly development lifecycle - 1 week of mutation testing per quarter
Each developer pushing 5 commits per day across an average of 7 contracts during the week of testing & running mutation testing for each commit
A total of 200 runs per year
For Olympix running on Olympix servers:
$2,000 fixed annual cost
No local machine processing costs
For tools running locally:
An 8-core machine
$0.032 cost per minute ($1.92 per hour) of github action time
It is important to note that as a team grows or as development velocity increases, the total number of runs will increase as well, driving up variable costs with tools running on local machines.
Continuous Development
Olympix
At Olympix, we are dedicated to maintaining our position at the forefront of smart contract security by investing millions of dollars in continuous development. Our commitment includes:
Comprehensive Exploit Analysis: We meticulously analyze every known exploit to assess whether our current tools could have prevented them. If gaps are identified, we enhance existing tools or develop new solutions to address these vulnerabilities.
Regular Updates: We introduce new mutant operators quarterly to stay ahead of emerging threats.
Advanced Testing Capabilities: In Q2 2025, we will release support for fuzz and fork tests within our mutation testing framework, enhancing our detection capabilities.
Static Analysis Enhancements: Our static analyzer receives quarterly updates, with new detectors and improvements to identify statically detectable bugs more effectively.
Intelligent Fuzzing: Recognizing that some bugs elude static analysis and mutation testing, we are developing an intelligent fuzzer, set to enter private alpha in 2025, with new fuzzing strategies released every quarter.
Slither
Slither is an open-source static analysis tool for Solidity and Vyper, initially developed by the security firm Trail of Bits. While it has been a valuable resource since its release in 2018, its development relies on contributions from the open-source community. Given that Trail of Bits is an audit firm, there is limited incentive for them to invest heavily in enhancing Slither, as doing so could potentially devalue their primary service offerings. Consequently, Slither may be more suitable for smaller projects rather than high-stakes, reputable deployments.
Vertigo-rs
Vertigo-rs is a mutation testing tool for Ethereum smart contracts, actively maintained by RareSkills. Originally forked from Joran Honig's Vertigo, this tool has been updated to support the Foundry framework. While RareSkills continues to maintain Vertigo-rs, the scope of its development is more limited compared to Olympix's extensive investment. Updates and new features are introduced as needed, but the scale and frequency of enhancements may not match the rapid advancements in smart contract vulnerabilities.
Conclusion
When it comes to mutation testing for smart contracts, Olympix is unmatched in effectiveness, efficiency, and innovation.
Mutant Operators: Olympix focuses on high-impact vulnerabilities, carefully selecting mutant operators that align with real-world exploits. Unlike Slither, which includes low-value mutations that introduce unnecessary noise, Olympix ensures that every mutation contributes to meaningful security testing.
Runtime & Cost: Olympix runs in minutes, while Slither and Vertigo-rs take hours or even days, consuming significant local compute resources. With unlimited runs at a fixed per year, Olympix eliminates both opportunity and infrastructure costs, making it the most scalable and cost-effective solution.
Continuous Development: Olympix invests millions of dollars to ensure its tools remain cutting-edge. We continuously analyze new exploits, introduce quarterly updates, and are set to release fuzz and fork test support in Q2 2025. In contrast, Slither is an open-source project with little incentive for advancement, and Vertigo-rs is maintained on a much smaller scale.
For security-conscious teams that need fast, scalable, and continuously improving mutation testing, Olympix is the clear leader—ensuring your smart contracts are resilient, audit-ready, and protected from the latest threats.
What’s a Rich Text element?
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.
Follow-up: Conduct a follow-up review to ensure that the remediation steps were effective and that the smart contract is now secure.
Follow-up: Conduct a follow-up review to ensure that the remediation steps were effective and that the smart contract is now secure.
In Brief
Remitano suffered a $2.7M loss due to a private key compromise.
GAMBL’s recommendation system was exploited.
DAppSocial lost $530K due to a logic vulnerability.
Rocketswap’s private keys were inadvertently deployed on the server.
Hacks
Hacks Analysis
Huobi | Amount Lost: $8M
On September 24th, the Huobi Global exploit on the Ethereum Mainnet resulted in a $8 million loss due to the compromise of private keys. The attacker executed the attack in a single transaction by sending 4,999 ETH to a malicious contract. The attacker then created a second malicious contract and transferred 1,001 ETH to this new contract. Huobi has since confirmed that they have identified the attacker and has extended an offer of a 5% white hat bounty reward if the funds are returned to the exchange.
