Mastering Invariant Testing in Foundry & Echidna

Invariant testing as kind of super-fuzzing

Fuzz Test = many randomly generated values and verify assertion
Invariant test : apply same idea but to the system as a whole, during an invariant test, the fuzzer run again all functions in all contracts (until we choose to constrain it).

Invariant testing is about ensuring internal consistency and adherence to predefined logical conditions, while fuzz testing is about assessing how software handles unexpected and potentially invalid external inputs, with a strong emphasis on identifying security issues and vulnerabilities.

Invariant testing campaigns have two dimensions, runs and depth:

runs: Number of times that a sequence of function calls is generated and run.
depth: Number of function calls made in a given run. Invariants are asserted after each function call is made. If a function call reverts, the depth counter still increments.

Target Contracts: The set of contracts that will be called over the course of a given invariant test fuzzing campaign. This set of contracts defaults to all contracts that were deployed in the setUp function, but can be customized to allow for more advanced invariant testing.

Target Senders: The invariant test fuzzer picks values for msg.sender at random when performing fuzz campaigns to simulate multiple actors in a system by default. If desired, the set of senders can be customized in the setUp function.

Target Selectors: The set of function selectors that are used by the fuzzer for invariant testing. These can be used to use a subset of functions within a given target contract.

Invariant test Helper Functions with Foundry

Target Contract Setup :

Target contracts can be set up using the following three methods:

Contracts that are manually added to the targetContracts array are added to the set of target contracts.
Contracts that are deployed in the setUp function are automatically added to the set of target contracts (only works if no contracts have been manually added using option 1).
Contracts that are deployed in the setUp can be removed from the target contracts if they are added to the excludeContracts array.

Contract LifeCycle

construction/initialization
Regular functioning
Optionally end state

Think of which properties remain true during each life phases

Ways to assert invariants :

Direct assertions ⇒ query a protocol smart contract and assert values are as expected :
```
  assertGe(
      token.totalAssets(),
      token.totalSupply()
  )
```
Ghost variables assertions : query a protocol smart contract and compare it against a value that has been persisted in the test environment (ghost variable) :
```
  assertEq(
      token.totalSupply(),
      sumBalanceOf
  )
```
Deoptimizing (Naive implementation assertions) : query a protocol smart contract and compare it against a naive and typically highly gas-inefficient implementation of the same desired logic :
```
  assertEq(
      pool.outstandingInterest(),
      test.naiveInterest()
  )
```

Cheap vs Expensive Invariant

cheap ⇒ a simple require in the code is enough
expensive ⇒ off chain

White / Black Box Invariant

white : with internal knowledge
black : without

Types of invariants :

Relationships between inter-related storage locations :
- sum of all values in a mapping X must equal Y stored elsewhere in storage (regular check in white box)
- all addresses present in EnumerableSet X must also exist as keys in mapping Y (regular check in white box)
Monetary value held by contract/protocol and solvency requirements :
- once token/reward/yield distribution is complete, the contract should have 0 balance (end of state in black box)
- contract should always have enough tokens to cover liabilities (regular, black box)
Logical invariants thatprevent invalid states :
- account with active borrow can’t exit market where they borrowed from (regular, black box)
- protocol should never enter a state where borrower can be liquidated but can’t repay (regular,black box)
Denial of service (DOS) from unexpected errors :
- liquidation should never revert with unexpected errors such as access array out of bounds, under/overflow etc (regular,white box)

How to write invariants :

start from the whitepaper
Think of and Summarize properties in plain english before writing any code (⇒ in a properties.md file for example)
Categorize Invariants using Certora :
- valid states
- State transitions
- Variable transitions
- High level properties
- Unit tests
Start writing properties in order priority (most high level first) ⇒ think FREI-PI (function requirement, effect interaction and protocol invariants)
Bound values to not waste fuzz runs
Check your logic against different / de-optimized implementations (don’t reproduce the same issue/behavior)
Use multiple actors for more realistic scenarios
Limit the number of target and selectors the fuzzer is calling (including state vars)
- if you’re using echidna use internal functions
Check both success & failure cases for maximum coverage with either try/catch or if/else
Think about how your invariants may change depending on the state of your contract/of the system
Always check the coverage ⇒ you might not be testing everything (turn on coverage on echidna)
Reduce input space of the fuzzer (as less inputs for the fuzzer as possible)
Test multiple runs, depth and seeds
Create unit test with failed runs to confirm a bug

Life Cycle in 5 days :

1st day : finding invariants, defining them
2nd day : set up echidna contracts, foundry invariant test
3,4,5 : optimisations, fine tuning, different scenarios

Handlers vs No Handlers

A handler contract is used to test more complex protocols or contracts. It works as a wrapper contract that will be used to interact or make calls to our desired contract.

It is particularly necessary when the environment needs to be configured in a certain way (i.e. a constructor is called with certain parameters).

How it works is that, in the setUp function in the test file, we deploy the handler contract that will make calls to the pool contract and set only this handler contract as the target contract in the test using the targetContract(address target) test helper function.

The idea is to wrap underlying functions from the target contract, satisfying preconditions to prevent trivial reverts. No handlers can also be a choice.

Rather than expose the target functions directly to the fuzzer, we'll instead point the fuzzer at our handler contract and add functions to the handler that delegate to target.

Because of this, only the functions of the handler contract would be called randomly by the fuzzer.

⇒ As soon as we introduce a handler, we are starting to introduce assumptions about the system under test. It's necessary to constrain the system in order to test it meaningfully, but it's also important to stop and consider the assumptions we're making along the way.

Foundry comes with test helper functions in the forge-std library that allows us to specify our target contracts, target artifacts, target selectors, and target artifacts selectors.

Some helper functions are

targetContract(address newTargetedContract_)
targetSelector(FuzzSelector memory newTargetedSelector_)
excludeContract(address newExcludedContract_).

To see all available test helper functions, see here and here.

Ghost Variables :

We can use ghost variables in our handler contract to track state that is not otherwise exposed by the contract under test. For example keeping track of the sum of all individual deposits into a contract using an accumulator variable

*** If you have deposits and withdraws it is still preferable to not increment/decrement 1 ghost variable but have 2 variables to keep accounting distinct.
To check before/after for example

How to not “kind of replicate” the contract logic with ghost variables in our handler ?

If any of our assumptions are wrong in both the contract under test and in our ghost variable logic that replicates it, we will simply be replicating bugs in the implementation in our tests.

In general, I think it's a good principle to rely on external state from the contract under test whenever possible.

Chimera

setup.sol is base setup ⇒ have ghost variables use to track important state for comparison of contract state during invariant checks
Properties.sol : where invariants are defined
- inherits from setup and from chimera assert
- Define invariants in Echidna/Medusa style with property_ for functions prefix
Create handlers : function handler_ prefix : target function.sol inherits from properties + chimera Base target functions
Create front-end :
- Echidna : xxxxCrypticTester.sol : inherits from targets functions + cryptic asserts
- Foundry : xxxCrypticTesterToFoundry.sol : inherits from targetsFunctions + chimera Foundry Asserts
  - wrap property_* to invariant_*

Foundry

This enables invariant testing to test aspects that we ”didn’t think of.”

References :

Invariant Testing : From 0 to Hero

Table of contents