6/22 Berkeley QANet Bug Fix Retro

Summary

• On June 22, 2022, a bug was discovered in the Berkeley QANet that halted the testnet and could have been exploited on mainnet.
• To prevent this type of attack from happening again, an emergency soft fork was issued to all block producers. More than 50% of the Mina network upgraded to the new version within three days. 
• A network halt is unlikely on mainnet and O(1) Labs have implemented new tests to reduce the likelihood of future similar bugs. 

Bug Discovery

On June 22, 2022, a bug was discovered on Mina’s Berkeley QANet when a community member (seemingly by accident) deployed a transaction that caused that network to halt. The community member was using an older version of snarkyJS. Upon initial analysis of the bug, O(1) Labs believed that a similar network halt wherein a bad actor could send this type of transaction to stop the network was theoretically possible on mainnet. If this type of action were taken on mainnet, additional transactions would not be able to occur. Out of an abundance of caution, ecosystem partner O(1) Labs took immediate action and was able to complete the fix within four weeks.

Technical Details 

On the BerkeleyQANet, an invalid transaction was included in a block, causing the network to halt. 

The zkApp transaction from an incompatible o1js (formerly SnarkyJS) version got accepted into the pool because the verifier incorrectly validated the proof. After it was added to the block, the prover correctly failed to verify the faulty/invalid zkApp proof, causing the network halt.

This was a data-validation issue rather than a cryptographic one and the Pickles cryptographic system was, and still is, sound. However, Mina nodes would accept proofs that were invalid before forwarding them on to Pickles.

The same check was also identified to be missing on mainnet. However, the issue on mainnet is different in that it is possible, although not straightforward and thus, very unlikely to have happened. An attacker could change the inputs to blockchain/transaction SNARK that doesn’t get caught during block verification/SNARK work verification when they are received. As a result, a faulty block could get into the canonical chain.

Network Upgrade Implementation

Similar to the last bug hotfix, engineers implemented a fix shortly after the initial hotfix discussion, and a phased upgrade rollout was executed.

Mina participants were notified about the upgrade via a phased approach, starting with block producers, and gradually expanded to the wider community to prioritize the safety of the network and reduce the likelihood of the bug being exploited. Over 50% of the network upgraded within three days, and over 60% of the network was upgraded within 5 days. The network has been working as expected since the fix was completed, which occurred in-line with the planned beta and stable upgrades.

Network Improvements

The goal of the soft fork was to prevent a potential attack before anyone could exploit the bug. O(1) Labs released an updated daemon for the majority of the stake so the network could upgrade before revealing what was known about the bug. 

Since the issue was identified, engineers have added a check to the code that Mina nodes use to validate gossip messages so that such invalid transactions cannot be included in any future network, including mainnet. In addition, other tests were added to make sure invalid proofs are handled correctly, and O(1) Labs engineers performed an audit to ensure no other checks were missing. 

All nodes should now be running 1.3.1.1 or 1.3.2beta2, or newer.

Thank you to the community members who identified the issues, especially Gareth Davies, and everyone who collaborated to upgrade quickly. 

If you want to contribute to further securing the network, please join us on Berkeley Testnet Alpha. The network needs members to send transactions and build zkApps to confidently test the environment in preparation for Mina’s mainnet.