9/14 Devnet Bug Hotfix Retro

By: Mina Foundation

Summary

• On September 14th, a bug was discovered on Mina’s devnet testing environment via the core protocol engineers’ standard testing practice. The bug was not discovered on Mina’s mainnet or exploited, and no funds were compromised.
• As a precaution, an emergency upgrade was released and also strongly recommended block producers on Mina’s mainnet to upgrade to the new network. More than 60% of the Mina network smoothly upgraded within 24 hours.
• Additionally, to ensure there are no other bugs of this type, the core protocol engineers added more aggressive tests to that area of the code.  Additional edge cases that could lead to similar transaction logic issues were discovered and a follow-up upgrade with a larger, comprehensive set of fixes, was shared on September 23. More than 50% of the Mina network was upgraded in just over 24 hours.
• Improvements will be made to the transaction pool to reduce the likelihood of future bugs.

Bug Discovery

At around 5pm PST on September 14th, a bug was discovered on Mina’s devnet environment. The bug was triggered by an automatic transaction-sending script that O(1) Labs – one of Mina’s ecosystem partners – runs as a standard testing practice on the devnet. Engineers discovered the bug after being notified by the devnet health alerting system — the same system used to monitor mainnet. The bug was identified promptly within a couple of hours of the alert going off, and although the bug had not been exploited, emergency hotfix discussions started the same day in an abundance of caution.

Technical Details

A block producer on devnet run by O(1) Labs experienced a glitch in its transaction pool, mistakenly choosing an invalid transaction to include in a block. This transaction moved more MINA than the amount that was available in the account. Other nodes confirmed this transaction. The receiving account did not exist, and the transferred amount was insufficient to pay to create that account, so nodes disregarded the full amount and only verified that the account contained enough to pay the fee. The transaction SNARK correctly rejected the transaction for using more funds than were held in the account, and prevented SNARK workers from proving the transaction. 

As a result of this invalid transaction, the devnet network is no longer processing new transactions and will need to be relaunched once this has been fully addressed. All of the caution and urgency in the incident response was intended to ensure that the mainnet did not confirm a similar bad transaction. We would like to highlight that this is not a trivial condition to recreate and is more likely to happen by accident than on purpose by an adversary.

Network Upgrade Implementation

Engineers implemented a fix shortly after the initial hotfix discussion, and a phased upgrade rollout was agreed upon. 

Mina participants were notified about the upgrade via a phased approach, starting with block producers and gradually expanding to the wider community, to prioritize the safety of the network and reduce the likelihood of the bug being exploited. 

More than 60% of the Mina network was upgraded within the first 24 hours. On September 23, a follow-up upgrade was shared with the whole community which encompassed a larger, more comprehensive set of bug fixes. More than 50% of the Mina network upgraded to this follow-up in just over 24 hours.

Network Improvements

Engineers have since fixed the transaction application logic to handle this case. They have also added tests to check that such transactions cannot be included in verified blocks, and if they are included (either maliciously or by accident) those blocks get rejected. The team at O(1) Labs is also working to add more checks to the transaction pool, to ensure that such a glitch is even less likely in the future.

Nodes still running earlier versions should upgrade to the newest releases as soon as possible. For participants currently on 1.1.5 or 1.1.7, please follow the release notes here. If you are currently running any of the 1.2.0 betas, please follow the release notes here. 

This is the first major hotfix on Mina’s mainnet since its launch six months ago and we’d like to thank our dedicated Block Producer community for their swift response, which ensured the situation remained controlled throughout. We are currently working on formalizing a new process for handling these types of situations and we welcome community members and developers’ continued contributions to Mina’s network security and ecosystem.