Adversarial Testnet Retrospective — Testworld

Highlights/Stats

• Testworld was the largest Proof of Stake testnet outside of ETH 2.0 with 4.6K participants and 1.2k concurrently connected to the network.
• 12.6K total challenges submitted, racking up to 20M testnet points.
• Participation resulted in 265 new Genesis Founding Members bringing the community to a total of 663 members for the launch of mainnet.
• 120 of the top block producing participants accepted delegation from the Mina Foundation.
• 1000+ validators on the network greatly increased iteration speed and launched 12 releases during Testworld.
• Influx of community members brought in over 30k on Telegram, 21k on Twitter, and 8k on Discord. Members are distributed in over 100 countries around the globe speaking in over 16 different languages.

Overview

Mina’s adversarial testnet “Testworld” was successfully launched to uncover vulnerabilities and ensure the network is resilient in the lead-up to mainnet. The participation was overwhelming, and it made Testworld the largest Proof of Stake testnet outside of ETH 2.0. In this final test candidate, 4.6K members engaged in pushing the boundaries of Mina’s testnet to help prepare for Mina’s mainnet launch, coming soon in Q1 of 2021. Almost 500 Testworld bounties were given out to participants, including token rewards, USD bug bounty rewards, delegations from Mina Foundation’s treasury, and Genesis token grants. The bug bounty program is still ongoing, see here for more information.

Additionally, Testworld enabled us to identify top candidates for the Genesis token program, bringing the total number of Genesis founding members to 650+. These members will become Mina’s first block producers in the upcoming mainnet launch, and contribute to securing the network and a more decentralized future.

Table of Contents

I. Bug Improvements

II. Our Findings

III. Testworld Winners

Bug Improvements

Thanks to robust participation by the Mina community, the team was able identify and address several bugs to help improve the function and stability of the network:

SYNCING WITH SUPER CATCHUP

• With this change, participants are able to sync much more quickly and are made more aware of the current progress. The team has made this the new default for later releases.

LIBP2P (OUR PEER TO PEER NETWORKING SYSTEM)

• Fixed the max peers count not being respected and issue where some users were being flagged by their ISP as running malware
• Fixed networking bugs that prevented connecting to O(1) Labs nodes #7519

ARCHIVE NODE RECOVERY AND RESILIENCY

• Retry logic for the archive-node #7707
• Standalone tool in the archive-node package for recovering block data from alternate sources #7695
• A variety of improvements for reliability and recoverability of archive node data #7492 #7354 #7447 #7456
• To learn more about how this system works in its entirety visit https://docs.minaprotocol.com/en

GRAPHQL, CLI, UX AND LOGGING

• Asynchronous best-chain GraphQL endpoint #7717
• ClientSDK feature for getting the Public Key for a given Private Key #7512
• Improved documentation around -log-levels #7635
• Commit SHA is now included in error and fatal logs #7693
• Rosetta API additions / improvements #7588 #7535 #7129
• New graphQL blocks endpoint #7548 #7592
• Improved logging for ivar.fill exceptions exceptions #7483 and #7476
• Improved documentation around -log-levels #7635
• Improved logging around – Out of Memory errors #7356 and Monitor.Error exceptions #7277

LOCATED AND FIXED A VARIETY OF CRASHES INCLUDING:

• When nodes try to process expired transactions #7705
• Segfault in CamlPointerlibrary #7669
• A snark worker crash due to invalid keys #7666
• A bug causing the zkApps demo to crash #6133
• Error/exception handling for commands in the transaction pool #7672
• Prover crashes due to logic bug in #7672
• Dangling_parent_reference crashes #7376
• Crashes due to long genesis ledger filenames #7248
• A variety of crashes should be less frequent / non-existent #7611 #7575 #7568 #7562

REDUCED MEMORY CONSUMPTION

• Improved garbage collection for lower memory use #7551

Additional improvements:

• Terraform workspaces support #7710
• Prometheus metrics improvements #7685 #7677 Infrastructure/automation improvements #7595 #7602 #7603 #7573 #7511 #7430
• coda advanced dump-staking-ledger now returns an error when the staking ledger is not available #7612
• Fixed issue with getting stuck at block height 1 #7494
• Fixed the issue causing max observed block length 1 even on healthy nodes #7500
• Fixed the logic around snark work with fees less than 0.5 #7485
• Fixed transaction replacement (for sending the same transaction with a higher fee) #7346

Testworld Keypair Vulnerability:

A testnet participant uncovered a testnet (not protocol) vulnerability, and was awarded a bounty for reporting an issue regarding the visibility of Testworld keypairs. Testworld keypairs are password protected, however all of the keypairs we initially distributed had the same (infamous) password: naughty blue worm.

For Testworld, iteration speed and ensuring all of the keys had the ability to produce at least one block was a top priority, and so the team pre-generated all technical participant keypairs to achieve these goals. There was also a recent switch of service providers from S3 to Google Cloud, but the team didn’t take enough time to fully audit the new solution and understand how keypair visibility had changed; which meant there wasn’t adequate protection on those keys to ensure they were only visible to their intended recipients. All of these factors combined produced a vulnerability which would have meant testnet participants had the ability to gain an unfair advantage over other users, which is why the team decided to pause and start a new network.

Points and standings weren’t affected, but to ensure the vulnerability didn’t cause issues down the road, the network was immediately shut down, the leaderboard paused and the community informed of the situation. The network was relaunched and new keypairs were generated using more secure infrastructure and unique passwords.

The team appreciates being notified of this situation and was glad for this learning experience. Fortunately, this issue was only related to testnet and it was addressed promptly before it affected any standings or members. Mainnet keypairs will be user-generated and will not be distributed by O(1) Labs, so this will not be an issue in the future

Several new features were also introduced during Testworld:

• Distinct signatures in “mainnet” builds vs. the testnet signatures used up until now #7738
• New validate-transaction command for verifying test transactions signed via other tools (like the ledger app) #7698
• Keypair validation tools #7604 #7617
• Mina-generate-keypair features for increased transparency/validation #7516 #7514 #7583

Our Findings

The team often hears positive feedback on the quality of our documentation. This is thanks to the Mina community who helped improve it during the lifetime of all testnets in the last 1.5 years. Thanks to you, Mina’s documentation became one of the most comprehensive and accessible resources available to learn about Mina.

In launching Testworld, the team was able to analyze and fix areas of the network that needed improvement. There were reports of blocks not being broadcasted through the network. From the preliminary investigation it appeared that blocks were being rejected for being a slot too late. Furthermore, some of these were not due to resource constraints that could cause block generation to be slow and therefore rejected. To understand the impact of this, the team introduced a new log file mina-rejected-blocks.log that has all the rejected blocks a node has seen. These are the blocks produced by other block producers that are generated well within the slot duration but end up being rejected by the node.

The team analyzed more than 680 rejected-blocks logs that community members sent and found that there were quite a few blocks being rejected that could have filled empty slots in the main chain. Here are some numbers: Number of slots for which 1 or more blocks were rejected due to invalid time (late by 1 or more slots) = 1263

There can be multiple blocks per slot and only one of them will get accepted. However, number of missed slots due to blocks being rejected for arriving late= 236 out of 3215.

So ~7.5% of slots missed because of the latency+block production time+validation time. The investigation on this is being continued by testing the block rejected rate in different network conditions by varying the number of nodes connected, gossip traffic in the network, etc.

There is also an issue #7446 describing a possible fix to alleviate this issue.

Until fixed, this requires the assumed honest stake to increase by 7.5%, if an adversary were to fix this issue and keep their fix privately on their own nodes. Applying it to the protocol’s initial constants’ implied honesty assumption of 65%, this increases the assumption by up to 70.27% until the issue is fixed, which is still within an acceptable threshold for launch.

Lastly, the team is pleased to share that the hardfork went smoothly despite this being the first time there was a hardfork on a testnet during epoch 3! The team gained more confidence that the hardforking system works well.

Testworld Winners

Testworld took place with the primary reason to ensure the security of the protocol. Almost 500 Testworld bounties were given out, including token rewards, USD bug bounty rewards, delegations from Mina Foundation’s treasury, and Genesis token grants. Participants were able to win by:

• Uncovering vulnerabilities in the network
• Accruing testnet tokens by completing technical & community challenges, &
• By scoring high points on the testnet leaderboard

Awards were given to the top block producers, users who submitted unconventional methods to optimize Block and SNARK production, as well as top rankers on the Testworld leaderboard. The bug bounty program for uncovering vulnerabilities in the network is still ongoing, see here for more information.

DELEGATION FROM MINA’S TREASURY

The MINA foundation will delegate 100% of its tokens post-mainnet to 120 block producers. 5% fees on Foundation delegation is allowed. 60 days after mainnet launch, the performance of the block producers on the network will be evaluated and the treasury of the Foundation will be redelegated accordingly – The Foundation reserves the right to redelegate in its sole discretion and may change the terms and conditions of its delegation policy at any time with or without prior notice. All of Mina’s block producers are encouraged to opt in for the delegation program. Sign up for the Mina newsletter to receive updates on this.

ACCUMULATE TESTNET TOKENS BY BLOCK PRODUCTION

The top 20 users who found unconventional ways to optimize their earning via Block Producing were rewarded with up to 40,000 tokens.

Participants demonstrated unique and effective approaches to run Block Producers and optimize uptime. Some of the creative strategies included were: users who made use of geological or network diversity, scripts to help with running the daemon, or other methods of keeping high uptime.

ACCUMULATE TESTNET TOKENS BY SNARK WORK

The top 20 users who found unconventional ways to optimize their earnings via SNARK production were rewarded with up to 40,000 tokens.

Participants applied an unique way of selling more SNARKs. Some examples of creative methods include programmatically optimizing for the best current snark fee, running the snark workers in a sophisticated way, and making use of high end hardware.

LEADERBOARD STANDINGS

Members who are ranked in the top 50 on the Testworld Leaderboard were rewarded with up to 40,000 tokens.

For a full list of all Testworld winners, follow this link: http://bit.ly/36WFmtO. For details on corresponding rewards, check out the Testworld page: http://minaprotocol.com/testworld.

The team is impressed with all the skills, creativity and excitement in the Mina community. Testworld was a huge success thanks to everyone’s participation and support. The Mina Foundation and O(1) Labs can’t wait to launch Mina’s mainnet together with you. It is planned for Q1 2021, and there is a lot of exciting news and events coming up. Please sign-up for the Mina newsletter to receive important updates.