Susucoin Hard Fork @ Block 125900

This is a behind-the-scenes look into how the Susucoin dev team identified an exploit, mobilized a community, built a patch, and deployed a hard fork.

December 18, 2018 @ 14:40 JST

Ishikawa-san messaged me to tell me he had received a warning from another coin dev that Susucoin was vulnerable to a difficulty algorithm exploit. The reporting coin dev gave some advice to Ishikawa-san about how to proceed. I was initially suspicious of the warning and began researching the exploit. 

December 18, 2018 @ 14:45 JST

I tell Ishikawa-san that a hard fork would be the only way to effectively fix this exploit if we were truly affected by it.

December 18, 2018 @ 15:21 JST

Ishikawa-san was confident that Susucoin was vulnerable to this exploit, which was effectively a >51% mining attack that, when paired with timestamp manipulation, could procure an unlimited amount of blocks in just a few hours. Note, Susucoin was not the only affected coin. Many coins have this same (or similar) code and are still vulnerable.

December 18, 2018 @ 15:26 JST

I decide that a hard fork is the only way to guarantee Susucoin's long-term safety.

December 18, 2018 @ 15:30 JST

I begin reading CoinExchange's policies about hard forking.

December 18, 2018 @ 15:39 JST

A plan is made. We would first do a thorough confirmation that Susucoin is affected by the expoit. In case of a positive confirmation, we would notify CoinExchange immediately and begin setting up a hard fork.

December 18, 2018 @ 15:48 JST

We discover that this same bug was already exploited in a different coin and the attacker, in just a few hours, was able to lower the difficulty from 500,000,000 to just 200 and stole many blocks during the attack. Ishikawa-san learns that at least BTC, LTC, BCH, and DASH all have similar bugs in their code, but are generally protected due to the difficulty of executing a >51% attack on their chains. Susucoin is much smaller and more vulnerable to such mining attacks, so our plan of action would need to be swift.

December 18, 2018 @ 15:51 JST

Ishikawa-san and I thoroughly confirm that Susucoin's codebase is affected by this bug.

December 18, 2018 @ 15:58 JST

I begin writing a letter to CoinExchange explaining the bug and asking them to put Susucoin into maintenance mode until we can get the new hard-forked client ready.

December 18, 2018 @ 16:06 JST

CoinExchange is notified that "Susucoin has a bug in our difficulty algorithm that allows potentially infinite blocks within just a few hours of a >50% (sic) hashrate selfish mining attack."

December 18, 2018 @ 16:06 JST

I notify the Discord channel that we are investigating a bug in Susucoin and will probably need to hard-fork. 

December 18, 2018 @ 16:10 JST

Ishikawa-san and I discuss hard-fork strategies. 

December 18, 2018 @ 17:10 JST

We receive confirmation from CoinExchange that they are reviewing our issue. 

December 18, 2018 @ 17:27 JST

Ishikawa-san and I bunker down and begin studying the exploit and how to best patch it for Susucoin. 

December 18, 2018 @ 17:49 JST

We decide to upgrade to either the LWMA-1 or LWMA-3 algorithms. We begin discussing the merits of each.

December 18, 2018 @ 18:18 JST

Ishikawa-san and I have a 20-minute-and-34-second phone call to decide that we will upgrade to zawy's LWMA-3 algorithm. 

December 18, 2018 @ 18:56 JST

Ishikawa-san prepares and begins compiling the new client with the LWMA-3 difficulty algorithm. 

December 18, 2018 @ 19:51 JST

Ishikawa-san confirms he is able to compile the new difficulty algorithm into Susucoin. We begin discussing hard-coded constants in the algorithm and how to adjust them for Susucoin. 

December 18, 2018 @ 19:59 JST

I ask anybody mining on the testnet to stop mining because we will test the hard fork in a few hours.

December 18, 2018 @ 20:16 JST

Ishikawa-san takes a break and I review the new difficulty algorithm code and begin compiling it in my environment. 

December 18, 2018 @ 20:18 JST

I confirm compiling and start checking for bugs in the implementation. 

December 18, 2018 @ 20:32 JST

I confirm the algorithm implementation is correct and begin setting up a hard fork on the testnet that we can use to test the difficulty algorithm with real mining hardware.

December 18, 2018 @ 20:39 JST

My neighbor crashes into my mailbox with her car. I go outside and talk to her. She gives me some bread and said she will pay for the damages.

December 18, 2018 @ 20:54 JST

I finish writing the testnet hard-fork code and send it to Ishikawa-san for review.

December 18, 2018 @ 21:11 JST

Ishikawa-san finishes his meal and reviews my hard-fork code.

December 18, 2018 @ 21:50 JST

Ishikawa-san approves my hard-fork code and we begin setting up the testnet hard fork on a few full nodes.

December 18, 2018 @ 22:52 JST

I invite the owner of People's Pool, zinntikumugai, to help us test the hard fork. I send him the code and he begins upgrading his pool to test the hard fork on the testnet.

December 18, 2018 @ 23:03 JST

Ishikawa-san and I finish compiling and setting up three (3) full nodes for testing the hard fork on the testnet. 

December 18, 2018 @ 23:12 JST

I mine testnet block #17081, hardforking the testnet.

December 18, 2018 @ 23:14 JST

I confirm with Ishikawa-san that the hard fork worked correctly on the testnet.

December 18, 2018 @ 23:34 JST

I send my main net hard-fork code to Ishikawa-san for review. 

December 18, 2018 @ 23:38 JST

Ishikawa-san says that at first glance the hard fork looks correct, but he needs a bit more time to check thoroughly.

December 18, 2018 @ 23:40 JST

I check CoinExchange and Susucoin was in maintenance mode already. I'm not sure exactly what time they put Susucoin into maintenance mode.

December 18, 2018 @ 23:51 JST

I point an ASIC at zinntikumugai's testnet pool and begin putting some hash into the algorithm to test that it is adjusting difficulty correctly. I enable some analytics testing and let the miner with analytics code run all night.

December 19, 2018 @ 09:03 JST

I notice irregularities in the results from my analytics code of the mining on testnet. I begin debugging the problem by first reviewing the analytics and then doing some experiments to pinpoint the problem.

December 19, 2018 @ 10:07 JST

I notice that the difficulty is dropping to almost 0 about every 25 blocks on the testnet. I know that the testnet has code which is supposed to drop the difficulty to 0 during certain circumstances. I begin analyzing if this drop in difficulty is because of the difficulty algorithm or the aforementioned "feature" of the testnet.

December 19, 2018 @ 13:32 JST

Ishikawa-san contacts me and we have a phone call to discuss the difficulty drop on the testnet. We generate many plots to see how the testnet difficulty is being reset. 

December 19, 2018 @ 15:56 JST

Ishikawa-san begins making a test suite with values from the testnet to check if this is a bug in the algorithm or a "feature" of the testnet.

December 19, 2018 @ 16:33 JST

I think the test suite code will take a long time to prepare, so I come up with an alternate way to verify whether this is a bug in the algorithm or a "feature" of the testnet. A simple printf in the code should tell us if the testnet is resetting the difficulty or not. 

December 19, 2018 @ 16:36 JST

We put the printf in the testnet code and redownload the testnet blockchain. 

December 19, 2018 @ 16:53 JST

We confirm the testnet was resetting the difficulty and it was not a bug in the new algorithm.

December 19, 2018 @ 18:33 JST

The Susucoin Discord chat erupts into discussion about hard forks and the future direction of mining algorithms for Susucoin. At the end of the discussion, I decide that we will just hard-fork Susucoin to implement the new difficulty algorithm and we can continue discussing mining algorithms at a different time.

December 19, 2018 @ 20:39 JST

After being distracted by the Discord chat and having dinner, Ishikawa-san makes a pull request to the Susucoin GitHub and I review it.

December 19, 2018 @ 20:52 JST

Ishikawa-san sends the testnet mining data to zawy, who takes a look at the raw data to confirm if there is anything wrong with the algorithm implementation. 

December 19, 2018 @ 23:04 JST

I merge Ishikawa-san's difficulty algorithm patch and begin preparing a patch with the hard-fork height targeted to approximately 26 hours after the pull request. I estimate that with compiling clients and testing, it should give end users about 24 hours to upgrade to the newest code. 

December 19, 2018 @ 23:30 JST

Ishikawa-san reviews and merges my hard-fork patch into the Susucoin master repository. We begin installing the hard fork on a few full nodes to test that everything is compiling and working correctly. I prepare patch notes and bump the Susucoin version. 

December 20, 2018 @ 00:29 JST

After our testing is finished, Steve begins compiling the Susucoin wallets for windows/mac. I announce that the hardfork code is available on github and Windows/Mac clients are being compiled.

December 20, 2018 @ 01:34 JST

Steve uploads the new Windows/Mac clients to the Susucoin website. I ask everybody on Discord and Twitter to update their Susucoin wallet and announce that the hard fork will happen in approximately 24 hours. 

December 21, 2018 @ 01:30 JST

Susucoin is hard-forked at block height 125900 to use the LWMA-3 difficulty algorithm.

December 21, 2018 @ 01:33 - 01:39 JST

Pools announce that they have successfully hard-forked and everything is working well.

Thank you to everyone who helped with the hard fork. 
Thanks to the community for upgrading their clients so quickly. 
Thanks to CoinExchange.io for taking swift action. 
Thanks to Ishikawa-san for helping us build and deploy the hard fork. 
Thanks to Zawy for helping us analyze and decide on the best course of action. 
Thanks to zinntikumugai for helping us test easily with his pool.