If you already know about Ravencoin and the vulnerability, you can just skip this background section. Ravencoin is an open-source project which started as a code-fork (copy) of Bitcoin’s code. Code changes were made to increase capacity and speed, a new name (Ravencoin) given, a different token issuance schedule for RVN built, and a new lottery mechanism for issuing new RVN (mining) was added. Ravencoin was launched as its own network on Bitcoin’s ninth birthday on Jan 3, 2018. On November 5th, 2018, the ability to create your own tokens, sub-tokens, and unique tokens was activated. Tags, restricted assets, and memo capabilities were activated on Feb 7th, 2020. The Ravencoin network consists of people voluntarily running code. This code finds and connects the nodes together into a big network capable of value transfer, and creates a tamper-proof ledger that everyone shares.
On the morning of June 29th, 2020, a discovery was made that the network was not operating as intended. Extra RVN were being minted outside of the 5000 RVN per block that are ordinarily created by the network in a once-per-minute competitive lottery.
A vulnerability was discovered in the consensus code of Ravencoin. As you can imagine, there is a lot of new code in Ravencoin to handle messaging, assets, tags, asset naming, etc. There is a special class of code that accepts or rejects transactions called consensus code because all nodes should arrive at a consensus (or mutual agreement). It is considerably more complex in Ravencoin than it is in Bitcoin because of the asset layer. It is the governance rules for what is allowed. It is the bouncer, and the security guard. It comes in two levels. One level protects the mempool. The mempool is where transactions wait to be put into blocks. It is possible for different nodes to have different stuff in their mempool. The other level is what is allowed into a block. This is where double-spends are caught. But in the case of Ravencoin, it is also where it checks to make sure a newly created asset has a unique name, and where it checks that 500 RVN is sent to a specific burn address to create an asset. There are lots of rules like this.
Important to this story is a rule that was in place to check that for any asset transaction, make sure that RVN output is 0. As originally written, this rule was for any asset transactions. A code change that gave one message for asset creation with non-zero RVN, and a different error for asset transfers with non-zero RVN was introduced, and because there wasn’t a case for asset re-issuance it allowed an output with RVN. It appears this was introduced intentionally.
There’s a reason we think the vulnerability was not a mistake but was introduced intentionally. For one, the GitHub account for WindowsCryptoDev was created just for submitting this pull request. Yes, I can hear the readers saying, “then why did you let the PR in?”. There were two reasons. First, the changes looked pretty innocuous and were disguised as a formatting change for custom error messages. And second, we are always requesting that more developers contribute to this project. We want this project to be decentralized, with lots of ideas, and lots of developer support. It was great to see someone pitching in. Note: We do have a tremendous amount of developer support, but most is external to the core code.
For those that aren’t developers, what happened is that a critical check was moved from outside the conditional (if statement) into two if statements which cover both cases (New Asset and Transfer Asset). The problem is that there were three cases (New Asset, Transfer Asset, and Reissue Asset). The re-issue asset wasn’t checked and so a RVN output could be created on a reissue and the consensus code would let it through.
So could it have been an accidental mistake by an inexperienced developer? Sure, anything is possible, but the evidence suggests the possibility of a carefully planned attack. There is an asset that was created two days after this PR that can be linked to the attacker through analyzing the blockchain and the addresses involved. The automated attack didn’t begin until May 9th and appears to have been crafted to stay hidden.
The attack stayed hidden for several reasons. It was designed to stay hidden. It required an asset, and then the asset to be re-issued. Lots of assets were created with lots of different names, and these were transferred around. From what we can tell, this was done just to obscure the reissuance of the asset. If you think about it, it might look suspicious to keep re-issuing the same asset, and someone might have looked into the reason. After a whole bunch of transfers, an asset like TESTING_GPU_TRANSFERS* would be reissued, which is where the transaction exploited the vulnerability and minted the excess RVN.
It was the team from CryptoScope that saw outputs of RVN when building an asset-aware explorer for RVN. The explorer is great, and you can try it at https://rvn.cryptoscope.io. CryptoScope has put out its own public post about this vulnerability and how they found it.
They asked us about these outputs and to them they looked like Illegal Supply. And they were right. These RVN were a result of the exploit where an asset is reissued, the input RVN is 0, and the output RVN is whatever the exploiter wanted it to be.
Because the vulnerability was obvious if you go to this link: https://rvn.cryptoscope.io/address/?address=Illegal%20Supply, we requested that CryptoScope take links off that go to the illegal supply page. CryptoScope said they could temporarily leave it on an unpublished domain and pause the published domain. That helped out a lot. There was concern from some of their developers that they were hiding this, and if it was an ‘inside job’, meaning us, then they were protecting the perpetrators. So we allowed them into more of the internal conversation on what we were doing, and let them know that we would let everyone know how helpful they were. They were very helpful. We also appreciate them building an asset-aware explorer for Ravencoin.
The code fix
This kicked off one of the longest weeks of my life. After we find out about it, we need to track down how. With the information made visible by the CryptoScope explorer, it didn’t take too long to figure out the vulnerability. The code fix was completed quickly, but we couldn’t publish it to the main repo because it would reveal the vulnerability and put RVN holders everywhere at risk.
We realized that since this is a tightening of the rules (disallowing the RVN issuance on re-issue) that it could be done as a soft fork. This means that only the nodes that mine blocks needed to be updated. This is a much smaller set of nodes. Finally, something breaking in our favor. But there are two big risks. One risk is a chain split where the network isn’t clear which is the real Ravencoin blockchain, and the unpatched nodes are willing to accept a chain that has the exploited transactions. For that reason, we need a dominant amount of hash power. How dominant? Well, that depends on how sure we want to be that we don’t create a different, equally disastrous outcome. 51% isn’t gonna cut it. How about 60%? Maybe, but if the attacker, who seems to be to be sophisticated decides to do a Nice Hash fight and keep the non-fixed code in play, it won’t be enough.
We settle on 70%, which is far less than we would normally use because the other 30% ends up with orphaned blocks, but under these emergency conditions that wasn’t the primary concern. In fact, seeing orphaned blocks is what we hope will get the unknown mining pools quickly transitioned to the new code.
Not Signaling the Attacker
We know there’s a risk that once the attacker sees a code change, or an attempt to slay their golden goose, that the Ravencoin network is at even greater risk.
We create a decoy change for the public open-source code. It was a small change that fixes the JSON encoding for one of the rpc calls related to the messaging feature. We put it into master and update the version number on both the public decoy and the private repo vulnerability fix. Now, I know that some of you might say, “but that’s what an evil team would do if they wanted to slip through malicious code.” Yes, we get that. The message that went to the miners let them know about the update and offered to provide the private repo for them to build-from-source. The decoy was just so that if the attacker was watching for an update, they would see a small change to an rpc call and not think their golden goose was being throttled.
At this point early in the week, we think we can just quietly let the mining pools update, and tell everyone later about the fixed vulnerability. We were wrong.
Ravencoin is very decentralized, and therefore impossible to just ‘push’ the fixed code to the mining pools. We can request that mining pools update, but somewhere between 48% and 57% is mined by known pools. The other is unknown, meaning we have no way, at all, of contacting them without public disclosure. Yes, we could do a limited public exposure in a mining chat room, or something, but that looks like we’re hiding a vulnerability, and in the crypto community that can go viral, because it looks evil, despite good intentions.
We decided that to get the mining pools updated and get Ravencoin secured, we needed to be public about the problem.
We also felt compelled to notify the exchanges before the public announcement, and suggest that they turn off deposits, and at their discretion maybe even pause trading and withdrawals. We didn’t want to completely stop the markets. It doesn’t seem right that you can’t sell when you get the bad news, and for those that have a high-risk tolerance and want to buy on the bad news, it doesn’t seem right to stop them either. So why stop deposits? Because there was a non-zero probability that the perpetrator creates such a large quantity of RVN that the RVN community has no choice but to re-start and roll-back the issuance, which would hurt exchanges that took in that RVN.
We sent individual messages to all the mining pools that we know. Most of them know us pretty well, but we offered the option for them to build-from-source, which means the code that fixes the vulnerability is sent to them so they can build it themselves and see any code changes that are being introduced. It requires mutual trust. The incentives here are aligned properly as the mining pool only benefits if there is value for the coin being mined, so there isn’t any incentive to hurt the network.
The timing of mining pools updating is challenging because they are spread around the world, so messages that are sent out may not be seen for another fourteen hours. And for some, we’re communicating through their support e-mail, and not directly to the person who can take quick action.
BIP9 activates the new code after 70% of the blocks are mined with the new code. But not just any 70% of the blocks. It was actually 1411 of 2016 blocks. Some of this code was inherited from Bitcoin which has a 2016 block cycle of two weeks. Ravencoin runs faster so it’s 2016 blocks, at one block per minute, is only 1.4 days. Ordinarily, it also waits for another 2016 blocks after locking-in to activate. This gives time for the other miners (the minority) to get on-board with the majority or be left behind. In this case, we choose to override that extra waiting period because of the emergency nature of the problem.
The Starting Gun
We had already been working on this for days trying to think of other ways to handle it. Could we slip it to the mining pools without the perpetrator knowing? Could we convince enough hash power to update? Is there any other way to communicate to the unknown pools without appearing sneaky, or worse, tipping off the attacker.
I didn’t sleep much on Tuesday and Wednesday from worrying about these questions and by Thursday my brain was mush. I wrote up the message to send to the exchanges, then the message for the mining pools, and finally the message for everyone else. We knew that as soon as we sent anything to exchanges or mining pools that the starting pistol had fired and it all needed to happen very quickly. News would spread, and the perpetrator would become aware that his golden goose was being killed. I was asked if I was ready to go, and I wasn’t. I was on my third cup of coffee, and my brain wasn’t functioning well — it was buzzing. We discussed holding off until Monday but thought better of it. If I could just rest for a few hours, I’d be ready to go. I took a short nap and after lunch, on Thursday we put together a checklist of who would do what. We requested help from some of the most trusted Ravencoin community members, and we began notifying the world starting with mining pools and quickly moving to exchanges. We knew that not all exchanges would immediately get the message and pause deposits, but we felt it necessary to notify them before telling everyone else which would put the perpetrator on notice.
One person was designated to watch for excessive issuance, or any change in the 2 hr cycle of creating 500,000 RVN, automatically splitting it into pieces, and sending it to an exchange to presumably be sold.
We used a shared Google spreadsheet to mark off each exchange that we’d contacted, or sent a message. As we’re sending out messages through e-mail, Discord, Telegram, support websites, etc., we’re all linked together in Zoom commenting about how great it would be if only there was crypto-based messaging system that was tied to a token, speaking of course about the messaging system built into Ravencoin.
Once most of the exchanges were notified, it was time to let everyone know there was a problem with Ravencoin, and send out a message to everybody. Now, I know some argue that everything except the original Bitcoin is a shitcoin and centralized. But it isn’t true. There are several Ravencoin Discord groups, but none are owned, operated, or moderated by any of the developers. A few of the developers have elevated accounts there. We started by publishing on my Medium account where I write a lot about Ravencoin, and other crypto-related topics, and I tweeted it out on my personal @tronblack account. From there it was picked up by the Ravencoin community and the word spread pretty quickly. I was also able to publish to a blog, and then those were posted on various Telegram, and Discord channels that are run by the community. The ravencoin.org site has a mailing list, and we sent it out to those subscribers.
It was now a race to get the network secured and stabilized. We had picked a pretty good time to start. By accident, the 2016 block cycle was just starting. It took a while before we saw the first block mined by the new code. There is a different version so the BIP9 code can distinguish blocks mined by the vulnerable code, and blocks mined by the fixed code. We also had the fixed code reject bad transactions before they even get to the mempool. As it turns out, this helped us.
It’s a pretty helpless feeling knowing that this amazing network that you’ve championed and poured your energy into is vulnerable to attack and you can only stand by and watch as the race between the network becoming secure and the perpetrator destroying it unfolds. We know that the network can absorb a few day’s worth of excess issuance if it stays at 500,000 RVN every two hours, but we also know that it can’t withstand a massive inflation event. We don’t know what size issuance would be unsurvivable, but we’ll know it when we see it.
Every two hours, we see another issuance and it’s still 500,000. We watch it split into varying sizes and get sent to some address that we thought was at Binance. It’s Thursday night. We have tomorrow off officially for the 4th of July, but I know I have to stay vigilant until this is either solved, or we kick over to Plan B. There isn’t anything I can do, so I check on twitter and find overwhelming positive support from so many amazing people I’ve met over the years because of this project.
I decided to try and sleep and set an alarm to go off every two hours so I can check on the issuance, and I set it up so I’ll get slack messages from the team. Sleep doesn’t work, so I get up to monitor the network, and it looks like it’s been almost four hours since the last issuance. It’s 3:18 in the morning and I slack the team to let them know that 240 blocks (4 hrs worth) have gone by without an issuance and we’re expecting them about every two hours. Has it stopped? I keep waiting for the next transaction and it doesn’t come. Now I’m full of adrenaline thinking that the perpetrator knows the network is being fixed, and I’m waiting for a massive inflation event.
I install the new raven-qt on the laptop so I can monitor the progress of the BIP9 activation. By 8 am we’re at 385 of 613, which is about 62%, but it’s climbing. We need 1411 of 2016 blocks. It looks like it may be possible to do this in one cycle. That’s the best-case scenario, and by this time there have been no more issuances from the automated script, and no massive inflation events. This might work. I start to think that mentioning law enforcement may have scared off the perpetrator. Maybe he’s thinking that he got away with it, has stopped the script, and is on the run. Since then, we’ve learned that the transactions were created, submitted, and rejected by updated software and didn’t make it to an unpatched mining pool to get into a block. The script was still running and submitting, but there was a bouncer at the door in the updated software protecting the mempool that kicked the abusive transactions to the curb.
I rest for a bit, but sleep doesn’t come easy because my mind is still racing. I think about what I can possibly do. I get up and tweet how you can help Ravencoin by mining to the pools that have updated. Any hash power that switches away from an unpatched mining pool to a patched one, counts for double in moving the percentage up. I don’t want to play favorites with pools or convince anyone to move away from an updated pool, so I ask any pools that are updated to list themselves.
About twenty minutes later I find out that twitter posts are being made about it being related to asset issuance. I think about what we’ve published and realize there are a couple of threads that can be followed to figure out the exploit. Great, now I’m worried about more attackers. I wasn’t too worried about someone just digging through the code and finding it. If we didn’t catch it, it is unlikely someone else will stumble across it. But if they get the CryptoScope explorer to show the illegal issuance, it is pretty obvious where to look because it shows the extra output, and all of the transactions are re-issuances of assets. I mentally kick myself for letting the CryptoScope URL remain in the public message to everyone.
I see some twitter posts asking me where the code is that caused the exploit. I ignore them. I’m happy to show, and even explain, everything once the network is secure, but not before.
By noon on Friday, we’ve reached 66.9% and it is still moving up. This means we must have at least one of the unknown pools, or miners have shifted their rigs to known good pools. It’s looking very likely we’ll make it in one 2016 block cycle.
At 2:30 we’re almost at 70%, and I’m getting more confident we’ll make it this cycle. Now, when will the cycle end? I do the math, which is pretty easy. Just have to take the blocks that are left in the cycle, and since each one is about a minute, I calculate the network will be secure at about 5:10 am. I check for new issuances, and there are none, so if this holds, there is an extra 297,000,002.63128620 RVN. I had estimated getting it fixed, absent a massive issuance, or a change in the perpetrator’s automated script, at about 315 million, so this is encouraging.
I eat some lunch and we cancel a Zoom meeting because there’s nothing to discuss beyond what we’ve put in slack and nothing more we can do but wait.
I see a LinkedIn request from Nathan at CoinDesk. He’s asking about the reason for contacting law enforcement. The gist of the question he wants answered is whether the perpetrator is doing anything illegal. Fair question, but not one I’m ready to discuss at that time. At this point, it appears to me like the perpetrator is scared and has stopped the script. I fantasize about him (or her) cowering in a corner, afraid to touch the computer for fear of leaving more digital footprints, waiting for a knock on the door. As it appears now, he (or she) may have forgotten his script is running.
I look at twitter and once again I’m amazed by the supportive community and well wishes from so many. Then I see a link to a YouTube video by Tone Vays and Jimmy Song. I like both of these guys, but they’re well-known Bitcoin maximalists that throw logic to the wind when comparing Bitcoin to ’shitcoins’ (in their vernacular). I watch anyway. Yep, as I expected, explaining how this could never happen to Bitcoin, because yada, yada, yada. Vays and Song completely ignore that the same thing already happened to Bitcoin in August 2018, but the vulnerability was caught before it was exploited. However, Bitcoin was vulnerable for a long time. In fact, because the vulnerability was present in Bitcoin since at least October 2017, Ravencoin devs had to make the same fix in the Ravencoin project after the vulnerability was revealed publicly. I think it would be instructive, so I’ve written up a comparison of our handling to the Bitcoin core developer’s handling and included it below. There are many similarities and a few interesting differences.
I check the CryptoScope explorer again, and at first, it looks like the automated script has started back up, as there’s a new issuance. But it’s different. A million RVN instead of 500,000. I think at first that he’s amped up the issuance to make up for lost time, but that doesn’t make sense, as there isn’t really a limit to the number of transactions or the qty of RVN that can be minted. I look at the asset that was reissued. It’s different. The asset being reissued is from two years ago. The earliest asset I could trace to the other attacker was January 17, 2020. This is a new attacker. At 11:19 pm on Friday, I jump on Slack and let the team know. But there’s another difference, the RVN are still in the output address. They haven’t been transferred to an exchange. Maybe this is someone just experimenting. Also, I wishfully think that someone who was involved in RVN two years ago is probably friendly to the project. I think maybe they’ll “give back” the RVN.
2:04 am Saturday morning, I get a message on Slack. “And now another.” I check CryptoScope, and sure enough another million. That’s two million. Ok, less likely they’ll give it back. I’m tired, but now I’m thinking that there might be a blackmail attempt like ‘give me X or I create billions of RVN’. I check my private messages from Discord, Telegram, Twitter, several e-mails, and Signal. Nothing of note.
I wait for more excess issuances, and see that one of our very trusted community members is in Discord is already researching the new transaction. They’ve figured out the linkage between the 6969 asset and an asset named OVERSTOCK.COM that someone tried to sell to Overstock after it was issued years ago. I end up confusing it with the OVERSTOCK asset which we did acquire using the first RVN-for-asset atomic swap on the Ravencoin platform. But we do find the name and contact information for this new perpetrator.
It’s four in the morning, and I do the math again, and it looks like it is going to be closer to 7:15 am before the network is secure. I post the info in Slack, but without calling out names or @channel so it doesn’t wake anyone up.
At 5:20 in the morning, we reach 1411 blocks of 2016 so it is definitely going to activate the fix, but not until 2016 blocks have elapsed. Two more hours. Two potential attackers are active with the knowledge and technical chops to destroy RVN.
At 6:06 I find another transaction. My heart sinks. I look at the amount. 2.8 million more RVN. Survivable, but wait the re-issuance is linked to a different token — XODUS that was recently created. Oh no, XODUS (exodus?) sounds like a message. Is this someone sending a message, and is going to nuke the project with a massive inflation event. And, now there’s a third attacker.
Just over an hour to go. I’m refreshing the CryptoScope page watching for any new issuances and hoping they’re low enough to keep the project viable without requiring a reboot, which involves a whole new complex set of challenges. I’m thinking that if this wasn’t the year 2020, there might be a chance, but the way this year is going, I don’t like our odds.
I’m watching every block, passing the time by recalculating the percentage of good blocks, and refreshing the list of excess issuance transactions. 7:15 comes and goes. Why are blocks slower than one minute? Does a watched pot never add a new block? Ok, I’m mixing metaphors, but suffice it to say it was a very long hour.
7:26:27 am, the final block needed to complete the 2016 block cycle. I check CryptoScope and no more exploit transactions. The network is secure, but I watch it anyway. I want to make sure blocks are still being added and that the activated code, which was tested, but never on testnet, is operating as imagined.
A message from one of the developers comes in over Slack. “Ravencoin is now Locked in and no longer exploitable”. We give it thumbs-up.
I check the status of the BIP9 activation, and it says “locked-in”, but not “activated” as it normally would for an activated feature. I’m worried. What if it isn’t activated? What if it is vulnerable for another 2016 blocks? I don’t think I’d survive that. What color does gray hair go to under high stress if it’s already gray? I check the code, and the way it is written, the status gives locked-in, but when asking if it is activated, it turns locked-in or an activated into activated. I breathe a little easier.
It’s over. The network is secure.
I’m worried about saying it publicly right away. What if it isn’t? I haven’t slept and I’m not thinking clearly. I let the team know we’ll watch the network for an hour or so, and then give the all-clear. I ask if anyone knows of any Ravencoin forks, and I notify RVC that there’s vulnerability. Then realize it forked before January 15th. I go to sleep.
At 11:36 am, I tweet out that the code fix has been adopted and give some general information about the extent of the excess issuance. I see an outpouring of support, and I know then that Ravencoin is going to survive and be even stronger.
I start writing up Ravencoin — Emergency Ended, a book-end medium post to go with the Ravencoin — Emergency Update medium post from 48 very long hours ago. I tweet it out at 4:08 pm, sleep for two hours, and go join my family for what’s left of the 4th of July.
There were lots of risks. The biggest risk, as we perceived it, was that the perpetrator would just blow up Ravencoin by creating 21 billion RVN, and sending them to various exchanges and mixing them into the Ravencoin ecosystem. This was the worst-case scenario. This would require full disclosure of what happened and a re-boot of Ravencoin as Ravencoin-Phoenix where the blockchain restarts at the block before the massive exploit. In this scenario, the exchanges get hurt because the trading that happened between the time of the massive inflation event and stopping everything, when the exchanges are trading and allowing the withdrawal of DOGE, DASH, or BTC. When the massive inflation RVN transaction disappears, those RVN and their downstream transactions disappear too.
This was an existential crisis. We knew that it could be the end, and all that would be left would be to fix and restart for the sake of the assets. One of the things that makes blockchains so robust is that EVERYBODY has a copy of the ledger and every transaction from genesis to failure is available everywhere if anything catastrophic happens. On restart, the ledger doesn’t work if it has been tampered with because of the nature of chaining the blocks together with hashes. Blockchains are literally tamper-proof as any tampering can be detected from genesis to a checkpoint known and validated by everyone. Plan B was a restart, but we didn’t spend too much time on that other than to request that exchanges pause deposits to protect themselves, and we watched for a massive issuance event to initiate a full-stop.
Another risk was a chain split, which made us move the percentage up that triggers the updated consensus rules. We chose 70% which is much lower than the 90% and 85% we’ve used for other upgrades. The way BIP9 works is that it will activate once enough miners are running the upgraded code. This is great for upgrades because it allows a choice, instead of a forced upgrade, and if there isn’t high enough support, then the upgrade doesn’t go through. We, of course, usually recommend whatever upgrade, otherwise, we wouldn’t have spent time building and testing it, but nothing we can do can force it. It is mining pools voluntarily running the code and consensus rules. In this case, we did two things. We lowered the threshold to 70%. Lower than 70% and we risk a chain split and perhaps a mining fight with a well-funded adversary. Higher than 70% and we risk waiting for way too long for unknown mining pools to adopt the change or even mining pools that won’t accept the fix for whatever reason.
A valid criticism was brought up. Why was law enforcement contacted? Aren’t the rules just enforced by code? It is a valid question and I’ll give you my thoughts, and I’m sure reasonable minds, even ones that I work with may disagree. The exploiter of the vulnerability probably didn’t break any laws. It seems to me to be the equivalent of walking into a fast-food restaurant and walking out with all the napkins, all the ketchup, all the condiments, all the straws, all the cups and lids, and every plastic fork and spoon. They are free and there is no one-per-customer law. It violates all social norms, seems immoral, and clearly goes beyond the intent of the restaurant in providing these items. So what would happen if someone did that every day? What would you do?
What if someone put tape over the door latch and then did the same thing, but they took everything from behind the counter too? Ignoring the trespassing laws, they created the conditions and went beyond the intent of the restaurant’s free items.
There probably aren’t any federal laws in place for this type of exploit, and I’m not suggesting that there should be. Resources have to be allocated and there are other fish to fry like those that sell spaghetti with meatballs and sauce without prominently declaring the presence of the sauce.
Let’s look at why ‘law enforcement’ was left in the message to everyone about the vulnerability. And why I think it should’ve been in the message even if a law enforcement agency had told us to pound sand. By the way, the particular law enforcement agency we contacted was very professional, has crypto and cyber chops, is interested, and showed interest in this open-source project. But ignoring that for the moment, having some pressure on the perpetrator works from a game theory point-of-view. Without the threat of being discovered, and caught, there may have been a lot more damage from excess issuance. Remember that we were working from near-zero knowledge about the adversary.
What wasn’t in the public message?
We chose to leave details about the exploit out. We did not reveal it was related to re-issuance. We did not point to the area in the public repo that allowed the vulnerability. We did not reveal the law enforcement agency we were talking with — at their request. We also chose to leave out the exact number of RVN, but gave a ballpark figure, and as it turned out, a slight over-estimate. We did not know if, or when the script would stop creating excess RVN, because there was no way to know when the attack would stop or be stopped.
After the fix
After the network was stabilized, I wrote up a quick medium post entitled Ravencoin — Emergency Ended which let the exchanges know to resume trading.
We’ve made public the temporary private repo.
We’re in the process of moving the commits to the public repo and building binaries where the code matches the binaries.
I also started jotting down thoughts about the events of the last week. These thoughts are included in this update about what happened.
Why ask the exchanges to pause?
One of the big risks was a massive, project ending, inflation event. During this vulnerability, many blockchain savvy minds could’ve figured out the vulnerability and maliciously created all the RVN. As I intimated in the public revelation, this would require a full-stop, and reboot of RVN at block X-1 where X is the block containing the massive inflation event. By leaving the exchanges open, those RVN could be sold, and traded, and the reboot would hurt either the exchange or the last holder of the orphaned RVN. By asking the exchanges to at least halt deposits, it reduced that potential impact.
We discussed asking the exchanges to pause trading and withdrawals as well, but it seemed wrong to lock the market. That feels too much like the NMS market wide circuit-breakers, and I think those are a mistake. To not allow one to trade on the volatile events of the moment is to take away a lot of what trading is about and that is price discovery. A live 24x7 market lets everyone discover the point at which supply and demand cross at the moment and under these, potentially adverse, conditions.
What might we have done differently
I’m not talking about what we would’ve done differently knowing the end outcome. That’s 20/20 hindsight or Monday morning quarter-backing. I’m talking about if we’d spent even more time analyzing the situation, what might we have done better.
One thing is publishing fewer breadcrumbs leading to the root cause of the vulnerability. While we moved the fix to a private repo (temporarily), and we didn’t discuss the cause of the vulnerability, we did put out the link to the explorer that could display it and mentioned that the vulnerability creating code was submitted by a community member. These are both bread crumbs that might have helped point the way for two other attackers to issue more RVN.
We might have used something other than BIP9 to activate which would’ve allowed activation as soon as 70% was reached instead of waiting for the completion of a 2016 block cycle. The advantage of using BIP9 is that it has been used before and so the code is known and tested. Also, any block percentage test requires counting some number of blocks.
Is Ravencoin decentralized? The answer to this is a definite YES. This factored into how we handled fixing this vulnerability. If we could have just let the miners know, as Bitcoin core devs did in September 2018, this could’ve been completely solved before letting the world know.
The fact that we had no way of communicating with two good-sized unknown pools, or with several smaller unknown pools left us with a near-zero chance of getting to a safe amount of mining hash running the good code. This lack of ability to contact these unknown pools, while an ongoing exploit was continuing, eliminated one of the most efficient methods of updating by quietly contacting the mining pools.
The nature of this exploited vulnerability is inflation. The RVN and Ravencoin assets could not be stolen, or transferred, but new RVN could be minted outside of the expected 5000 per block coinbase transaction.
Through the exploit, an additional 301,804,400.51605642 were minted beyond the regular RVN coinbase. Since then, the last exploiter burned their 2803988.94326435, and the second exploiter burned exactly 1098000 RVN.
This leaves a total of 297,902,411.57279207 RVN as extra issuance.
So how does this get handled? Here are three options that we, or the Ravencoin community, have come up with.
- Leave it be. 21,297,902,411.57279207 will eventually exist instead of approaching 21 billion.
- Pull forward the halving about 41.375 days (59580 blocks). Those days of mining would be sliced out, so all future halvings are 59580 blocks earlier. This would bring supply into alignment.
- Pull forward mining by 82.75 days (119160 blocks), so the first halving starts earlier but then goes back on the same schedule as before. So those 119,160 blocks would be mined at 2500 RVN per block instead of 5000 RVN per block to compensate for the extra RVN, bringing supply into alignment.
- Set mining the coinbase to 0 when the supply reaches 21,000,000,000. Most mined coins arrive at 0 coinbase, but long into the future. This has coinbase mining rewards stopping at around year 26, which seems way too soon. I’ve included a spreadsheet that shows how this isn’t a great option.https://docs.google.com/spreadsheets/d/1kArrxGh4KmaRR_CKL84xCeZgZfrP6TSjOIrt-Uy6FQo/edit?usp=sharing
The supply of a crypto-currency is sacrosanct. This isn’t the @#$%! Federal Reserve.
My preference would be to get supply back to the expected supply. My preference is option 2, but I’m open to other suggestions. If more RVN from the remaining two perpetrators that haven’t returned it yet, is burned, we can adjust these numbers accordingly.
These discussions can begin immediately. It is up to the community to decide, and we can code a solution for BIP9 adoption. This does affect the miners as it is related to the mining reward, so it seems very appropriate to enable it via counting mined blocks that adopt the change. Option 1 is the default if nothing is changed.
Following the Money (RVN)
Where is the excess RVN now? Bitcoin and Ravencoin work similarly in that every transaction is recorded forever on a permanent ledger. This allows everybody to see the addresses where these RVN went.
Starting with the excess supply, we can follow these RVN. Viewing the CryptoScope explorer Illegal Supply, anyone can see all the RVN that shouldn’t exist and every subsequent transfer.
I’d encourage anyone to follow these tokens. We’ve followed the RVN and found that they get split and end up in these addresses:
rJtgQB79VdYNCtXEg3L31QATPYU4xQFGxG (OKEx) — 7,117,984 (minus 21,187.52 before attack) = 7,096,796 RVN
RAekzFLJDfLpaTfMonPNEvahWVYvBu2iE8 (Binance) — 116,313,813 (minus 372,044.67 before attack) = 115,941,768.33 RVN
Total deposited in these addresses: 361,100,757 RVN
Total deposited during attack date range: 312,778,999.33 RVN (This is more than the amount from exploit ended at 297,902,411.57279207 RVN).
I asked @unclear how the CryptoScope RVN explorer could be certain that the addresses tagged as belonging to Binance accounts were actually Binance accounts. Here’s an excellent post showing how these associations were made: https://medium.com/@cryproscope/solus-explorer-what-is-the-solus-address-grouping-function-f8dda05c427e
Three of these four crypto-addresses appear to be owned and assigned by Binance to individual accounts. The green address in the post from CryptoScope is the address of a test where 3 RVN was deposited into a newly created Binance account to see where it ended up. You can think about this like sending a specially marked rubber duck down a river (Binance assigned deposit address) and finding it in a lake to be sure that the river feeds into the lake (Binance hot-wallet where we see the other RVN end up).
Thank you to @unclear from CryptoScope for the time and energy spent researching this.
It will require the participation of these exchanges to investigate further. The exchanges have done nothing wrong, but it would be wonderful if they could help for the sake of crypto.
As you would expect with excess issuance being sold into the crypto markets, it impacted the price. It appears as though the excess RVN was sent to an exchange. We assume it was traded for something else, perhaps DOGE or BTC.
As you’d expect, the price of RVN is starting to go back up (2020–07–07). Considering the nature of the vulnerability, and the risk, the price remained surprisingly stable throughout the emergency. I attribute this to the Ravencoin community recognizing the value of the project and being able to look past the temporary supply disruption.
I’ve been assured that none of the core developers sold during the emergency.
Comparing the Bitcoin and Ravencoin approach
Trigger warning: If you are a Bitcoin maximalist, please skip this section for your own sanity. I’m lookin’ at you Tone. 😉
I’ll compare the approach we took with two different Bitcoin vulnerabilities that are of the same type as the one that challenged us.
2010–184+ billion BTC inflation — soft fork solution. This was exploited, but it was early enough in Bitcoin’s growth cycle that it could be easily remedied. The limited miners were contacted and the fix applied at the miner or mining pool level. I only bring this one up because it was of a size that the issuance would destroy the Bitcoin economy if left unfixed. This was done as a quiet fix.
2018 (August) — Bitcoin inflation vulnerability — sneak fix to miners as a soft fork. This sneak fix was done by masking the nature of the real fix and only discussing a DDOS fix. We were not able to take this approach because of the decentralization of mining there was about 44% of the mining hash power being mined by unknown pools combined with the knowledge that the threat actor would see this update. It appears the Bitcoin vulnerability was present from Nov 1, 2016, to Sept 17, 2018. Thankfully, because I really like Bitcoin and hold some, a friendly developer caught the Bitcoin vulnerability before it was exploited. So how was this fixed without most people knowing about it? This Bitcoin vulnerability was fixed on the down-low by keeping the real fix secret, similar to our approach, and quietly contacting enough mining pools to run it under the guise of the fix being primarily for DDOS vulnerability. The Ravencoin team took the approach of not publishing the fix to the public GitHub, which would reveal the details of the vulnerability. In order to get a majority of the mining hash power, it necessitated a public statement and exposure to notify the unknown mining pool hash power of the critical update. As with Bitcoin, it is up to each individual mining pool whether to update, up to each individual miner which pool they use, and up to every individual which software to run.
After the BTC vulnerability fix was quietly deployed, the information was made public. Until deployed to enough mining nodes, it was kept secret. Some altcoin developers, including Ravencoin’s developers, quickly fixed the same, now public, vulnerability in their own coin’s source code. About 14 other alt-coins did not do it quickly enough, and were inflated into oblivion. Some of those coins re-launched on a pre-exploit block and survive today, and some were so damaged that they never recovered. Was this the responsible approach by the team that fixed BTC? I have mixed feelings on this as I would’ve liked a heads-up after BTC was fixed, but before the vulnerability information was published widely. This opens up the question of what responsibility or courtesy should developers in the crypto space show to other projects. A Bitcoin maximalist maintains a very narrow view of the crypto space and the worth of other projects. They don’t always see how other types of value besides bitcoin can also be issued, tracked, traded, and transferred using the same technology.
There’s a video out by Tone Vays and Jimmy Song, and knowing what you know now, you can judge it for yourself. https://www.youtube.com/watch?v=VdmnXQB33_c
We tried to figure out how to mitigate the impact of the worst-case scenario while making code changes and disclosures for a quick resolution. It turned out well in the end, but there was no guarantee of a successful outcome. I’m relieved that it worked out as well as it did. I’m also in awe of the overwhelmingly positive response from the Ravencoin community. Just because of the way the crypto-space operates and the benefits of robust debate, I’m expecting criticism of our approach. It wasn’t perfect, but it ended well.
Should there be more eyes on the code?
Yes. This is an open-source project. Except for two times, both for emergency fixes, the code is public and before it goes to master, and before it gets built into downloadable binaries. Anyone can review the code, but few do. This is true of most open-source projects. There is talk of the community funding a third-party audit and I would support that.
What is being done to keep this from happening again?
Extra scrutiny for consensus code is the obvious low-hanging fruit. This vulnerability was inserted, but there are other ones that could exist or could be introduced accidentally through coding mistakes. I think coding mistakes are more common, and so putting more eyes on routine commits is another change. We are also putting in a plan to better vet unknown contributors.
*Correction: This post formerly associated a token that may not have been involved in the exploit.