|Security Incident Recap|
In this article, I will share a recap of what occurred in the past two weeks, including lessons learned, stress dealt with, and wisdom gained. - CZ
A group of hackers was able to gain control of a number of user accounts and made some large withdrawal requests in such a way that they bypassed our pre-withdrawal risk management checks. Our post-withdrawal risk monitoring system caught the actions immediately and suspended all subsequent withdrawals. While things are crystal clear in hindsight, at that moment, we weren’t 100% sure as to what exactly happened. Was it an actual user action, a glitch in the system, or maybe a hack? As we were still evaluating the situation at the time, we decided to proceed with caution. I put out a tweet saying the withdrawal servers are in unscheduled maintenance mode, while the team continued to investigate what happened. After confirming it was a hack, more questions followed:
- How much did the hackers withdraw?
- Were there previous withdrawals that we didn’t notice?
- How many other accounts did the hackers have?
- What other risks are involved?
- How did the hackers know our risk management rules so precisely? Do we have a mole?
- What do we need to do to get the withdrawal system online again?
While the team was investigating the above, there were further questions that needed to be answered:
- How should we communicate?
- What would the community reaction be?
- How much reputational damage would we suffer?
In tough moments, we always choose to rely on our ethos: Users First and Be Transparent.
After the initial incident, we decided to put out a notification across all our channels regarding the security incident. By then, we were relatively certain that there was only a single affected transaction. All our other wallets were safe. We were cautious that the hackers may still have control to additional accounts that we were not fully aware of. Further withdrawals still posed a risk, and we needed to make a few significant changes to the system before we could re-enable withdrawals for our users. This security incident notification stated an estimation of one week of suspension on withdrawals.
In our technology world, you can never accurately estimate how long changes might take. It is quite different when you compare it to repeated predictable work. But our users and community needed an estimate, and once communicated, it became a target deadline for our team to deliver. I did not know how the community would react to a one week of withdrawal suspension, but luckily, being transparent paid off and we received tremendous support from our amazing community.
Lesson: During a crisis, constant transparent communication is key.
We had already previously scheduled a video AMA just a couple hours later. I thought it would be appropriate to keep it, as lots of people would have questions, which turned out to be the right thing to do.
Seeing me live did put a lot of our community at ease. The livestream was analyzed to death, including body language analysis which I thought was a very good thing. It truly shows how the crowd will work as a hive mind on different aspects of the analysis. The body language analysis results were very positive, which is reassuring.
Lesson: Get on a live video stream during a crisis. Your users deserve to know, not just what happened, but how you are doing and handling it, including allowing them to judge your mental state for themselves.
Before the AMA, I had been up all night and I was really feeling the effects. So, I took a 15-minute nap just before the AMA. Upon waking up, my team told me there was an interesting proposal from a Bitcoin Core developer. I read it for a few seconds. It involved something called a “reorg”. While I know it’s technically possible for a rollback in a 51% attack scenario, it never occurred to me that it is also technically possible to change one transaction and keep all other transactions intact, while hugely incentivizing the miners. The discussion was already pretty hot on Twitter, so I mentioned it in the AMA as something that was suggested. Little did I know, it was a taboo topic. Lesson learned.
I am not gonna deny it. My first reaction was: “F***!”, the second and third reactions were also the same. A few moments after that, I began to come to terms with it, “Well that sucks! What should we do now? Lots of people are waiting for me, some for instructions, some for information and some for reassurance. Lots to do, let’s just get on with it.”
Checking with the team, they were already a couple of steps ahead of me, implementing additional security measures to further ring-fence our systems and discussing all available options. The entire team was online. I have seen this mode before, it’s called “War Mode”. Luckily, our team is accustomed to high pressured situations, and our urge to fight was stronger than ever. A few of them even gave me a pat on the back for planning to do the livestream AMA. A few variations of “Balls of Steel, Boss” came up a few times. They were cheering me on, I knew that was a good sign.
After 10 seconds of the “F***, F***, F***” state, I did a quick mental calculation. 7000 BTC, fine, I know we have more than that in our own BTC funds alone. There is enough. Mentally, a second calculation eased my thinking. The amount was about the same as a quarterly burn we did about a year ago.
Also, this is not a single outlay of cash percentage-wise by Binance. Back in September 2017, when the Chinese government issued a letter banning ICOs and “recommending” projects to return money to investors. The news alone caused many tokens to drop below their ICO prices, and many project teams couldn't return the whole amount to users. While BNB stayed strong at about 6x of the ICO price, Binance did help a number of projects raise money on our platform, and they were affected by this policy. So we did a quick calculation: if we were to help cover the loss for our users and for those projects, it would cost us roughly US$6,000,000. Putting this in perspective, while we only raised US$15,000,000 two months prior, we spent a bunch of money and were barely cash flow neutral at the time. We decided to do it anyway. I was in a moving subway when the team called me, and we made that decision together in less than 5 minutes. That was more than 35% of all the cash we had at that time. The goodwill that decision generated eventually brought us many users from China and all over the world and helped to fuel our growth. So, this time, this $40m represented a much smaller % of our cash reserves, plus we had the #SAFU fund that could fully cover it.
We thus announced that we would cover the entire loss in full.
Lesson: Money can always be earned later, do the right thing first.
We got tremendous community support, from people defending us, to people helping us answering questions in the community, on Twitter, Telegram, and Facebook. The Binance Angels (our volunteers) have been running at full steam on multiple communities, addressing questions and reassuring our users around the clock. Thank you, Thank you, We thank you!
Many partners jumped in to help. Analytics teams started to help us track the stolen funds, e.g. Peck Shield, Whale Alert, etc. Exchanges and wallet services offered to block any deposits associated with the hacker addresses. Some of them may be perceived as our “competitors” by some people, but I am impressed at how the entire community came together and stood united at a time of need.
We also received numerous offers for help from law enforcement agencies around the world. This is a result of working with them closely in the past, usually helping them to solve cases. Now, they offered their help to us in return.
Lesson: Being transparent makes it easier for others to help you.
I got 40+ new leads from various security experts/consultants/companies offering to help. While some clearly intended to help, many were simply trying to sell their services. While all help is fully appreciated, the timing was actually a little off. It would not be good for me to schedule 40 calls during the week when our system is partially down. Some even flat out suggested that we give them full access to our servers so that they can help us do forensics, we politely decline. Moving on...
One Quarter in a Week
Our team pushed on, day and night. In places where we congregate in small temp “offices”, we had Ikea temp beds rolled out. I won’t go into the details here, as we don’t disclose our security practices. But to bring the system back online in one week, all of our teams did more than a quarter’s worth of work in that one week.
A Blessing in Disguise
Speaking with various team members, and as correctly analyzed by community members, such as Gautam Chhugani, this incident may actually be a good thing for us in the long run. Security is a never-ending practice. There are always more things to do in security, and we have implemented many of them in this last week and will continue to implement more in the future. Given this incident, Binance has actually become far more secure than before, not just in the affected areas, but as a whole.
We always maintain constant and transparent communication with our community during the crisis. We believe this to be a strong factor that contributed to the support we got from the community in return. One clear measure is the BNB price: it dropped a bit on the initial news, but not nearly as much as one would have expected, and even before we resumed withdrawals, it has already made a strong come back and hit all-time highs (in USD) again.