Security Investigation Retrospective

On January 5, 2016, we issued a password reset for all Linode customers during our investigation into the unauthorized access of three customer accounts. We have been working with federal authorities on these matters and their criminal investigations are ongoing. Today we are sharing our findings and those of the third-party security firm we retained to assist us with this investigation.

Before diving in we’d like to reassure you that your account information is safe. We found only three customers affected by this incident and have resolved these issues with them directly.

What Happened

This is a complex retrospective of two separate investigations, one in July and another in December. While the cases share similarities, we have no evidence to support the two being related. Nevertheless, here is a full timeline of the events as we have come to understand them.

On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notification confirming a root password reset for one of their Linodes. Our initial investigation showed the unauthorized login was successful on the first attempt and resembled normal activity.

On July 12, in anticipation of law enforcement’s involvement, the customer followed up with a preservation request for a Linode corresponding to an IP address believed to be involved in the unauthorized access. We honored the request and asked the customer to provide us with any additional evidence (e.g., log files) that would support the Linode being the source of malicious activity. Neither the customer nor law enforcement followed up and, because we do not examine customer data without probable cause, we did not analyze the preserved image.

On the same day, the customer reported that the user whose account was accessed had lost a mobile device several weeks earlier containing the 2FA credentials required to access the account, and explained that the owner attempted to remotely wipe the device some time later. In addition, this user employed a weak password. In light of this information, and with no evidence to support that the credentials were obtained from Linode, we did not investigate further.

On December 9 an independent security researcher contacted us. The researcher claimed to be tracking an individual who had stolen credentials from numerous other service providers. The researcher wanted to make us aware that the individual may have made attempts to use these stolen credentials to log in to some of our customers’ accounts.

Our initial investigation concluded that the IPs provided had, in fact, been used to log into three accounts on the first attempt. In other words, the user arrived to the Linode Manager login page with the credentials necessary to log in, just as any regular user would. That same day we contacted the customers and received confirmation from each that the activities were suspicious. We also confirmed that none of these accounts had multi-factor authentication enabled and all had employed weak passwords.

On December 13 we started necessary fleet-wide Xen Security Advisory (XSA) maintenance, rebooting servers in their local nighttime hours around the clock. Although unrelated to the investigation, this continued through December 18 and was a significant resource constraint.

On December 14, although we had discovered no evidence of an intrusion into our infrastructure, we began interviewing third-party security firms and contacted multiple law enforcement agencies. We also dedicated all available internal resources to the effort and began scrutinizing our environment to identify any evidence of abuse or misuse.

On December 17, because of the similarities between this case and the one from July, we reopened the July case and concluded we now had sufficient reason to examine the image retained from that investigation.

Linode uses TOTP to provide two-factor authentication. This is an algorithm that uses a secret key stored on, and shared between, our server and the customer’s two-factor authentication app (such as Google Authenticator). The algorithm generates a time-sensitive code that the user provides during login as an additional authentication component. We encrypt these secret keys when storing them in our database.

After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code. Though the credentials found were unrelated to any of the unauthorized Linode Manager logins made in December, the discovery of this information significantly changed the seriousness of our investigation.

On December 21 our third-party security partner joined the investigation. This team proceeded with a forensic analysis to identify any unauthorized system-level activity that may have permitted an intruder to access customer credentials contained in our database. The team also searched for evidence of web application misuse that would have provided lateral movement from one Linode Manager account to another. Additionally, the team initiated a targeted vulnerability assessment with the objective of identifying a possible attack vector for obtaining access to the Linode database.

On December 25 DDoS attacks against our infrastructure began. Although we do not have any evidence to support these attacks being related to the incidents of unauthorized access, the attacks necessitated resources be pulled away from our investigation. This, combined with employees being away for the holidays, created additional challenges for our support and operations teams.

On January 5 our security partner concluded its investigation and we issued the password reset. Our internal security team continued to review our infrastructure for the next several weeks and developed a detailed plan for improving our overall security.

Findings

The findings of our security partner’s investigation concluded there was no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. Furthermore, the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access.

Linode’s security team did discover a vulnerability in Lish’s SSH gateway that potentially could have been used to obtain information discovered on December 17, although we have no evidence to support this supposition. We immediately fixed the vulnerability.

Other theories considered that could explain the unauthorized access include an external compromise, such as the weak passwords previously mentioned being used with other online services, or phishing attacks against those users.

What We’re Doing About it

We are using what we’ve learned to make comprehensive improvements to our infrastructure, including areas unrelated to the incident. Here are a few of the things we’ve been working toward:

Authentication microservice: Several of our applications (such as the Linode Manager and Lish) perform user authentication. Previously, these applications performed this function by having access to the credential information directly within our database, and then performing comparisons themselves. We’ve developed a new approach that involves a carefully secured and monitored microservice that maintains ownership of all customer credentials. Under this method, when an application requires user authentication, the microservice is able to validate the credentials by returning a simple “yes” or “no.” Applications won’t have access to the credential information. In fact, the databases that power our infrastructure won’t contain credential information at all when the rollout of this microservice is complete. Also, customer passwords, previously stored as salted SHA-256 hashes with thousands of rounds, will be stored using bcrypt and will be upgraded seamlessly on a subsequent login.

Linode Manager notifications: We will be working to enhance notifications our customers receive about their respective account activity, including alerts to login attempts from new IP addresses and failed logins.

CC Tokenization: Although our investigation yielded no evidence of credit card information being accessed, we are taking advantage of our payment processor’s tokenization feature to remove the risk associated with storing credit card information.

Policy: We’ve been developing multiple policies derived from the NIST framework on topics ranging from clean-desk to password standards. A significant new policy is the creation of “security zones” for sensitive elements of infrastructure, like our database and authentication servers. The results of those efforts have greatly reduced the number of employees that have access to sensitive systems and data.

Hiring: In addition to the changes outlined above, we are hiring a senior-level security expert to join our company and lead a larger team of engineers focused full-time on security. This team will not only ensure we are following current best practices but will also expand our written policies, formalize our provisioning procedures and fundamentally ensure our policies are supported by process and accountability.

A New Linode API: Our most important long-term strategy is a rewrite of our legacy ColdFusion codebase that gives us the opportunity for a fresh start and to apply the lessons we have learned over the past 13 years. To do this, we’ve been building a new Linode API that’s stateless, RESTful, and implemented in Python. We’ve been working on this for the past many months and will announce a public alpha of the new API in the next few weeks.

Open-source Linode Manager: This new API will be the foundation for all things coming in the future, including an open-source Linode Manager which will replace the current manager.

Looking Ahead

We recognize we have room for improvement when it comes to communication and transparency. XSAs and persistent DDoS attacks throughout December notwithstanding, we should have communicated the nature and extent of the DDoS attacks and this security incident to our customers sooner. To say we were resource constrained at the time would be a fair assessment. Still, we could have done better and have since made procedural changes to ensure a team member is appointed to important events like these in the future to facilitate frequent and transparent communication with our customers.

We are incredibly grateful for the customers who have supported us throughout these events. We’ve heard your recommendations and felt the support you’ve provided over the past few months. Know that we continue listening and acting on your feedback.

We’ll conclude by saying how sorry we are if we’ve let you down. We value the trust you’ve placed in us as your hosting provider and are committed to earning that trust every day. We hope the details provided here clear up some misinformation and demonstrate our willingness to address opportunities for improvement, do the right thing, and increase communication and transparency with you, our customers.