SHAPESHIFT CYBERATTACK POSTMORTEM PREPARED FOR: SHAPESHIFT AG
Ledger Labs Inc. Michael Perklin, CBP, CISSP, CISA, EnCE, ACE
[email protected]
ShapeShift Cyber-Attack Postmortem On 2016-04-09 at 13:00 EDT, ShapeShift CEO Erik Voorhees contacted Ledger Labs Inc. (LLI) with a request for digital investigative assistance. Erik stated that following the security incident that ShapeShift experienced on 2016-04-07, ShapeShift was hacked a second time earlier that morning. In addition, ShapeShift experienced a third breach on 2016-03-14 by an internal employee. Following a brief call with some members of the ShapeShift team, the decision was made to send Michael Perklin of LLI to ShapeShift’s headquarters. Michael boarded a flight and immediately attended ShapeShift’s headquarters to begin his investigation. Michael conducted interviews of Erik Voorhees and other ShapeShift staff to gather preliminary statements and build a preliminary timeline of events.
Established Facts The following facts were established following the preliminary investigation: •
•
•
•
•
•
Following the 2016-03-14 breach (which will be referred to as Breach1 of Infrastructure1), entirely new SSH keys and passwords were created by all team members in order to securely build the new infrastructure; Following the 2016-04-07 breach (Breach2 of Infrastructure2), some team members changed computers to protect against malware that may have existed on those machines. Entirely new SSH keys and passwords were created by all team members; The Breach2 attacker used the alias Rovion Vavilov, and was the same attacker that breached ShapeShift’s servers during the 2016-04-09 attack (Breach3 of Infrastructure3). Breaches 2 and 3 sent coins to the same addresses, and an email was sent by the Breach2 attacker to Erik Voorhees with a single “:)” minutes after Breach3 was completed; Rovion was able to learn the IP addresses of both Infrastructure2 and Infrastructure3 Infrastructure3 despite these IPs being known only by ShapeShift staff. Infrastructure3’s Infrastructure3’s IP address was identified within hours of being established by the ShapeShift team; Rovion was able to access Infrastructure3 through firewall rules that restricted access to the ShapeShift HQ static IP address; ShapeShift received an independent report that showed internal ShapeShift communications had been previously compromised;
Digital Forensic Investigation Following the preliminary investigation, a digital-forensic investigation was launched to analyze both Infrastructure2 and Infrastructure3 for artifacts that would provide insight into the breaches. Infrastructure1 Analysis The first server analyzed was the machine that ran the core ShapeShift exchange code. For the purposes of this report, the server will be referred to as Simpson. A bitstream disk image was captured using a combination of the dd, netcat and bzip2 commands. The sha1 hash of the disk image was calculated as ac0a4d287a1227d7c805f7d9b83141dadfdf682f and was verified following the transfer. Analysis of the Ubuntu operating system configuration identified that there was no logging or auditing configured beyond the default configuration that ships with Ubuntu. This limited the amount of artifacts recorded by the operating system, and limited the amount of evidentiary data available. An analysis of /var/log/auth.log revealed that the logs were wiped at some time prior to 2016-04-10 at 23:05 UTC. A forensic recovery of the previous contents of auth.log was attempted and the deleted log file was successfully recovered from unallocated space. Analysis of the recovered log data revealed artifacts of select actions executed by Rovion. Rovion executed the following commands on Simpson on 2016-04-09 between 10:18:54 and 10:23:08 UTC: • • • • • • • • • •
/bin/mv slave /bin/udevd-bridge /bin/mv init /etc/init/udevd /etc/init/udevd-bridge.conf -bridge.conf /bin/chown root.root /bin/udevd-bridg /bin/udevd-bridge e /bin/chmod 755 /bin/udevd-bridg /bin/udevd-bridge e /bin/chown root.root /etc/init/udevd/etc/init/udevd-bridge.conf bridge.conf /bin/chmod 644 /etc/init/udevd/etc/init/udevd-bridge.conf bridge.conf /usr/bin/touch -d Apr 14 2014 /etc/init/udevd-bridge.conf /etc/init/udevd-bridge.conf /usr/bin/touch -d Jan 26 08:38 /bin/udevd-bridge /sbin/status udevd-bridge /sbin/start udevd-bridge
The above commands show the installation of a rootkit at /bin/udevd-bridge. A sha1 hash was calculated for this file which yielded the value d94f219c7b71745c11e84d652819ff1f0c4b6582. The contents of the /etc/init/udevd-bridge.conf file ensured the rootkit would launch every time the system was booted. Rovion named the rootkit to appear like a system file on the Ubuntu operating system, and further hid the rootkit by modifying the timestamps to match similar system files. Rovion then launched the rootkit with the /sbin/start udevd-bridge command.
Prior to these commands, the auth.log showed the last login to the server was completed via public key authorization on 2016-04-09 at 8:44:37 UTC by a key with 9d:35:fd:6d:60:a8:6e:2e:56:f3:d0:a :6e:2e:56:f3:d0:ac:07:79:7f:cb c:07:79:7f:cb. This is RSA fingerprint 9d:35:fd:6d:60:a8 consistent with a known employee login to Simpson via VPN connection from the employee’s home. This SSH connection remained open until 13:04 UTC, suggesting the employee’s session was used to breach the Simpson server. Although significantly more analysis was performed, the operating system’s configuration prevented additional artifacts from being generated following user and system actions, resulting in insufficient evidentiary data. Infrastructure2 Analysis Following the analysis of Simpson, a bitstream image of the server that ran ShapeShift’s core exchange code from Infrastructure2 was obtained using a combination of dd, gzip2, and netcat. For the purpose of this report, this machine will be identified as Lenny. The sha1 hash of the disk image was calculated as 41b5bc88fd7ab4ef0844069df51fc281caa59338. Analysis of Lenny’s Ubuntu operating system’s configuration revealed that – similar to Simpson – there was no logging or auditing configured beyond the default configuration that ships with Ubuntu. Analysis of the /var/log/auth.log file showed tampering via overwriting unlike Simpson which had its log deleted. The last few lines of the log were overwritten with NULL (0x00) bytes, preventing digitalforensic recovery. Analysis of the /bin folder identified the installation of the same rootkit identified on the Simpson server at the same path: /bin/udevd-bridge. Although significantly more analysis was performed, no data artifacts were identified that could help identify how the back-door was placed on Lenny, or who performed it.
Analysis Since direct evidence of a specific attack vector was not found during the digitalforensic investigation, an analysis of the available facts was performed to identify all possible attack vectors that fit the facts. It was noted that the attacker was not only able to compromise both infrastructures fairly quickly, but they were able to identify their IP addresses equally as fast. The following attack vectors were possible avenues of attack sorted in order of probability: 1. An(other) employee with access to both Simpson and Lenny performed or assisted with both attacks; 2. A Remote Access Trojan (RAT) was installed on a laptop belonging to an employee with access to both Simpson and Lenny. The compromised laptop
allowed Rovion to obtain the location of both new infrastructures and obtain an SSH key for access; 3. A vulnerability exists in the ShapeShift source code that launched a reverse shell to a machine under Rovion’s control. This source code ran on both Simpson and Lenny, and upon reaching out to Rovion’s machine, told him their IP addresses; 4. A vulnerability exists within one of the services running on the Ubuntu operating system that was exploited to open a reverse shell to a machine under Rovion’s control. This service ran on both Simpson and Lenny. Rovion obtained the IP addresses of both machines via a communications channel breach (i.e. email, Slack, etc.);
Chat with Rovion Towards the end of the investigation while remediation was underway, Erik Voorhees received communications from Rovion with a request to exchange his stolen ether for bitcoin. Erik made the decision to conduct this trade in exchange for information with the hope of identifying the entry point of compromise. Following this interaction, Rovion provided information that he alleged was true. Although there is no way to identify whether the information was actually true, the information provided corroborated a number of facts uncovered during the investigation that would only be known by Rovion, and provided a number of other facts that were consistent with the conditions observed during the attacks. These included: • •
Rovion installed a rootkit at /bin/udevd-bridge Rovion purchased information from a previous employee which allowed him to conduct the attacks on ShapeShift. This information included: o The public/external IP address of ShapeShift HQ’s office o The username/password of ShapeShift HQ’s office router’s admin interface, and the port upon which the admin interface was listening. This would allow Rovion to remap ports to internal machines at will. o The internal IP address of a ShapeShift employee’s machine within the ShapeShift HQ network Details of an AquaConnect installation on a ShapeShift employee’s o laptop that was installed by the prior employee prior to his departure on March 14th, along with the username/password/port with which to connect o An SSH key belonging to a ShapeShift employee that granted access to Simpson
CCSS Assessment Following the digital forensic investigation, LLI performed an assessment of the ShapeShift infrastructure against the CryptoCurrency Security Standard (CCSS). (CCSS). The assessment identified a number of aspects in which ShapeShift’s controls were sufficient for obtaining Level 3, however others were identified that left ShapeShift’s infrastructure as uncertified. Since CCSS requires unanimous compliance of all aspects in order to achieve a security level, ShapeShift’s resultant level was 0 – Uncertified . LLI drafted a list of security controls that must be implemented in order for ShapeShift’s infrastructure to be graded as CCSS Level 1. LLI staff worked with ShapeShift staff to implement these controls.
Corrective Action Although evidence of the specific attack vector could not be found due to both the destruction of evidence by Rovion and the lack of logging configured on the servers, a number of corrective actions would prevent each of the above attack vectors from being exploited, thereby dramatically increasing ShapeShift’s security. This section outlines the corrective actions taken by the ShapeShift team under LLI’s guidance: Computing Hardware Replacement All ShapeShift employees who had access to both Infrastructure1 and Infrastructure2 received brand new computing hardware to ensure any RATs, backdoors, or malware installed on their machines would not persist. Communication Channel Replacement All existing communication channels between ShapeShift employees have been – or are in the process of being – replaced. This includes email accounts, Slack accounts, and GitHub accounts. In addition, all employees were given instructions on how to securely communicate company secrets including IP addresses, API keys, SSH public keys, shared passwords, etc. which were distilled in a company-wide security policy. Cryptographic Key Replacement All employees generated new GPG keys that were protected by strong passwords. Employees also generated new SSH keys that were also protected by strong passwords. Employees were given instruction on the proper use of these keys when accessing production and development servers, and were trained on the proper protocols for the communication of secrets between users.
Production Environment Refresh ShapeShift’s production environment was completely rebuilt using brand new cloud computing accounts. Different operating systems were chosen to house ShapeShift’s new infrastructure to enhance the default security posture of the system. Infrastructure Infrastructure deployment scripts were built using Ansible that ensured firewall rules were put in place prior to the operating system’s connection to the Internet, and ensured user accounts, permissions, required software, module dependencies, and configuration files were placed in the appropriate places within the production environment in a secure and repeatable manner that was not subject to human error. Finally, a number of tools were removed from production servers to hinder the installation of unauthorized applications and tools, such as compilers. Enhanced Logging on Production Servers Enhanced logging capabilities were configured on all production systems that captured relevant actions performed by both the operating systems and all users on those systems. Furthermore, all production servers were configured to send log entries to an off-site logging server to ensure that evidentiary data could not be destroyed following any future breaches. These changes enhanced ShapeShift’s logging configuration to comply with CCSS Level 3. Additional Firewall Rules A number of firewall rules were added to further restrict access to the production environment servers. A bastion server was configured that would marshal access to each of the production servers within the production environment. Outbound firewall rules were added to prevent production servers from reaching out over the Internet to other servers. This prevents the establishment of reverse shells and the initiation of downloads of malware and other tools that would aid an attack. Security Policies Ledger Labs drafted an Employee Security Policy and an Infrastructure Security Policy that identify security procedures and protocols for the use of ShapeShift assets. Employees are required to read and sign the policies and submit identification to ShapeShift’s Human Resources department. This control helps ShapeShift achieve compliance with CCSS Level 2.
Future Enhancements Although ShapeShift staff implemented numerous controls that enhanced security, a few of LLI’s recommendations were deferred for future implementation. These recommendations are outlined here: Multi-Signature Architecture Although this is required for CCSS Level 2 and not Level 1, LLI recommends that ShapeShift’s architecture be re-architected to require multiple signatures. This would prevent a single compromised employee from being able to misappropriate funds on his or her own. End-users should be presented with a P2SH address (or equivalent for its coin type) that is built from a script that requires 3 signatures – 2 signatures from online signing agents that exist external to ShapeShift’s infrastructure, and a 3rd offline recovery key that is stored safely and securely. Deterministic Keys Although Deterministic Keys is another CCSS Level 2 requirement and not Level 1, LLI recommends ShapeShift’s architecture be re-architected to make use of deterministic seeds. This practice would allow public-facing servers to calculate information they need without communicating with private servers. Furthermore, deterministic keys would remove the requirement to perform regular backups of keys since a single backup at the time of implementation would be able to create any key used by ShapeShift. Key Backups, Environmental Protection, and Backup Access Controls ShapeShift’s current backup practices are not compliant with CCSS Level 1. In the near future, key backups will become CCSS Level 1 compliant with the implementation implementation of deterministic deterministic seeds. Once these backups are created, they can be stored in fire-proof / water-proof containers with tamper-evident seals in a location that requires strong access controls to ensure the backup strategy complies with CCSS Level 3 Data Sanitization Policy A Data Sanitization Policy (DSP) should be drafted to ensure media on end-of-life equipment is destroyed in a way that prohibits digital-forensic recovery of confidential information.
Conclusion The digital-forensic investigation, combined with the CCSS assessment, identified a number of opportunities to increase ShapeShift’s security. Ledger Labs worked closely with ShapeShift’s team to carry out these changes prior to the re-launch of the site. Where changes could not be made immediately, immediately, ShapeShift has planned to effect them in the near future once the exchange has been re-launched to the public. The type of breach experienced by ShapeShift has been observed in other bitcoin exchanges in the industry. One notable difference in this case is that usernames, passwords, and email addresses were not compromised alongside the compromised servers due to ShapeShift’s unique information-less exchange architecture. Ledger Labs will continue to work with ShapeShift to ensure the highest level of security for their exchange.
Michael Perklin, CBP, CISSP, CISA, EnCE, ACE Head of Security and Investigative Investigative Services Ledger Labs Inc.
