SHAPESHIFT CYBERATTACK POSTMORTEM

PREPARED FOR: SHAPESHIFT AG

Ledger Labs Inc.

Michael Perklin, CBP, CISSP, CISA, EnCE, ACE
michael@ledgerlabs.io

ShapeShift Cyber-Attack Postmortem

On 2016-04-09 at 13:00 EDT, ShapeShift CEO Erik Voorhees contacted Ledger Labs
Inc. (LLI) with a request for digital investigative assistance. Erik stated that following
the security incident that ShapeShift experienced on 2016-04-07, ShapeShift was
hacked a second time earlier that morning. In addition, ShapeShift experienced a
third breach on 2016-03-14 by an internal employee. Following a brief call with
some members of the ShapeShift team, the decision was made to send Michael
Perklin of LLI to ShapeShifts headquarters. Michael boarded a flight and
immediately attended ShapeShifts headquarters to begin his investigation.
Michael conducted interviews of Erik Voorhees and other ShapeShift staff to gather
preliminary statements and build a preliminary timeline of events.

Established Facts
The following facts were established following the preliminary investigation:

Following the 2016-03-14 breach (which will be referred to as Breach1 of

Infrastructure1), entirely new SSH keys and passwords were created by all
team members in order to securely build the new infrastructure;
Following the 2016-04-07 breach (Breach2 of Infrastructure2), some team
members changed computers to protect against malware that may have
existed on those machines. Entirely new SSH keys and passwords were
created by all team members;
The Breach2 attacker used the alias Rovion Vavilov, and was the same
attacker that breached ShapeShifts servers during the 2016-04-09 attack
(Breach3 of Infrastructure3). Breaches 2 and 3 sent coins to the same
addresses, and an email was sent by the Breach2 attacker to Erik Voorhees
with a single :) minutes after Breach3 was completed;
Rovion was able to learn the IP addresses of both Infrastructure2 and
Infrastructure3 despite these IPs being known only by ShapeShift staff.
Infrastructure3s IP address was identified within hours of being established
by the ShapeShift team;
Rovion was able to access Infrastructure3 through firewall rules that
restricted access to the ShapeShift HQ static IP address;
ShapeShift received an independent report that showed internal ShapeShift
communications had been previously compromised;

Digital Forensic Investigation

Following the preliminary investigation, a digital-forensic investigation was launched
to analyze both Infrastructure2 and Infrastructure3 for artifacts that would provide
insight into the breaches.
Infrastructure1 Analysis
The first server analyzed was the machine that ran the core ShapeShift exchange
code. For the purposes of this report, the server will be referred to as Simpson. A
bitstream disk image was captured using a combination of the dd, netcat and
bzip2 commands. The sha1 hash of the disk image was calculated as
ac0a4d287a1227d7c805f7d9b83141dadfdf682f and was verified following the
transfer.
Analysis of the Ubuntu operating system configuration identified that there was no
logging or auditing configured beyond the default configuration that ships with
Ubuntu. This limited the amount of artifacts recorded by the operating system, and
limited the amount of evidentiary data available.
An analysis of /var/log/auth.log revealed that the logs were wiped at some time
prior to 2016-04-10 at 23:05 UTC. A forensic recovery of the previous contents of
auth.log was attempted and the deleted log file was successfully recovered from
unallocated space.
Analysis of the recovered log data revealed artifacts of select actions executed by
Rovion. Rovion executed the following commands on Simpson on 2016-04-09
between 10:18:54 and 10:23:08 UTC:

/bin/mv slave /bin/udevd-bridge

/bin/mv init /etc/init/udevd-bridge.conf
/bin/chown root.root /bin/udevd-bridge
/bin/chmod 755 /bin/udevd-bridge
/bin/chown root.root /etc/init/udevd-bridge.conf
/bin/chmod 644 /etc/init/udevd-bridge.conf
/usr/bin/touch -d Apr 14 2014 /etc/init/udevd-bridge.conf
/usr/bin/touch -d Jan 26 08:38 /bin/udevd-bridge
/sbin/status udevd-bridge
/sbin/start udevd-bridge

The above commands show the installation of a rootkit at /bin/udevd-bridge. A

sha1 hash was calculated for this file which yielded the value
d94f219c7b71745c11e84d652819ff1f0c4b6582. The contents of the
/etc/init/udevd-bridge.conf file ensured the rootkit would launch every time
the system was booted. Rovion named the rootkit to appear like a system file on the
Ubuntu operating system, and further hid the rootkit by modifying the timestamps
to match similar system files. Rovion then launched the rootkit with the
/sbin/start udevd-bridge command.

Prior to these commands, the auth.log showed the last login to the server was
completed via public key authorization on 2016-04-09 at 8:44:37 UTC by a key with
RSA fingerprint 9d:35:fd:6d:60:a8:6e:2e:56:f3:d0:ac:07:79:7f:cb. This is
consistent with a known employee login to Simpson via VPN connection from the
employees home. This SSH connection remained open until 13:04 UTC, suggesting
the employees session was used to breach the Simpson server.
Although significantly more analysis was performed, the operating systems
configuration prevented additional artifacts from being generated following user
and system actions, resulting in insufficient evidentiary data.
Infrastructure2 Analysis
Following the analysis of Simpson, a bitstream image of the server that ran
ShapeShifts core exchange code from Infrastructure2 was obtained using a
combination of dd, gzip2, and netcat. For the purpose of this report, this machine
will be identified as Lenny. The sha1 hash of the disk image was calculated as
41b5bc88fd7ab4ef0844069df51fc281caa59338.
Analysis of Lennys Ubuntu operating systems configuration revealed that similar
to Simpson there was no logging or auditing configured beyond the default
configuration that ships with Ubuntu. Analysis of the /var/log/auth.log file
showed tampering via overwriting unlike Simpson which had its log deleted. The
last few lines of the log were overwritten with NULL (0x00) bytes, preventing digitalforensic recovery.
Analysis of the /bin folder identified the installation of the same rootkit identified
on the Simpson server at the same path: /bin/udevd-bridge.
Although significantly more analysis was performed, no data artifacts were
identified that could help identify how the back-door was placed on Lenny, or who
performed it.

Analysis
Since direct evidence of a specific attack vector was not found during the digitalforensic investigation, an analysis of the available facts was performed to identify all
possible attack vectors that fit the facts. It was noted that the attacker was not only
able to compromise both infrastructures fairly quickly, but they were able to identify
their IP addresses equally as fast. The following attack vectors were possible
avenues of attack sorted in order of probability:
1. An(other) employee with access to both Simpson and Lenny performed or
assisted with both attacks;
2. A Remote Access Trojan (RAT) was installed on a laptop belonging to an
employee with access to both Simpson and Lenny. The compromised laptop

allowed Rovion to obtain the location of both new infrastructures and obtain
an SSH key for access;
3. A vulnerability exists in the ShapeShift source code that launched a reverse
shell to a machine under Rovions control. This source code ran on both
Simpson and Lenny, and upon reaching out to Rovions machine, told him
their IP addresses;
4. A vulnerability exists within one of the services running on the Ubuntu
operating system that was exploited to open a reverse shell to a machine
under Rovions control. This service ran on both Simpson and Lenny. Rovion
obtained the IP addresses of both machines via a communications channel
breach (i.e. email, Slack, etc.);

Chat with Rovion

Towards the end of the investigation while remediation was underway, Erik
Voorhees received communications from Rovion with a request to exchange his
stolen ether for bitcoin. Erik made the decision to conduct this trade in exchange for
information with the hope of identifying the entry point of compromise.
Following this interaction, Rovion provided information that he alleged was true.
Although there is no way to identify whether the information was actually true, the
information provided corroborated a number of facts uncovered during the
investigation that would only be known by Rovion, and provided a number of other
facts that were consistent with the conditions observed during the attacks. These
included:

Rovion installed a rootkit at /bin/udevd-bridge

Rovion purchased information from a previous employee which allowed him
to conduct the attacks on ShapeShift. This information included:
o The public/external IP address of ShapeShift HQs office
o The username/password of ShapeShift HQs office routers admin
interface, and the port upon which the admin interface was listening.
This would allow Rovion to remap ports to internal machines at will.
o The internal IP address of a ShapeShift employees machine within the
ShapeShift HQ network
o Details of an AquaConnect installation on a ShapeShift employees
laptop that was installed by the prior employee prior to his departure
on March 14th, along with the username/password/port with which to
connect
o An SSH key belonging to a ShapeShift employee that granted access
to Simpson

CCSS Assessment
Following the digital forensic investigation, LLI performed an assessment of the
ShapeShift infrastructure against the CryptoCurrency Security Standard (CCSS). The
assessment identified a number of aspects in which ShapeShifts controls were
sufficient for obtaining Level 3, however others were identified that left ShapeShifts
infrastructure as uncertified. Since CCSS requires unanimous compliance of all
aspects in order to achieve a security level, ShapeShifts resultant level was 0
Uncertified.
LLI drafted a list of security controls that must be implemented in order for
ShapeShifts infrastructure to be graded as CCSS Level 1. LLI staff worked with
ShapeShift staff to implement these controls.

Corrective Action
Although evidence of the specific attack vector could not be found due to both the
destruction of evidence by Rovion and the lack of logging configured on the servers,
a number of corrective actions would prevent each of the above attack vectors from
being exploited, thereby dramatically increasing ShapeShifts security. This section
outlines the corrective actions taken by the ShapeShift team under LLIs guidance:
Computing Hardware Replacement
All ShapeShift employees who had access to both Infrastructure1 and
Infrastructure2 received brand new computing hardware to ensure any RATs,
backdoors, or malware installed on their machines would not persist.
Communication Channel Replacement
All existing communication channels between ShapeShift employees have been or
are in the process of being replaced. This includes email accounts, Slack accounts,
and GitHub accounts.
In addition, all employees were given instructions on how to securely communicate
company secrets including IP addresses, API keys, SSH public keys, shared
passwords, etc. which were distilled in a company-wide security policy.
Cryptographic Key Replacement
All employees generated new GPG keys that were protected by strong passwords.
Employees also generated new SSH keys that were also protected by strong
passwords.
Employees were given instruction on the proper use of these keys when accessing
production and development servers, and were trained on the proper protocols for
the communication of secrets between users.

Production Environment Refresh

ShapeShifts production environment was completely rebuilt using brand new cloud
computing accounts. Different operating systems were chosen to house
ShapeShifts new infrastructure to enhance the default security posture of the
system.
Infrastructure deployment scripts were built using Ansible that ensured firewall
rules were put in place prior to the operating systems connection to the Internet,
and ensured user accounts, permissions, required software, module dependencies,
and configuration files were placed in the appropriate places within the production
environment in a secure and repeatable manner that was not subject to human
error.
Finally, a number of tools were removed from production servers to hinder the
installation of unauthorized applications and tools, such as compilers.
Enhanced Logging on Production Servers
Enhanced logging capabilities were configured on all production systems that
captured relevant actions performed by both the operating systems and all users on
those systems. Furthermore, all production servers were configured to send log
entries to an off-site logging server to ensure that evidentiary data could not be
destroyed following any future breaches. These changes enhanced ShapeShifts
logging configuration to comply with CCSS Level 3.
Additional Firewall Rules
A number of firewall rules were added to further restrict access to the production
environment servers. A bastion server was configured that would marshal access to
each of the production servers within the production environment.
Outbound firewall rules were added to prevent production servers from reaching
out over the Internet to other servers. This prevents the establishment of reverse
shells and the initiation of downloads of malware and other tools that would aid an
attack.
Security Policies
Ledger Labs drafted an Employee Security Policy and an Infrastructure Security
Policy that identify security procedures and protocols for the use of ShapeShift
assets.
Employees are required to read and sign the policies and submit identification to
ShapeShifts Human Resources department. This control helps ShapeShift achieve
compliance with CCSS Level 2.

Future Enhancements
Although ShapeShift staff implemented numerous controls that enhanced security,
a few of LLIs recommendations were deferred for future implementation. These
recommendations are outlined here:
Multi-Signature Architecture
Although this is required for CCSS Level 2 and not Level 1, LLI recommends that
ShapeShifts architecture be re-architected to require multiple signatures. This
would prevent a single compromised employee from being able to misappropriate
funds on his or her own. End-users should be presented with a P2SH address (or
equivalent for its coin type) that is built from a script that requires 3 signatures 2
signatures from online signing agents that exist external to ShapeShifts
infrastructure, and a 3rd offline recovery key that is stored safely and securely.
Deterministic Keys
Although Deterministic Keys is another CCSS Level 2 requirement and not Level 1,
LLI recommends ShapeShifts architecture be re-architected to make use of
deterministic seeds. This practice would allow public-facing servers to calculate
information they need without communicating with private servers. Furthermore,
deterministic keys would remove the requirement to perform regular backups of
keys since a single backup at the time of implementation would be able to create
any key used by ShapeShift.
Key Backups, Environmental Protection, and Backup Access Controls
ShapeShifts current backup practices are not compliant with CCSS Level 1. In the
near future, key backups will become CCSS Level 1 compliant with the
implementation of deterministic seeds.
Once these backups are created, they can be stored in fire-proof / water-proof
containers with tamper-evident seals in a location that requires strong access
controls to ensure the backup strategy complies with CCSS Level 3
Data Sanitization Policy
A Data Sanitization Policy (DSP) should be drafted to ensure media on end-of-life
equipment is destroyed in a way that prohibits digital-forensic recovery of
confidential information.

Conclusion
The digital-forensic investigation, combined with the CCSS assessment, identified a
number of opportunities to increase ShapeShifts security. Ledger Labs worked
closely with ShapeShifts team to carry out these changes prior to the re-launch of
the site. Where changes could not be made immediately, ShapeShift has planned to
effect them in the near future once the exchange has been re-launched to the
public.
The type of breach experienced by ShapeShift has been observed in other bitcoin
exchanges in the industry. One notable difference in this case is that usernames,
passwords, and email addresses were not compromised alongside the
compromised servers due to ShapeShifts unique information-less exchange
architecture. Ledger Labs will continue to work with ShapeShift to ensure the
highest level of security for their exchange.

Michael Perklin, CBP, CISSP, CISA, EnCE, ACE

Head of Security and Investigative Services
Ledger Labs Inc.