Tales from the cloud trenches: Raiding for AWS vaults, buckets and secrets

In this post, we explore a campaign we've witnessed in the wild across several AWS environments. Specifically, we will be reporting on attacker techniques that we've noticed from a cluster of malicious IP addresses targeting AWS Secrets Manager, AWS S3 and AWS S3 Glacier.

Key points and observations

• We observed malicious automated activity attempting to enumerate victims’ AWS secrets, S3 buckets, and S3 Glacier vaults.
• While we cannot confirm the initial access vector, based on the use of long-term access keys (AKIA) and the automated activity, we surmise that the attacker was likely using leaked access keys.
• The attacker was using a mix of residential proxies and a VPN client, specifically the Cloudflare Warp VPN.
• The attacker was using an unusual user agent that indicated they were signing their own AWS API requests, something we have not observed in previous attacks.
• Despite successfully being able to list objects in several S3 buckets, there were no follow-up data exfiltration attempts that we could observe, even though S3 data events were enabled.

Attacker activity: Pivoting on IP addresses and enumerating vaults, buckets, and secrets

Over a period from 2024-05-23 to 2024-05-27, we identified abnormal behavior in a customer’s AWS environment during a threat hunt. We observed the IP address 148[.]252.146.75 attempting the API calls ListSecrets and ListVaults in a single AWS environment. Our IP addresses are enriched by internal and external threat intelligence sources. We were able to enrich the IP address in question to determine that it was potentially a residential proxy on the Vodafone network (a UK mobile provider).

Pivoting on this IP address, we were able to identify additional activity in another AWS environment, which included attempts to enumerate AWS S3 buckets with the ListBuckets API call and then determine their contents by using the ListObjects API call across any available buckets. Based on the event times we observed, we are confident that this activity was automated.

We did not identify any follow-up actions to retrieve secrets (GetSecretValue or BatchGetSecretValue) or retrieve objects from S3 buckets (GetObject). We were able to confirm that S3 data events were enabled.

There are a couple of reasons why an attacker may not exfiltrate data after discovering:

• They are running a broad automated campaign against many AWS environments, and as such may assess what data is available before attempting to exfiltrate.
• They may be testing what level of access an AWS identity has to determine resale value in a marketplace.

Notable AWS service: S3 Glacier

While AWS S3 is a common target for attackers, this is the first time we have observed an attacker targeting data stored in S3 Glacier vaults, which are well suited for backup data storage. While we only observed failed enumeration attempts, based on available AWS documentation, we assume we would see two subsequent Initiate Job API calls after enumeration, one to retrieve a list of the available vault archives, and the second to retrieve the specific archive. An archive can then be downloaded using the Get Job Output API call.

 "awsRegion": "us-west-2",
  "evevntName": "ListVaults",
  "eventCategory": "Management",
  "level": "Error",
  "eventVersion": "1.08",
  "eventSource": "glacier.amazonaws.com",
  "readOnly": true,
  "eventType": "AwsApiCall",
  "error": {
    "kind": "AccessDenied",
    "message": "User: arn:aws:iam::<ACCOUNT ID>:user/<IAM USER> is not authorized to perform: glacier:ListVaults on resource: arn:aws:glacier:us-west-2:<ACCOUNT ID>:vaults/ because no identity-based policy allows the glacier:ListVaults action"
  },

Notable network infrastructure: CloudFlare WARP VPN

Attackers commonly use VPNs to mask their true geo-location. The Cloudflare WARP client is a free VPN (subscription models are available) that allows the user to tunnel their traffic through Cloudflare’s network. A possible advantage for an attacker in using this client is that AWS API calls from Cloudflare may appear less suspicious on the surface in comparison with some of the hosting providers used by other VPN services.

Notable user agent: Signing API requests

python-requests/<requests version> auth-aws-sigv4/<library version>

Based on this code snippet, the user agent identified in this attack is likely to have been generated by the Python library requests-auth-aws-sigv4. . This library allows the user to sign AWS API requests manually. In the automated attacks we have observed in the past, the user agents typically indicate the use of the AWS CLI or Boto3 (AWS SDK for Python). AWS uses Sigv4 to sign API requests. This signing is handled automatically by Boto3. At this time, we have been unable to discern a specific reason why the attacker would want to manage this manually, but this may be a good indicator of suspicious activity in your environment if the usage of this library is not expected.

Summary of attacker activity

Discovery:

• T1580 - Cloud Infrastructure Discovery
• T1619 - Cloud Storage Object Discovery

Detection opportunities

Enumeration techniques on commonly used services like AWS S3 and AWS Secrets Manager can be difficult to detect with a high confidence. Here are some suggestions to help to identify this type of activity:

• Use indicators in the IoC section to detect this specific campaign. If you are able, it may be helpful to enrich IP addresses for the specific WARP client if API calls from CloudFlare are expected.
• Identify ListSecret/ListVault API calls across multiple regions in a short time period. In the data we have, the attacker attempted to run each of these API calls across 17 regions in under a minute.
• Identify spikes in AccessDenied errors for the following API calls: ListSecrets, ListBuckets, ListObjects, ‘ListSecrets. In circumstances where the attacker does not have the correct permissions, there may be an anomalous spike in AccessDenied errors.

How Datadog can help

Datadog Cloud SIEM and Cloud Security Management (CSM) come with the following out-of-the-box rules to identify suspicious activity relevant to these attacks in an AWS environment. The Cloud SIEM rules help identify potential threats, while the CSM rules help identify long-lived or stale access keys. Long-lived access keys tend to carry a higher risk of being associated with a compromise.

• A user received an anomalous number of AccessDenied errors
• Compromised AWS IAM User Access Key
• Access keys should be rotated every 90 days or less
• Inactive IAM access keys older than 1 year should be removed

Conclusion

During the development of this blog, Lacework released a blog detailing the same campaign. They captured additional AWS API calls indicating that the target of the campaign may have been Amazon Bedrock and the possible misuse of LLM resources.

We think this campaign warrants a closer look by detection and response teams as the exfiltration of production LLM data and resources from your cloud environment could be highly impactful to business operations.

Indicators of compromise