Following attackers’ (Cloud)trail in AWS: Methodology and findings in the wild

As part of our security research, we regularly perform threat hunting activities—i.e., proactively searching for, and possibly identifying, malicious activity—in cloud environments. Threat hunting helps us identify new threats and improve the content available in Datadog’s security products.

Recently, our threat hunting sessions have focused primarily on AWS environments. In this post, we detail some of our methodology to proactively identify malicious activity by investigating logs in AWS Cloudtrail, which centralizes logs documenting user activity and API usage across an organization’s AWS infrastructure. We also share our findings on malicious activity we’ve observed in the wild, along with actionable queries and indicators of compromise for those who want to search their own environments for signs of the threat activity we’ve observed.

While some of these attacks reflect patterns that have been observed before and are starting to be well-understood, we believe it's critical to continuously challenge hypotheses based on what's really happening in the wild.

Goals of threat hunting in CloudTrail

Threat hunting is generally expected to drive the following outcomes:

• Finding breaches and malicious activity
• Fueling the design and creation of new detection rules
• Building subject matter expertise on threat activity in a particular type of environment
• Documenting learnings to share with others

Every hunt session starts with a hypothesis about attacker behavior and attempts to verify this idea in the real world using available data sources. The goal, much like the scientific method, is to reject or confirm the hypothesis. In an effort to better understand AWS threat actors, we performed several hunt sessions based on the following hypotheses:

• An attacker may attempt to create an IAM user in order to persist in an environment they’ve breached
• An attacker may attempt to leverage Amazon SES for email spam purposes
• An attacker who has compromised an environment with private EC2 instances may attempt to access it by creating or importing EC2 key pairs and modifying security groups
• An attacker who has compromised AWS credentials may attempt to enumerate the permissions assigned to a stolen identity

Findings

Creating IAM users is a popular persistence vector

We identified that shortly after compromising an AWS environment, attackers frequently attempt to create an IAM user. In particular, we assess with high confidence that the following IAM user names have been used to create "backdoor" IAM users:

EmperorsToolsShops32
one_piece
Policy-role-16
Policy-role-6
s3.dev
s3.simprosys
flashx7860
sesadministrator
ses-admin
ses_BackupUser
sesadminsmtp
roxaot
devlops
produc-sns-net-usr
ampIify-dev
root
jSDSsajsnhjjjjjjwyyw
ses_legion
ses_master
MailSES
s3-ftp

To identify suspicious identities, we looked at several elements such as:

• High volumes of access denied errors from a specific identity
• IAM user creation events from EC2 instances (where the role session name starts with i-)
• Access denied errors that occurred when creating an IAM user, especially when the same IAM user name was attempted to be created across multiple environments
• IAM user creation events from identities that had never created IAM users in the past
• IAM user names with grammatical errors or slight deviations of a common word

Then, we analyzed the related activity to confirm maliciousness with a high level of confidence.

The "root" user

As indicated in the list above, we've seen numerous cases of an attacker attempting to create an IAM user named "root".

{
  "awsRegion": "us-east-1",
  "eventSource": "iam.amazonaws.com",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "error": {
    "kind": "AccessDenied",
    "message": "User: arn:aws:iam::REDACTED:user/REDACTED is not authorized to perform: iam:CreateUser on resource: arn:aws:iam::REDACTED:user/root because no identity-based policy allows the iam:CreateUser action"
  },
  "eventName": "CreateUser"
}

This strikes us as an attempt to stay under the radar, as people may think that this IAM user is the AWS default root user (which is, in reality, not an IAM user but a concept of its own).

The Emperor Tools toolkit

In one case, the malicious IAM user (EmperorsToolsShops32) is related to a toolkit, Emperors Tools, that's sold through an online shop.

The threat actor selling this toolkit has several Telegram channels, including one where they provide "free samples" of compromised AWS and RDP credentials:

It appears that the PHP script the actor uses to receive stolen files is available on GitHub. Using Shodan and Google, we were able to identify at least several compromised, legitimate websites that the actor is likely using as an intermediary.

Funnily enough, the attacker's upload script is vulnerable to a straightforward arbitrary file upload vulnerability that would easily allow remote code execution.

Persist, but check first

In several cases, we witnessed the attacker checking for the presence of a specific malicious IAM user before attempting to create it.

{
  "eventName": "GetUser",
  "errorCode": "AccessDenied",
  "errorMessage": "User: arn:aws:iam::[redacted]:user/[redacted] is not authorized to perform: iam:GetUser on resource: user admindvelop because no identity-based policy allows the iam:GetUser action"
}

We witnessed the following IAM user names being checked:

administrator
adminprod
adminprodz
admindvelop

Creating security groups

We identified an actor who, after initial compromise, attempted to create a security group called Java_Ghost. We noticed this activity in several independent environments:

{
  "awsRegion": "us-east-1",
  "eventName": "CreateSecurityGroup",
  "eventCategory": "Management",
  "eventSource": "ec2.amazonaws.com",
  "requestParameters": {
    "groupName": "Java_Ghost",
    "groupDescription": "TEST"
  }
}

A similar activity was also noticed by a Reddit user in May 2022, whose account was compromised.

Considering that all attempts to create this security group failed due to a lack of permissions, we can't know for sure the intent of the attacker. However, it seems reasonable to assume that they were creating a security group before attempting to run EC2 instances.

Creating EC2 key pairs

The creation of a EC2 key pair is a known technique to maintain access to an AWS environment via EC2 instances. During our research, we witnessed one attacker attempt a DescribeInstances API call with a filter for a specific key-name before attempting to create a key pair. We inferred that this was likely an automated function to determine if the AWS account had been compromised previously.

{
  "eventName": "DescribeInstances",
  "requestParameters": {
    "filterSet": {
      "items": [{
        "valueSet": {
          "name": "key-name",
          "items": [{
              "value": "xg1"
          }]
        }
      }]
    }
  }
}

{
  "eventName": "CreateKeyPair",
  "requestParameters": { 
    "keyFormat": "pem", 
    "keyName": "xg1" 
  }
}

temp_key_pair
xg1 

Most common enumeration techniques

In the table below, we’ve compiled the most common enumeration API calls discovered throughout our hunting activity. At this stage, the attacker has compromised credentials, but they don't know where they’ve landed in the breached environment. Their goal is to gain situational awareness and understand the value and potential of the compromised account.

The usage pattern of these API calls leads us to believe that the discovery is automated, and not performed by a human—at least not at first. For instance, in several cases in which the compromised IAM user had the word s3 in it, the attacker did not attempt to list or access S3 buckets. These events are also logged within a very short period of time, reinforcing our assumption that their discovery is automated.

Detection Opportunities

As an outcome of our threat hunt, we used our findings to create several high-confidence detections. These are presented in a CloudTrail SQL format, allowing any team to search their CloudTrail logs for potential threats through AWS CloudTrail Lake.

Atomic indicator IAM user creation

SELECT
  eventTime,
  sourceIPAddress,
  userIdentity.arn as user,
  requestParameters['userName'],
  eventName
FROM
  <event-data-store-id>
WHERE
  eventSource = 'iam.amazonaws.com' AND eventName = 'CreateUser' 
  AND requestParameters['userName'] IN ('EmperorsToolsShops32', 'one_piece', 'Policy-role-16', 'Policy-role-6', 's3.dev', 's3.simprosys', 'flashx7860', 'sesadministrator', 'ses-admin', 'ses_BackupUser', 'sesadminsmtp', 'roxaot', 'devlops', 'produc-sns-net-usr', 'ampIify-dev', 'root', 'jSDSsajsnhjjjjjjwyyw', 'ses_legion', 'ses_master', 'MailSES', 's3-ftp')

SELECT 
  eventTime,
  sourceIPAddress,
  userIdentity.arn AS user,
  errorMessage,
  eventName
FROM 
  <event-data-store-id>
WHERE 
  eventSource = 'iam.amazonaws.com' AND eventName = 'CreateUser' 
  AND (errorMessage LIKE '%/EmperorsToolsShops32%' OR errorMessage LIKE '%/one_piece%' OR errorMessage LIKE '%/Policy-role-6%' OR errorMessage LIKE '%/Policy-role-16%' OR errorMessage LIKE '%/s3.dev%' OR errorMessage LIKE '%/flashx7860%' OR errorMessage LIKE '%/sesadministrator%' OR errorMessage LIKE '%/ses-admin%' OR errorMessage LIKE '%/ses_BackupUser%' OR errorMessage LIKE '%/sesadminsmtp%' OR errorMessage LIKE '%/roxaot%' OR errorMessage LIKE '%/devlops%' OR errorMessage LIKE '%/produc-sns-net-usr%' OR errorMessage LIKE '%/adminprod%' OR errorMessage LIKE '%/adminprodz%' OR errorMessage LIKE '%/admindvelop%' OR errorMessage LIKE '%/ampLify-dev%' OR errorMessage LIKE '%/root%' OR errorMessage LIKE '%/jSDSsajsnhjjjjjjwyyw%' OR errorMessage LIKE '%/ses_legion%' OR errorMessage LIKE '%/ses_master%' OR errorMessage LIKE '%/MailSES%' OR errorMessage LIKE '%/s3-ftp%')

Atomic indicator Security Group creation

SELECT 
  eventTime,
  sourceIPAddress,
  userIdentity.arn AS user,
  eventName
FROM
  <event-data-store-id>
WHERE 
  eventSource = 'ec2.amazonaws.com' 
  AND eventName = 'CreateSecurityGroup'
  AND requestParameters['groupName'] = 'Java_Ghost'

Atomic indicator EC2 keypair creation

SELECT 
  eventTime,
  sourceIPAddress,
  userIdentity.arn AS user,
  eventName,
  requestParameters['keyName']
FROM
  <event-data-store-id>
WHERE 
  eventSource = 'ec2.amazonaws.com' 
  AND eventName = 'CreateKeyPair'
  AND requestParameters['keyName'] IN ('xg1', 'temp_key_pair')

EC2 instances attempting to create IAM users

SELECT 
  eventTime, 
  sourceIPAddress, 
  userIdentity.sessioncontext.sessionissuer.arn as roleArn, 
  split(userIdentity.arn, '/')[3] as instanceId, 
  requestParameters['userName'], eventName 
FROM 
  <event-data-store-id>
WHERE 
  eventSource = 'iam.amazonaws.com' AND eventName = 'CreateUser' 
  AND split(userIdentity.arn, '/')[3] LIKE 'i-%'

Principals with a high number of access denied errors

SELECT
  useridentity.arn,
  COUNT(*) AS "Total access denied errors",
  COUNT(DISTINCT eventName) AS "Unique failed event names",
  array_distinct(array_sort(zip_with(array_agg(eventSource), array_agg(eventName), (source, event) -> concat(split_part(source, '.', 1), ':', event))) AS "Event names with permission denied")
FROM
  <event-data-store-id>
WHERE errorCode = 'AccessDenied'
GROUP BY useridentity.arn
HAVING COUNT(DISTINCT concat(eventSource, ':', eventName)) > 1
ORDER BY "Unique failed event names" DESC

How Datadog can help

Datadog Cloud SIEM includes several detection rules that help teams detect some of the types of malicious activity we encountered in our AWS threat hunting. These include:

• Amazon SNS enumeration attempt by previously unseen user
• User travel was impossible in AWS CloudTrail IAM log
• Compromised AWS IAM user access key
• AWS security group created, modified, or deleted
• AWS IAM user created with AdministratorAccess policy attached
• Tor client IP address identified within AWS environment
• The AWS managed policy AWSCompromisedKeyQuarantineV2 has been attached
• AWS IAM AdministratorAccess policy was applied to a user

In addition, you can use the following Datadog logs queries to identify activity matching one of the indicators of compromise described in this post.

• Potentially malicious IAM user creation (open in-app logs search):

source:cloudtrail @evt.name:CreateUser (@requestParameters.userName:(EmperorsToolsShops32 OR one_piece OR Policy-role-6 OR Policy-role-16 OR s3.dev OR s3.simprosys OR flashx7860 OR sesadministrator OR ses-admin OR ses_BackupUser OR sesadminsmtp OR roxaot OR devlops OR produc-sns-net-usr OR adminprod OR adminprodz OR admindvelop OR ampLify-dev OR root OR jSDSsajsnhjjjjjjwyyw OR ses_legion OR ses_master OR MailSES OR s3-ftp) OR @error.message:(*/EmperorsToolsShops32* OR */one_piece* OR */Policy-role-6* OR */Policy-role-16* OR */s3.dev* OR */s3.simprosys* OR */flashx7860* OR */sesadministrator* OR */ses-admin* OR */ses_BackupUser* OR */sesadminsmtp* OR */roxaot* OR */devlops* OR */produc-sns-net-usr* OR */adminprod* OR */adminprodz* OR */admindvelop* OR */ampLify-dev* OR */root* OR */jSDSsajsnhjjjjjjwyyw* OR */ses_legion* OR */ses_master* OR */MailSES* OR */s3-ftp))

• Potentially malicious security group creation (open in-app logs search):

source:cloudtrail @evt.name:CreateSecurityGroup @requestParameters.groupName:"Java_Ghost"

• Potentially malicious keypair creation (open in-app logs search):

source:cloudtrail @evt.name:Create KeyPair @requestParameters.keyName:(xg1 OR temp_key_pair)

Conclusion

We are always trying to learn how the cloud threat landscape is evolving, and through the threat hunting activities described in this post, we identified some unique behaviors and validated some of the other amazing research out there, which we have referenced below. We hope you can use some of these indicators to find—or, hopefully, not find—badness in your own environment.

References

• https://www.chrisfarris.com/post/public-access-key-2023/
• https://www.cadosecurity.com/legion-an-aws-credential-harvester-and-smtp-hijacker/
• https://permiso.io/blog/s/unmasking-guivil-new-cloud-threat-actor/
• https://permiso.io/blog/s/approach-to-detection-androxgh0st-greenbot-persistence/
• https://sysdig.com/blog/cloud-breach-terraform-data-theft/
• https://orca.security/wp-content/uploads/2023/06/Orca-Security-2023-Honeypotting-in-the-Cloud-Report-Attacker-Tactics-and-Techniques-Revealed.pdf

Annex: Indicators of compromise

These indicators have been observed between August 6 2023, and September 7 2023.