We have identified a cross-tenant vulnerability in Amazon Web Services (AWS) that exploits AWS AppSync. This attack abuses the AppSync service to assume IAM roles in other AWS accounts, which allows an attacker to pivot into a victim organization and access resources in those accounts. This blog post describes how we discovered the vulnerability, a proof of concept showing how we performed sts:AssumeRole into roles that trust the AppSync service, and our disclosure process with the AWS team.
The information was shared with AWS who have since remediated this vulnerability. AWS has also investigated and confirmed that it has not been used against other AWS customers. The official security bulletin from AWS can be found here.
Disclosure timeline
September 1, 2022: We identified the vulnerability and reported it to AWS, who then confirmed they received the report.
September 2, 2022: AWS reproduced the attack and confirmed the vulnerability's impact.
September 6, 2022: AWS pushed a fix to the AppSync service, and we confirmed it remediated the vulnerability.
November 21, 2022: Coordinated disclosure with AWS.
Background
Our researchers and practitioners proactively analyze cloud technologies for vulnerabilities and the attack surface they possess. This allows us to see how attackers could use these technologies to compromise organizations. Through this research, we can provide our customers cutting-edge detection rules that keep them one step ahead of cyber threats.
One way we prioritize which technologies to research is through customer usage. AppSync is one of the more popular Datadog integrations, so we scoped a research project to assess it and the project resulted in the following vulnerability.
AppSync is a popular AWS service that allows developers to quickly create GraphQL and Pub/Sub APIs. When creating a GraphQL API in AWS AppSync, a developer will have to create a data source. A data source is something that stores or has access to the data the GraphQL API is interacting with. Examples include Lambda functions, DynamoDB, RDS, and external APIs.
In addition to the predefined data sources and resolvers (functions used to interact with the data source), AppSync offers the ability to invoke AWS APIs directly. This allows developers to create integrations with AWS services that do not have a predefined resolver. For example, if a developer wanted to create a GraphQL API to interact with Amazon S3, they could configure AppSync directly in their production environment to perform the interaction.
To authorize the actions AppSync will perform, the developer creates a role (or AppSync can automatically create it on their behalf) with the required IAM permissions. This role will have a trust policy that allows the AppSync service to assume the role, and it looks like the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "appsync.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}Using the S3 example, if a developer was building that API, they would create a role with the S3 permissions they need and allow AppSync to assume that role. When that GraphQL API is called, AppSync will assume the role, perform the AWS API call, and interpret the results.
As security researchers, this introduces an interesting question: can we somehow trick the AppSync service to assume a role in an account we don’t control and access its resources?
Confusing the deputy
This type of vulnerability would be an example of the confused deputy problem, where a less-privileged entity (the attacker) convinces a more-privileged entity or service (AppSync) to perform some action on its behalf. We wanted to find a way to confuse AppSync to assume roles in other accounts.
AWS safeguards against this type of attack by validating the role’s Amazon Resource Name (ARN), a unique identifier for an AWS resource. During the creation of a data source, the API will look at the provided ARN and determine if it is in the same AWS account. If it is not, the API will throw an error.
nick.frichette@host /tmp% aws appsync create-data source \
> --api-id example123example123exampl \
> --type HTTP \
> --http-config file://http.json \
> --service-role-arn arn:aws:iam::111111111111:role/example
An error occurred (AccessDeniedException) when calling the CreateDataSource operation: Cross-Account pass role is not allowed.By inspecting the request, one can see that the ARN is passed in the serviceRoleArn parameter:
{
"name": "custom_data_source",
"type": "HTTP",
"serviceRoleArn": "arn:aws:iam::111111111111:role/example",
"httpConfig": {
"endpoint": "https://sts.us-east-1.amazonaws.com/",
"authorizationConfig": {
"authorizationType": "AWS_IAM",
"awsIamConfig": {
"signingRegion": "us-east-1",
"signingServiceName": "sts"
}
}
}
}While testing the validation of this ARN, we identified that the API would accept JSON payloads with properties that used mixed case. For example, the API expected httpConfig but would not throw an error if hTtPcOnFiG was provided.
This finding revealed that a serviceRoleArn provided with a different casing would bypass the validation, allowing us to provide an ARN of a role in a different AWS account. For example, here is a request and response using the normal serviceRoleArn case:
And here is an example with a non-standard servicerolearn case:
By bypassing the ARN validation, we were able to create AppSync data sources tied to roles in other AWS accounts. This would allow an attacker to interact with any resource associated with a role which trusts the AWS AppSync service in any account.
Accessing resources in other AWS accounts
With this vulnerability in hand, we could create data sources in our own account which pointed to resources in other AWS accounts.
To demonstrate this attack, imagine a hypothetical company called EventPlanners that specializes in managing meeting invitations for clients. This company has a GraphQL API for scheduling meetings, collaboration, and controls to ensure that private events are only viewable by the right users. All information is stored in DynamoDB. This example is based on the Event App sample project from AWS AppSync.
To allow the GraphQL API to interact with the DynamoDB table, an IAM role is created in the victim account that provides the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"dynamodb:DeleteItem",
"dynamodb:GetItem",
"dynamodb:PutItem",
"dynamodb:Query",
"dynamodb:Scan",
"dynamodb:UpdateItem"
],
"Resource": [
"arn:aws:dynamodb:us-east-1:677301038893:table/AppSyncEventTable-xrpYoWZk",
"arn:aws:dynamodb:us-east-1:677301038893:table/AppSyncEventTable-xrpYoWZk/*"
]
}
]
}This role is then used in the DynamoDB Resolver to access the database. Because this role has a trust relationship with the AppSync service, the vulnerability can be used to assume into their account.
Let’s take a look from an attacker’s perspective. Say, for example, an attacker was able to phish an EventPlanners employee and gain access to internal documentation. The attacker does not have sufficient privilege to access production databases, but they can see the ARNs of resources in the production AWS account.
Using the vulnerability, the attackers could compromise the EventPlanners database by creating their own AppSync API and data source pointing to the role in the EventPlanners account.
{
"name": "malicioussource",
"type": "HTTP",
"servicerolearn": "arn:aws:iam::677301038893:role/service-role/appsync-ds-ddb-yoczrt-AppSyncEventTable-xr",
"httpConfig": {
"endpoint": "https://dynamodb.us-east-1.amazonaws.com/",
"authorizationConfig": {
"authorizationType": "AWS_IAM",
"awsIamConfig": {
"signingRegion": "us-east-1",
"signingServiceName": "dynamodb"
}
}
}
}This would allow the attacker to interact with this data source as if they owned it. From here, the attackers could create a new resolver and specify a request mapping template to call dynamodb:Scan:
Now the attacker can call the API to view the entirety of the EventPlanners database:
And with that, the entirety of the database is now in the attacker’s control.
Detection
Detecting this attack can be challenging because the associated CloudTrail logs indicate that it is coming from the AppSync service itself. Here is a snippet of the AssumeRole event in CloudTrail:
"eventVersion": "1.08",
"userIdentity": {
"type": "AWSService",
"invokedBy": "appsync.amazonaws.com"
},
"eventTime": "2022-09-03T21:20:18Z",
"eventSource": "sts.amazonaws.com",
"eventName": "AssumeRole",
"awsRegion": "us-east-1",
"sourceIPAddress": "appsync.amazonaws.com",
"userAgent": "appsync.amazonaws.com",
…As a result, if an attacker knew both the ARN of the role associated with AppSync to assume and the resources they were interested in, it would appear as if AppSync is performing its normal functions. For example, if the attacked role regularly invoked s3:ListBuckets, it may be difficult to determine if this was a normal operation or one carried out by a malicious actor.
That being said, there are some indicators of compromise to look out for. First, an attacker may find/enumerate a role ARN but not know what permissions are associated with it. As a result, an attacker may attempt to brute force the permissions. This activity would result in a large number of AccessDenied events where the source is appsync.amazonaws.com. Additionally, an attacker’s behavior may trigger anomalous detections. For example, even if a role was highly privileged, having s3:* permissions, an attacker may perform an action which is an anomaly. If a role had never called s3:ListBuckets, and it suddenly has, this may be an indicator of compromise.
For Datadog customers, we have queried relevant log sources to determine if any of our customers may have been impacted and confirmed none were. If you are concerned about this vulnerability and its effect on your organization, please reach out to your AWS relationship manager.
Conclusion
This vulnerability in AWS AppSync allowed attackers to cross account boundaries and execute AWS API calls in victim accounts via IAM roles that trusted the AppSync service. By using this method, attackers could breach organizations that used AppSync and gain access to resources associated with those roles. After finding this vulnerability, we contacted the AWS Security Team who swiftly remediated the issue.
Acknowledgements
We’d like to give a major thank you to Dan Urson and the AWS Security Outreach Team for being awesome to work with, as always.
