What if IAM Users could freely generate AWS API keys without restriction? What if security admins lacked visibility into API key creation and could not audit who used API keys?
While this scenario doesn't apply to AWS, it's a stark reality for Google Cloud HMAC Keys. This blog outlines three vulnerabilities surfaced from how Google Cloud handles user-associated HMAC keys.
1. Vulnerability #1 - Insufficient Logging
2. Vulnerability #2 - Unmanagable Long-Term Credentials
3. Vulnerability #3 - Unauditable Long-Term Credentials
TLDR;
- HMAC keys serve a practical purpose. They can be used to create Sigv4 signed headers used to authenticate against the Cloud Storage XML API. for up to 7 days after initial creation.
- Google Cloud Audit Logs do not record HMAC key creation or deletion events when associated with user accounts.
- No Google Cloud API is available to administrators, preventing them from auditing the existence of HMAC keys associated with user accounts.
- No Cloud IAM permissions are available to restrict the creation, deletion or usage of HMAC keys.
- This issue has been reported via Google’s Vulnerability Reward Program and they have closed the issue without providing a fix citing that the reported behavior is working as intended.
Using HMAC Keys
The APIs for creation and deletion of HMAC keys associated with user accounts are not accessible externally. These actions can only be performed through the cloud console private API service (cloudconsole-pa), accessed via the interoperability tab of the storage console. The identified URLs below represents the endpoints within the cloud console used for creating and deleting user-associated HMAC keys when accompanied by a valid login cookie.
- A POST request to `/v3/entityServices/StorageEntityService/entities/STORAGE_USER_HMAC/CREATE` creates an HMAC key used in the Sigv4 signing process
- A POST request to `/v3/entityServices/StorageEntityService/entities/STORAGE_USER_HMAC/DELETE` deletes an HMAC associated with an user.
Google Cloud users are able to create a seemingly endless quantity of the resource type: “type.googleapis.com/google.internal.cloud.console.clientapi.storage.UserHmac”
No upper limit of HMAC keys was identified in testing, and no rating limiting was encountered.
Vulnerability #1 - Insufficient Logging
Cloud Audit Logs do not capture actions mediated through the cloud console private API service (cloudconsole-pa). Consequently, there is no logging of HMAC key creation or deletion linked to user accounts. This absence of logs hampers defenders' ability to alert or monitor the creation of HMAC keys for user accounts, posing a persistence risk, or their deletion, presenting a denial of service risk.
Vulnerability #2 - Unmanagable Long-Term Credentials
Users with Google Cloud access can create HMAC keys, requiring at least the resource manager.projects.get permission. However, there are no Cloud IAM permissions to manage HMAC keys for self or other users, hindering administrators from controlling their creation. As a result, this advisory does not provide recommendations to mitigate the risk of HMAC key exposure through limiting permission assignment.
Vulnerability #3 - Unauditable Long-Term Credentials
Cloud Administrators face challenges in managing HMAC keys within their organizations, lacking visibility into which user accounts have generated these keys and whether they are actively being used to access storage objects. Additionally, there's a lack of functionality to revoke keys associated with other users, restricting their ability to enforce security policies effectively.
Incident response teams rely on Cloud Logging to monitor Cloud Storage object access, but they lack specific indicators to determine if HMAC keys are being utilized in these access attempts. While various containment actions, such as suspending or deleting compromised user accounts, may initially seem effective by rejecting previously created Sigv4 signed headers, reactivating or recreating the same user allows the reuse of credentials unless they have expired.
Furthermore, removing Cloud IAM Roles can revoke access to affected storage resources. However, it's important to note that reassigning roles does not invalidate previously created Sigv4 signed headers, allowing them to continue functioning even after the role change.
HMAC Key Abuse Case
An attacker in possession of a Google user account with access to a Google Cloud Project and minimally the `resource manager.projects.get` permission is able to generate HMAC keys in the interoperability tab in the storage console.
An attack path leveraging HMAC keys may play out as follows:
1. An attacker compromises a Google user account
2. Generates an HMAC Key for the user
3. Uses the HMAC key to generate a series of Sigv4 signed headers with a 7-day expiration.
4. Exfiltrates data from Google Cloud Storage until the header signature expires.
5. Identification of the malicious storage access and/or response is hindered due to vulnerabilities #1, #2 and #3 outlined above.
6. Attempts at containment may be either ineffective such as changing the compromised users password or temporary if an affected user is suspended but reactivated later.
Proof of Concept
Functionality is not exposed in the Google Cloud SDK for converting a user-associated HMAC key to usable credentials through the Sigv4 signing process. As such, we’ve provided a simple python helper script to do just that. It takes in an access key id, secret, object and bucket name and returns a curl request with a signed authorization header which will be used to retrieve the targeted GCS object via the XML API.
The proof of concept is heavily influenced by an article by Rosy Parmar on using HMAC to authenticate to Google Cloud.
Recommendations
In light of the numerous abuses associated with user-specific HMAC keys, presented here are a series of recommendations that enhance GCP HMAC key management, logging, and lifecycle.
1. Permissions & APIs
- Introduce APIs and associated permissions that will empower administrators, giving them the authority to create and delete user-specific HMAC keys as needed.
- Create APIs and associated permissions that allow Google Workspace administrators to audit HMAC keys across all users in their Organization, their usage, and the ability to delete keys for others.
2. Org Policy Constraints
- Create an Org Policy Constraint allowing policy administrators to prevent the creation of user-associated HMAC keys.
3. Logging
- Write to Admin Activity admin logs the creation and deletion of all HMAC keys including those associated with users.
- Indicate in Cloud Logs when the Cloud Storage XML API is accessed using a Sigv4 header credential.
4. Credential Management
- Limit the number of HMAC keys any user can create to a maximum of two.
- Introduce a user-configured expiration data for HMAC keys.
- Only display HMAC keys to users once, as they are created, and do not secrets to the UI in subsequent requests.
Disclosure Timeline
2/07/24 [Kat Traxler]: Reported to Google’s Vulnerability Reward Program under the headline "HMAC keys generated for users pose security risks, notably due to the absence of logging events upon their creation or deletion.”
2/07/24 [Google]: Confirms receipt of the report
2/08/24 [Google]: Indicated they were closing the issue due to lack of details. Requested attack scenario and proof of concept.
02/09/24 [Kat Traxler]: Responded with a more detailed walkthrough of attacker actions given the characteristics of HMAC keys and a promise to write a PoC
2/12/24 [Google]: Status changed to ‘Assigned, Re-opened, Triaged’
2/16/24 [Google]: The Google Bug Hunter Team recapped the reported vulnerability details, requested an example of a Sigv4 signed header and asked if credentials can be used to access anything besides the Google Cloud Storage XML API.
2/18/24 [Kat Traxler]: Responded to the series of questions and confirmed that credentials generated from HMAC keys can only be used to access the Google Cloud Storage XML API.
2/19/24 [Kat Traxler]: Provided the Google Bug Hunter Team with a link to Github-hosted PoC allowing them to generate their own signed headers with HMAC keys.
2/28/24 [Google]: The issue was “identified as an Abuse Risk and triaged to our Trust & Safety team.”
2/28/24 [Google]: Responded with a thank you for the report and indicated the team was analyzing the submission.
04/01/24 [Kat Traxler]: “Following up on this report. It's about 4 weeks since we last talked. As a reminder, I plan on disclosing this issue around the first week of June. I'd like to use the time we have to coordinate with the service team to implement missing logging and IAM controls. Without new events being logged and new IAM controls, the disclosure will only contain details of the persistence mechanism without any guidance for customers on how to prevent or detect it. No one likes that kind of disclosure blog. *Please let me know how I can help elevate this issue or coordinate with responsible parties.”
4/02/24 [Google]: “🎉 Nice catch! I've filed a bug with the responsible product team based on your report. We'll work with the product team to ensure this issue is addressed. We'll let you know when the issue was fixed.”
4/11/24 [Google]: “Google Vulnerability Reward Program panel has decided that the security impact of this issue does not meet the bar for a financial reward.”
6/04/24 [Google]: “Our systems show that the bug we created based on your report has been closed without providing a fix. This may have happened for various reasons: the risk impact might be too small to warrant a fix, there might be other mitigating factors, or simply the product is not maintained anymore. The exact status is INTENDED_BEHAVIOR. This decision has been made by the relevant product teams and does not affect your VRP reward amount or Leaderboard position. We can't provide more details in this automated notification, but we'll be happy to answer your questions regarding this decision.“