LinkedIn
Twitter
While at BSides Las Vegas this summer, I attended an excellent presentation by Blake Hudson titled, Pipeline Pandemonium: How to Hijack the Cloud and Make it Rain Insecurity. Blake provided a deep dive on the various security risks and vulnerabilities that can exist in CI/CD pipelines. He also walked through how to execute some offensive techniques for exploiting said vulnerabilities.
One of the key takeaways was how easy it is for attackers to steal secrets from CI/CD pipelines if they gain access to them. Stolen secrets such as API keys, service account keys, SSH keys, database credentials, and private certificates can allow attackers to do significant damage to an organization including disrupting critical services, exfiltrating data, or deploying ransomware.
Many organizations still rely on long-lived credentials within their CI/CD pipelines. Long-lived credentials that are seldom (or never) rotated increase a company’s risk and can provide attackers with extended or indefinite access to sensitive systems & data. In contrast, a short-lived credential is like granting a temporary access code to a workload that only works for a finite period of time.
In this blog series, I’ll demonstrate how to improve an organization’s security posture by eliminating the need for long-lived credentials (service account keys) in a CI/CD pipeline. Specifically, I’ll walk through an example of how to configure Workload Identity Federation with my Detection-as-Code pipeline that’s used to manage my rules in Google Security Operations (SecOps). We’ll also look into monitoring & detection considerations and how to use Google SecOps to detect the usage and creation of service account keys.
What are Service Account Keys?
In Google Cloud, a service account is a special type of account that represents a non-human user, such as an application, service, or workload. A service account is identified by its email address, which is unique to the account. For example, google-secops-dac@project-123456.iam.gserviceaccount.com.
Applications can use service accounts to carry out authorized actions in your Google Cloud environment. When an application authenticates as a service account, it has access to all resources that the service account has permission to access.
Unlike regular user accounts, service accounts don’t have passwords. Instead, service accounts can use RSA key pairs for authentication. If you know the private key of a service account's key pair, you can interact with Google Cloud APIs on the service account's behalf.
Because the private key lets you authenticate as the service account, having access to the private key is similar to knowing a user's password. The private key is known as a service account key.
Service Account Key Risks
Service account keys should remain private and can become a security risk if not managed carefully. As per Google Cloud’s documentation, you should use a more secure alternative for authentication whenever possible. We’ll look at an example of a more secure authentication method in part two.
Some risks associated with service account keys are as follows:
- Compromised Credentials – Leaked or stolen keys can grant attackers unauthorized access to your cloud resources, potentially leading to a data breach or service disruption. Keys can be inadvertently exposed in places such as code repositories, wikis, or instant messaging platforms.
- Privilege Escalation – If a service account has excessive permissions, a compromised key can allow attackers to escalate their privileges and gain access to even more sensitive resources. This can happen if the key grants access to resources beyond its intended purpose or if it can be used to impersonate a higher-privileged account.
- Non-Repudiation: A threat actor that authenticates using a service account key and carries out actions as a service account makes it harder for the security team to trace actions back to a specific individual.
Example Service Account Key Usage in a CI/CD Pipeline
At this point, we understand what service account keys are and some of the security risks associated with them. In this section, I’ll provide an overview of a CI/CD pipeline that utilizes a service account key before implementing a more secure alternative, Workload Identity Federation.
In my lab environment, I’ve implemented Detection-as-Code to manage my rules in Google SecOps. I have a GitHub repository that contains my rule logic and a set of GitHub Actions workflows (CI/CD jobs) to manage my rules via the Google SecOps API.
The GitHub Actions workflows use a Google Cloud service account key for authentication to Google SecOps’ API before carrying out actions such as creating, updating, or enabling/disabling rules. The service account key is stored in the GitHub repository in the form of a secret.
The diagram below shows the interactions between the service account key stored in the GitHub repository, the GitHub Actions workflows, and the Google SecOps API.
Using GitHub and GitHub Actions to manage rules in Google SecOps
Let’s take a look at how an attacker can access secrets such as service account keys from CI/CD pipeline tools. The developers of these tools implement basic safeguards to prevent secrets from being printed to the console output. The image below shows an attempt to print the secret, “HOGWARTS_API_KEY”. Note that the secret is masked as “***” in the console output. That’s good.
By executing a variant of one of the techniques from the conference talk I mentioned at the beginning of this post, the example output below shows that the secret, “Alohamora62442” was printed to the console.
To summarize, determined attackers can exploit weaknesses in CI/CD pipelines to gain access to sensitive secrets like service account keys. This highlights the critical need for more secure authentication mechanisms that minimize reliance on long-lived credentials.
Wrap Up
That brings us to the end of this post where we reviewed the security risks associated with long-lived credentials. I’ll see you over in part two where I’m going to implement an authentication method that’s more secure than using service account keys: Workload Identity Federation.
Additional Resources
- Pipeline Pandemonium: How to Hijack the Cloud and Make it Rain Insecurity – Blake Hudson
- Best practices for managing service account keys – Google Cloud
- Best practices for using service accounts – Google Cloud