Protecting Your CI/CD from Build-Time Threats

DevOps has transformed software delivery, but with rapid deployments come increased security risks. As a DevOps engineer, I’ve seen firsthand how small security oversights can lead to major vulnerabilities.

In this article, I’ll share real-world experiences—some hard-learned lessons—and provide actionable strategies and tools to secure your DevOps pipeline effectively.

Understanding DevOps Security Risks

Security is often overlooked when implementing DevOps pipelines, where speed and automation take priority. However, failing to address security from the start can lead to serious consequences. I once encountered a scenario where a misconfigured CI/CD pipeline inadvertently exposed sensitive API keys in a public repository. Within hours, unauthorized actors attempted to exploit the credentials, emphasizing just how quickly security oversights can be targeted.

So, what exactly are we securing against? DevOps pipelines often handle highly sensitive information, including customer data, application secrets, and infrastructure configurations.

Whether deploying to on-premise environments, hybrid setups, or the cloud, security threats come from multiple directions—both internal and external. Key risks include:

Now that we have covered some of the threats that can affect any DevOps pipeline and workflows, let’s get started with these tools and strategies for safer deployments

Secure Your CI/CD Pipelines

Your CI/CD pipeline is a critical entry point for attackers. If your pipeline is compromised, you risk deploying malicious artifacts directly into production, which could expose sensitive data, disrupt services, damage customer trust and financial loss.

For our CI/CD Pipelines, what are we securing against?

I’ve personally seen how seemingly small oversights can lead to security incidents. In one case, a build process pulled an outdated package from an external repository, introducing a known vulnerability into production.

In another instance, a misconfigured pipeline exposed sensitive logs containing API keys, which could have been exploited for unauthorized access. These experiences reinforced the importance of proactive security measures in CI/CD pipelines.

Here’s how to secure your CI/CD pipelines effectively from these threats coming from inside and outside of your organization:

Implement Image Scanning in CI/CD

Container security is crucial in modern DevOps workflows. A single vulnerable image can expose your entire infrastructure to attacks. One of my deployments once suffered from vulnerabilities hidden in container images. We deployed an image with an outdated OpenSSL version, which exposed our cluster to potential exploits. Since then, I always integrate image scanning into CI/CD pipelines to catch vulnerabilities before deployment.

To prevent such issues, you can use tools like Trivy, Anchore, or Clair to scan images for vulnerabilities before pushing them to production. These tools help automate security checks, ensuring that vulnerable images never make it to your registry.

Running a Basic Trivy Scan

This command analyzes the my-app:latest container image and reports any detected vulnerabilities. If issues are found, you can take corrective action before the image is used in a production environment.

Output of a Trivy Scan

Below is an example of Trivy’s output when scanning an image with vulnerabilities:

This output provides useful details, including:

Example: Integrating Trivy in a CI/CD pipeline (GitHub Actions)

To automate vulnerability scanning in your CI/CD pipeline, you can integrate Trivy directly into GitHub Actions. This ensures that every new commit triggers an image scan before deployment.

The following workflow sets up Trivy in a GitHub Actions pipeline. It builds upon the previous example where we scanned an image manually using trivy image my-app:latest. Now, we automate this process so that every push runs a security scan on the latest Docker image.

This GitHub Actions workflow triggers on every push event, initiating the security scan of the Docker image. The first step, actions/checkout@v2, checks out the repository code, making it ready for subsequent steps. The second step uses the Trivy Action (aquasecurity/trivy-action@master), which scans the specified image (your-image:latest) for vulnerabilities.

The results are displayed in a table format, making it easy to view any issues before the image is deployed. Integrating image scanning in CI/CD workflows ensures that vulnerabilities are identified early and not pushed to production, reducing security risks in your deployments.

Example output of a CI/CD Trivy Scan

Once the workflow executes, the output looks like this:

In this case, Trivy has detected a medium-severity vulnerability in libxml2. If left unpatched, this issue could potentially allow an attacker to exploit XML parsing weaknesses, leading to unauthorized access or data leaks. To mitigate this risk, the recommended action is to update libxml2 to a patched version using the latest package manager update or selecting a secure base image.

Enforce Signed and Trusted Images

Ensuring that only trusted images make it into your production environment is critical for supply chain security. Attackers can inject malicious images into your pipeline if you don’t enforce image provenance, potentially leading to compromised applications and unauthorized access.

To mitigate this risk, you should implement image signing and verification in your CI/CD workflows. Cosign, part of the Sigstore project, allows you to sign images with cryptographic keys and verify their authenticity before deployment.

Use the code like the following to sign an image using Cosign:

Then, once you have signed it, use the following code to verify the signed image:

The cosign sign command signs the Docker image (my-app:latest) using a private key (cosign.key), ensuring that the image has been authorized by the owner. This step guarantees that the image has not been tampered with or replaced by an unauthorized party.

The cosign verify command checks the signature of the image using the corresponding public key (cosign.pub). If the image has been altered in any way or the signature does not match, this verification step will fail, alerting you to potential security risks.

Restricting Pipeline Permissions

A common security misconfiguration in CI/CD pipelines is running them with excessive privileges. This can lead to unintended access to sensitive resources or even allow attackers to escalate their privileges. To mitigate this risk, it’s best to follow the principle of least privilege by restricting access wherever possible.

Best practices for restricting CI/CD pipeline permissions include:

To illustrate this, here’s an example of how to create a dedicated Kubernetes service account for GitHub Actions:

In the example provided, we are creating a service account in Kubernetes specifically for GitHub Actions. The apiVersion: v1 defines the version of the Kubernetes API that the configuration uses, and kind: ServiceAccount indicates that you are creating a service account resource.

The metadata section defines the name (github-actions-sa) of the service account and the namespace (ci-pipeline) where it will reside

Secure the CI/CD Pipeline with Policy as Code

One of the most effective ways I’ve found to enforce security in your CI/CD pipeline is through policy as code. When you use tools like Open Policy Agent (OPA) and Kyverno, you can define security policies that are automatically enforced throughout the deployment process.

In one of my projects, I once deployed a CI/CD pipeline that lacked stringent policies, leading to an insecure deployment. The pipeline allowed excessive permissions for service accounts, which meant that a compromised build process could modify critical infrastructure resources. Additionally, there were no restrictions on container images, allowing unverified or potentially malicious images to be deployed into production. This oversight introduced serious security risks, including the possibility of privilege escalation and supply chain attacks.

We later integrated OPA with Kubernetes Admission Controllers to enforce security policies on incoming manifests, preventing deployments with excessive privileges. The impact was immediate, misconfigurations were caught before they reached production.

Practical Code Example

Imagine you are responsible for securing a Kubernetes cluster and want to ensure that only signed container images are deployed. Without proper enforcement, developers might inadvertently—or maliciously—use unsigned or unverified images, increasing the risk of supply chain attacks.

To mitigate this, you can use Kyverno to enforce a policy that only allows signed images from a trusted repository. This ensures that all container images running in production are verified and have not been modified by unauthorized sources.

Here’s how you can achieve this using Kyverno:

Here’s what this policy accomplishes:

In the provided practical code example, we are using Kyverno to enforce container image security. The apiVersion: kyverno.io/v1 specifies the version of the Kyverno API being used, and kind: ClusterPolicy indicates that a cluster-wide policy is being created. The metadata section defines the policy name (require-signed-images).

Under the spec section, the validationFailureAction: Enforce setting ensures that if a resource does not meet the policy criteria, it is rejected.

The rules section defines the policy itself, where the name: check-image-signature rule applies to all Pod resources.

The policy then validates that the containers in the pods are using images signed by a trusted repository, in this case, gcr.io/signed-repo/*. If the container image does not match the required pattern, the policy will prevent the deployment from proceeding, thus ensuring that only signed images are used in production.

Security misconfigurations in Kubernetes often stem from human error, such as using unverified container images. A policy like this eliminates that risk by automating security enforcement at the cluster level. Even if a developer mistakenly tries to deploy an unsigned image, the policy blocks it, ensuring that only trusted, validated software enters production. This approach minimizes vulnerabilities and strengthens the overall security posture of your Kubernetes environment.

Secrets Management: Avoid Hardcoded Credentials

Where I work as a DevOps engineer, I once witnessed a deployment failure because an API key was accidentally committed to a public repository. Within minutes, attackers started abusing it. This incident highlighted a critical lesson—robust secrets management is non-negotiable to prevent credential leaks and unauthorized access.

To avoid this issue and ensure API keys are correctly assigned to the right environment (e.g., production vs. development), teams should implement the following tools and strategies for secrets management.

Tools & Strategies for Secrets Management

Use HashiCorp Vault for Secrets Management

A secure way to manage API keys and other secrets is by storing them in HashiCorp Vault, which acts as a centralized, encrypted storage system. Instead of hardcoding credentials in your application code, Vault enables secure retrieval and access control for secrets.

Example: Storing and Retrieving Secrets with HashiCorp Vault

Let’s say we need to store and access an API key securely. First, we store the secret in Vault. You can use this command to store your secrets in Vault:

This command stores the secret API key (my-super-secret-key) in Vault under the path secret/api-key.Now, to retrieve this secret programmatically in Python, we can use the hvac library:

What This Code Does

When you use Vault instead of hardcoding credentials, we eliminate the risk of exposing API keys in code repositories. Additionally, Vault’s access control mechanisms ensure that only authorized applications and users can retrieve secrets, further strengthening security.

Use GitGuardian CLI to Scan for Secrets

Accidentally committing secrets—like API keys or database credentials— to your user repository can lead to security breaches, unauthorized access, and even financial loss. To prevent this, teams should proactively scan their code for exposed secrets before pushing changes to a repository.

One effective approach is to integrate GitGuardian CLI (ggshield) into your workflow. GitGuardian helps detect hardcoded secrets in real time, preventing accidental leaks before they reach a remote repository.

Example: Scanning for Secrets with GitGuardian CLI

First, install GitGuardian CLI:

Next, run a pre-commit scan to check for secrets before committing code:

What This Does

Why This Matters

When developers integrate GitGuardian CLI into pre-commit hooks, they can automate secret scanning and prevent security incidents before they happen. This ensures that sensitive data never leaves a developer’s machine, reducing the risk of credential leaks in Git repositories.

Final Thoughts

Securing a DevOps pipeline requires proactive measures at every stage of software delivery. As we’ve covered in this Part 1, CI/CD environments present multiple attack vectors, from credential leaks and insider threats to supply chain attacks and insecure dependencies. Without the right security controls, a single misconfiguration can expose your entire infrastructure.

When teams implement these strategies, they can significantly reduce risks before deployment. Additionally, proper secrets management with tools like HashiCorp Vault and GitGuardian CLI helps prevent accidental credential leaks, ensuring sensitive data remains protected.

However, security doesn’t stop at CI/CD. Even a perfectly secured pipeline won’t protect workloads once they are deployed into Kubernetes or the cloud. Runtime security, access controls, and monitoring become the next critical layers of defense.

In Part 2, we will focus on hardening Kubernetes deployments, enforcing strict access control, securing runtime environments, and implementing real-time threat monitoring. With these additional measures, you can extend security beyond the pipeline and ensure your cloud-native infrastructure remains resilient against evolving threats.

Let’s move forward and get started with Part 2: Hardening Kubernetes and Cloud Security – Strengthening Deployments and Runtime Protection.