Hardening Kubernetes security requires a layered approach: enforce RBAC with least-privilege roles, restrict pod-to-pod communication using NetworkPolicy resources, prevent containers from running as root with Pod Security Admission, scan infrastructure-as-code templates for misconfigurations before deployment, and monitor running containers in real time for suspicious behavior. This guide covers each layer with practical YAML examples and tool recommendations including Falco, Checkov, and Kubernetes-native security controls.<\/p>\n\n\n\n
Strengthening Deployments and Runtime Protection<\/em><\/strong><\/p>\n\n\n\n In Part 1<\/a> of this series, we explored the foundational aspects of securing a DevOps pipeline, including CI\/CD security best practices, image scanning, enforcing signed images, restricting pipeline permissions, and managing secrets effectively. These strategies focused on protecting the software supply chain from build-time threats and credential leaks.<\/p>\n\n\n\n However, securing the CI\/CD pipeline is only the beginning. Once applications are deployed, they need strong security controls within Kubernetes and cloud environments to prevent lateral movement, privilege escalation, and runtime threats.<\/p>\n\n\n\n In Part 2, we shift our focus to securing containerized deployments and cloud infrastructure. We will cover Kubernetes security hardening, real-time monitoring, infrastructure as code (IaC) security, and shifting security left to catch misconfigurations early. These strategies will help your team secure workloads in production and respond to potential threats before they lead to a security breach.<\/p>\n\n\n\n In one of our company projects, an engineer mistakenly granted a container unnecessary root privilege<\/strong>, which allowed an attacker to escalate their access and compromise the system. This incident reinforced the importance of restricting permissions and controlling communication between containers in Kubernetes.<\/strong><\/p>\n\n\n\n Containers<\/a> and Kubernetes<\/a> environments require special attention to security. There are several basic strategies that you need to understand when you are hardening your container\/Kubernetes environments against internal and external threats:<\/p>\n\n\n By default, Kubernetes allows all pods to communicate freely, which can expose sensitive services to unintended access. To enforce stricter security, we can use a The following Kubernetes This Kubernetes NetworkPolicy<\/a> restricts communication between pods. It targets “frontend” pods ( The policy uses the In one of our deployments, a developer accidentally launched a container with root privileges<\/strong>, which exposed the system to potential privilege escalation attacks. Running containers as root is a major security risk<\/strong>, as it allows attackers to gain control over the host machine.<\/p>\n\n\n\n To mitigate this, Kubernetes provides Pod Security Policies (PSP)<\/strong> and Pod Security Admission (PSA)<\/strong> to enforce strict security controls on pod execution.<\/p>\n\n\n\n The following Apply this policy to your cluster to enforce security standards. This Kubernetes Additionally, it ensures that the container’s root filesystem is read-only ( When you apply this security policy, organizations can enforce container security best practices and reduce the risk of privilege escalation attacks. While PSP is deprecated in Kubernetes 1.21+, its successor, Pod Security Admission (PSA), provides similar enforcement mechanisms for restricting insecure configurations.<\/p>\n\n\n\n In a past project, a misconfigured RBAC policy accidentally gave all developers full admin privileges<\/strong> on a Kubernetes cluster. This led to accidental deletions, unexpected configuration changes, and security risks<\/strong>. While RBAC provides a structured way to define access controls, it does not inherently prevent misconfigurations. Instead, combining RBAC with regular policy reviews, automated scanning tools, and least-privilege enforcement helps reduce the risk of mistakes like this.<\/p>\n\n\n\n To limit unintended privilege escalation, RBAC should be carefully implemented alongside policy validation tools that detect misconfigurations before they become security risks. Additionally, organizations should regularly audit roles, enforce strict access policies, and implement monitoring solutions to catch excessive permissions early.<\/p>\n\n\n\n How to Lock Down Permissions:<\/strong><\/p>\n\n\n Imagine you have a development team where certain users only need to view resources in the production namespace but should not be able to modify anything. To enforce this, you can define a Here\u2019s how you can do it:<\/p>\n\n\n\n This The This RBAC policy prevents unauthorized changes<\/strong> while still allowing users to access the information they need.<\/p>\n\n\n\n Real-time monitoring and alerting are essential for detecting suspicious activity in a Kubernetes cluster. Falco<\/a>, an open-source runtime security tool, helps detect unexpected behaviour by monitoring system calls and alerting on potential security violations.<\/p>\n\n\n\n Suppose you want to monitor your cluster for unauthorized container activity, such as unexpected privilege escalations or file access attempts. You can quickly deploy Falco using Helm, which simplifies installation and management.<\/p>\n\n\n\n Here\u2019s how you can install Falco in your cluster:<\/p>\n\n\n\n Once installed, Falco starts monitoring system calls and detecting policy violations. Below is an example of Falco detecting a privilege escalation attempt:<\/p>\n\n\n\n You can customize Falco\u2019s detection rules to match your security requirements, filtering out false positives and tailoring alerts to your environment. Integrating Falco with logging solutions like Elasticsearch, Grafana, or SIEM tools further enhances visibility and incident response capabilities.<\/p>\n\n\n\n Tracking security events in your Kubernetes cluster is crucial for detecting unauthorized changes and ensuring compliance. Audit logging<\/strong> helps you monitor actions such as the creation, deletion, and modification of key resources. By defining an audit policy<\/strong>, you can control which events are logged and at what level of detail.<\/p>\n\n\n\n Suppose you want to log changes to critical Kubernetes resources\u2014such as Pods, Deployments, and Services\u2014to detect potential security threats or misconfigurations. The following audit policy helps you achieve this:<\/p>\n\n\n\n The Log Entry for Creating a Pod<\/strong><\/p>\n\n\n\n Log Entry for Deleting a Deployment<\/strong><\/p>\n\n\n\n Log Entry for Updating a Service<\/strong><\/p>\n\n\n\n Log Entry for Patching a Pod<\/strong><\/p>\n\n\n\n Even with strong security measures in place, incidents can still occur. I once dealt with a situation where an application was exploited, but due to inadequate monitoring, we didn\u2019t detect the breach for hours. Real-time threat detection and automated response are crucial<\/strong> for minimizing damage and improving security resilience.<\/p>\n\n\n\n The following are a few strategies and tools that you can employ when monitor your DevOps pipelines.<\/p>\n\n\n To quickly detect and respond to suspicious activity in your AWS environment, you can use AWS GuardDuty, <\/strong>a threat detection service that continuously monitors for malicious activity<\/strong>. The following command enables GuardDuty in your AWS account:<\/p>\n\n\n\n Once enabled, GuardDuty continuously scans AWS CloudTrail logs, VPC Flow Logs, and DNS logs for anomalies, such as:<\/p>\n\n\n Here\u2019s an example output when GuardDuty detects suspicious activity:<\/p>\n\n\n\n In this example, GuardDuty has detected unauthorized API calls made by a compromised IAM (AWS Identity and Access Management) user attempting to list S3 buckets. The finding includes details such as:<\/p>\n\n\n When you integrate GuardDuty with AWS Security Hub or SIEM tools, you can automate responses to detected threats\u2014such as isolating compromised instances or triggering alerts to security teams.<\/p>\n\n\n\n This proactive monitoring approach ensures that security incidents are detected and addressed swiftly, minimizing potential damage.<\/p>\n\n\n\n The concept and methods of Infrastructure as Code<\/a> (IaC) are powerful, yet come with inherent risks\u2014just as it streamlines deployments, it can just as easily propagate security misconfigurations at scale. I once encountered a Terraform script that, due to a single oversight, inadvertently made an S3 bucket publicly accessible, exposing sensitive data to the internet. This incident reinforced the importance of proactive security scanning<\/strong> in IaC workflows.<\/p>\n\n\n\n The following are a few tools and techniques to help you to secure your IaC environment.<\/p>\n\n\n When working with Terraform, it\u2019s critical to catch security misconfigurations before <\/strong>applying changes. tfsec<\/a> is a static analysis tool that scans Terraform code for potential security risks.<\/p>\n\n\n\n This command checks for insecure configurations in Terraform<\/a> files and provides actionable insights to fix them.<\/p>\n\n\n\n For example, if a Terraform script accidentally sets an S3 bucket to public:<\/p>\n\n\n\n Integrating Early in my DevOps career, I made the mistake of assuming security checks at deployment were enough. The reality? Fixing security issues at deployment is costly and time-consuming<\/strong>. The key is to shift security left<\/strong>\u2014integrating security into every phase of the DevOps pipeline, from code commit to production.<\/strong><\/p>\n\n\n\n Some techniques and tools to employ:<\/p>\n\n\n To prevent hardcoded secrets and misconfigurations<\/strong> from being committed, set up a pre-commit hook<\/strong> in your repository.<\/p>\n\n\n\n By adding a This ensures no secrets or misconfigurations are committed:<\/p>\n\n\nContainer and Kubernetes Security: Hardening Your Deployments<\/h2>\n\n\n\n
Explore: <\/strong>Deploying Docker apps to Kubernetes with Jenkins<\/a><\/p>\n\n\n\nStrategies for Containers and Kubernetes Security<\/h3>\n\n\n\n
\n
Kubernetes Network Policy to Restrict Pod Communication<\/h3>\n\n\n\n
NetworkPolicy<\/code> to limit which pods can talk to each other.<\/p>\n\n\n\nNetworkPolicy<\/code> ensures that only frontend pods can communicate with backend pods, blocking all other inbound connections:<\/p>\n\n\n\napiVersion: networking.k8s.io\/v1\nkind: NetworkPolicy\nmetadata:\n name: restrict-internal-traffic\nspec:\n podSelector:\n matchLabels:\n app: frontend\n policyTypes:\n - Ingress\n ingress:\n - from:\n - podSelector:\n matchLabels:\n app: backend<\/pre>\n\n\n\n
app: frontend<\/code>) and allows incoming traffic only from “backend” pods (app: backend<\/code>).<\/p>\n\n\n\nIngress<\/code> type, meaning it controls incoming traffic, ensuring that “frontend” pods can only receive traffic from “backend” pods, improving security by isolating other pods. When you limit pod communication, <\/strong>this policy reduces attack surfaces <\/strong>and prevents unauthorized access <\/strong>between services. It\u2019s an essential step in hardening Kubernetes deployments <\/strong>and mitigating lateral movement in the event of a breach.<\/p>\n\n\n\nUse Pod Security Policies (PSP) or Pod Security Admission (PSA)<\/h3>\n\n\n\n
Example: Enforcing a Restricted Pod Security Policy<\/h4>\n\n\n\n
PodSecurityPolicy<\/code> prevents containers from running as root and enforces additional security restrictions:<\/p>\n\n\n\napiVersion: policy\/v1beta1\nkind: PodSecurityPolicy\nmetadata:\n name: restricted\nspec:\n privileged: false\n allowPrivilegeEscalation: false\n runAsUser:\n rule: MustRunAsNonRoot\n readOnlyRootFilesystem: true\n volumes:\n - 'configMap'\n - 'emptyDir'\n - 'secret'<\/pre>\n\n\n\n
What This Policy Does<\/h4>\n\n\n\n
PodSecurityPolicy<\/code> (PSP) configuration enhances the security of your cluster by preventing containers from running with elevated privileges. Specifically, it enforces that containers cannot run as root (runAsUser: MustRunAsNonRoot<\/code>) and blocks privilege escalation (allowPrivilegeEscalation: false<\/code>).<\/p>\n\n\n\nreadOnlyRootFilesystem: true<\/code>), which reduces the risk of unauthorized modifications. The volumes<\/code> section restricts the types of volumes that can be mounted, allowing only configMap, emptyDir<\/code>, and secret<\/code> to limit access to potentially sensitive data. This policy helps mitigate security risks associated with running containers with excessive permissions.<\/p>\n\n\n\nWhy This Matters<\/h4>\n\n\n\n
Carefully Employ Role-Based Access Control (RBAC)<\/h3>\n\n\n\n
\n
Example: Restricted role binding<\/h4>\n\n\n\n
RoleBinding<\/code> that assigns a read-only role to a specific developer.<\/p>\n\n\n\napiVersion: rbac.authorization.k8s.io\/v1\nkind: RoleBinding\nmetadata:\n name: DeveloperReadAccess\n namespace: production\nsubjects:\n- kind: User\n name: developer-read-access\n apiGroup: rbac.authorization.k8s.io\nroleRef:\n kind: Role\n name: read-only\n apiGroup: rbac.authorization.k8s.io<\/pre>\n\n\n\n
RoleBinding<\/code> example defines a restricted role for a specific user, developer-read-access<\/code> in the production<\/code> namespace. It grants the user access with a read-only<\/code> role, meaning the user can only view resources and cannot modify them.<\/p>\n\n\n\nroleRef<\/code> section specifies that the user is assigned the read-only<\/code> role, which is a predefined role within the rbac.authorization.k8s.io<\/code> API group. By using this role binding, you ensure that users are only granted the minimum permissions required for their tasks, adhering to the principle of least privilege. This setup enhances security by limiting access to critical resources.<\/p>\n\n\n\nSimple Talk is brought to you by Redgate Software<\/h3>\n
Deploy Security Monitoring and Incident Response Tools<\/h3>\n\n\n\n
Example: Deploying Falco for Runtime Security Monitoring<\/h4>\n\n\n\n
helm repo add falcosecurity https:\/\/falcosecurity.github.io\/charts\nhelm install falco falcosecurity\/falco<\/pre>\n\n\n\n
11:34:21.123456123: Warning Privilege Escalation Detected (user=root, process=sudo) \n11:34:22.654321654: Notice Unexpected File Modification (\/etc\/passwd edited by unknown process) <\/pre>\n\n\n\n
Set Up Kubernetes Audit Logging<\/h3>\n\n\n\n
Example: Configuring an Audit Policy to Track Key Events<\/h4>\n\n\n\n
apiVersion: audit.k8s.io\/v1\nkind: Policy\nrules:\n - level: Metadata\n verbs: [\"create\", \"delete\", \"update\", \"patch\"]\n resources:\n - group: \"\"\n resources: [\"pods\", \"deployments\", \"services\"]<\/pre>\n\n\n\n
Breakdown of the Audit Policy ConfigurationSpecify the Audit API Version and Policy Kind<\/h4>\n\n\n
\n
apiVersion: audit.k8s.io\/<\/code>v1 ensures compatibility with the Kubernetes audit logging system.<\/li>\n\n\n\nkind: Policy<\/code> defines an audit logging policy.<\/li>\n<\/ul>\n<\/div>\n\n\nLogging Level<\/h4>\n\n\n
\n
level: Metadata<\/code> setting ensures that Kubernetes logs metadata about API requests but not full request\/response bodies.<\/li>\n\n\n\nMonitor Key Kubernetes Events<\/h4>\n\n\n
\n
[\"create\", \"delete\", \"update\", \"patch\"]<\/code> section ensures that any changes to resources are logged.<\/li>\n<\/ul>\n<\/div>\n\n\nresources<\/code> list targets Pods, Deployments, and Services, which are critical for cluster operations and security. Below are sample log entries for each action (create, delete, update and patch):<\/p>\n\n\n\n{\n \"kind\": \"Event\",\n \"apiVersion\": \"audit.k8s.io\/v1\",\n \"level\": \"Metadata\",\n \"verb\": \"create\",\n \"user\": \"admin\",\n \"objectRef\": {\n \"resource\": \"pods\",\n \"namespace\": \"production\",\n \"name\": \"nginx-pod\"\n },\n \"timestamp\": \"2025-03-06T12:45:30Z\"\n}<\/pre>\n\n\n\n{\n \"kind\": \"Event\",\n \"apiVersion\": \"audit.k8s.io\/v1\",\n \"level\": \"Metadata\",\n \"verb\": \"delete\",\n \"user\": \"devops-engineer\",\n \"objectRef\": {\n \"resource\": \"deployments\",\n \"namespace\": \"staging\",\n \"name\": \"backend-service\"\n },\n \"timestamp\": \"2025-03-06T13:15:22Z\"\n}<\/pre>\n\n\n\n{\n \"kind\": \"Event\",\n \"apiVersion\": \"audit.k8s.io\/v1\",\n \"level\": \"Metadata\",\n \"verb\": \"update\",\n \"user\": \"automation-bot\",\n \"objectRef\": {\n \"resource\": \"services\",\n \"namespace\": \"production\",\n \"name\": \"payment-gateway\"\n },\n \"timestamp\": \"2025-03-06T14:05:10Z\"\n}<\/pre>\n\n\n\n{\n \"kind\": \"Event\",\n \"apiVersion\": \"audit.k8s.io\/v1\",\n \"level\": \"Metadata\",\n \"verb\": \"patch\",\n \"user\": \"sre-team\",\n \"objectRef\": {\n \"resource\": \"pods\",\n \"namespace\": \"dev\",\n \"name\": \"web-app\"\n },\n \"timestamp\": \"2025-03-06T15:30:45Z\"\n}<\/pre>\n\n\n\nMonitoring and Incident Response: Always Be Ready<\/h2>\n\n\n\n
\n
Example: Setting Up AWS GuardDuty for Threat Detection<\/h3>\n\n\n\n
aws guardduty create-detector \u2013enable<\/pre>\n\n\n\n
How This Works<\/h4>\n\n\n
\n
aws guardduty create-detector<\/code> \u2013 This command initializes GuardDuty, enabling it to analyze logs and detect potential threats.<\/li>\n\n\n\n--enable<\/code> \u2013 Ensures that GuardDuty starts monitoring right away without requiring additional configuration.<\/li>\n<\/ul>\n<\/div>\n\n\n\n
Example GuardDuty Findings<\/h4>\n\n\n\n
{\n \"schemaVersion\": \"2.0\",\n \"accountId\": \"123456789012\",\n \"region\": \"us-east-1\",\n \"id\": \"abcd1234-ef56-7890-gh12-ijklmnopqrst\",\n \"type\": \"UnauthorizedAccess:IAMUser\/AnomalousBehavior\",\n \"severity\": 6.0,\n \"createdAt\": \"2025-03-06T14:10:00Z\",\n \"updatedAt\": \"2025-03-06T14:15:00Z\",\n \"resource\": {\n \"resourceType\": \"AWS::IAM::User\",\n \"accessKeyDetails\": {\n \"userName\": \"compromised-user\",\n \"accessKeyId\": \"AKIAEXAMPLE\"\n }\n },\n \"service\": {\n \"serviceName\": \"guardduty\",\n \"eventFirstSeen\": \"2025-03-06T14:05:00Z\",\n \"eventLastSeen\": \"2025-03-06T14:10:00Z\",\n \"action\": {\n \"actionType\": \"AWS_API_CALL\",\n \"apiCallDetails\": {\n \"api\": \"ListBuckets\",\n \"serviceName\": \"s3.amazonaws.com\",\n \"remoteIpDetails\": {\n \"ipAddressV4\": \"203.0.113.42\",\n \"country\": \"Unknown\"\n }\n }\n }\n }\n}<\/pre>\n\n\n\n\n
Infrastructure as Code (IaC) Security<\/h2>\n\n\n\n
\n
Example: Using tfsec for Terraform Security Scanning<\/h3>\n\n\n\n
tfsec .\/terraform<\/pre>\n\n\n\n
What this does<\/h4>\n\n\n
\n
tfsec .\/terraform<\/code> \u2013 Runs a security scan on all Terraform configuration files in the .\/terraform<\/code> directory.<\/li>\n\n\n\nresource \"aws_s3_bucket_public_access_block\" \"example\" {\n bucket = aws_s3_bucket.example.id\n block_public_acls = false # Security risk: Public ACLs allowed\n}<\/pre>\n\n\n\ntfsec<\/code> will flag this misconfiguration and suggest enforcing stricter access controls.<\/p>\n\n\n\nWhy This Matters<\/h4>\n\n\n\n
tfsec<\/code> into CI\/CD pipelines ensures that security best practices are applied before deployment, reducing the risk of exposing sensitive infrastructure due to simple configuration errors.<\/p>\n\n\n\nShifting Security Left: Catching Issues Early<\/h2>\n\n\n\n
\n
Example: Enforcing Security with Pre-commit Hooks<\/h3>\n\n\n\n
Step 1: Add a
.pre-commit-config.yaml <\/code>file<\/h4>\n\n\n\n# .pre-commit-config.yaml\nrepos:\n - repo: https:\/\/github.com\/pre-commit\/pre-commit-hooks\n rev: v3.4.0\n hooks:\n - id: detect-secrets\n - id: check-yaml<\/pre>\n\n\n\n
.pre-commit-config.yaml<\/code> file, you can specify repositories and hooks to run before committing code. In this case, the configuration adds the pre-commit-hooks<\/code> repository and specifies two hooks: detect-secrets<\/code> to identify hardcoded secrets in the code and check-yaml<\/code> to validate YAML file syntax. These checks are run automatically whenever you try to commit, helping prevent sensitive data from being committed and ensuring code quality.<\/p>\n\n\n\nStep 2: Install and Enable Pre-commit<\/h4>\n\n\n\n
# .pre-commit-config.yaml\nrepos:\n - repo: https:\/\/github.com\/pre-commit\/pre-commit-hooks\n rev: v3.4.0\n hooks:\n - id: detect-secrets # Scans for hardcoded secrets\n - id: check-yaml # Ensures YAML syntax is valid<\/pre>\n\n\n\n
\n
pip install pre-commit<\/code> \u2013 Installs the pre-commit tool.<\/li>\n\n\n\npre-commit install<\/code> \u2013 Enables hooks in your repository.<\/li>\n\n\n\npre-commit run --all-files<\/code> \u2013 Runs security checks on all files.<\/li>\n<\/ul>\n<\/div>\n\n\n