Securing and Monitoring AKS

Securing and Monitoring Azure Kubernetes Service (AKS)

Introduction: Why AKS Security Matters

As organizations transition from traditional virtual machine-based architectures to containerized environments, the complexity of managing security increases exponentially. Azure Kubernetes Service (AKS) provides a managed environment for deploying and scaling containers, but the responsibility for securing the cluster remains a shared effort between the cloud provider and the operator. Security in AKS is not a one-time configuration; it is a continuous lifecycle of hardening, monitoring, and auditing.

Why is this so critical? In a standard virtual machine, you might only worry about the OS and the application. In Kubernetes, you have the control plane, the worker nodes, the container runtime, the networking stack, and the application code itself. A single misconfiguration in a Kubernetes manifest or a overly permissive role-based access control (RBAC) policy can lead to lateral movement, data exfiltration, or the compromise of the entire cluster. This lesson will guide you through the essential layers of securing AKS, from the API server to the pods running your workloads.

1. Hardening the Control Plane and API Server

The Kubernetes API server is the brain of your cluster. It is the entry point for all administrative commands, whether from the kubectl CLI or automated CI/CD pipelines. If an attacker gains unauthorized access to your API server, they essentially own the cluster.

Restricting Access to the API Server

By default, the AKS API server has a public endpoint. While this is convenient for development, it is a significant security risk in production environments. You should always aim to restrict access to this endpoint.

• Authorized IP Ranges: You can limit access to the API server to specific IP addresses (e.g., your corporate VPN or office network). This prevents attackers from even attempting to authenticate from unknown locations.
• Private Clusters: For high-security requirements, you should deploy a Private AKS cluster. In this configuration, the API server is accessible only via a private IP address within your virtual network, effectively removing it from the public internet entirely.

Callout: Public vs. Private API Endpoints A public API endpoint is exposed to the internet, relying entirely on RBAC and authentication to keep it secure. A private endpoint uses Azure Private Link to ensure that traffic stays on the Microsoft backbone network, making it invisible to the public internet and significantly reducing the attack surface.

RBAC and Identity Management

Role-Based Access Control (RBAC) is the primary mechanism for controlling who can do what in your cluster. You should adhere to the principle of least privilege, ensuring that users and service accounts only have the permissions necessary to perform their specific tasks.

Instead of using the default cluster-admin role, create custom roles that scope permissions to specific namespaces. Furthermore, integrate Azure Active Directory (Azure AD) with AKS. This allows you to manage Kubernetes access using your existing corporate identities, enabling features like multi-factor authentication (MFA) and conditional access policies.

2. Securing the Node Level

While the control plane manages the cluster, the worker nodes are where your actual containers execute. Securing these nodes is just as important as securing the API server.

Choosing the Right OS

AKS supports both Linux and Windows nodes. For most workloads, Linux is the standard. However, you should use an optimized, container-focused operating system like Azure Linux (CBL-Mariner) or Ubuntu. These distributions have a smaller footprint and fewer installed packages, which reduces the number of potential vulnerabilities (the "attack surface").

Node Auto-Upgrade and Security Patching

Security vulnerabilities are discovered in Linux kernels and container runtimes regularly. If you do not patch your nodes, you leave the cluster open to known exploits.

• Automatic Upgrades: Enable automatic node image upgrades in AKS. This ensures that your nodes are regularly patched with the latest security updates from Microsoft.
• Planned Maintenance: Use maintenance windows to ensure that upgrades do not disrupt your business-critical applications.

Warning: Ignoring Node Patches Failing to patch your worker nodes is one of the most common ways clusters are compromised. An attacker who gains a foothold in a container can use a kernel exploit to "break out" of the container and gain root access to the host node. Always keep your node images updated.

3. Network Security Policies

Kubernetes, by default, allows all pods to communicate with all other pods in the cluster. This "flat network" model is dangerous. If one container is compromised, the attacker can easily scan the rest of the cluster for other vulnerabilities.

Implementing Network Policies

Network policies act like a firewall for your pods. You can define rules that control traffic flow based on labels, namespaces, and port numbers.

Example: Deny All Traffic by Default You should start by creating a "Default Deny" policy for every namespace. This ensures that no traffic is allowed unless explicitly permitted.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

Example: Allowing Traffic Between Specific Services Once you have a default deny policy, you can selectively allow traffic. For example, you might want to allow your frontend pods to talk to your backend pods, but nothing else.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

4. Container Image Security

The security of your cluster is only as good as the code running inside it. If you use an image with a known vulnerability, the rest of your security configuration might be bypassed.

Container Registry Best Practices

• Private Registries: Never pull images from public repositories without vetting them. Use Azure Container Registry (ACR) to store and manage your images.
• Image Scanning: Enable vulnerability scanning in ACR (often via Microsoft Defender for Containers). This automatically scans your images for known CVEs (Common Vulnerabilities and Exposures) and alerts you if a high-risk vulnerability is found.
• Content Trust: Use image signing to ensure that the images you pull have not been tampered with. This ensures that only images built by your trusted CI/CD pipelines can be deployed.

Tip: Minimize Image Size Use "distroless" images or multi-stage builds to create minimal container images. These images contain only your application and its dependencies, removing shells, package managers, and other tools that an attacker would find useful for reconnaissance after gaining access.

5. Monitoring and Auditing

You cannot secure what you cannot see. Monitoring in AKS involves two distinct layers: monitoring the cluster health and monitoring the security events.

Azure Monitor and Container Insights

Container Insights is the native way to monitor AKS. It collects metrics from the nodes, the Kubelet, and the pods. You should set up alerts for high CPU/memory usage, which can sometimes indicate a denial-of-service attack or a compromised crypto-mining container.

Kubernetes Audit Logs

The Kubernetes API server generates audit logs that record every request made to the cluster. These logs are invaluable for forensic analysis after a security incident.

• Enable Audit Logging: Ensure that you have enabled diagnostic settings to send your API server audit logs to an Azure Log Analytics Workspace.
• Analyze Logs: Use Kusto Query Language (KQL) to search for suspicious activities, such as an unusual number of failed login attempts or unauthorized attempts to list secrets.

Example KQL Query for Suspicious API Activity:

AzureDiagnostics
| where Category == "kube-audit"
| where responseStatus_code_d >= 400
| project TimeGenerated, verb_s, requestUri_s, user_s, responseStatus_code_d
| sort by TimeGenerated desc

This query helps you identify all failed requests to the API server, which is a great starting point for detecting brute-force attacks or misconfigured service accounts.

6. Secrets Management

One of the most common mistakes in Kubernetes is storing sensitive information like API keys, database passwords, or certificates in plain text within configuration files or environment variables.

Using Azure Key Vault

Kubernetes Secrets are, by default, only base64 encoded, not encrypted at rest (unless you specifically configure encryption at rest with a Customer Managed Key). For production, you should move your secrets out of the cluster and into a dedicated secrets manager.

• Azure Key Vault Provider for Secrets Store CSI Driver: This allows you to mount secrets stored in Azure Key Vault directly into your pods as volumes. The application reads the secret from the file system, and the secret never exists as a native Kubernetes object.
• Workload Identity: Use Azure AD Workload Identity to allow your pods to authenticate to Azure Key Vault using an Azure Managed Identity rather than using a static secret (like a Service Principal client secret).

7. Comparison: Traditional vs. Cloud-Native Security

8. Common Pitfalls and How to Avoid Them

Pitfall 1: Running Containers as Root

By default, many container images run as the root user. If an attacker compromises the application, they are already running as root inside the container.

• Solution: Always set a non-root user in your Dockerfile.

  RUN addgroup -S appgroup && adduser -S appuser -G appgroup
  USER appuser
  

Pitfall 2: Overly Permissive RBAC

Granting cluster-admin to every developer or service account is a recipe for disaster.

• Solution: Use Roles and RoleBindings to limit access to specific namespaces. Regularly audit your cluster roles to ensure they are still necessary.

Pitfall 3: Lack of Resource Limits

Without resource limits, a single pod can consume all the memory or CPU on a node, effectively crashing other pods (a self-inflicted Denial of Service).

• Solution: Always define resources.limits and resources.requests in your deployment manifests.

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

Pitfall 4: Exposing Services to the Internet Unnecessarily

Every LoadBalancer service creates a public IP address.

• Solution: Use an Ingress Controller (like NGINX or Azure Application Gateway Ingress Controller) to route traffic through a single entry point. Use internal load balancers for services that do not need to be public.

9. Step-by-Step: Securing a New AKS Cluster

If you are setting up a new cluster, follow this checklist to ensure a secure baseline:

1. Enable Microsoft Defender for Containers: This provides real-time threat protection for your AKS clusters and scans your container images.
2. Use Private Clusters: When creating the cluster, ensure the --enable-private-cluster flag is set to keep the API server off the public internet.
3. Enable Azure AD Integration: Use the --enable-azure-rbac flag to delegate Kubernetes authorization to Azure AD.
4. Configure Log Analytics: Create a workspace and enable diagnostic logs for the API server, Kubelet, and cluster autoscaler.
5. Set Up Network Policies: Choose azure or calico as your network policy engine during cluster creation.
6. Apply Security Contexts: Define securityContext in your pod specs to enforce runAsNonRoot: true and readOnlyRootFilesystem: true.

10. Advanced Concepts: Policy as Code

As your cluster grows, manual security reviews become impossible. "Policy as Code" allows you to automatically enforce security standards across the entire cluster.

Azure Policy for Kubernetes

Azure Policy for Kubernetes (built on Gatekeeper) allows you to define rules that are enforced by the Kubernetes admission controller. For example, you can create a policy that rejects any deployment that attempts to run a container as root or any deployment that does not have resource limits defined.

Example Policy Logic:

• "Deny all deployments that do not have a resource limit set."
• "Deny all deployments that pull images from untrusted registries."
• "Require all pods to have specific labels for cost tracking."

By using these policies, you move from "detective" security (finding issues after they happen) to "preventative" security (stopping issues before they are deployed).

11. Incident Response in AKS

Even with the best security, incidents can happen. Having a plan is essential.

1. Isolation: If a pod is compromised, use kubectl cordon and kubectl drain to remove the node from the cluster so it can be investigated without further traffic.
2. Forensics: Use kubectl cp to extract logs or files from the container for analysis.
3. Snapshotting: If using persistent volumes, take a snapshot of the disk attached to the node for offline analysis.
4. Recovery: Once the vulnerability is identified, patch the image or update the configuration and redeploy the workload.
5. Audit: Review the audit logs to determine the entry point of the attacker and close that gap.

12. Frequently Asked Questions (FAQ)

Q: Is Kubernetes "secure by default"? A: No. Kubernetes is designed for flexibility and ease of use, which often conflicts with security. It requires significant configuration to make it production-ready from a security standpoint.

Q: Should I use public images from Docker Hub? A: Only if you have a process to scan them. It is safer to pull public images into your own private ACR, scan them, and then deploy them from there.

Q: How do I handle emergency patches? A: Have a CI/CD pipeline that can trigger a deployment to all clusters within minutes. If you have to manually update each cluster, your response time will be too slow during a major security event.

Q: What is the difference between a Network Policy and an Azure Network Security Group (NSG)? A: NSGs operate at the virtual network level (IP/Port). Network Policies operate at the Kubernetes pod level (Label/Namespace). You need both: NSGs for broad infrastructure traffic and Network Policies for granular pod-to-pod communication.

Key Takeaways

1. Shared Responsibility: Understand that while Microsoft secures the underlying infrastructure of the AKS control plane, you are responsible for securing the pods, the network policies, and the identities used to access the cluster.
2. Least Privilege is Mandatory: Always use RBAC to limit permissions. Never use cluster-admin for daily tasks, and always use managed identities for pods to access Azure resources.
3. Default Deny: Start with a "deny all" networking posture and only open ports and paths that are explicitly required by your application architecture.
4. Visibility is Security: Enable diagnostic logs and use tools like Container Insights and Microsoft Defender for Containers to get real-time visibility into what is happening inside your cluster.
5. Automate Everything: Use Policy as Code (Azure Policy) to prevent insecure configurations from being deployed in the first place, rather than trying to fix them after the fact.
6. Stay Updated: Automate node image upgrades and image scanning to minimize the impact of known vulnerabilities (CVEs).
7. Secrets Management: Never store secrets inside your container images or Kubernetes YAML files. Use the Secrets Store CSI Driver to integrate with Azure Key Vault.

By following these practices, you transform AKS from a potentially vulnerable platform into a resilient, enterprise-grade environment. Security in Kubernetes is not about reaching a "perfect" state, but about building a system that is difficult to break, easy to monitor, and quick to recover when threats emerge.