AKS Manifest Files

Mastering AKS Manifest Files: A Comprehensive Guide

Introduction: The Blueprint of Your Infrastructure

When you begin working with Azure Kubernetes Service (AKS), you quickly realize that managing individual containers manually is impossible at scale. You cannot simply log into a virtual machine and run docker run commands for every instance of your application. Instead, you need a declarative way to tell the cluster what it should look like. This is where Kubernetes manifest files come into play.

A manifest file is essentially a blueprint. It is a text file written in YAML format that describes the desired state of your application. When you submit this file to the Kubernetes API server, the cluster takes on the responsibility of making reality match your description. If you say you want three replicas of a web server running, Kubernetes monitors the cluster, detects if one crashes, and automatically restarts it to ensure the count remains at three.

Understanding how to write, structure, and maintain these files is the single most important skill for an AKS developer. Without them, you are just clicking buttons in a portal. With them, you are practicing infrastructure-as-code, ensuring that your deployments are repeatable, version-controlled, and transparent. In this lesson, we will dissect the anatomy of these files, explore best practices for managing them, and look at how to handle complex configurations in a production environment.

The Anatomy of a Kubernetes Manifest

Every manifest file, regardless of its complexity, follows a standard structure defined by the Kubernetes API. While there are many different types of objects you can define, they all share four primary fields that the system requires to process the request.

1. The Four Essential Fields

• apiVersion: This tells the cluster which version of the Kubernetes API you are using to create the object. For example, apps/v1 is standard for deployments, while v1 is used for services or pods.
• kind: This specifies the type of object you are creating. Are you making a Deployment, a Service, a ConfigMap, or a Secret? The kind determines how the API server validates the data.
• metadata: This is where you label your object. You provide a name, a namespace, and any labels or annotations that help you organize your resources.
• spec: This is the "meat" of the file. It defines the desired state. For a Deployment, this includes the container image, the number of replicas, and the port mappings.

Callout: Declarative vs. Imperative It is important to distinguish between declarative and imperative management. Imperative commands (like kubectl run) tell the cluster what to do right now. Declarative manifests (using kubectl apply) tell the cluster what the end state should be. Declarative is preferred in production because it allows you to track changes in Git, perform rollbacks, and maintain a history of your infrastructure configuration.

A Basic Deployment Example

Let’s look at a simple Nginx deployment. This file tells AKS to pull the Nginx image and ensure that two copies are running at all times.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80

In this example, the selector field is crucial. It acts as a bridge between the Deployment controller and the Pods. The Deployment looks for any Pods that have the label app: nginx. This is how the system keeps track of which pods belong to which deployment.

Understanding Services: Exposing Your Application

A Deployment creates pods, but pods are ephemeral. They come and go as they fail, are updated, or are rescheduled. Because they have changing IP addresses, you cannot rely on them for direct communication. A Service acts as a stable load balancer in front of your pods.

The Service Manifest

A Service manifest identifies pods based on the same labels used in the Deployment. Here is how you expose the Nginx deployment we just created:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

By setting the type to LoadBalancer, AKS will automatically provision an Azure Load Balancer. This gives your application a public-facing IP address that remains stable even if the underlying pods are deleted and recreated.

Configuration Management: ConfigMaps and Secrets

Hardcoding configuration values inside your application code is a major security and maintenance risk. If you change your database URL, you should not have to rebuild your container image. Instead, you should inject these values at runtime using ConfigMaps and Secrets.

ConfigMaps

ConfigMaps are designed for non-sensitive data, such as environment variables, configuration files, or command-line arguments.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_URL: "db.example.com"
  LOG_LEVEL: "info"

Secrets

Secrets are similar to ConfigMaps but are intended for sensitive data like API keys, connection strings, or passwords. While ConfigMaps store data in plain text, Secrets are base64-encoded (though note that in Kubernetes, this is not true encryption at rest; you should use Azure Key Vault integration for production-grade security).

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  password: "super-secret-password"

Note: Always use the stringData field in your YAML files when defining secrets manually. Kubernetes will automatically encode the values for you, saving you the trouble of manually base64-encoding strings.

Best Practices for Maintaining Manifests

As your application grows, you will eventually find yourself with dozens, or even hundreds, of manifest files. If you do not follow strict organizational rules, you will quickly lose control of your cluster.

1. Use Version Control (GitOps)

Treat your manifests like source code. Store them in a Git repository. Every change should be a pull request that requires review. This provides a clear audit trail of who changed what and when. If a deployment causes an outage, you can simply revert the commit in Git to restore the previous state.

2. Organize by Environment

Use directories to separate your environments. A common structure looks like this:

• /base: Contains common manifests used across all environments.
• /overlays/dev: Contains patches for the development environment.
• /overlays/prod: Contains patches for the production environment (e.g., higher replica counts, more CPU/RAM).

3. Avoid Large, Monolithic Files

While you can put multiple resource definitions in a single file by separating them with ---, it is often cleaner to keep one resource per file. This makes it easier to track changes and prevents accidental modifications to unrelated components.

4. Always Define Resource Requests and Limits

One of the most common mistakes in AKS is failing to define resource requirements. If you do not tell Kubernetes how much CPU and memory your pod needs, the scheduler cannot make informed decisions about where to place the pod.

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

• Requests: The minimum amount of resources the pod is guaranteed.
• Limits: The maximum amount of resources the pod is allowed to consume. If a pod exceeds its memory limit, the system will kill it.

Common Pitfalls and How to Avoid Them

Even experienced developers fall into traps when working with manifests. Here are the most frequent issues and how to steer clear of them.

Pitfall 1: Over-using the latest tag

Using image: myapp:latest is a recipe for disaster. If your deployment restarts and pulls a new version of the "latest" image, you might inadvertently introduce breaking changes without a proper deployment process. Always use specific version numbers or git commit hashes for your image tags.

Pitfall 2: Forgetting Health Checks

Kubernetes needs to know if your application is healthy. If you don't provide a livenessProbe and a readinessProbe, Kubernetes will assume your container is healthy as long as the process is running—even if the application is hung or returning errors.

livenessProbe:
  httpGet:
    path: /health
    port: 80
  initialDelaySeconds: 3
  periodSeconds: 3

Pitfall 3: Not Using Namespaces

If you run all your applications in the default namespace, you will eventually run into naming collisions. If two teams try to name their service api, one will overwrite the other. Use namespaces to logically isolate teams, projects, or environments within the same cluster.

Callout: The Power of Namespaces Namespaces are not just for organization; they are also a primary tool for resource quotas. You can limit the total CPU and memory available to an entire namespace, ensuring that a development workload cannot consume all the resources intended for production.

Step-by-Step: Deploying to AKS

Let’s walk through the process of taking a set of manifests and applying them to your AKS cluster.

1. Preparation: Ensure your kubectl context is set to your AKS cluster. You can check this by running kubectl config current-context.
2. Validation: Before applying, use kubectl apply -f <file> --dry-run=client to check if your YAML is valid without actually creating the resources.
3. Application: Run kubectl apply -f . to apply all files in your current directory.
4. Verification: Use kubectl get pods to see if your pods are in the Running state. If they are in CrashLoopBackOff or Pending, use kubectl describe pod <pod-name> to investigate the error.
5. Troubleshooting: If a pod fails to start, check the logs with kubectl logs <pod-name>. This will show you the application's standard output, which is usually the first place to look for runtime errors.

The Role of Tooling: Helm and Kustomize

As you move beyond simple applications, writing raw YAML for every environment becomes tedious. You will find yourself copying and pasting files and changing only a few lines. This is where templating tools come in.

Kustomize

Kustomize is built into kubectl. It allows you to define a "base" configuration and then apply "overlays" for different environments. You don't have to touch the base files; you just create a small file that describes the difference (e.g., "in production, set replicas to 5").

Helm

Helm is the package manager for Kubernetes. It uses charts, which are packages of pre-configured Kubernetes resources. Instead of writing raw YAML, you use a values.yaml file to inject variables into templates. This is the industry standard for deploying complex, multi-component applications like databases or message brokers.

Security Considerations for Manifests

Security is not an afterthought in Kubernetes; it must be baked into your manifest files.

1. Run as Non-Root

By default, many containers run as the root user. This is a significant security risk. If a container is compromised, the attacker has root access to the container filesystem. Always specify a security context to force the container to run as a non-privileged user.

securityContext:
  runAsUser: 1000
  runAsGroup: 3000
  allowPrivilegeEscalation: false

2. Network Policies

By default, all pods in a namespace can talk to all other pods. This is rarely what you want. Use Network Policies to create a "zero-trust" environment where pods can only communicate with the specific services they need.

3. Read-Only Filesystems

If your application doesn't need to write to the container's root filesystem, mark it as read-only. This prevents attackers from installing malicious tools or modifying application files if they manage to execute code inside the container.

Handling Secrets: The Azure Key Vault Integration

Earlier, I mentioned that Kubernetes secrets are not truly secure. In a production AKS environment, you should never store sensitive data directly in your YAML files. Instead, use the Azure Key Vault Secrets Store CSI Driver.

This driver allows you to mount secrets from Azure Key Vault directly into your pod as a volume or an environment variable. The secret never touches your Git repository. The pod authenticates with Key Vault using a Managed Identity, retrieves the secret at runtime, and keeps it in memory. This is the gold standard for security in the Azure ecosystem.

Troubleshooting Workflow: A Proactive Approach

Even with perfect manifests, things will go wrong. When they do, follow this systematic approach:

1. Check the Status: kubectl get pods
   • Pending: Usually means insufficient resources or a bad node selector.
   • CrashLoopBackOff: The application is starting and then failing. Check logs.
   • ImagePullBackOff: The cluster cannot find or access the image. Check the image name and registry credentials.
2. Describe the Object: kubectl describe <kind> <name>
   • This shows the events associated with the resource. It will tell you if the scheduler rejected the pod or if a health check failed.
3. Check the Logs: kubectl logs <pod-name>
   • This is the standard output of your application. Ensure your app logs to stdout/stderr.
4. Check Connectivity: kubectl exec -it <pod-name> -- /bin/sh
   • This allows you to jump inside the container and test network connectivity or verify file paths.

Warning: Never use kubectl exec to modify the state of a running container. Any changes you make inside the container will be lost the moment the pod restarts. Always make changes in the manifest file and redeploy.

Advanced Manifest Concepts: Sidecars and Init Containers

Sometimes, a single container isn't enough. You might need a helper process to handle log shipping, authentication, or configuration generation.

Init Containers

Init containers run to completion before the main application container starts. They are perfect for waiting for a database to be ready or downloading configuration files from a remote source.

initContainers:
- name: wait-for-db
  image: busybox
  command: ['sh', '-c', 'until nc -z db-service 5432; do sleep 2; done;']

Sidecars

Sidecars run alongside your main application container in the same pod. They share the same network namespace and can communicate over localhost. This is common for service meshes like Istio or Linkerd, where a proxy container handles all network traffic for your application.

Best Practices Checklist for Your Team

To wrap up, here is a checklist you can use when reviewing your team's manifests:

• Resource Requests/Limits: Are they defined? Are they realistic?
• Health Checks: Are liveness and readiness probes configured?
• Security Context: Are we running as non-root?
• Version Control: Is this file in Git?
• Image Tags: Are we using specific versions (not latest)?
• Namespaces: Is this resource in the correct namespace?
• Secrets: Are we using Key Vault or another external manager?

Summary: Key Takeaways

1. Declarative Management: Always use manifest files rather than manual commands. This creates a source-of-truth in Git and allows for repeatable deployments.
2. Separation of Concerns: Use Deployments for pods, Services for networking, and ConfigMaps/Secrets for configuration. Keep these definitions separate to maintain clarity.
3. Resource Constraints: Never deploy a pod without CPU and memory requests and limits. This is vital for cluster stability and cost management.
4. Security First: Use non-root users, read-only filesystems, and external secret management like Azure Key Vault to protect your application.
5. Observability: Implement liveness and readiness probes to ensure Kubernetes can accurately manage the lifecycle of your pods.
6. Tooling is Essential: As your infrastructure grows, adopt tools like Kustomize or Helm to manage environment-specific configurations without duplicating code.
7. Systematic Troubleshooting: Use the kubectl lifecycle—get, describe, logs—to diagnose issues quickly, and always remember that the manifest, not the live container, is the place to fix problems.

Mastering manifest files is a journey. Start by writing simple deployments, then gradually introduce more complex configurations like Init containers, Network Policies, and external secret stores. By treating your infrastructure as a well-documented, versioned, and secure set of files, you transform your AKS cluster from a complex black box into a predictable, high-performance environment for your applications.