Custom Images for SAP

Custom Images for SAP Infrastructure: A Comprehensive Guide

Introduction: Why Custom Images Matter for SAP

In the context of SAP infrastructure, the compute layer is the foundation upon which your entire application stack rests. Whether you are running SAP S/4HANA, SAP NetWeaver, or SAP BW/4HANA, the consistency, security, and performance of your underlying operating system (OS) are critical. Traditionally, organizations relied on manual OS installation or generic cloud marketplace images. However, as SAP environments grow in complexity and scale, these manual methods become significant bottlenecks. This is where custom images—often referred to as "Golden Images"—come into play.

A custom image is a pre-configured snapshot of an OS that includes your organization's specific hardening policies, security agents, monitoring tools, networking configurations, and SAP-specific OS tunings. By using custom images, you transition from "manual server builds" to "automated infrastructure deployment." This shift is not merely about convenience; it is about risk mitigation. When every SAP application server or database node is deployed from the same verified, hardened source, you eliminate the "configuration drift" that often leads to mysterious production outages and security vulnerabilities.

This lesson explores the lifecycle of custom images for SAP, starting from the initial build process, moving through the hardening and testing phases, and concluding with deployment strategies in a cloud-native environment. We will look at how to maintain these images over time, ensuring that your SAP estate remains compliant and performant as software versions evolve.

The Anatomy of an SAP-Ready Image

Before jumping into the technical implementation, we must define what makes an image "SAP-ready." SAP has stringent requirements for the OS layer, which differ significantly from standard web server or general-purpose compute requirements.

Key Components of an SAP Image

An SAP-ready custom image typically contains the following layers:

• Operating System Base: A supported version of Linux (e.g., SUSE Linux Enterprise Server for SAP Applications or Red Hat Enterprise Linux for SAP Solutions). These specific versions include kernel parameters tuned for SAP workloads.
• SAP Kernel Dependencies: Installation of necessary libraries, such as libstdc++, compat-sap-c++, and specific glibc versions that the SAP installer (SWPM) expects to find.
• Security Hardening: Implementation of CIS (Center for Internet Security) benchmarks, disabling unnecessary ports, and configuring robust SSH access controls.
• Cloud-Specific Agents: Integration of cloud provider agents (e.g., Azure VM Extensions, AWS SSM Agent, or Google Guest Environment) to ensure the compute instance can communicate with the cloud control plane.
• Monitoring and Logging: Pre-installed agents for SAP-specific monitoring (such as the SAP Host Agent) and centralized logging tools (like Splunk, ELK, or cloud-native logging services).
• Performance Tuning: Pre-configured tuned profiles or specific kernel parameter modifications (e.g., vm.swappiness, fs.file-max) that are documented in SAP Notes.

Callout: The "Golden Image" Philosophy The Golden Image is a version-controlled, immutable artifact. It should never be modified after deployment. If a configuration change is required, you do not patch the running server; instead, you update the base image, build a new version, and trigger a rolling update of your infrastructure. This ensures that your production environment is always predictable and reproducible.

Building Your Custom Image Pipeline

Creating a custom image manually is prone to human error. To achieve a professional standard, you must implement an automated pipeline. Most mature infrastructure teams use tools like HashiCorp Packer to define image builds as code.

Step-by-Step: Defining an Image Build with Packer

Packer allows you to create identical machine images for multiple cloud platforms from a single source configuration file. Below is a conceptual example of a build configuration for an SAP-hardened SLES image.

# Example Packer HCL configuration snippet
source "amazon-ebs" "sap-sles" {
  ami_name      = "sap-sles-15-sp4-hardened-${formatdate("YYYYMMDD", timestamp())}"
  instance_type = "m5.large"
  region        = "us-east-1"
  source_ami_filter {
    filters = {
      virtualization-type = "hvm"
      name                = "suse-sles-15-sp4-*"
      root-device-type    = "ebs"
    }
    owners = ["amazon"]
  }
  ssh_username = "ec2-user"
}

build {
  sources = ["source.amazon-ebs.sap-sles"]

  provisioner "shell" {
    inline = [
      "sudo zypper install -y sap-sles-15-sp4-packages",
      "sudo sysctl -w vm.swappiness=10",
      "sudo systemctl enable sapinit"
    ]
  }
}

Explanation of the Build Process

1. Source Selection: The configuration identifies a base image provided by the cloud vendor. It is vital to use the "for SAP" variants provided by vendors like SUSE, as these include specific repository access for SAP-related patches.
2. Provisioning: The provisioner block runs the commands required to transform the generic OS into an SAP-hardened OS. You should place your installation scripts in a version-controlled repository (like Git) and execute them via Packer.
3. Validation: After the scripts finish, Packer shuts down the instance and creates an image. You should integrate automated testing (e.g., using InSpec or Goss) to verify that the image meets your security and performance requirements before it is marked as "Ready for Production."

Best Practices for SAP Image Management

Managing custom images can quickly become a mess if you do not have a clear strategy for versioning and lifecycle management.

1. Versioning and Tagging

Never use a generic name like sap-base-image. Instead, adopt a strict semantic versioning scheme. For example, sap-sles15sp4-v20231012-001. This allows you to track exactly when an image was built and what security patches it contains. Tags are equally important; tag your images with metadata such as OS_Version, SAP_Kernel_Level, Security_Baseline_Date, and Environment_Scope (e.g., Sandbox, QA, Production).

2. The "Immutable Infrastructure" Pattern

Resist the urge to log into production servers to fix issues. If a server is missing a required library, fix it in the image build pipeline and redeploy. While this sounds daunting for a multi-terabyte SAP HANA database, it is the standard for application servers and web dispatchers. For databases, focus on ensuring the image is perfectly tuned for disk performance and I/O latency.

3. Automated Security Scanning

Integrate vulnerability scanning into your image pipeline. Tools like Trivy or Clair can scan your custom images for known vulnerabilities before they are promoted to your compute gallery. If a build contains a "Critical" or "High" vulnerability, the pipeline should fail automatically, preventing the image from being used.

Warning: The "Configuration Drift" Trap Avoid the "Snowflake Server" syndrome. If you allow administrators to manually install packages or modify kernel parameters on individual SAP instances, you lose the ability to guarantee the state of your infrastructure. If a manual change is made, that server is no longer identical to the image it was built from, making troubleshooting during an SAP incident significantly more difficult.

Comparison: Marketplace Images vs. Custom Images

Implementing SAP-Specific OS Tuning

SAP workloads are unique because they demand high memory throughput and low-latency disk I/O. Your custom image must reflect this.

Kernel Parameters for SAP HANA

When building your image, ensure that /etc/sysctl.d/99-sap.conf is populated with the correct values. While these values can be applied at runtime, baking them into the image ensures that every new instance is ready for the SAP HANA installer the moment it boots.

# Example content for /etc/sysctl.d/99-sap.conf
# These settings should be verified against the latest SAP Notes
kernel.shmmax = 18446744073699999999
kernel.shmall = 18446744073699999999
kernel.sem = 2000 2000000 2000 1024
vm.max_map_count = 2147483647
fs.file-max = 2000000

Networking and Storage

SAP systems rely heavily on high-speed networking for internal communication (e.g., between SAP Web Dispatchers and Application Servers). Your custom image should include the necessary drivers for high-performance networking (such as ENA on AWS or Accelerated Networking on Azure). Furthermore, if you use specific storage protocols like NFS for SAP shared filesystems (/sapmnt, /usr/sap/trans), ensure that the necessary client tools (like nfs-common or cifs-utils) are pre-installed and configured to optimize mount options.

Troubleshooting Common Image Pitfalls

Even with a robust pipeline, issues will arise. Being aware of the most common mistakes will save you significant downtime.

1. The "Missing Dependency" Error

The most common error during an SAP installation on a new image is a missing shared library. SAP’s Software Provisioning Manager (SWPM) performs a prerequisite check before starting. If your custom image is missing a library like libnsl.so.1, the installer will fail.

• The Fix: Maintain a "prerequisite manifest" file in your image pipeline that lists every library SAP requires. Use your CI/CD tool to verify these exist in the image before it is tagged as "Production Ready."

2. Time Synchronization Issues

SAP applications are extremely sensitive to time drift. If the system clock on an application server drifts from the database server, you will see authentication errors and transaction failures.

• The Fix: Ensure that your custom image comes pre-configured with a reliable time synchronization service (e.g., chrony or ntpd) pointing to internal, high-precision NTP servers.

3. Cloud-Init Failures

Cloud-init is the standard way that cloud instances configure themselves on the first boot (setting hostnames, injecting SSH keys, etc.). If your custom image has a corrupted cloud-init configuration, the instance will boot but fail to initialize correctly.

• The Fix: Always test a fresh instance deployment from your new image in a sandbox environment before promoting it to production. Never assume an image is "good" simply because it built successfully.

Note: The Importance of SAP Notes SAP Note 1275776 (Linux) and similar platform-specific notes are your primary source of truth. Always cross-reference your custom image kernel parameters and library requirements with the most recent SAP Notes. SAP frequently updates these requirements as kernel versions evolve.

Advanced Image Lifecycle Management

Once you have established a build pipeline, you must consider the "End of Life" (EOL) for your images. An image that was perfect six months ago may now be vulnerable or outdated.

The Rolling Update Strategy

To maintain a secure environment, implement a rolling update cycle. Every quarter (or more frequently, depending on your security policy), rebuild your golden image with the latest OS patches and security updates. Once the new image is validated, start replacing your SAP instances one by one. By using infrastructure automation tools (like Terraform or Ansible), you can automate the replacement of an old application server with a new one built from the latest image, ensuring that no server in your environment is ever more than a few months old.

Image Galleries and Versioning

Most cloud providers offer a "Compute Gallery" or "Image Gallery" service. Use these to organize your images. A gallery allows you to:

1. Replicate images across different regions for disaster recovery.
2. Manage permissions, ensuring only the SAP Basis team can modify the images.
3. Deprecate old versions, preventing developers from accidentally using an outdated, insecure image for new deployments.

Security Hardening: The "Zero Trust" Approach

In an SAP environment, the database contains the crown jewels of the organization. Your custom image must be hardened to the highest standard.

Key Hardening Steps

• Disable Unused Services: Remove or disable services that are not required for SAP, such as avahi-daemon, cups, or bluetooth.
• SSH Access: Disable root login via SSH and enforce key-based authentication. If possible, integrate with an identity provider (like Active Directory or LDAP) to manage access.
• File System Security: Ensure that sensitive directories (like /usr/sap and /sapmnt) have strict permissions. Consider mounting critical partitions with noexec, nodev, and nosuid flags where appropriate.
• Logging: Configure rsyslog or journald to forward all logs to a centralized, write-only logging server. This ensures that even if an attacker gains root access, they cannot wipe their tracks.

Practical Example: Automating the Build

Let’s walk through a typical workflow for an SAP Basis engineer using a modern DevOps toolchain.

1. Code Change: A developer updates the provision.sh script in the Git repository to include a new security patch required by the latest SAP Note.
2. CI Trigger: The Git push triggers a Jenkins or GitHub Actions pipeline.
3. Build: The pipeline invokes Packer, which spins up a temporary instance, runs the provision.sh script, and installs the required packages.
4. Test: The pipeline runs an InSpec test suite to confirm the presence of the new patch.
5. Publish: If the test passes, Packer creates a new AMI (Amazon Machine Image) or Managed Image and tags it with the version number.
6. Notification: The pipeline sends a Slack or email notification to the Basis team informing them that a new "Golden Image" is available for use.

This process removes all manual effort and ensures that the image is documented, tested, and ready for use.

Key Takeaways for SAP Infrastructure Teams

1. Images as Infrastructure-as-Code: Treat your SAP images as code. Use tools like Packer to define your build process in a version-controlled repository. This ensures that every server you deploy is identical and predictable.
2. Security is Non-Negotiable: Hardening your OS is not an "add-on." It is a fundamental requirement. Use automated scanning tools to ensure your images meet compliance standards before they reach the production environment.
3. Automate to Eliminate Drift: Configuration drift is the enemy of stability. By using immutable images, you force yourself to fix issues at the source (the image build) rather than patching individual servers. This leads to a much more stable environment.
4. SAP-Specific Tuning: Never use a generic OS image for SAP. Always ensure your image includes the specific kernel parameters, libraries, and performance tunings required by SAP HANA and the SAP application stack.
5. Lifecycle Management: Images have an expiration date. Implement a quarterly schedule to rebuild and refresh your golden images to incorporate the latest OS patches and security updates.
6. Testing is Mandatory: Never deploy an image to production without running automated tests. A build that completes successfully does not mean the configuration is correct for your SAP workload.
7. Version Everything: Use a clear semantic versioning scheme for your images. This allows you to track, audit, and roll back to previous versions if a new image introduces an unexpected issue.

By following these principles, you move away from the high-stress, manual world of server administration and into a world of automated, reliable, and secure SAP infrastructure. The initial effort to set up a pipeline pays for itself many times over in reduced incident response time and improved system uptime.

Common Questions (FAQ)

Q: Can I use the same image for both SAP HANA and SAP Application Servers?

A: While it is technically possible, it is generally recommended to have separate images. SAP HANA has very specific storage and I/O requirements that differ from the application layer. Keeping these images separate allows you to optimize each one for its specific task.

Q: How often should I rebuild my Golden Images?

A: At a minimum, every quarter. However, if a critical security vulnerability is announced (e.g., a kernel-level exploit), you should trigger an immediate rebuild and rolling update of your infrastructure.

Q: What if I need to make a quick "emergency" change to a production server?

A: Avoid it at all costs. If you must make a change, document it thoroughly and make sure the same change is applied to your image build pipeline immediately. The goal is to reach a state where you never have to make a manual change again.

Q: Do I need a different image for each cloud provider?

A: Yes. While the OS configuration might be similar, the cloud-specific agents and drivers (like the AWS ENA driver or Azure Linux Agent) vary. Using Packer allows you to maintain one set of configuration scripts while targeting multiple cloud platforms.