On-Demand vs Provisioned

Module: Implementation and Integration

Lesson: On-Demand vs. Provisioned Model Deployment

Introduction: Why Deployment Strategy Matters

When you finish training a machine learning model, you have achieved only half of the goal. The true value of a model is realized only when it is deployed into a production environment where it can make predictions for end-users or other software systems. One of the most critical decisions you will face during this phase is choosing the right deployment pattern: On-Demand (often referred to as Serverless or Pay-per-Request) or Provisioned (often referred to as Dedicated or Fixed-Capacity).

This choice is not merely a technical preference; it is a financial and operational decision that dictates how your infrastructure handles traffic, how much you pay, and how quickly your users receive their results. If you choose the wrong strategy, you might find yourself paying for idle servers that do nothing, or worse, facing a system outage during a traffic spike because your infrastructure could not scale fast enough. In this lesson, we will dissect both patterns, examine their technical underpinnings, and provide a framework for selecting the right one for your specific use case.

Understanding On-Demand Deployment

On-Demand deployment is a model where the infrastructure required to run your model is only allocated when a request is received. Think of this like a taxi service: you do not own a car, but you pay for a ride only when you need to go somewhere. In the cloud computing world, this is commonly associated with "Serverless" functions, such as AWS Lambda, Google Cloud Functions, or Azure Functions, coupled with model serving frameworks that support dynamic scaling.

How It Works

When a request arrives at your API endpoint, the cloud provider spins up an instance of your model container (or a container that loads the model into memory), processes the request, returns the prediction, and then shuts down the environment after a short period of inactivity. This is highly efficient for workloads that are sporadic or unpredictable.

Advantages of On-Demand

• Cost Efficiency for Low Traffic: Since you do not pay for idle time, you only incur costs when the model is actually performing inference.
• Zero-Touch Scaling: The cloud provider handles the scaling logic automatically. If you go from zero requests to a thousand requests in a minute, the provider spins up more instances to handle the load without you needing to manage server clusters.
• Operational Simplicity: You do not need to patch operating systems, manage Kubernetes clusters, or monitor server health. You focus entirely on the code and the model artifact.

Callout: The "Cold Start" Problem A significant drawback of On-Demand deployment is the "cold start." Because the infrastructure is not running constantly, the first request after a period of inactivity forces the provider to initialize the environment, load the model into memory, and start the runtime. This can lead to latency spikes that might be unacceptable for real-time applications.

Understanding Provisioned Deployment

Provisioned deployment involves reserving dedicated computing resources—such as virtual machines, bare-metal servers, or Kubernetes pods—that remain running 24/7. This is analogous to owning your own vehicle. It is always available in your driveway, ready for you to drive at a moment's notice. You pay for the vehicle regardless of whether it is being driven or sitting idle.

How It Works

You configure a specific number of instances to run your model. These instances are kept in a "warm" state, meaning the model is already loaded into memory. When a request hits your load balancer, it is immediately routed to one of these pre-warmed instances. If the traffic exceeds the capacity of your initial pool, you typically configure an auto-scaler to add more instances, but the core foundation remains always-on.

Advantages of Provisioned

• Predictable Latency: Since the model is always loaded and the environment is warm, you avoid cold starts. This is essential for applications where response time must be consistent, such as high-frequency trading or real-time user interfaces.
• Resource Control: You have complete control over the hardware configuration. You can choose specific GPU types, memory sizes, and CPU architectures that are perfectly tuned for your model's computational requirements.
• Cost Predictability at Scale: While expensive for low traffic, provisioned resources often become cheaper than on-demand services when you have a high, steady baseline of traffic, as you can utilize reserved instance pricing.

Comparing the Two Strategies

To help you decide, let's look at the key differences in a structured format.

Implementation: Practical Examples

Example 1: Deploying On-Demand with AWS Lambda

When deploying a model on-demand, you typically package your model artifact (like a .pkl or .onnx file) within a container image.

# Example of a simplified Lambda handler for an On-Demand model
import json
import joblib

# Load model outside the handler to take advantage of execution context reuse
model = joblib.load("model.pkl")

def lambda_handler(event, context):
    try:
        data = json.loads(event['body'])
        prediction = model.predict([data['features']])
        return {
            'statusCode': 200,
            'body': json.dumps({'prediction': prediction.tolist()})
        }
    except Exception as e:
        return {'statusCode': 500, 'body': str(e)}

Why this works: The model is loaded once when the container initializes. Subsequent requests during the same execution window reuse this memory, reducing the latency for those specific calls. However, if the container shuts down due to inactivity, the next request will trigger a reload, causing the cold start.

Example 2: Deploying Provisioned with Kubernetes

In a provisioned environment, you define a Deployment file that tells your cluster exactly how many replicas to maintain.

# Kubernetes deployment specification
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-service
spec:
  replicas: 3 # Always keep 3 instances running
  selector:
    matchLabels:
      app: model-service
  template:
    metadata:
      labels:
        app: model-service
    spec:
      containers:
      - name: model-container
        image: my-model-repo:v1
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"

Why this works: By setting replicas: 3, you guarantee that even if one instance fails, two others are available to serve requests. This configuration provides high availability and consistent performance, as the instances are never turned off.

Step-by-Step Selection Process

Choosing between these patterns should follow a logical workflow. Do not guess; use your data to drive the decision.

1. Analyze Traffic Patterns: Review your historical logs or projected usage. Is your traffic consistent throughout the day, or does it spike at specific hours?
2. Define Latency Requirements: What is the maximum acceptable time for a prediction? If you need sub-100ms response times for a smooth user experience, on-demand might be risky due to cold starts.
3. Evaluate Model Size: Large models (e.g., LLMs or deep learning computer vision models) take a long time to load into memory. On-demand deployment is often impractical for these because the cold start time could be several seconds or even minutes.
4. Calculate Total Cost of Ownership (TCO): Use a calculator to estimate the monthly cost of a provisioned cluster vs. the estimated cost of the number of requests you expect to process on-demand.
5. Test for "Cold Start" Impact: Deploy a prototype to an on-demand service and simulate a period of inactivity. Measure the latency of the first request after the break. If it exceeds your SLA, move to provisioned.

Note: Many modern cloud platforms offer "Provisioned Concurrency" for on-demand services. This is a hybrid approach where you pay a small fee to keep a set number of instances "warm," effectively eliminating cold starts while maintaining the ability to auto-scale beyond that baseline.

Best Practices and Industry Standards

Regardless of the model you choose, there are universal rules that apply to production deployments.

• Separate Model Artifacts from Code: Never hardcode your model weights into your application code. Use an object storage service (like S3 or GCS) to store the model and load it at runtime. This allows you to update the model without redeploying the entire application logic.
• Implement Health Checks: In provisioned environments, ensure your load balancer is checking the health of your model containers. If a container is stuck in a loop or out of memory, the load balancer should stop sending traffic to it.
• Version Your Models: Always tag your models with version numbers (e.g., v1.2.0). If a deployment goes wrong, you should be able to roll back to the previous version in seconds.
• Monitor Performance Metrics: Track not just CPU and memory, but also inference latency and prediction distributions. If the average latency starts creeping up, it is a sign that your model needs optimization or your infrastructure needs more resources.
• Resource Limits: Always set resource limits (CPU/Memory). If you don't, a single runaway process could consume all resources on a node, crashing other services sharing that same hardware.

Common Pitfalls and How to Avoid Them

Even experienced engineers fall into these traps. Here is how to keep your deployment stable.

1. The "Big Model" Mistake

Developers often try to deploy massive models to serverless environments. If your model file is 5GB, the time it takes to download and load that model into RAM will consistently trigger the maximum timeout of your serverless function.

• Solution: Use model compression techniques (quantization, pruning) or move to a provisioned environment where the model is already pre-loaded on persistent storage volumes.

2. Ignoring Scaling Limits

On-demand services have "concurrency limits." If your application goes viral and you hit your account limit, the provider will start rejecting requests.

• Solution: Always check your account quotas. If you expect a major launch, contact your cloud provider to proactively increase your concurrency limits.

3. Failing to Handle "Thundering Herds"

If you have a provisioned setup and suddenly receive a massive spike, your auto-scaler might not be fast enough to spin up new pods.

• Solution: Use "buffer" capacity. Configure your auto-scaler to keep your average CPU utilization at 60% rather than 90%. This gives you a 30% overhead to handle sudden spikes while the new instances are booting up.

4. Hard Dependencies on External Services

If your model calls an external API to fetch features during inference, and that API goes down, your model will fail.

• Solution: Always implement retries with exponential backoff for external calls. If the external service is not strictly necessary for the prediction, consider returning a cached or default value.

Advanced Considerations: The Hybrid Approach

In many sophisticated production environments, the distinction between on-demand and provisioned is blurring. Many organizations use a tiered approach:

• Tier 1 (Core Models): High-traffic, mission-critical models are hosted on Provisioned Kubernetes clusters to ensure 99.99% availability and low latency.
• Tier 2 (Experimental/Batch Models): Internal tools, data science experiments, or low-priority batch processing jobs are hosted on On-Demand functions. This saves thousands of dollars annually while allowing data scientists to iterate rapidly without infrastructure overhead.

This tiered strategy is the industry standard for mature MLOps teams. It recognizes that not every model deserves the same level of investment in infrastructure. By categorizing your models based on their business impact and traffic profile, you can optimize both your budget and your operational workload.

Summary and Key Takeaways

Choosing between On-Demand and Provisioned deployment is a balancing act between cost, performance, and complexity. There is no "one size fits all" solution. Instead, you must align your technical choice with your business requirements.

Key Takeaways:

1. Understand Your Traffic: Use historical data to determine if your traffic is spiky (On-Demand) or steady (Provisioned).
2. Mind the Latency: If you require sub-millisecond, consistent responses, avoid the cold-start penalties of On-Demand deployment.
3. Cost vs. Complexity: On-Demand is cheaper for low-traffic applications, but Provisioned offers better control and predictability for high-scale enterprise applications.
4. The Cold Start Factor: Always account for model initialization time. If your model takes too long to load, it will fail in a purely serverless environment.
5. Infrastructure as Code (IaC): Regardless of the method, manage your infrastructure through code (Terraform, CloudFormation, etc.) to ensure that your deployments are repeatable and auditable.
6. Monitoring is Non-Negotiable: You cannot optimize what you do not measure. Monitor latency, throughput, and error rates from day one.
7. Start Simple: If you are unsure, start with an On-Demand setup. It is easier to migrate to a Provisioned setup once you have traffic data than it is to manage a complex Provisioned cluster from day one for a product that has no users yet.

By applying these principles, you will be able to build a deployment pipeline that is not only efficient but also resilient enough to support your models as they grow and evolve with your business. Remember that deployment is an iterative process; do not be afraid to switch strategies as your application matures and your traffic patterns change.

FAQ: Frequently Asked Questions

Q: Can I mix both approaches? A: Yes, and it is common. You might have a front-end service that handles user requests using a provisioned model, while a background data-cleaning job runs on an on-demand function.

Q: Does "Serverless" mean there are no servers? A: No. It simply means the servers are managed by the provider. You are still running on physical hardware, but you are abstracted away from the maintenance and patching of that hardware.

Q: How do I handle model updates in a Provisioned environment? A: Use a "Blue-Green" deployment strategy. Spin up a new set of instances with the new model (Green), test them, and then slowly shift traffic from the old instances (Blue) to the new ones. This ensures zero downtime during updates.

Q: What is the most common mistake beginners make? A: Over-provisioning. Beginners often spin up massive, expensive clusters "just in case," wasting significant budget. Start small, monitor your metrics, and scale up only when your data indicates you need to.

Q: Is Kubernetes overkill for my model? A: Often, yes. If you are a small team or have a single model, managed services (like AWS SageMaker, Google Vertex AI, or Azure ML) can provide the benefits of provisioned hosting without the operational overhead of managing a full Kubernetes cluster.