Kabelo Dike / AI Stack Mental Model

The Infrastructure and Serving Layer

· Layer L3 of 5

Having explored the foundation model as a static set of mathematical weights, we step down to Layer 3: Infrastructure and Serving.

A model file sitting on a hard drive cannot answer user requests. To make it useful, an operational serving layer must load those weights into GPU memory, schedule concurrent requests, optimize memory bandwidth, and expose stable network endpoints. If the Application Layer is where tasks are orchestrated and Layer 4 is the raw intelligence, Layer 3 is the high-performance engine room that makes running that intelligence economically viable and reliable at scale.

What “inference engineering” actually means

Serving an AI model is fundamentally different from serving a traditional web application or database:

  • The ingress/egress translation (tokenizer as encoder and decoder): A GPU cannot process strings or raw text directly. Before any computation touches the model weights, an independent, high-speed CPU program — the Tokenizer — runs at the front door:
    • Encoding (string → token IDs): It uses a deterministic algorithm (like Byte Pair Encoding) to slice input text into chunks and map them to an array of integer IDs. This integer array is what gets metered by rate limiters and passed to the GPU.
    • Decoding (token IDs → characters): As the GPU generates output integer IDs one by one, the tokenizer instantly decodes those numbers back into characters and words so they can stream to the client.
  • Compute and memory bound: In a standard web service, a server processes an HTTP request by querying a database or running light business logic, which takes minimal CPU time. In contrast, generating tokens requires streaming billions of floating-point parameters through high-bandwidth memory for every single token produced.
  • Two discrete execution phases: Once encoded, every inference request runs in two distinct mathematical modes:
    • Prefill (prompt processing): The engine consumes your entire input prompt at once. This phase is compute-bound and highly parallelized across GPU cores.
    • Decode (token generation): The engine generates tokens sequentially, one at a time. This phase is memory-bandwidth bound, as the entire model’s weights must be accessed in memory to predict just one token.

KV caching: the foundation of serving efficiency

In transformer models, each new token depends on the mathematical state of every preceding token. Without an optimization technique, generating token 100 would require recalculating the attention math for tokens 1 through 99 all over again.

  • Reusing attention state: The serving engine calculates intermediate Key and Value vectors for each token once and stores them in GPU RAM, known as the KV Cache. When generating the next token, the engine simply reads the cached state rather than recomputing the past tokens from scratch.
  • The memory problem: The KV cache grows linearly with context length and the number of concurrent users. On busy production servers, GPU memory often runs out not from the model weights themselves, but from the massive memory footprint of active KV caches.
  • Paged attention: Modern serving engines manage GPU memory like an operating system manages virtual memory, breaking the KV cache into non-contiguous blocks to prevent memory fragmentation and allow vastly higher concurrency.

Traffic management: API gateways, keys, and rate limits

Between the Application Layer and the raw GPU workers sits the API Gateway:

  • API keys as contracts: An API key is an authentication token passed in the HTTP request header. It tells the serving platform who is calling the model, verifies project authorization, and maps the request to an account ledger for billing.
  • Token-based rate limiting: Traditional APIs throttle traffic by requests per minute (RPM). AI serving engines must rate limit by both requests per minute and tokens per minute (TPM). A single request with a 100,000-token document consumes vastly more GPU memory and compute than a two-sentence chat, so quota systems must budget token volume dynamically.
  • Dynamic batching: GPUs are inefficient when processing one request at a time. Serving runtimes continuously group incoming requests from different users into dynamic batches, feeding them through the model simultaneously to maximize hardware saturation without introducing noticeable latency.

Routing and cluster topology

At scale, a model is rarely hosted on a single machine. The serving layer manages routing across clusters of accelerators:

  • Model parallelism (tensor and pipeline): Leading foundation models can exceed hundreds of gigabytes in size, making them too large to fit into the memory of a single GPU. Serving frameworks split the model across multiple chips connected by ultra-fast interconnects, coordinating parallel matrix math across the cluster.
  • Worker routing and cache awareness: Advanced routers inspect incoming prompts to identify which GPU worker already has that prompt’s prefix cached in memory. Routing requests to workers that already hold the relevant KV cache avoids redundant prefill compute, drastically reducing both latency and cost.

The architectural boundary

The Infrastructure and Serving Layer does not care about chat sessions, business guardrails, or UI formatting. Its sole contract is operational: accept incoming token arrays over network protocols, schedule them across physical accelerators, optimize GPU memory, and stream completed tokens back under strict latency and throughput SLAs.