Production AI inference on Google Cloud

Record the workload envelope

Measure request and response size, expected concurrency, peak pattern, latency target, accelerator need, model load time, data locality, and availability objective. Separate steady traffic from bursts. State whether requests are synchronous, asynchronous, streaming, or batch. Request and response size: Percentiles from live or rehearsal logs. Concurrency: Arrival rate multiplied by average latency (Little's law). Peak pattern: Ratio of peak traffic to steady traffic, and ramp speed. Latency budget: 50th, 95th, and 99th percentile targets per request class. Accelerator need: GPU or TPU class, memory footprint, and throughput tests. Model load time: Cold-start measurement in the target environment. Availability objective: Error budget and permitted recovery time. Measure latency in percentiles, not averages. An average hides the tail that users feel. State the percentile, the window, and the request class for every target. A 95th percentile (p95) of 800 milliseconds for an interactive request means something different from a p95 of 30 seconds for a document-processing job. Record accelerator needs precisely. Note which graphics processing unit (GPU) or Tensor Processing Unit (TPU) class the model requires, how much memory it occupies, and whether request batching changes that requirement. A model that fits one L4 with batching can need a larger class at single-request latency. Also define the quality boundary. For a predictive model, record the evaluation set and threshold. For a generative system, include retrieval quality, groundedness, policy checks, tool behavior, and human review.

Select a serving path

Vertex AI online prediction provides managed endpoints for custom trained models. It supports standard prediction requests, raw requests for custom payloads, and explanation requests for configured models. Managed services reduce some platform work, but the team still owns the model contract, evaluation, data path, access, monitoring, and cost decisions. Cloud Run can suit containerized inference when the workload fits its runtime and accelerator options. Google Kubernetes Engine (GKE) can suit teams that need Kubernetes-level control, specialized scheduling, or a shared serving platform. Batch prediction can be a better fit when the result does not need an online response. Vertex AI online prediction: Managed endpoints for custom models; Provisioned-capacity cost and quota lead time. Vertex AI batch prediction: Large jobs without an online deadline; Scheduling, input staging, and output handling. Cloud Run with GPU: Containerized services that scale with traffic; GPU availability by region and cold starts. GKE: Custom scheduling or a shared serving platform; Cluster operations and upgrade burden. Do not select from a feature checklist alone. Run a representative load and failure test. Map each option to the team's operating model. A managed endpoint can reduce infrastructure work while leaving model packaging, evaluation, access, observability, quota planning, and release decisions with the delivery team. A container service can simplify application integration while requiring careful work on instance lifecycle and accelerator capacity. A Kubernetes platform can offer control while adding a larger platform surface to operate. Price each path with the real traffic shape. Node-based endpoints bill for provisioned capacity whether or not requests arrive. Request-based services can scale to zero but charge per use and can add cold-start latency. A path that looks cheaper at average load can cost more at your actual peak-to-steady ratio. Choose one primary path and one fallback pattern. A fallback can be a prior model, a smaller model, a cached result, a deterministic rule, or a manual workflow. It should follow the product's safety and freshness requirements.

Design model and prompt artifacts

Version the model artifact, serving container, tokenizer, generation configuration, prompt, tool definitions, retrieval configuration, and safety rules that affect the response. Treat these as one release set. A model version alone cannot reproduce a generative result when the surrounding configuration has changed. Give the release set a single identifier and store it in a registry. The identifier should resolve to the model weights or publisher version, container digest, tokenizer, generation settings, prompt template, retrieval configuration, and safety filters. When an incident asks which system produced an answer, that identifier is the answer. Separate the model contract from the platform. The request schema, response schema, error shapes, and timeouts should be testable against a local container. Teams that test the contract only through the managed endpoint discover packaging problems during deployment, where they are most expensive. For custom containers, define health behavior, startup time, model-loading failure, shutdown handling, and the response when an accelerator is unavailable. Keep the container contract small and test it outside the platform before deployment. For managed publisher models, record the selected model identifier and configuration. Review availability, quotas, regional support, data handling, and release notes in the current Google Cloud documentation before changing live traffic.

Design the full request path

Authenticate the caller and authorize the action. Validate size, type, and schema before expensive work. Fetch approved features or retrieval context with a time budget. Call the model through a versioned contract. Apply output validation and policy rules. Return a stable response or defined fallback. Record measures without exposing sensitive input. Allocate the end-to-end latency budget across the stages before building. If the p95 target is 800 milliseconds and the model needs 500 under load, validation, retrieval, and output checks share the remaining 300. A retrieval call with no time budget can consume the whole allowance and leave the model unreachable. Set timeouts as a hierarchy. The caller's deadline should exceed the path budget, which should exceed the model call timeout, which should exceed any single dependency call. A dependency timeout longer than the caller's patience converts a slow dependency into a failed request with wasted compute.

Test before release

Test warm and cold behavior, peak traffic, dependency delay, model errors, malformed input, quota exhaustion, and regional failure. Measure percentile latency, timeout rate, saturation, cost per accepted result, and quality by important segment. Inject failures deliberately before users find them: For a generative model, evaluate prompt and model versions together. Record retrieval sources and tool calls. Use shadow traffic or a restricted rollout when practical. Keep a rollback route to the prior version or a manual workflow. Shadow traffic sends a copy of live requests to the candidate version without returning its answers. It measures behavior on real input with no user risk. A canary returns the candidate's answers to a small, defined slice of traffic. Use shadow first when quality is uncertain, then canary with guardrail metrics and an automatic rollback trigger. Build a load profile from expected arrival patterns rather than an average request rate. Test sustained traffic, abrupt bursts, large inputs, long outputs, concurrent tool calls, and a dependency that consumes most of the time budget. Measure queue time separately from model time. Define a quality evaluation set that represents the operating task. Include common requests, rare but important cases, malformed inputs, unsupported requests, and cases that require refusal or human review. For retrieval systems, measure whether the required evidence was retrieved before grading the generated answer. Run a cost test with the same request distribution. Report cost per accepted result and cost by important request class. A lower endpoint price can still cost more if it produces longer outputs, more retries, or additional review.

GPU or TPU capacity unavailable, or quota exhausted at deploy time.
Model server crash and restart during traffic.
Malformed, oversized, or adversarial input.
Dependency timeout at the retrieval or feature step.
A sudden burst beyond the autoscaling rate.
Loss of a region or a zonal dependency.

Understand the cost drivers

Inference cost follows the shape of the workload, not its average. The main drivers: Batch instead of online: Higher throughput per GPU-hour; Results arrive later. Smaller or distilled model: Lower latency and capacity cost; Quality must be re-evaluated. Scale-to-zero floor: No idle cost; Cold starts on the first requests. Caching repeated requests: Fewer model calls; Freshness and consent constraints. Report cost per accepted result, split by request class. An answer that fails policy checks after full generation costs the same compute as a good one and delivers nothing.

Provisioned capacity: node-based endpoints bill for the replica-hours reserved, including idle time.
Request volume: request-based services bill per call and per unit of compute time.
Output length: for generative models, long answers cost more tokens, more time, and more review.
Retries: a client retry after a slow response can double the load during an incident.
Evaluation and review: human review, reranking, and policy checks belong in the unit cost.

Plan capacity and quotas

Document the quotas that can limit deployment or traffic. Confirm accelerator availability, endpoint limits, request limits, build and artifact limits, and regional dependencies. A load test in one project does not prove that target-project quotas are ready. GPU and TPU classes have regional availability, quota, and sometimes reservation lead times. Confirm the specific class in each deployment region, and record how long a quota increase took the last time the team requested one. Include that lead time in the release calendar. Set a capacity review trigger based on saturation, queue time, error rate, and forecast traffic. Include quota increase lead time in the release plan. When autoscaling is used, monitor scale-out time and the requests affected during that interval. For latency-sensitive traffic, set a scaling floor. Vertex AI endpoints support a minimum replica count, and Cloud Run supports minimum instances. Both trade idle cost for the removal of cold starts from the user path. Measure scale-out time under rehearsal and count the requests that arrive during it.

Design fallback and failure behavior

Choose the fallback per failure class, and record which class triggered it. A timeout might route to a smaller model. A model-server failure might return a cached answer where freshness allows. A policy-check failure should return a refusal or a review path, never an unchecked answer. Keep fallbacks observable. A silent fallback repairs the error rate and hides a quality regression. Track fallback rate, fallback type, and outcome quality as first-class metrics, and alert when the rate crosses a threshold. Test the fallback itself. A fallback path that has never carried load is an assumption, and it often depends on the same failing service as the primary path.

Operate the endpoint

Create alerts for error rate, latency, saturation, dependency failure, quality drift, and unexpected cost. Keep deployment, rollback, and incident runbooks. Assign owners for the service, model, data, security, and business measure. Monitor the signals that cover the whole path: Review incidents across the full request path. A quality problem can begin in retrieval, feature freshness, policy configuration, a tool response, or the model. Keep trace identifiers across services and record only the input details needed for diagnosis under the approved data policy. Use a release record that connects the artifact set, evaluation result, load test, cost test, change approval, deployment, and first live observation. That record makes rollback and later comparison practical.

Request volume, error rate, and latency percentiles per request class.
Saturation: GPU or TPU utilization, replica count, and queue time.
Dependency health: retrieval latency, feature freshness, and tool errors.
Quality drift: evaluation scores on sampled live traffic.
Fallback rate and fallback type.
Cost per accepted result against the budget.

Release readiness checklist

The workload envelope is measured, with percentile targets and a named owner.
The serving path survived a representative load and failure test.
The release set has one identifier covering model, container, prompts, retrieval, and policy.
Timeouts form a hierarchy inside the caller's deadline.
Quotas and GPU or TPU availability are confirmed in every deployment region.
Fallbacks are defined per failure class, tested under load, and observable.
Cost per accepted result is measured by request class.
Rollback returns to a recorded prior release set.
Alerts cover saturation, quality drift, fallback rate, and unexpected cost.

Primary sources

Official references for the tools and services covered in this guide.