Optimize and Serve Vision Models with Quantization & TensorRT

Real production pain looks like: good numbers on a researcher’s workstation, but p95 latency spikes and cost balloons when the model lands in a multi-tenant cluster or on an edge device; post-deploy surprises (preproc CPU stalls, dynamic shapes, inappropriate batch sizing) break your SLO even before you start pruning weights. You need a repeatable baseline, an optimization plan that preserves your key slice metrics, and a deployment story that includes compiled engines and validated runtime configs.

Contents

• When to optimize: baselines and SLOs
• Quantization and pruning: practical recipes and pitfalls
• Compiling and tuning with TensorRT and ONNX
• Serving strategies with Triton and autoscaling
• Practical checklist for immediate implementation

When to optimize: baselines and SLOs

Start by measuring the problem on the hardware and workload you actually care about. Capture:

• Accuracy on production-like slices (mAP, top-1/top-5, per-class recall) using a held-out validation set that mirrors production distribution.
• Latency distribution (p50, p95, p99), throughput (images/sec), and GPU/CPU utilization under representative traffic. Use trtexec for low-level engine benchmarking and perf_analyzer for server-level workloads when you plan to use Triton.

Define concrete success criteria before changing the model. Examples you can adopt immediately:

• p95 latency improvement ≥ 2× or p95 < X ms (domain-specific).
• Accuracy drop ≤ 0.5 absolute top-1 (or a chosen business threshold).
• Cost per 1M inferences reduced by Y% (use the cost formula in the checklist below).

Make the baseline artifact reproducible: version the raw model, export a canonical ONNX or model file, capture the exact pre/post-processing code as preprocess.py/postprocess.py, and store a short perf script that reproduces the numbers (use the same client workload and flags). This “artifact + perf script” is the golden baseline you’ll compare optimizations against.

Quantization and pruning: practical recipes and pitfalls

Quantization and pruning are powerful, but they behave differently and demand different validation.

Quantization (PTQ vs QAT)

• Prefer a quick post‑training quantization (PTQ) pass to test the performance envelope—use FP16 first (FP16 almost always reduces memory and speeds up TensorCore-backed GPUs) and then try INT8 for extra gains. TensorRT supports FP16/INT8 and uses per-channel weight scales for conv/FC weights—this reduces per-layer quantization error for convolution layers.
• Calibration matters. For typical ImageNet-style CNNs, TensorRT docs note that a few hundred representative images (≈500 is a commonly cited practical number) are often sufficient to generate useful INT8 dynamic ranges for activations. Cache that calibration table and reuse it across builds when possible.
• When accuracy degrades with PTQ, run Quantization-Aware Training (QAT) to recover quality. QAT inserts fake-quantize operations so the model learns to be robust to quantization noise; PyTorch’s QAT flows have shown strong recovery relative to PTQ, especially on harder models. QAT is more engineering work but often required for sub-1% accuracy loss targets.

Pruning (structured vs unstructured)

• Unstructured pruning (removing individual weights) cuts parameter count but rarely translates to GPU speedups on its own because sparse patterns are irregular and need special kernels or libraries. Classic work shows large parameter reduction is possible but not always practical for speed without runtime support.
• Structured sparsity (channel, filter, block pruning) removes whole compute units (filters, channels, or fixed patterns) and maps efficiently to GPUs. NVIDIA’s Ampere/Hopper families expose a 2:4 fine-grained structured sparsity pattern that can give up to ~2× effective throughput for supported ops when you match that pattern in training/pruning and use TensorRT/cuSPARSELt optimized paths. Generate the sparse pattern during training or via a sparse retraining workflow to restore accuracy.
• Practical rule: for GPU speed, prefer structured pruning or platform-supported sparsity patterns; reserve unstructured pruning for storage/transfer/edge-memory wins unless you have a sparse GEMM runtime.

Pitfalls to watch for

• BatchNorm folding and operator fusions must happen before quantization; otherwise dynamic ranges and fused operations can produce unexpected errors. TensorRT fuses layers and you should calibrate after fusion or use calibration flows that are compatible with your fused graph.
• ONNX operator coverage and operator semantics mismatches can cause small numeric divergences that magnify after quantization. Sanitize ONNX and compare numeric outputs (tools below).

Compiling and tuning with TensorRT and ONNX

A practical compile-tune pipeline (repeatable, automated) looks like:

1. Export a canonical ONNX artifact from your training framework (torch.onnx.export() is the recommended path for PyTorch exports). Make the export deterministic: fixed opsets, explicit batch dims, and known input shapes where possible.
2. Sanitize and simplify the ONNX model with onnx-simplifier or use Polygraphy to compare backends and isolate mismatches before compilation. Polygraphy can run onnxruntime vs TensorRT and highlight per-layer differences.
3. Build a TensorRT engine with explicit optimization profiles to support dynamic shapes you need. Example Python snippet to create an optimization profile:

# Python / TensorRT (conceptual)
profile = builder.create_optimization_profile()
profile.set_shape("input", (1,3,224,224), (8,3,224,224), (32,3,224,224))
config.add_optimization_profile(profile)

TensorRT chooses kernels per-profile; build engines for the shape ranges that reflect production traffic.

1. Use trtexec to benchmark and to serialize engines; use a timing cache to reduce rebuild time. trtexec doubles as a fast profiler and engine generator. Example trtexec usage to build FP16 or INT8 engines:

# FP16 engine
trtexec --onnx=model.onnx --saveEngine=model_fp16.plan --fp16 --workspace=4096

# INT8 engine (requires calibration cache or calibrator)
trtexec --onnx=model.onnx \
        --minShapes=input:1x3x224x224 --optShapes=input:8x3x224x224 --maxShapes=input:32x3x224x224 \
        --int8 --calib=/path/to/calib_cache \
        --saveEngine=model_int8.plan --workspace=4096

TensorRT exposes timing caches and serialized engines; reusing them saves minutes of build time and avoids long, noisy autotuning steps during CI. ONNX Runtime’s TensorRT execution provider also highlights the benefit of caching (timing cache, engine cache) to reduce session startup time dramatically.

Calibration notes

• Build calibration tables using a representative sample set and a calibrator (examples exist in the TensorRT samples). Cache and version those calibration artifacts. Calibrating before layer fusion tends to produce portable caches; calibrating after fusion may not be portable across platforms or TensorRT versions.

Validation during compile

• Use polygraphy run to compare the compiled engine against ONNX/float32 outputs on a handful of tricky inputs (corner cases, low-light images, occlusions). Run regression tests at p95 and mAP for the target slices.

Serving strategies with Triton and autoscaling

When you need production-grade serving across many models or versions, the Triton Inference Server is the pragmatic choice: it natively hosts TensorRT engines, ONNX models, TorchScript, TensorFlow graphs, and more from a model repository layout, and exposes an HTTP/gRPC API plus Prometheus metrics for autoscaling.

Practical deployment patterns

• Put compiled TensorRT *.plan files in a Triton model repository with a config.pbtxt to control instance_group, max_batch_size, and dynamic_batching. Example minimal config.pbtxt:

name: "resnet50"
platform: "tensorrt_plan"
max_batch_size: 32
input [
  { name: "input_0" data_type: TYPE_FP32 dims: [3,224,224] }
]
output [
  { name: "output" data_type: TYPE_FP32 dims:  }
]
instance_group [
  { count: 2 kind: KIND_GPU }
]
dynamic_batching {
  preferred_batch_size: [4,8,16]
  max_queue_delay_microseconds: 1000
}

• Use Triton’s perf_analyzer for load-testing server-level behavior (batching effects, concurrency trade-offs, and network overhead). perf_analyzer reproduces client-side behavior and reports p50/p90/p95/p99 and throughput under realistic loads.

Autoscaling and metrics

• Scrape Triton’s /metrics Prometheus endpoint and drive HPA/KEDA with custom metrics such as in_flight_requests, avg_queue_delay, or gpu_utilization. Triton provides these metrics natively on the metrics endpoint. Autoscale on the metric that best predicts SLO breaches (often request queue length or p95 latency) rather than raw GPU utilization alone.

Packing and sharing GPUs

• Use multiple model instances per GPU for small models, and tune instance_group.count to trade latency for throughput. Prefer colocating models that share pre/post-processing CPU patterns to reduce host-side overhead. Test with perf_analyzer and watch server-side metrics (queue_time, compute_input, compute_infer, compute_output) to find hotspots.

Practical checklist for immediate implementation

Below is a compact, actionable checklist and a few snippets you can run now.

1) Baseline & gating

• Export baseline artifact: model.onnx, preprocess.py, postprocess.py, perf_script.sh.
• Capture: Top-1/top-5, mAP per slice, p50/p95/p99 latency, throughput (infer/sec), GPU utilization, memory footprint.
• Set accept criteria: e.g., p95_target, max_accuracy_drop, cost_reduction_target.

2) Quick wins (order matters)

• Enable FP16 inference first (often safe on NVIDIA GPUs). Benchmark with trtexec --fp16.
• Add mixed precision in training or use quantization-aware training if FP16 causes unacceptable loss.

3) Quantization protocol

• Run PTQ INT8 calibrations with a representative sample (~100–1,000 images; ~500 is a practical starting point for ImageNet-scale conv nets). Save calib_cache and version it.
• If PTQ breaks critical slices, schedule a short QAT fine-tune (1–10 epochs depending on model size) with fake-quantize ops. Track validation metrics per epoch.

4) Pruning protocol

• Choose structured pruning for GPUs (channel/filter/block) or target the platform-supported 2:4 pattern if you intend to use AMPERE/Hopper sparse acceleration. Retrain (or fine-tune) after pruning to recover accuracy.
• Benchmark both dense+quantized and sparse+quantized flows; sparse speedups require library/runtime support (cuSPARSELt / TensorRT ASP flows).

5) Compile & tune

• Export sanitized ONNX (torch.onnx.export() with dynamo=True or the recommended exporter) and run Polygraphy to check parity.
• Build TensorRT engines with optimization profiles that represent production shape ranges and save the serialized engine and timing cache. Use trtexec to iterate quickly.
• Leverage --useCudaGraph in trtexec/runtime if you have stable input shapes and need ultra-low latency.

6) Serve & autoscale

• Put the compiled plan into Triton model repo with config.pbtxt assigning proper instance_group and dynamic_batching.
• Load-test with perf_analyzer and collect metrics from Triton /metrics. Create HPA/KEDA rule(s) on a chosen metric (queue size or p95 latency).

7) Validation & rollback

• Run a production-mirroring canary: route a percent of traffic to the new optimized model; compare per-slice metrics (latency and accuracy). Measure drift, and have a rollback criterion (e.g., >0.5 absolute accuracy hit on any monitored slice or 2× p95 regression).
• Store the engine, calibration cache, and config.pbtxt in the model registry; tag with the exact TensorRT/Triton/container versions so the artifact is reproducible.

Helpful formulas and snippets

• Cost per inference (simple): cost_per_inference = (instance_hourly_cost / 3600) / throughput_per_sec
• p95 calculation (Python):

import numpy as np
lat_ms = np.array([...])  # list of per-request latencies in ms
p95 = np.percentile(lat_ms, 95)

Quick pointers for edge deployment

• For Jetson and other embedded targets, use JetPack's bundled TensorRT and test on-device early; ONNX Runtime and TensorRT are available for Jetson (JetPack) and often are the easiest path to rapid iteration. Export, compile, test latencies on the actual SOM (system-on-module), and profile for CPU bottlenecks (preproc) before claiming GPU wins.

Important: Always tie an optimization to a measurable, versioned artifact (model.plan / calib_cache / config.pbtxt) and an automated perf test. That combination is what makes model optimization safe and repeatable.

Measure, validate, and write down the trade-off you are willing to accept between accuracy and latency. Apply the smallest change that meets the SLO (FP16 → INT8 → structured sparsity → QAT) and keep the full experimental record in version control so you can reproduce the wins on new hardware generations.

Sources:
NVIDIA TensorRT Developer Guide - Core TensorRT concepts: precision modes (FP32/FP16/INT8), optimization profiles, trtexec usage and performance benchmarking; guidance on engine building and runtime tuning.

Performing Inference In INT8 Precision (TensorRT docs) - Details on INT8 calibration, calibrator APIs, calibration cache portability, and practical notes (recommended calibration sample sizes).

Triton Model Repository (NVIDIA Triton docs) - Model repository layout, config.pbtxt fields, platform-specific model files, and version policies.

Triton Performance Analyzer (perf_analyzer) guide - How to benchmark Triton-served models, options for realistic input data, and comparing batching/concurrency trade-offs.

Quantization-Aware Training for Large Language Models (PyTorch blog) - Practical QAT workflows, reasons to prefer QAT over PTQ in some cases, and PyTorch QAT tooling notes.

ONNX Runtime — TensorRT Execution Provider - Details on using TensorRT as an ONNX Runtime EP, engine/timing caches, and the speedups from caches.

Accelerating Inference with Sparsity Using the NVIDIA Ampere Architecture and NVIDIA TensorRT - Explanation of 2:4 structured sparsity, sparse Tensor Cores and practical sparse retraining workflow and speedups.

Learning both Weights and Connections for Efficient Neural Network (Han et al., 2015) - Foundational pruning methodology and empirical results showing large parameter reductions with retraining.

Polygraphy documentation (NVIDIA) - Tooling to compare backends, sanitize ONNX, and debug TensorRT/ONNX numeric mismatches.

Exporting a PyTorch model to ONNX (PyTorch docs) - Recommended ONNX export practices and the torch.onnx.export() API for stable ONNX artifacts.

Triton Metrics (Prometheus) — Triton docs - Available Triton Prometheus metrics, endpoint details, and configuration options.

Exploiting Ampere Structured Sparsity with cuSPARSELt (NVIDIA blog) - cuSPARSELt library overview for sparse GEMM and integration points for sparse acceleration on Ampere GPUs.