TensorRT FP16
It remained inside the accuracy budget, used the least measured board power, and produced the highest FPS/W on the tested RTX 5070.
Computer vision · model deployment
YOLO26 for dense retail detection
I fine-tuned YOLO26-s on SKU-110K, then built GPU, CPU, and browser inference paths with TensorRT, ONNX Runtime, and WebGPU. I measured validation accuracy, model latency, and GPU board power.
01 · Live demo
Browser inference
WebGPU / WASM · on-deviceThe page loads the deployed 38 MB FP32 ONNX model and runs it locally with WebGPU or the WASM CPU fallback. Use a sample shelf, the video, or a local file. Selected media is not uploaded.
Initialising detector
connecting…
Select a shelf or drop an image/video
Nothing is uploaded
Nothing is uploaded
Data handling: the page uses WebGPU when available and falls back to WebAssembly on the CPU. Your selected media stays in this tab.
0.5716
FP32 ONNX reference accuracy
mAP@50–95
9.3
Highest measured GPU efficiency
TensorRT FP16 · reciprocal FPS/W
1.8 ms
Lowest measured GPU p50
TensorRT FP8
22.9 ms
Controlled browser p50
WebGPU · RTX 5070
02 · Benchmarks
Accuracy, latency, and GPU power
TensorRT FP16 met the 2% relative mAP-loss limit and had the highest measured GPU efficiency. FP8 reduced p50 latency by 0.1 ms, but used 20 W more board power and increased relative mAP loss from 0.06% to 1.12%. TensorRT INT8 exceeded the limit.
| Deployment path | mAP@50–95 | Relative Δ | p50 | Reciprocal FPS* | Power | FPS/W |
|---|---|---|---|---|---|---|
| PyTorch · GPU FP32 | 0.5720 | — | 12.9 ms | 77 | 66 W | 1.2 |
| TensorRT · GPU FP16selected | 0.5713 | −0.06% | 1.9 ms | 536 | 58 W | 9.3 |
| TensorRT · GPU FP8 | 0.5652 | −1.12% | 1.8 ms | 560 | 78 W | 7.2 |
| TensorRT · GPU INT8exceeds limit | 0.5393 | −5.65% | 2.7 ms | 374 | 88 W | 4.2 |
| ORT · CPU FP32 | 0.5716 | — | 52.5 ms | 19 | — | — |
| ORT · CPU INT8 | 0.5675 | −0.72% | 37.7 ms | 27 | — | — |
| ONNX · browser WebGPU† | 0.5716† | — | 22.9 ms | 44 | — | — |
* Reciprocal FPS is 1,000 ÷ p50 latency, not measured end-to-end video throughput. † Browser accuracy is the validated FP32 ONNX reference, not a separate in-browser mAP run. Figures are from one hardware setup.
Accuracy loss vs FP32 ONNX
Lower is better · track maximum = 6% · budget = 2%
2% acceptance boundary
GPU efficiency
Reciprocal FPS per measured board watt · higher is better
INT8 behavior was runtime-specific
ORT used per-channel QDQ and retained 94 detection-head nodes in FP32. TensorRT used a different build policy, so the rows are not equivalent INT8 treatments.
TensorRT INT8 affected mAP@50–95 more than mAP@50
The larger change in the stricter localization metric is consistent with less precise boxes, but these measurements do not establish the cause.
03 · Inference flow
Browser execution path
Each input is letterboxed to 640 × 640 without changing its aspect ratio. The padding and scale are retained so the model's output coordinates can be mapped back to the source image.
loadImage()Read the sourceKeep its original width and height for the final coordinate transform.
preprocess()Letterbox to 640²Pad with 114 gray, convert RGB to 0–1 floats, then arrange NCHW.
session.run()Execute the graphWebGPU first; WebAssembly CPU fallback if GPU access is unavailable.
decode()Decode model outputKeep confidence ≥ 0.25 and reverse the scale plus padding offsets.
drawDetections()Render detectionsScale source coordinates to the visible canvas and draw boxes locally.
Post-processing boundary: the end-to-end YOLO26 export already returns the selected [1,600,6] detections. The browser does not run a separate NMS implementation.
04 · Implementation
Training, export, and evaluation
I trained one detector, retained an FP32 reference, and built each target-specific deployment path. I applied a 2% relative mAP-loss limit when comparing the results. These are complete deployment paths, not a precision-only ablation.
01
Set the evaluation criteria
Require relative mAP@50–95 loss at or below 2%. Compare p50 model latency and available power measurements among passing configurations.
Reason
Dense, overlapping products require accurate localization. The accuracy budget prevents latency alone from determining the selected configuration.
Measurement boundary
Power was recorded only for GPU paths, so CPU and browser energy efficiency cannot be ranked.
02
Fine-tune for dense shelves
YOLO26-s · SKU-110K · 80 epochs · 640 px · automatic batch size · mosaic disabled for the final 10 epochs.
Inspect train.py ↗Decision
max_det was raised from 300 to 600. Dense scenes average roughly 150 objects and can exceed the default cap; clipping candidates would make recall look worse before deployment even begins.
Why close mosaic
Mosaic improves diversity early. Turning it off for the final 10 epochs lets the model finish on natural image boundaries rather than synthetic tile seams.
03
Preserve a portable reference
Export FP32 ONNX with static batch 1, 640 × 640 input, opset 19, graph simplification, and a 600-detection output cap.
Inspect export_onnx.py ↗Why FP32 first
The uncompressed export is an accuracy reference and a portable input for ONNX Runtime, ModelOpt FP8, and the browser. Quantizing later keeps each target's policy explicit.
Why static
The benchmark measures single-frame latency at one input size. A fixed graph reduces shape overhead and is easier to compile for WebGPU.
04
Optimize for each runtime
TensorRT built GPU paths; ORT produced CPU QDQ INT8; onnxruntime-web executed the FP32 graph in the browser.
Inspect deployment code ↗CPU INT8 choice
Use per-channel weight scales and leave the Detect head in FP32 after whole-graph quantization collapsed confidences. This retained FP32 computation in the block that failed under whole-graph quantization.
Important confound
TensorRT FP16/INT8 export directly from the checkpoint, FP8 starts from ONNX, and ORT changes quantization scope. The rows compare complete paths, not bit width in isolation.
05
Benchmark the deployment paths
Accuracy: held-out 588-image / ~91k-box validation split. Latency: fixed 640² tensors, discarded warmups, sequential session.run calls.
Protocol
GPU: 30 warmups + 200 runs with CUDA synchronization and NVML sampling every 4 ms. CPU: 10 + 100 runs. WebGPU: 12 + 40 runs in headed Chromium.
Interpretation
Use p50 for a typical single request and p95 for tail behavior. The published page reports p50; raw logs and p95 outputs are not versioned.
05 · Engineering details
Toolchain, constraints, and limitations
The project records runtime-specific implementation problems, blocked configurations, and evidence gaps alongside the benchmark results.
Toolchain by stage
TRAINPython 3.11 · PyTorch 2.11 + CUDA 12.8 · Ultralytics 8.4.87
EXPORTONNX opset 19 · onnxslim · NumPy
GPUTensorRT 11.1 · NVIDIA ModelOpt 0.45 · NVML
CPUONNX Runtime · static QDQ · per-channel weights
BROWSERonnxruntime-web 1.22 · WebGPU / WASM · Canvas API
VERIFYPlaywright · headed Chromium benchmark · headless end-to-end demo test
Observed failure modes
- 01INT8 detection head collapsed.Quantizing the entire ORT graph drove confidences to zero. The fix was to exclude 94 head nodes and quantize the backbone/neck per channel.
- 02TopK broke symbolic shape inference.The NMS-free head’s fixed top-600 selection blocked ORT’s symbolic pass, so preprocessing skips that pass on an already simplified static graph.
- 03FP4 was blocked by the host toolchain.ModelOpt’s kernel could not load across the installed CUDA 12.8 PyTorch and CUDA 13.0 compiler split. FP4 is unmeasured, not a negative result.
- 04Headless Chromium hid hardware WebGPU.The controlled browser benchmark therefore runs headed and requests WebGPU only, avoiding a silent WASM measurement.
What the evidence supports
- FP16 is the best measured GPU deployment on this RTX 5070 under the stated budget.
- ORT INT8 improves CPU p50 by about 1.39× while staying within the accuracy budget.
- The NMS-free export works client-side and removes a separate browser NMS stage.
Limits of the current evidence
- No repeated training seeds, uncertainty intervals, raw benchmark logs, or training curves are versioned.
- GPU power is board power without idle subtraction; CPU and browser power were not measured.
- There is no matched TensorRT head-exclusion ablation and no NMS-based detector baseline.
- Browser latency depends on the visitor’s device; the live number is not the controlled RTX 5070 result.