Skip to content
All tags

#tensorrt-llm

2 posts
tech guide Self-Hosted Inference

Self-Hosted Inference Overview: When Running Your Own Models Makes Sense

The key question in self-hosted inference isn't how fast the engine is — it's your GPU utilization. A fully saturated A100 costs ~$0.70 per million output tokens; at 10% utilization that becomes $7, more than most cloud APIs. This overview maps seven tools across three layers to help you decide which layer you need.

#self-hosted#llm-inference#vllm#sglang#tensorrt-llm#triton#ray-serve#gpu
tech deep-dive Self-Hosted Inference

TensorRT-LLM: The Compile-for-Performance NVIDIA-Only LLM Inference Engine

TensorRT-LLM is NVIDIA's open-source LLM inference library (Apache 2.0). It offline-compiles model weights and compute graphs into optimized TensorRT engines, then serves them with custom CUDA kernels, in-flight batching, and multi-dimensional parallelism. The cost: NVIDIA GPUs only, compilation takes tens of minutes, and switching models or quantization means rebuilding.

#tensorrt-llm#nvidia#llm-inference#gpu#self-hosted#cuda