Skip to content
All tags

#triton

2 posts
tech guide Self-Hosted Inference

Self-Hosted Inference Overview: When Running Your Own Models Makes Sense

The key question in self-hosted inference isn't how fast the engine is — it's your GPU utilization. A fully saturated A100 costs ~$0.70 per million output tokens; at 10% utilization that becomes $7, more than most cloud APIs. This overview maps seven tools across three layers to help you decide which layer you need.

#self-hosted#llm-inference#vllm#sglang#tensorrt-llm#triton#ray-serve#gpu
ai deep-dive Reading Stanford CS336

CS336 Lecture 6: Benchmark and Profile Before Writing a Triton Kernel

Lecture 6 turns GPU principles into kernels: benchmark scaling across shapes, profile actual calls and time, then implement GeLU, softmax, reductions, and tiled matrix multiplication in Triton. Speed begins with measuring correctly.

#cs336#triton#gpu#cuda#performance