Skip to content
All tags

#llm-serving

2 posts
tech deep-dive Self-Hosted Inference

TGI: HuggingFace's LLM Inference Server, and Why It Entered Maintenance Mode

Text Generation Inference (TGI) is HuggingFace's own LLM inference server, built in Rust and Python. It pioneered continuous batching and Flash Attention in open-source inference engines. The GitHub repository was archived on March 21, 2026, and HuggingFace recommends migrating to vLLM or SGLang. TGI still matters: it defined the architectural baseline that successor engines inherited, and many HuggingFace Inference Endpoints still run it.

#tgi#huggingface#llm-serving#inference#gpu#self-hosted
tech deep-dive Self-Hosted Inference

Self-Hosting Inference with SGLang: RadixAttention, OpenAI APIs, and Multi-GPU Serving

SGLang is an inference engine for generative models. RadixAttention reuses KV cache across shared prefixes, while OpenAI-compatible APIs, structured output, and multi-GPU parallelism support production LLM serving; it is not a complete product backend.

#sglang#llm-serving#inference#gpu#self-hosted#openai-api