Skip to content
All tags

#triton-inference-server

2 posts
tech guide Technology Choices in the AI Era

How to Pick a Self-Hosted Inference Server: From Ollama to Xinference, Six Tools and Their Trade-Offs

Self-hosted inference servers fall into three layers: execution engine (llama.cpp), serving engine (vLLM, SGLang), and model management platform (Ollama, Xinference, Triton). Picking the right layer matters more than picking the right tool — ask where your bottleneck is before deciding where to add complexity.

#llm-inference#self-hosting#vllm#sglang#ollama#xinference#triton-inference-server#ray-serve#gpu
tech deep-dive Self-Hosted Inference

NVIDIA Triton Inference Server: Multi-Framework Models, Dynamic Batching, and Pipelines

Triton Inference Server serves TensorRT, ONNX, PyTorch, and other models through consistent HTTP and gRPC APIs. Its defining tools are the model repository, dynamic batching, instance groups, and ensembles—not LLM-specific KV-cache scheduling.

#triton-inference-server#nvidia#inference#model-serving#gpu#self-hosted