Skip to content
All tags

#xinference

2 posts
tech guide Technology Choices in the AI Era

How to Pick a Self-Hosted Inference Server: From Ollama to Xinference, Six Tools and Their Trade-Offs

Self-hosted inference servers fall into three layers: execution engine (llama.cpp), serving engine (vLLM, SGLang), and model management platform (Ollama, Xinference, Triton). Picking the right layer matters more than picking the right tool — ask where your bottleneck is before deciding where to add complexity.

#llm-inference#self-hosting#vllm#sglang#ollama#xinference#triton-inference-server#ray-serve#gpu
tech deep-dive Technology Choices in the AI Era

Xinference: One Platform to Manage LLM, Embedding, Speech, and Image Models

Xinference wraps vLLM, SGLang, llama.cpp, Transformers, and MLX under a single management layer, using a Web UI and OpenAI-compatible API to manage LLMs, embedding, rerank, speech, and image models — suited for self-hosted deployments that need multiple model types to coexist. But the management layer's parsing logic also creates a larger attack surface than pure serving engines (CVE-2026-61539 is a case study).

#xinference#llm-inference#self-hosting#vllm#sglang#model-serving#gpu#openai-api