Skip to content
All tags

#openai-api

3 posts
tech deep-dive Technology Choices in the AI Era

Xinference: One Platform to Manage LLM, Embedding, Speech, and Image Models

Xinference wraps vLLM, SGLang, llama.cpp, Transformers, and MLX under a single management layer, using a Web UI and OpenAI-compatible API to manage LLMs, embedding, rerank, speech, and image models — suited for self-hosted deployments that need multiple model types to coexist. But the management layer's parsing logic also creates a larger attack surface than pure serving engines (CVE-2026-61539 is a case study).

#xinference#llm-inference#self-hosting#vllm#sglang#model-serving#gpu#openai-api
ai deep-dive

Cerebras Inference: Know the Bottleneck Before Putting Wafer-Scale Speed in an Agent Loop

Cerebras can dramatically accelerate generation on supported models, but agent latency still depends on prefill, tool I/O, model quality, and platform compatibility.

#cerebras#inference#wafer-scale-engine#ai-agent#llm#openai-api
tech deep-dive Self-Hosted Inference

Self-Hosting Inference with SGLang: RadixAttention, OpenAI APIs, and Multi-GPU Serving

SGLang is an inference engine for generative models. RadixAttention reuses KV cache across shared prefixes, while OpenAI-compatible APIs, structured output, and multi-GPU parallelism support production LLM serving; it is not a complete product backend.

#sglang#llm-serving#inference#gpu#self-hosted#openai-api