SGLang
An Apache-2.0 inference engine and OpenAI- and Anthropic-compatible model server for LLM, vision-language and diffusion models, built for large-scale serving.
SGLang is an Apache-2.0 inference engine and model server for LLMs, vision-language and diffusion models, with OpenAI-compatible endpoints and an Anthropic-compatible /v1/messages endpoint Claude Code can target directly. About 37,600 GitHub stars as of October 2026; latest release v0.5.22. LMSYS-hosted; core contributors launched RadixArk in May 2026.
SGLang is the serving engine you reach for when the workload looks like agents rather than chat: long shared prefixes, tool loops, RL rollouts, and clients that expect either the OpenAI or the Anthropic wire format. The README describes it as "an open-source inference framework for large language, vision-language, and diffusion models, optimized for agentic workloads, large-scale serving, and RL rollouts", and says the project is "hosted by LMSYS, a non-profit open-source organization".
Two numbers frame its momentum: about 37,600 GitHub stars as of October 2026, and a release cadence of roughly every two weeks, with v0.5.22 published on 2026-10-19 carrying "950 PRs from 245 contributors".
Highlights
- Three model families, one package. LLMs, vision-language models and, through SGLang Diffusion, image and video generation, all in the
sglangPython package. - OpenAI-compatible endpoints. Chat completions, completions, vision and embeddings, so existing clients keep working.
- Anthropic-compatible endpoint.
/v1/messagesand/v1/messages/count_tokensare registered automatically on every server, so the Anthropic SDKs and agentic CLIs such as Claude Code work without changes. - Wide hardware support. NVIDIA (B200 through RTX 30/40/50, DGX Spark, Jetson Orin), AMD Instinct MI300X to MI355X, Google TPU v6e and v7, Intel Arc and Xeon, Apple Silicon via Metal and MLX, Huawei Ascend and Moore Threads.
- Agent and RL ecosystem. SpecForge for training speculative-decoding draft models; RL frameworks Miles, slime, AReaL, Tunix and verl; orchestration with SMG, RBG, llm-d, Ray Serve and NVIDIA Dynamo.
- SGLang Omni. Audio model serving for text-to-speech and speech recognition.
In an AI-assisted workflow
The quickstart targets Linux with an NVIDIA GPU and a CUDA 13-compatible driver:
docker pull lmsysorg/sglang:latest
docker run --rm --gpus all --ipc=host -p 30000:30000 lmsysorg/sglang:latest \
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
# or without Docker:
uv pip install --prerelease=allow sglang
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000Verify with a curl to http://localhost:30000/v1/chat/completions. The same server also answers on /v1/messages, which is the part that matters for coding agents: point Claude Code at it and the Anthropic SDK or CLI needs no changes, and the docs call out the CLAUDE_CODE_ATTRIBUTION_HEADER setting as required for good prefix-cache reuse. The Claude Code tips guide covers the client side. For a chat front end, Open WebUI or AnythingLLM can sit on the OpenAI-compatible endpoint.
WARNING
SGLang loads Hugging Face checkpoints, not GGUF files, and it targets multi-GPU and multi-node serving (tensor and expert parallelism, PD disaggregation, speculative decoding). For a single Mac or CPU box, llama.cpp, Ollama or LM Studio are the simpler path; Apple Silicon support only arrived in v0.5.22.
NOTE
The default CUDA version moved to 13.0 across SGLang, sgl-kernel and the Docker images in the v0.5.11 era, with PyTorch upgraded from 2.9 to 2.11. The homepage still documents a CUDA 12.9 install through extra index URLs if your driver is older.
Good to know
License and governance. Apache-2.0, and the LICENSE file, the GitHub API and the README badge agree. The open-source project remains under sgl-project and LMSYS; on 2026-05-05 core contributors launched RadixArk with a 100 million dollar seed round at a 400 million dollar post-money valuation, led by Accel and co-led by Spark Capital, with NVentures, AMD, MediaTek and Databricks among the participants, to "grow SGLang and Miles" and "build managed infrastructure for teams developing AI at scale". RadixArk's post does not mention LMSYS and the README still says LMSYS-hosted; no formal stewardship transfer has been stated on a primary source. No prices are published for the managed platform.
2026 changes. The January blog post on SGLang-Diffusion reported the diffusion engine (Flux, Qwen-Image, Wan2.2) running "up to 2.5x faster than our initial release". v0.5.10 (April) enabled piecewise CUDA graphs by default and added elastic expert parallelism, HiSparse sparse attention and LoRA for MoE layers. Day-0 support posts followed for DeepSeek-V4 (April), Inkling (July) and Kimi K3 (July). v0.5.22 (October) says models load "10x faster with the new InstantTensor loader", claims up to 27 percent more throughput from PD disaggregation, adds the Apple Silicon PyTorch backend, and lists new models including EmbeddingGemma 2 and HunyuanImage-3.0. SGLang Summit 2026 runs November 12 to 13 in San Francisco.
Against the alternatives: vLLM is the closer comparison, a sibling Apache-2.0 Python engine with a bigger community and a place in more hosted stacks; SGLang's case is prefix caching for agent traffic, the diffusion engine in the same package, the Anthropic endpoint and the hardware table. llama.cpp is a different job: a C/C++ engine for GGUF models on laptops, CPUs and single-GPU boxes. The best local LLM tools roundup draws the line where local ends and serving begins, and the open-source LLM roundup lists the models worth serving on either engine.
Frequently asked questions
- Is SGLang free?
- Yes. SGLang is Apache-2.0 and the project sells nothing itself. RadixArk, the company its core contributors launched in May 2026, builds managed infrastructure on top of it; we did not find a public price list for that, so no figures are quoted here.
- SGLang vs vLLM?
- Both are Python GPU serving engines with OpenAI-compatible servers under Apache-2.0. SGLang differentiates on RadixAttention prefix caching for agentic and RL workloads, a built-in diffusion engine for image and video in the same package, an Anthropic-compatible /v1/messages endpoint for Claude Code, and the widest hardware table (NVIDIA, AMD, TPU, Intel, Apple Silicon, Ascend). vLLM has the larger community and is the default in more hosted stacks. We do not cite benchmark numbers between them.
- Can Claude Code use SGLang?
- Yes. Every SGLang server registers an Anthropic-compatible /v1/messages endpoint (plus /v1/messages/count_tokens) with no extra flag, so the Anthropic SDKs and agentic CLIs such as Claude Code can talk to it without changes. The docs cover pointing Claude Code at the server, including the CLAUDE_CODE_ATTRIBUTION_HEADER setting that is required for good prefix-cache reuse.
- Does SGLang run on a Mac?
- As of v0.5.22 (October 2026) SGLang runs on Apple Silicon Macs with the standard PyTorch backend, alongside MLX. It is still built for multi-GPU datacenter serving, though; for one laptop or a CPU box, llama.cpp, Ollama or LM Studio are the simpler path.
Filed under
sglang · inference · llm-serving · open-source · anthropic-compatible · diffusion
Related
- Best Open-Source LLMs in 2026: The Open-Weight Families ComparedThe open-weight LLM families current in October 2026, from DeepSeek and Qwen to Gemma 4 and Muse, with sizes, context, the exact licence and where to run them.
- AnythingLLMAn MIT-licensed private AI app for chatting with documents and running agents with MCP tools on any local or cloud LLM, as a desktop app or Docker server.
- Open WebUISelf-hosted AI workspace that fronts Ollama and any OpenAI-compatible API with chat, RAG, MCP tools, agents and enterprise auth, via pip, Docker or Kubernetes.
- vLLMA high-throughput, memory-efficient inference and serving engine for LLMs, with PagedAttention, continuous batching, and an OpenAI-compatible API server.
- GGUFGGUF is the single-file binary model format used by llama.cpp and other GGML-based runtimes, storing quantized tensors plus typed metadata in one file.
- LLM Inference EngineerUse this agent to serve and optimize self-hosted LLM inference — sizing GPUs, configuring a serving engine like vLLM (continuous batching, PagedAttention, tensor parallelism), applying quantization, and tuning throughput and tail latency against a cost and p95 budget. Examples — "serve Llama-3-70B at p95 under 2s on our GPUs", "our self-hosted model is slow and the GPUs sit half-idle — raise throughput", "quantize this model to fit one GPU without wrecking quality".
- Best Tools for Running LLMs Locally in 2026The local LLM stack ranked by job — Ollama, LM Studio, Jan, llama.cpp, vLLM, and LiteLLM — with a verdict each and the line where local inference stops paying.
- OllamaAn open-source tool to run open-weight LLMs locally with a single command, including a local OpenAI-compatible API.