SGLang

High-performance serving framework for large language models and multimodal models

Achieve low-latency, high-throughput inference from a single GPU to large distributed clusters with RadixAttention, continuous batching, and advanced optimizations.

e.currentTarget.style.backgroundColor = '#d4621f'} onMouseLeave={(e) => e.currentTarget.style.backgroundColor = '#b14421'}> Get started API reference View on GitHub

Quick start

Get SGLang running in minutes

Install SGLang using pip: ```bash theme={null} pip install "sglang[all]" ``` For AMD ROCm GPUs, use `pip install "sglang[all]" --index-url https://download.pytorch.org/whl/rocm6.2` Start the SGLang server with a model: ```bash theme={null} python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 30000 ``` Or use the CLI: ```bash theme={null} sglang serve meta-llama/Meta-Llama-3.1-8B-Instruct --port 30000 ``` Use the OpenAI-compatible API to send requests: ```python theme={null} import openai client = openai.Client( base_url="http://localhost:30000/v1", api_key="EMPTY" ) response = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "What is the capital of France?"} ], temperature=0.7 ) print(response.choices[0].message.content) ``` ``` The capital of France is Paris. ```

Key features

Everything you need for high-performance LLM serving

Efficient prefix caching that automatically reuses KV cache across requests Zero-overhead CPU scheduler for optimal GPU utilization Multi-dimensional parallelism: tensor, pipeline, expert, and data parallelism Fast JSON decoding with compressed finite state machines Serve multiple LoRA adapters in a single batch Support for FP4, FP8, INT4, AWQ, and GPTQ quantization

Explore by topic

Deep dive into specific areas

Understand SGLang's architecture and design Configure and run the serving backend Build prompts with constrained generation 100+ supported LLM and multimodal models Structured outputs, LoRA, speculative decoding Performance tuning and optimization techniques

Hardware & deployment

Run SGLang on your infrastructure

Supported hardware

• NVIDIA GPUs (GB200, B300, H100, A100, Spark)
• AMD GPUs (MI355, MI300)
• Intel Xeon CPUs
• Google TPUs
• Ascend NPUs

Deployment options

→ Docker containers → Kubernetes clusters → Cloud platforms (AWS, GCP, Azure) → Multi-node distributed setup

Resources

Additional help and information

Common questions and answers Configuration reference Debugging and common issues Blog posts, papers, and talks

Join the community

SGLang powers over 400,000 GPUs worldwide and is trusted by leading enterprises. Get help, share feedback, and connect with other users.

e.currentTarget.style.backgroundColor = '#d4621f'} onMouseLeave={(e) => e.currentTarget.style.backgroundColor = '#b14421'}> Join Slack GitHub Weekly dev meeting