Launch Server

Learn how to launch and configure the SGLang inference server

Overview

SGLang provides a high-performance inference server that can be launched using the sglang serve command. The server supports various deployment modes including HTTP, gRPC, and disaggregated prefill-decode architectures.

Basic Usage

Starting the Server

The simplest way to launch a server:

sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct

Common Launch Options

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --tp-size 2 \
  --mem-fraction-static 0.8

Server Modes

HTTP Server (Default)

The default server mode provides OpenAI-compatible HTTP endpoints:

sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct

Available Endpoints:

• /v1/chat/completions - Chat completions API
• /v1/completions - Text completions API
• /v1/embeddings - Embeddings generation
• /health - Health check endpoint
• /get_model_info - Model information

gRPC Server

For lower latency in high-throughput scenarios:

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --grpc-mode

Disaggregated Prefill-Decode

SGLang supports separating prefill and decode into different instances for optimized resource utilization.

Prefill Server:

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend mooncake

Decode Server:

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mooncake

Parallelism Options

Tensor Parallelism

Split model across multiple GPUs:

sglang serve \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp-size 4

Pipeline Parallelism

Distribute model layers across GPUs:

sglang serve \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp-size 2 \
  --pp-size 2

Data Parallelism

Run multiple replicas for increased throughput:

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --dp-size 4 \
  --load-balance-method round_robin

Performance Optimization

Memory Management

mem-fraction-static · float · default: "auto"

Fraction of GPU memory to allocate for static usage (model weights + KV cache). Default is automatically calculated based on GPU memory and model size.

chunked-prefill-size · int · default: "auto"

Maximum number of tokens to process in a single prefill batch. Larger values improve throughput but require more memory.

max-total-tokens · int · default: "null"

Maximum total number of tokens in the KV cache pool. Limits memory usage.

CUDA Graph Optimization

CUDA graphs reduce kernel launch overhead:

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --cuda-graph-max-bs 256

cuda-graph-max-bs · int · default: "auto"

Maximum batch size for CUDA graph capture. Higher values enable batching more requests but require more memory. Set to 0 to disable.

disable-cuda-graph · bool · default: "false"

Disable CUDA graph optimization entirely.

Radix Attention Cache

Accelerate requests with shared prefixes:

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --enable-cache-report

disable-radix-cache · bool · default: "false"

Disable the radix attention cache (prefix caching).

enable-cache-report · bool · default: "false"

Include cache hit rate statistics in API responses.

Multi-Node Deployment

For distributed training across multiple machines:

# Node 0 (rank 0)
sglang serve \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp-size 8 \
  --nnodes 2 \
  --node-rank 0 \
  --dist-init-addr 192.168.1.100:5000

# Node 1 (rank 1)
sglang serve \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp-size 8 \
  --nnodes 2 \
  --node-rank 1 \
  --dist-init-addr 192.168.1.100:5000

Quantization

Reduce memory usage with quantization:

# FP8 Quantization
sglang serve \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --quantization fp8

# AWQ 4-bit Quantization
sglang serve \
  --model-path TheBloke/Llama-2-13B-AWQ \
  --quantization awq

# GPTQ Quantization
sglang serve \
  --model-path TheBloke/Llama-2-13B-GPTQ \
  --quantization gptq

Supported quantization methods:

• fp8 - FP8 quantization for reduced memory
• awq - Activation-aware Weight Quantization
• gptq - GPTQ quantization
• marlin - Marlin sparse format
• bitsandbytes - 8-bit and 4-bit quantization

Monitoring and Logging

Enable Metrics

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --enable-metrics

Metrics are exposed at http://localhost:30000/metrics in Prometheus format.

Request Logging

sglang serve \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --log-requests \
  --log-level info

log-requests · bool · default: "false"

Log all incoming requests and responses.

log-level · string · default: "info"

Set logging verbosity. Options: debug, info, warning, error.

Health Checks and Warmup

Server Warmup

By default, the server runs warmup requests to initialize CUDA graphs and caches:

skip-server-warmup · bool · default: "false"

Skip the warmup phase on server startup.

warmups · string · default: "null"

Specify custom warmup functions (comma-separated) to run before server starts. Example: --warmups=warmup_name1,warmup_name2

Health Endpoint

Check server health:

curl http://localhost:30000/health

Environment Variables

SGLang respects several environment variables:

• CUDA_VISIBLE_DEVICES - Control which GPUs are used
• NCCL_SOCKET_IFNAME - Network interface for multi-node communication
• SGLANG_USE_MODELSCOPE - Download models from ModelScope instead of HuggingFace
• HF_TOKEN - HuggingFace authentication token for gated models

Python API

You can also launch the server programmatically:

from sglang import Engine

engine = Engine(
    model_path="meta-llama/Llama-3.1-8B-Instruct",
    tp_size=2,
    mem_fraction_static=0.8
)

# Use the engine
response = engine.generate(
    prompt="Hello, how are you?",
    sampling_params={"temperature": 0.8, "max_new_tokens": 128}
)

print(response["text"])

See Also

• Server Arguments - Complete reference of all server arguments
• OpenAI Compatible API - HTTP API documentation
• Sampling Parameters - Control generation behavior