ServerArgs

Configuration options for SGLang server

ServerArgs

The ServerArgs class contains all configuration options for launching an SGLang server or engine. These arguments control model loading, memory management, parallelism, kernel backends, and optimization settings.

Usage

from sglang import Engine
from sglang.srt.server_args import ServerArgs

# Option 1: Pass arguments directly to Engine
engine = Engine(
    model_path="meta-llama/Llama-3.1-8B-Instruct",
    tp_size=2,
    mem_fraction_static=0.85
)

# Option 2: Create ServerArgs first
server_args = ServerArgs(
    model_path="meta-llama/Llama-3.1-8B-Instruct",
    tp_size=2,
    mem_fraction_static=0.85
)
engine = Engine(server_args=server_args)

Model and Tokenizer

model_path · str · required

Path to the model on Hugging Face Hub or local filesystem.

model_path="meta-llama/Llama-3.1-8B-Instruct"
# or
model_path="/path/to/local/model"

tokenizer_path · Optional[str] · default: "None"

Path to tokenizer. Defaults to model_path if not specified.

tokenizer_mode · str · default: "auto"

Tokenizer mode. Options: "auto", "slow", "fast".

skip_tokenizer_init · bool · default: "False"

Skip tokenizer initialization. Useful when passing pre-tokenized input_ids.

load_format · str · default: "auto"

Model weight loading format.

Options: "auto", "pt", "safetensors", "npcache", "dummy", "gguf", "bitsandbytes"

trust_remote_code · bool · default: "False"

Trust remote code when loading models from Hugging Face.

context_length · Optional[int] · default: "None"

Maximum context length. Auto-detected from model config if not specified.

revision · Optional[str] · default: "None"

Model revision (branch, tag, or commit) to use from Hugging Face.

HTTP Server

host · str · default: "127.0.0.1"

Server host address.

port · int · default: "30000"

Server port number.

api_key · Optional[str] · default: "None"

API key for authentication.

served_model_name · Optional[str] · default: "None"

Model name to report in API responses. Defaults to model_path.

Data Type and Quantization

dtype · str · default: "auto"

Data type for model weights and computation.

Options: "auto", "float16", "bfloat16", "float32"

quantization · Optional[str] · default: "None"

Quantization method.

Options: "awq", "fp8", "gptq", "marlin", "bitsandbytes", "gguf", and more.

quantization="awq"  # For AWQ quantized models
quantization="fp8"  # For FP8 quantization

kv_cache_dtype · str · default: "auto"

Data type for KV cache.

Options: "auto", "fp8_e4m3", "fp8_e5m2", "bfloat16", "float16"

Using FP8 for KV cache can significantly reduce memory usage.

Memory and Scheduling

mem_fraction_static · Optional[float] · default: "None"

Fraction of GPU memory to use for model weights and KV cache.

Auto-calculated based on GPU memory capacity if not specified.

mem_fraction_static=0.85  # Use 85% of GPU memory

max_total_tokens · Optional[int] · default: "None"

Maximum total tokens in the KV cache pool.

This is the maximum number of tokens that can be cached across all requests.

max_running_requests · Optional[int] · default: "None"

Maximum number of requests to process simultaneously.

max_queued_requests · Optional[int] · default: "None"

Maximum number of requests to queue when busy.

chunked_prefill_size · Optional[int] · default: "None"

Chunk size for chunked prefill.

Auto-calculated based on GPU memory capacity if not specified.

• Small GPUs (<20GB): 2048
• Medium GPUs (20-60GB): 4096
• Large GPUs (>60GB): 8192+

max_prefill_tokens · int · default: "16384"

Maximum tokens for prefill phase.

schedule_policy · str · default: "fcfs"

Scheduling policy. Options: "fcfs" (first-come-first-serve), "lpm" (longest-prefix-match).

enable_priority_scheduling · bool · default: "False"

Enable priority-based request scheduling.

Parallelism

tp_size · int · default: "1"

Tensor parallelism size (number of GPUs for model parallelism).

tp_size=4  # Use 4 GPUs with tensor parallelism

dp_size · int · default: "1"

Data parallelism size (number of independent model replicas).

dp_size=2  # Run 2 independent replicas

pp_size · int · default: "1"

Pipeline parallelism size (number of pipeline stages).

nnodes · int · default: "1"

Number of nodes in a multi-node setup.

node_rank · int · default: "0"

Current node rank (0 to nnodes-1).

Kernel Backends

attention_backend · Optional[str] · default: "None"

Attention kernel backend.

Options: "flashinfer", "flashinfer", "triton", "torch_native", "fa3" (FlashAttention-3)

Auto-selected based on hardware if not specified.

sampling_backend · Optional[str] · default: "None"

Sampling backend. Options: "flashinfer", "pytorch"

grammar_backend · Optional[str] · default: "None"

Structured generation backend.

Options: "xgrammar", "outlines", "llguidance", "none"

CUDA Graph Optimization

disable_cuda_graph · bool · default: "False"

Disable CUDA graph optimization.

cuda_graph_max_bs · Optional[int] · default: "None"

Maximum batch size for CUDA graph capture.

Auto-calculated based on GPU memory:

• Small GPUs: 8-24
• Medium GPUs: 32-160
• Large GPUs: 256-512

disable_cuda_graph_padding · bool · default: "False"

Disable padding in CUDA graph batch sizes.

Speculative Decoding

speculative_algorithm · Optional[str] · default: "None"

Speculative decoding algorithm.

Options: "EAGLE", "STANDALONE", "NGRAM"

speculative_draft_model_path · Optional[str] · default: "None"

Path to draft model for speculative decoding.

speculative_num_steps · Optional[int] · default: "None"

Number of speculative decoding steps.

speculative_num_draft_tokens · Optional[int] · default: "None"

Number of draft tokens to generate per step.

LoRA

enable_lora · Optional[bool] · default: "None"

Enable LoRA adapter support.

max_lora_rank · Optional[int] · default: "None"

Maximum LoRA rank to support.

lora_paths · Optional[Union[List[str], List[dict]]] · default: "None"

Paths to LoRA adapters to pre-load.

lora_paths=["path/to/lora1", "path/to/lora2"]

max_loaded_loras · Optional[int] · default: "None"

Maximum number of LoRA adapters to keep loaded.

lora_backend · str · default: "csgmv"

LoRA kernel backend. Options: "triton", "csgmv", "torch_native"

Expert Parallelism (MoE)

ep_size · int · default: "1"

Expert parallelism size for Mixture-of-Experts models.

moe_runner_backend · str · default: "auto"

MoE kernel backend.

Options: "auto", "triton", "flashinfer_cutlass", "deep_gemm"

moe_a2a_backend · str · default: "none"

All-to-all communication backend for MoE.

Options: "none", "deepep", "mooncake"

Logging and Monitoring

log_level · str · default: "info"

Logging level. Options: "debug", "info", "warning", "error"

log_requests · bool · default: "False"

Log all requests and responses.

show_time_cost · bool · default: "False"

Show time cost for each request.

enable_metrics · bool · default: "False"

Enable Prometheus metrics.

enable_trace · bool · default: "False"

Enable OpenTelemetry tracing.

otlp_traces_endpoint · str · default: "localhost:4317"

OpenTelemetry collector endpoint.

Advanced Options

disable_radix_cache · bool · default: "False"

Disable radix cache (prefix caching) optimization.

random_seed · Optional[int] · default: "None"

Random seed for reproducibility. Auto-generated if not specified.

stream_interval · int · default: "1"

Token interval for streaming responses.

download_dir · Optional[str] · default: "None"

Directory for downloading models from Hugging Face.

enable_torch_compile · bool · default: "False"

Enable PyTorch compilation for model optimization.

device · Optional[str] · default: "None"

Device to use. Options: "cuda", "cpu", "npu". Auto-detected if not specified.

Configuration Examples

Basic Configuration

from sglang import Engine

engine = Engine(
    model_path="meta-llama/Llama-3.1-8B-Instruct",
    tp_size=1,
    log_level="info"
)

Production Configuration

from sglang import Engine

engine = Engine(
    model_path="meta-llama/Llama-3.1-70B-Instruct",
    tp_size=4,
    mem_fraction_static=0.90,
    max_running_requests=1000,
    chunked_prefill_size=8192,
    enable_metrics=True,
    enable_torch_compile=True,
    log_level="warning"
)

Quantized Model

from sglang import Engine

engine = Engine(
    model_path="TheBloke/Llama-2-70B-AWQ",
    quantization="awq",
    tp_size=2,
    kv_cache_dtype="fp8_e4m3"
)

Multi-LoRA Configuration

from sglang import Engine

engine = Engine(
    model_path="meta-llama/Llama-3.1-8B-Instruct",
    enable_lora=True,
    max_lora_rank=64,
    lora_paths=[
        "adapter1",
        "adapter2",
        "adapter3"
    ],
    max_loaded_loras=10
)

Data Parallelism

from sglang import Engine

engine = Engine(
    model_path="meta-llama/Llama-3.1-8B-Instruct",
    dp_size=4,  # 4 replicas
    load_balance_method="round_robin"
)

Speculative Decoding

from sglang import Engine

engine = Engine(
    model_path="meta-llama/Llama-3.1-70B-Instruct",
    tp_size=4,
    speculative_algorithm="EAGLE",
    speculative_draft_model_path="meta-llama/Llama-3.1-8B-Instruct",
    speculative_num_steps=5
)

Multi-Node Configuration

# Node 0
from sglang import Engine

engine = Engine(
    model_path="meta-llama/Llama-3.1-70B-Instruct",
    tp_size=8,
    nnodes=2,
    node_rank=0,
    dist_init_addr="node0:12345"
)

# Node 1
engine = Engine(
    model_path="meta-llama/Llama-3.1-70B-Instruct",
    tp_size=8,
    nnodes=2,
    node_rank=1,
    dist_init_addr="node0:12345"
)

See Also

• Engine - Main inference engine
• Runtime - HTTP server wrapper
• SamplingParams - Sampling configuration