> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/sgl-project/sglang/llms.txt
> Use this file to discover all available pages before exploring further.

# Llama Models

> Usage guide for Meta's Llama model series in SGLang

Meta's Llama series represents one of the most widely-used families of open-source large language models, ranging from 7B to 400B parameters across Llama 2, Llama 3, and Llama 4 generations.

## Overview

[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation with industry-leading performance. SGLang has provided first-class support and optimizations for Llama models since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).

### Supported Llama Models

* **Llama 4 Scout** (109B) - Latest generation
* **Llama 4 Maverick** (400B) - Largest Llama model
* **Llama 3.x series** (1B, 3B, 8B, 70B) - Previous generation
* **Llama 2 series** (7B, 13B, 70B) - Foundation models
* **Llama Vision** (11B, 90B) - Multimodal variants
* **Specialized variants**: Classification, Embedding, Reward models

## Quick Start

### Basic Launch Command

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-1B-Instruct \
  --host 0.0.0.0 \
  --port 30000
```

### Llama 4 Launch (8xH100/H200)

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tp 8 \
  --context-length 1000000
```

## Llama 4 Configuration

### Hardware Recommendations

| Model                    | Hardware | Context Length | Notes                                  |
| ------------------------ | -------- | -------------- | -------------------------------------- |
| Scout (109B)             | 8×H100   | Up to 1M       | Adjust `--context-length` to avoid OOM |
| Scout (109B)             | 8×H200   | Up to 2.5M     | Extended context support               |
| Scout (109B) + Hybrid KV | 8×H100   | Up to 5M       | With `--swa-full-tokens-ratio`         |
| Scout (109B) + Hybrid KV | 8×H200   | Up to 10M      | Maximum supported context              |
| Maverick (400B)          | 8×H200   | Up to 1M       | Full precision                         |
| Maverick (400B)          | 8×B200   | -              | Optimal performance                    |

### Configuration Tips

#### Attention Backend Auto-Selection

SGLang automatically selects the optimal attention backend based on your hardware:

* **Blackwell GPUs** (B200/GB200): `trtllm_mha`
* **Hopper GPUs** (H100/H200): `fa3` (FlashAttention 3)
* **AMD GPUs**: `aiter`
* **Intel XPU**: `intel_xpu`
* **Other platforms**: `triton` (fallback)

To override auto-selection:

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tp 8 \
  --attention-backend fa3
```

#### Context Length Management

Adjust `--context-length` to avoid GPU out-of-memory issues:

```bash theme={null}
# Scout on 8×H100 - up to 1M tokens
--context-length 1000000

# Scout on 8×H200 - up to 2.5M tokens
--context-length 2500000
```

#### Hybrid KV Cache

Enable hybrid KV cache for extended context lengths using Llama 4's local attention layers:

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tp 8 \
  --context-length 5000000 \
  --swa-full-tokens-ratio 0.8  # Ratio of SWA layer KV tokens (default: 0.8, range: 0-1)
```

#### Chat Template

For chat completion tasks, add the Llama 4 chat template:

```bash theme={null}
--chat-template llama-4
```

#### Multimodal Support

For Llama Vision models:

```bash theme={null}
--enable-multimodal
```

## EAGLE Speculative Decoding

Llama 4 Maverick (400B) supports [EAGLE speculative decoding](https://docs.sglang.io/advanced_features/speculative_decoding.html#EAGLE-Decoding) for accelerated inference.

### Launch with EAGLE

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path nvidia/Llama-4-Maverick-17B-128E-Eagle3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --trust-remote-code \
  --tp 8 \
  --context-length 1000000
```

**Note**: The Llama 4 EAGLE draft model (`nvidia/Llama-4-Maverick-17B-128E-Eagle3`) only recognizes conversations in chat mode.

## Benchmarks

### Accuracy Test (MMLU Pro)

SGLang achieves accuracy matching or exceeding official benchmarks:

| Model                              | Official Benchmark | SGLang | Hardware |
| ---------------------------------- | ------------------ | ------ | -------- |
| Llama-4-Scout-17B-16E-Instruct     | 74.3               | 75.2   | 8×H100   |
| Llama-4-Maverick-17B-128E-Instruct | 80.5               | 80.7   | 8×H100   |

### Running Accuracy Tests

#### Llama-4-Scout

```bash theme={null}
# Launch server
python -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --port 30000 \
  --tp 8 \
  --mem-fraction-static 0.8 \
  --context-length 65536

# Run lm_eval
lm_eval --model local-chat-completions \
  --model_args model=meta-llama/Llama-4-Scout-17B-16E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
  --tasks mmlu_pro \
  --batch_size 128 \
  --apply_chat_template \
  --num_fewshot 0
```

#### Llama-4-Maverick

```bash theme={null}
# Launch server
python -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --port 30000 \
  --tp 8 \
  --mem-fraction-static 0.8 \
  --context-length 65536

# Run lm_eval
lm_eval --model local-chat-completions \
  --model_args model=meta-llama/Llama-4-Maverick-17B-128E-Instruct,base_url=http://localhost:30000/v1/chat/completions,num_concurrent=128,timeout=999999,max_gen_toks=2048 \
  --tasks mmlu_pro \
  --batch_size 128 \
  --apply_chat_template \
  --num_fewshot 0
```

## Llama 3.x Models

Llama 3.x models (1B, 3B, 8B, 70B) are also fully supported:

```bash theme={null}
# Llama 3.2 1B (lightweight)
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-1B-Instruct \
  --port 30000

# Llama 3.1 8B (popular size)
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000

# Llama 3.1 70B (multi-GPU)
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp 4 \
  --port 30000
```

## Specialized Llama Variants

SGLang supports specialized Llama model variants:

### Embedding Models

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-1B-Embedding \
  --port 30000
```

### Classification Models

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-1B-Classification \
  --port 30000
```

### Reward Models

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-1B-Reward \
  --port 30000
```

## Llama Vision (Multimodal)

Llama 3.2 includes vision-enabled variants (11B, 90B). See the [Multimodal Models guide](multimodal) for detailed usage.

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \
  --enable-multimodal \
  --port 30000
```

## Advanced Features

### EAGLE Decoding for Llama 3

```bash theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --speculative-algorithm EAGLE \
  --speculative-draft-model-path <eagle-draft-model> \
  --speculative-num-steps 3 \
  --tp 4
```

### Quantization

SGLang supports various quantization methods for Llama models:

```bash theme={null}
# FP8 quantization
--quantization fp8

# AWQ quantization
--quantization awq

# GPTQ quantization
--quantization gptq
```

## Resources

* [Official Llama 4 Model Card](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md)
* [SGLang Llama 4 Roadmap](https://github.com/sgl-project/sglang/issues/5118)
* [SGLang v0.4.5 Release Notes](https://github.com/sgl-project/sglang/releases/tag/v0.4.5)
* [Meta Llama Website](https://llama.meta.com/)

## Troubleshooting

### Out of Memory (OOM)

Reduce `--context-length`:

```bash theme={null}
--context-length 512000  # Reduce from 1M to 512K
```

Or reduce memory fraction:

```bash theme={null}
--mem-fraction-static 0.8  # Reduce from default 0.9
```

### Slow Model Loading

Increase timeout:

```bash theme={null}
--watchdog-timeout 1200  # Increase to 20 minutes
```

Enable parallel weight loading:

```bash theme={null}
--model-loader-extra-config '{"enable_multithread_load": true}'
```