Instructions to use local-inference-lab/GLM-5.3-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use local-inference-lab/GLM-5.3-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="local-inference-lab/GLM-5.3-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("local-inference-lab/GLM-5.3-NVFP4")
model = AutoModelForCausalLM.from_pretrained("local-inference-lab/GLM-5.3-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use local-inference-lab/GLM-5.3-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "local-inference-lab/GLM-5.3-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/local-inference-lab/GLM-5.3-NVFP4

SGLang

How to use local-inference-lab/GLM-5.3-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "local-inference-lab/GLM-5.3-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "local-inference-lab/GLM-5.3-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "local-inference-lab/GLM-5.3-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use local-inference-lab/GLM-5.3-NVFP4 with Docker Model Runner:
```
docker model run hf.co/local-inference-lab/GLM-5.3-NVFP4
```

Exclude BF16 MTP layer 78 from ModelOpt FP4

by bullpoint - opened Sep 10

base: refs/heads/main

←

from: refs/pr/2

Discussion Files changed

-0

bullpoint

Sep 10

Add model.layers.78* to quantization_config.ignore so ModelOpt-aware loaders leave the built-in multi-token prediction (MTP) layer unquantized. The checkpoint declares 78 transformer layers and one MTP layer at index 78. Its 768 routed expert weight tensors in model-00084-of-00084.safetensors are BF16, so applying NVFP4 quantization to those modules does not match the stored weights. The exclusion lets vLLM load the target model and its MTP layer from the same model directory.

This configuration-only change preserves quantization settings for transformer layers 0–77 and does not modify any checkpoint tensors.

Status: implemented. Validation:

Parsed the modified JSON and verified that adding the single ignore entry is the only change.
Inspected the safetensors header: all 768 routed MTP expert weight tensors are BF16. The downloaded shard's recorded SHA-256 matches the repository's shard at the PR base revision.
No server startup or inference qualification was performed for this PR.

Exclude BF16 MTP layer 78 from ModelOpt FP41f3bb90c

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment