Update README.md

README.md

@@ -6,7 +6,26 @@ pipeline_tag: text-generation
 base_model:
 - Qwen/Qwen3-Coder-30B-A3B-Instruct
 ---
+# Qwen3-Coder-30B-A3B-Instruct-AWQ
 
+## Method
+Quantised using [vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor.git), [nvidia/Llama-Nemotron-Post-Training-Dataset](https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset) and the following configs:
+```
+recipe = [
+    AWQModifier(
+        ignore=["lm_head", "re:.*mlp.gate$", "re:.*mlp.shared_expert_gate$"],
+        scheme="W4A16",
+        targets=["Linear"],  
+    ),
+]
+```
+## Inference
+
+### vllm
+Please load the model into vllm and sglang as float16 data type for AWQ support and use `tensor_parallel_size <= 2` i.e.,
+```
+vllm serve cpatonn/Qwen3-Coder-30B-A3B-Instruct-AWQ --dtype float16 --tensor-parallel-size 2 --pipeline-parallel-size 2
+```
 # Qwen3-Coder-30B-A3B-Instruct
 <a href="https://chat.qwen.ai/" target="_blank" style="margin: 2px;">
     <img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>