Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+inference/examples/images/carrots.jpeg filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
+quantization_config.json filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,5 @@
+.cache/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+inference/*_output.txt

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 DeepSeek
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,18 +1,143 @@
 ---
 license: mit
-base_model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp
-base_model_relation: quantized
-quantized_by: turboderp
-tags:
-- exl3
+library_name: transformers
+pipeline_tag: image-text-to-text
 ---
 
-EXL3 quants of [deepseek-ai/DeepSeek-V4-Flash-Vision-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp/tree/main)
+# DeepSeek-V4-Flash-Vision-Exp
 
-⚠️ Needs ExLlamav3 v1.4.9 (or `dev` branch)
+<!-- markdownlint-disable first-line-h1 -->
+<!-- markdownlint-disable html -->
+<!-- markdownlint-disable no-duplicate-header -->
 
-[2.04 bits per weight](https://huggingface.co/turboderp/DeepSeek-V4-Flash-Vision-Exp-exl3/tree/2.04bpw)    
-[2.27 bits per weight](https://huggingface.co/turboderp/DeepSeek-V4-Flash-Vision-Exp-exl3/tree/2.27bpw)    
-[2.52 bits per weight](https://huggingface.co/turboderp/DeepSeek-V4-Flash-Vision-Exp-exl3/tree/2.52bpw)    
-[2.77 bits per weight](https://huggingface.co/turboderp/DeepSeek-V4-Flash-Vision-Exp-exl3/tree/2.77bpw)    
-[3.04 bits per weight](https://huggingface.co/turboderp/DeepSeek-V4-Flash-Vision-Exp-exl3/tree/3.04bpw)    
+<div align="center">
+  <img src="https://github.com/deepseek-ai/DeepSeek-V2/blob/main/figures/logo.svg?raw=true" width="60%" alt="DeepSeek-V4" />
+</div>
+<hr>
+<div align="center" style="line-height: 1;">
+  <a href="https://www.deepseek.com/" target="_blank" style="margin: 2px;">
+    <img alt="Homepage" src="https://github.com/deepseek-ai/DeepSeek-V2/blob/main/figures/badge.svg?raw=true" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+  <a href="https://chat.deepseek.com/" target="_blank" style="margin: 2px;">
+    <img alt="Chat" src="https://img.shields.io/badge/🤖%20Chat-DeepSeek%20V4-536af5?color=536af5&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+<div align="center" style="line-height: 1;">
+  <a href="https://huggingface.co/deepseek-ai" target="_blank" style="margin: 2px;">
+    <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DeepSeek%20AI-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+  <a href="https://twitter.com/deepseek_ai" target="_blank" style="margin: 2px;">
+    <img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-deepseek_ai-white?logo=x&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+<div align="center" style="line-height: 1;">
+  <a href="LICENSE" style="margin: 2px;">
+    <img alt="License" src="https://img.shields.io/badge/License-MIT-f5de53?&color=f5de53" style="display: inline-block; vertical-align: middle;"/>
+  </a>
+</div>
+
+## Introduction
+
+We are excited to introduce **DeepSeek-V4-Flash-Vision-Exp**, our first experimental multimodal model in the DeepSeek-V4 family. It builds on the DeepSeek-V4-Flash architecture by incorporating visual modules and undergoing continued training to unlock visual understanding capabilities.
+
+Compared to DeepSeek-V4-Flash-0731, DeepSeek-V4-Flash-Vision-Exp achieves substantial improvements on its multimodal agent capabilities, while maintaining comparable performance on text-only agent tasks.
+
+<div align="center">
+
+| Benchmark | DeepSeek-V4-Flash-Vision-Exp | DeepSeek-V4-Flash-0731 | Opus-4.8 |
+| :--- | :---: | :---: | :---: |
+| **Text Agent Capabilities** | | | |
+| Terminal Bench 2.1 | 83.9 | 82.7 | 85.0 |
+| NL2Repo | 57.7 | 54.2 | 69.7 |
+| Cybergym | 75.3 | 76.7 | 78.3 |
+| DeepSWE | 59.3 | 54.4 | 58.0 |
+| Toolathlon-Verified | 75.9 | 70.3 | 76.2 |
+| DSBench-Hard | 63.6 | 59.6 | 71.7 |
+| AutomationBench (Public) | 25.7 | 25.1 | 27.2 |
+| **Multimodal Agent Capabilities** | | | |
+| ApexBench (Pass@1) | 36.5 | 26.2† | 39.4 |
+| Agents' Last Exam | 27.3 | 25.2† | 25.7 |
+| Chartography | 64.3 | - | 65.0 |
+| ZeroBench (Pass@5) | 35.0 | - | 34.0 |
+
+</div>
+
+Notes:
+
+1. For the text agent benchmarks above, DeepSeek models are evaluated with the minimal mode of DeepSeek Harness as the agent framework, using the `max` reasoning effort level with `temperature = 1.0, top_p = 0.95`.
+2. † For ApexBench and Agents' Last Exam, DeepSeek-V4-Flash-0731 ignores the multimodal elements in the input.
+
+
+## Repository layout
+
+This repository contains the tokenizer, prompt encoding reference, and a
+minimal PyTorch inference implementation for DeepSeek-V4 Flash Vision. The
+reference inference covers the vision encoder and aligner, DFlash attention,
+MoE, Hyper-Connections, and the DSpark forward path.
+
+```text
+.
+├── encoding/                  # OpenAI-style messages -> model prompt
+├── inference/                 # weight conversion and minimal inference
+│   └── examples/              # equivalent TXT and JSON vision prompts
+├── config.json                # Hugging Face model metadata
+├── generation_config.json
+├── model.safetensors.index.json
+├── tokenizer.json
+└── tokenizer_config.json
+```
+
+`encoding/` and `inference/` deliberately remain separate: prompt formatting
+does not depend on PyTorch, while inference imports the sibling encoding module
+with an explicit Python path. No symlinks are required.
+
+The tokenizer files are regular files so that the repository can be uploaded
+to Hugging Face without relying on local filesystem symlinks. The large model
+shards are described by `model.safetensors.index.json` and are not duplicated
+inside the source checkout used to assemble this repository.
+
+## Prompt encoding
+
+See [`encoding/README.md`](encoding/README.md). Both OpenAI-style JSON content
+blocks and the compact `<image>path</image>` TXT notation are supported. The two
+examples under `inference/examples/` encode to identical prompts and token IDs.
+
+## Minimal inference
+
+See [`inference/README.md`](inference/README.md) for dependency installation,
+checkpoint conversion, and TXT/JSON inference commands.
+
+## How to Run with vLLM
+
+For example, the command below serves the model with vLLM on a single 4×GB300 node. 
+See the [vLLM recipe](https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) for detailed instructions and other hardware configurations.
+
+```bash
+docker run --gpus all \
+  vllm/vllm-openai:deepseekv4-flash-vision deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
+  --kv-cache-dtype fp8 \
+  --block-size 256 \
+  --tensor-parallel-size 4 \
+  --tool-call-parser deepseek_v4 \
+  --enable-auto-tool-choice \
+  --reasoning-parser deepseek_v4 \
+  --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' \
+  --speculative-config '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}'
+```
+
+## How to Run with SGLang
+Enable DSpark with --speculative-algorithm DSPARK and do not set a separate --speculative-draft-model-path as the target and draft weights therefore come from the same checkpoint. See the [SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4#hw=b200&variant=flash-vision&quant=fp4&strategy=low-latency&nodes=single) for detailed instructions, benchmarks and other hardwares configurations.
+
+```
+sglang serve \
+  --model-path deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
+  --tp 4 \
+  --speculative-algorithm DSPARK \
+  --mem-fraction-static 0.85 \
+  --host 0.0.0.0 \
+  --port 30000
+```
+
+## License
+
+This repository is licensed under the [MIT License](LICENSE).

config.json

@@ -0,0 +1,146 @@
+{
+    "architectures": [
+        "DeepseekV4ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 0,
+    "eos_token_id": 1,
+    "expert_dtype": "fp4",
+    "hc_eps": 1e-06,
+    "hc_mult": 4,
+    "hc_sinkhorn_iters": 20,
+    "head_dim": 512,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "index_head_dim": 128,
+    "index_n_heads": 64,
+    "index_topk": 512,
+    "initializer_range": 0.02,
+    "max_position_embeddings": 1048576,
+    "model_type": "deepseek_v4",
+    "moe_intermediate_size": 2048,
+    "n_routed_experts": 256,
+    "n_shared_experts": 1,
+    "norm_topk_prob": true,
+    "num_attention_heads": 64,
+    "num_experts_per_tok": 6,
+    "num_hidden_layers": 43,
+    "num_hash_layers": 3,
+    "num_key_value_heads": 1,
+    "num_nextn_predict_layers": 3,
+    "o_groups": 8,
+    "o_lora_rank": 1024,
+    "q_lora_rank": 1024,
+    "qk_rope_head_dim": 64,
+    "quantization_config": {
+        "quant_method": "exl3",
+        "version": "1.4.8",
+        "bits": 2.27,
+        "head_bits": 4,
+        "calibration": {
+            "rows": 250,
+            "cols": 2048
+        },
+        "out_scales": "always",
+        "codebook": "mul1",
+        "original_quantization_config": {
+            "activation_scheme": "dynamic",
+            "fmt": "e4m3",
+            "quant_method": "fp8",
+            "scale_fmt": "ue8m0",
+            "weight_block_size": [
+                128,
+                128
+            ]
+        },
+        "vision_bits": 5,
+        "mtp_bits": 2
+    },
+    "rms_norm_eps": 1e-20,
+    "rope_scaling": {
+        "beta_fast": 32,
+        "beta_slow": 1,
+        "factor": 16,
+        "original_max_position_embeddings": 65536,
+        "type": "yarn"
+    },
+    "rope_theta": 10000,
+    "routed_scaling_factor": 1.5,
+    "scoring_func": "sqrtsoftplus",
+    "sliding_window": 128,
+    "swiglu_limit": 10.0,
+    "tie_word_embeddings": false,
+    "topk_method": "noaux_tc",
+    "torch_dtype": "bfloat16",
+    "transformers_version": "5.0.0",
+    "use_cache": true,
+    "vocab_size": 129280,
+    "compress_rope_theta": 160000,
+    "compress_ratios": [
+        0,
+        0,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        128,
+        4,
+        0,
+        0,
+        0
+    ],
+    "dspark_block_size": 5,
+    "dspark_noise_token_id": 128799,
+    "dspark_target_layer_ids": [
+        40,
+        41,
+        42
+    ],
+    "dspark_markov_rank": 256,
+    "vision_n_layers": 32,
+    "vision_dim": 1024,
+    "vision_n_heads": 16,
+    "vision_inter_dim": 2816,
+    "vision_patch_size": 14,
+    "vision_rope_theta": 10000.0,
+    "vision_downsample_ratio": 3,
+    "vision_max_n_token": 384,
+    "vision_min_pixels": 147456,
+    "vision_max_wh_ratio": 8
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "do_sample": true,
+  "temperature": 1.0,
+  "top_p": 1.0,
+  "transformers_version": "4.46.3"
+}

model-00001-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8a6ac64f34d25edfff4139e68445dfa9abbc6108295fc1a63346a7ff2217471
+size 8579457394

model-00002-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:636aaf9418514569dba42c8a61c17c6a6136a82d611bca8e5af3eb1d02bc97d7
+size 8403210866

model-00003-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:352a5372e327ef029a6b3f16b9c50e77cb2b3e0d9214f9038884fc15ce12c0b7
+size 8486922049

model-00004-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68465dd43c34cd77fd9e2adf8fb906eebcbcc00ff42240b0f143155805d01260
+size 8494594044

model-00005-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:038a797c1f2a4749e8428eb2fbedc15714be0b2134981b83c97f7380e4233b42
+size 8486931532

model-00006-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19192701bbeec1429c196ce6257586d0676f7e85b55a8f910c878e6879b8c152
+size 8494594044

model-00007-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ba86b8bb358acc75dac1f53dd4b397e22e0f03a5ed068bf2da25d5c62840805
+size 8486931532

model-00008-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5bc6a0de36a183b2177064c61ce67d1bc2359370be92181902ed045dcc7cc7be
+size 8494594044

model-00009-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bbf07895b337807c1123c3f681c0ece836a3109a64160a0af50cfc1d90a13f64
+size 6701237460

model-00010-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fed80b28198893bb5634b19fd91770d8e6907a798824782386690a3d80c464fe
+size 7779510962

model-00011-of-00011.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:942c283e80b084fcb3e8adbd899b7e11e95f0cf3b612f93dd4943f58a15cf4d3
+size 6085752384

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d4c7884b72a31ef582224abcae4f156f39e8a16f55e2fc3946cee73b913c9d7
+size 11773908

quantization_config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a9368695b8e5e6b797c64dba672d5998636503f60e9fee114d70caae5194385
+size 39576557

tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "add_bos_token": false,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<｜begin▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "legacy": true,
+  "model_max_length": 1048576,
+  "pad_token": {
+    "__type": "AddedToken",
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sp_model_kwargs": {},
+  "unk_token": null,
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}