Upload README.md

README.md

@@ -1,3 +1,64 @@
 ---
 license: cc-by-nc-sa-4.0
+datasets:
+- HumanF-MarkrAI/Korean-RAG-ver2
+language:
+- ko
+tags:
+  - Retrieval Augmented Generation
+  - RAG
+  - Multi-domain
 ---
+
+# MarkrAI/RAG-KO-Mixtral-7Bx2-v2.0
+
+# Model Details  
+
+## Model Developers  
+MarkrAI - AI Researchers
+
+## Base Model  
+[DopeorNope/Ko-Mixtral-v1.4-MoE-7Bx2](https://huggingface.co/DopeorNope/Ko-Mixtral-v1.4-MoE-7Bx2).  
+
+## Instruction tuning Method  
+Using QLoRA.  
+```
+4-bit quantization
+Lora_r: 64
+Lora_alpha: 64
+Lora_dropout: 0.05
+Lora_target_modules: [embed_tokens, q_proj, k_proj, v_proj, o_proj, gate, w1, w2, w3, lm_head]
+```
+
+## Hyperparameters  
+```
+Epoch: 5
+Batch size: 64
+Learning_rate: 1e-5
+Learning scheduler: linear
+Warmup_ratio: 0.06
+```
+
+## Datasets
+Private datasets: [HumanF-MarkrAI/Korean-RAG-ver2](https://huggingface.co/datasets/HumanF-MarkrAI/Korean-RAG-ver2)  
+```
+Aihub datasets 활용하여서 제작함.  
+```
+
+## Implmentation Code
+```
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+repo = "MarkrAI/RAG-KO-Mixtral-7Bx2-v2.0"
+markrAI_RAG = AutoModelForCausalLM.from_pretrained(
+        repo,
+        return_dict=True,
+        torch_dtype=torch.float16,
+        device_map='auto'
+)
+markrAI_RAG_tokenizer = AutoTokenizer.from_pretrained(repo)
+```
+
+# Model Benchmark
+- Coming soon...