Upload folder using huggingface_hub

README.md

@@ -0,0 +1,154 @@
+---
+base_model: microsoft/unixcoder-base
+language:
+- code
+library_name: transformers
+pipeline_tag: text-classification
+tags:
+- code
+- multi-task
+- structured-data
+- code-quality
+- content-type
+- regression
+- classification
+- starcoderdata
+datasets:
+- bigcode/starcoderdata
+---
+
+# StarCoderData Code Classifier
+
+Multi-task code classification model for filtering large-scale code datasets. Built to aggressively curate training data for a 1B parameter structured-data-focused code model.
+
+## Model Details
+
+- **Base model:** microsoft/unixcoder-base (125M params)
+- **Architecture:** Shared encoder + three task-specific linear heads
+- **Training data:** 191,776 code samples from bigcode/starcoderdata, labeled by GPT-5-nano (Batch API)
+- **Languages:** Python, JavaScript, TypeScript, Java, Go, Rust, SQL, Shell (25K per language)
+- **Training:** 3 epochs, batch size 16, lr 2e-5, AMP (bf16), torch.compile
+
+## Tasks
+
+| Task | Type | Output | Loss |
+|------|------|--------|------|
+| Code Quality (1-5) | Regression | Float | MSE |
+| Structured Data Relevance (0-3) | Regression | Float | MSE |
+| Content Type (9 classes) | Classification | Softmax | CrossEntropy |
+
+## Test Set Performance
+
+### Code Quality (1-5 scale)
+
+| Metric | Score |
+|--------|-------|
+| MAE | 0.598 |
+| Rounded Accuracy | 55.3% |
+| Spearman r | 0.575 |
+
+| Level | Precision | Recall | F1 |
+|-------|-----------|--------|----|
+| 1 - Broken/gibberish | 0.72 | 0.35 | 0.47 |
+| 2 - Functional but poor | 0.28 | 0.23 | 0.25 |
+| 3 - Decent | 0.49 | 0.67 | 0.56 |
+| 4 - Good | 0.68 | 0.63 | 0.65 |
+| 5 - Excellent | 0.08 | 0.00 | 0.01 |
+
+### Structured Data Relevance (0-3 scale)
+
+| Metric | Score |
+|--------|-------|
+| MAE | 0.421 |
+| Rounded Accuracy | 66.7% |
+| Spearman r | 0.807 |
+
+| Level | Precision | Recall | F1 |
+|-------|-----------|--------|----|
+| 0 - None | 0.73 | 0.70 | 0.71 |
+| 1 - Minor | 0.43 | 0.48 | 0.45 |
+| 2 - Significant | 0.74 | 0.75 | 0.75 |
+| 3 - Primary focus | 0.67 | 0.59 | 0.63 |
+
+### Content Type (9 classes)
+
+| Metric | Score |
+|--------|-------|
+| Accuracy | 87.5% |
+| Macro F1 | 0.678 |
+
+| Type | Precision | Recall | F1 | Support |
+|------|-----------|--------|----|---------|
+| library | 0.89 | 0.92 | 0.91 | 7,990 |
+| application | 0.83 | 0.75 | 0.79 | 3,404 |
+| test | 0.92 | 0.93 | 0.93 | 1,818 |
+| config | 0.77 | 0.68 | 0.72 | 309 |
+| tutorial | 0.56 | 0.37 | 0.45 | 227 |
+| data | 0.45 | 0.59 | 0.51 | 129 |
+| generated | 0.66 | 0.49 | 0.56 | 316 |
+| script | 0.90 | 0.93 | 0.91 | 4,970 |
+| other | 0.75 | 0.20 | 0.32 | 15 |
+
+## Why These Scores Are Acceptable
+
+This model is designed as a **coarse filter**, not a precise labeler. The intended workflow is:
+
+1. Run this model on the full StarCoderData (~250B tokens)
+2. Apply threshold filters (e.g., quality >= 3 AND structured_data >= 2)
+3. Train a 1B parameter model on the filtered subset
+
+For this filtering use case, what matters is **rank ordering**, not exact classification:
+
+- **Structured data (Spearman 0.81):** The model's strongest dimension. It reliably separates code with heavy structured data usage (APIs, schemas, serialization) from code without it. At the filtering threshold of structured_data >= 2, the model achieves 0.75 F1 — meaning the filtered subset will be genuinely rich in structured data patterns.
+
+- **Quality (Spearman 0.58):** The weakest dimension, but still useful for filtering. The model struggles most with the quality 2-3 boundary (decent vs. poor) and virtually ignores quality 5 (only 1.2% of training data). However, for the intended filter of quality >= 3, the model has decent precision at levels 3-4 (0.49-0.68). The key insight: false positives at the boundary (quality-2 code scored as 3) are tolerable because the structured data filter provides a second gate. Code that passes both filters is unlikely to be low quality.
+
+- **Content type (87.5% accuracy):** Strong performance on the high-volume categories that matter most for filtering: library (0.91 F1), script (0.91 F1), test (0.93 F1), and application (0.79 F1). The weaker categories (tutorial, data, generated, other) have low support — together they represent only 3.5% of the data. Even with lower recall on these rare types, the model will still flag enough of them for filtering decisions.
+
+- **Errors are symmetric, not catastrophic.** A quality MAE of 0.60 means predictions are typically off by less than one level. A file scored as quality 4 is almost certainly quality 3-5, not quality 1. This is precisely the behavior needed for threshold-based filtering — the model rarely makes predictions that are off by more than one level.
+
+## How to Improve
+
+The primary bottleneck is **training data volume and class balance**, not model capacity:
+
+1. **Scale up the GPT-5-nano labeling set.** The current model was trained on 192K labeled samples (~$20 via Batch API). Doubling to 400K samples (~$40) would particularly help quality levels 2 and 5, where the model struggles most. Level 5 (excellent code) had only 2,345 training examples — far too few for the model to learn the pattern.
+
+2. **Oversample rare classes.** Content types like tutorial (2,197 samples), data (1,413), and generated (2,975) are underrepresented. A targeted labeling run that specifically seeks out these types — e.g., filtering by filename patterns like `*_test.*`, `*.generated.*`, `tutorial*` — would improve recall on rare types without relabeling the entire dataset.
+
+3. **Increase max token length.** The current model uses 512 tokens, but code files often need more context to assess quality. Increasing to 1024 or 2048 tokens (UniXcoder supports up to 1024) would give the model more signal, particularly for quality assessment where style and documentation patterns emerge over longer spans.
+
+4. **Add a second training round with hard examples.** After running inference on the full StarCoderData, sample files where the model is least confident (prediction near the decision boundary, e.g., quality between 2.5 and 3.5) and send those to GPT-5-nano for labeling. Training on these hard cases would sharpen the model's performance exactly where filtering decisions are made.
+
+## Usage
+
+```python
+from train_starcoderdata import CodeClassifierModel, load_model
+from transformers import AutoTokenizer
+import torch
+
+model_dir = "models/starcoderdata-classifier"
+tokenizer = AutoTokenizer.from_pretrained(model_dir)
+model = load_model(model_dir)
+model.eval()
+
+code = "def hello(): print('world')"
+inputs = tokenizer(code, return_tensors="pt", truncation=True, max_length=512)
+with torch.no_grad():
+    quality, structured_data, content_type_logits = model(
+        inputs["input_ids"], inputs["attention_mask"]
+    )
+
+print(f"Quality: {quality.item():.1f}")
+print(f"Structured Data: {structured_data.item():.1f}")
+print(f"Content Type: {['library','application','test','config','tutorial','data','generated','script','other'][content_type_logits.argmax()]}")
+```
+
+## Files
+
+- `config.json`, `model.safetensors` — UniXcoder encoder weights (HuggingFace format)
+- `classifier_heads.pt` — Quality, structured data, and content type head weights
+- `tokenizer.json`, `tokenizer_config.json` — Tokenizer
+- `label_config.json` — Label definitions and task types
+- `test_metrics.json` — Full test set metrics
+- `training_history.csv` — Per-epoch training/validation metrics
+- `checkpoint.pt` — Full training checkpoint (for resume)

checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2aa8bda24f4129cdebea06b3a254d582859bb335b9a5f15011f3a7e23a17106
+size 1506793407

classifier_heads.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:427dbb95238f4428e390a2887f716229fd4090d59695f5848a327669315b8f11
+size 36993

config.json

@@ -0,0 +1,31 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "RobertaModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 1026,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.1.0",
+  "type_vocab_size": 10,
+  "use_cache": true,
+  "vocab_size": 51416
+}

label_config.json

@@ -0,0 +1,27 @@
+{
+  "content_types": [
+    "library",
+    "application",
+    "test",
+    "config",
+    "tutorial",
+    "data",
+    "generated",
+    "script",
+    "other"
+  ],
+  "quality_range": [
+    1,
+    5
+  ],
+  "structured_data_range": [
+    0,
+    3
+  ],
+  "num_content_types": 9,
+  "tasks": {
+    "quality": "regression",
+    "structured_data": "regression",
+    "content_type": "classification"
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83a946ad4457ba05d0fb29ea4ab7524872021277a02cd131ce9f01a79b2231cd
+size 503741264

test_metrics.json

@@ -0,0 +1,14 @@
+{
+  "loss": 1.3610109102735926,
+  "quality_mae": 0.598224937915802,
+  "quality_mse": 0.6799516677856445,
+  "quality_rounded_acc": 0.552508082177495,
+  "quality_spearman": 0.5751793155320672,
+  "structured_data_mae": 0.4209136366844177,
+  "structured_data_mse": 0.3287431299686432,
+  "structured_data_rounded_acc": 0.6668578579622484,
+  "structured_data_spearman": 0.8065141417877283,
+  "content_type_accuracy": 0.8753258942538326,
+  "content_type_macro_f1": 0.6776609396601136,
+  "combined_mae": 0.5095692873001099
+}

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "errors": "replace",
+  "is_local": false,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "RobertaTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}

training_history.csv

@@ -0,0 +1,4 @@
+epoch,train_loss,val_loss,val_quality_mae,val_quality_mse,val_quality_rounded_acc,val_quality_spearman,val_structured_data_mae,val_structured_data_mse,val_structured_data_rounded_acc,val_structured_data_spearman,val_content_type_accuracy,val_content_type_macro_f1,val_combined_mae
+1,1.9338482332460847,1.448065088578718,0.6309319138526917,0.7191991209983826,0.522108666179998,0.553817859073393,0.45338281989097595,0.35454854369163513,0.6431848993638544,0.79499171167133,0.8623944102617582,0.6234108515025513,0.5421573668718338
+2,1.3254653735931108,1.3402525051073595,0.6033948659896851,0.6770005226135254,0.5472937741161747,0.5729696168341741,0.42503005266189575,0.32683703303337097,0.6662842840755032,0.8050313624661184,0.8754823234956721,0.6905006297568385,0.5142124593257904
+3,1.1299075901272126,1.3563477016171384,0.6024025678634644,0.6801310777664185,0.5458337678590051,0.5730787938638631,0.4216791093349457,0.33049410581588745,0.6724893106684743,0.8050240322085739,0.8755866096568985,0.6923274552582193,0.512040838599205