Add Mantis-50M-Base (kash8080, 49.3M)
Mantis-50M-Base: 49,322,240 params, pretrained from scratch, open weights, standard Qwen3ForCausalLM (no remote code). It is the final checkpoint of its run (step 34,000), fixed as the release before any final eval.
- Model: https://huggingface.co/kash8080/Mantis-50M-Base
- Code: https://github.com/kash8080/mantis-50m
Scores (lm-eval 0.4.12 zero-shot acc_norm, fp32, official ArithMark-3 script, default settings). Raw files are in the repo's evals/.
| HellaSwag | ARC-E | ARC-C | PIQA | ArithMark-3 | II |
|---|---|---|---|---|---|
| 34.24 | 44.61 | 26.37 | 65.89 | 35.30 | 18.36 |
Follow-up to #275. That thread asked about Mantis-50M, which scored 85.70 on ArithMark with our synthetic arithmetic but only 31 to 41% on ASDiv/SVAMP. I am not submitting that model. This one resumes from its checkpoint before the decay (step 27,200) and reruns the decay on public data only.
Synthetic data still in the training history. The stable phase included 7% synthetic arithmetic from our first generator (code: src/slm50m/arith/generator.py, sample: v1-sample config of https://huggingface.co/datasets/kash8080/Mantis-50M-arith). It was removed for the decay. Zero exact and zero 13-gram matches with ArithMark-3 over ~54,000 generated problems. It was written with ArithMark's skill in mind, from general English word lists, never from benchmark items.
What it left behind:
- It did not raise ArithMark: 37.3 before the decay, 35.30 now.
- The model forgot most of it in the decay. Its own templates fell from 98.5% to 35.2% (chance 25%), and its validation loss rose from 0.549 to 2.04.
- GPT-3 arithmetic fell from 16.6 to 0.6 by exact match. Part of that is format: the model now writes working first. On 100 random 2-digit sums it wrote the right sum within 24 tokens on 62, so some weak addition remains. I can't say how much of that comes from the first generator.
Outside checks on this model. ASDiv/SVAMP held-out (232 problems no training source paraphrases, chance 25): 8 to 30% across formats. MathQA 23.5 (chance 20). MMLU elementary math 26.7 (chance 25). Details are on the model card.
The other training data includes Orca-Math, which contains most of ASDiv and SVAMP, and OpenMathInstruct-2 (GSM8K-type rows).
Since it has synthetic data in its history, please run it on your hidden sets. If it doesn't pass, I'd be grateful for the score so the card can state it.
"It was written with ArithMark's skill in mind" Generally this is not allowed however, this model passes hidden sets and as you pointed out, seemingly didnt impact am3 scores, so welcome to the leaderboard and congrats on #1!