Spaces:
Running
Running
Add claude 4.6 note
Browse files- app.py +6 -0
- model_sizes.csv +2 -0
app.py
CHANGED
|
@@ -103,6 +103,12 @@ ArmBench-LLM is a comprehensive benchmark for evaluating large language models o
|
|
| 103 |
markdown += f" - {display_name} ({metric_desc})\n"
|
| 104 |
|
| 105 |
markdown += """
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
## Submission Guide
|
| 107 |
|
| 108 |
To submit your model for evaluation:
|
|
|
|
| 103 |
markdown += f" - {display_name} ({metric_desc})\n"
|
| 104 |
|
| 105 |
markdown += """
|
| 106 |
+
## Important Note on Claude 4.6 Models
|
| 107 |
+
|
| 108 |
+
We observed significant reliability issues with Claude 4.6 during evaluation, which produced inconsistent and non-reproducible results. To preserve the integrity of the benchmark, all Claude 4.6 models have been excluded from the leaderboard.
|
| 109 |
+
|
| 110 |
+
You can learn more about the issue in [this GitHub issue](https://github.com/anthropics/claude-code/issues/17357).
|
| 111 |
+
|
| 112 |
## Submission Guide
|
| 113 |
|
| 114 |
To submit your model for evaluation:
|
model_sizes.csv
CHANGED
|
@@ -23,3 +23,5 @@ openai/gpt-5.4-mini, -
|
|
| 23 |
openai/gpt-5.4-nano, -
|
| 24 |
x-ai/grok-4.20-beta, -
|
| 25 |
qwen/qwen3.5-27b,27B
|
|
|
|
|
|
|
|
|
| 23 |
openai/gpt-5.4-nano, -
|
| 24 |
x-ai/grok-4.20-beta, -
|
| 25 |
qwen/qwen3.5-27b,27B
|
| 26 |
+
anthropic/claude-3.7-sonnet, -
|
| 27 |
+
anthropic/claude-sonnet-4, -
|