bugdaryan commited on
Commit
135cf18
·
1 Parent(s): d13da3b

Add claude 4.6 note

Browse files
Files changed (2) hide show
  1. app.py +6 -0
  2. model_sizes.csv +2 -0
app.py CHANGED
@@ -103,6 +103,12 @@ ArmBench-LLM is a comprehensive benchmark for evaluating large language models o
103
  markdown += f" - {display_name} ({metric_desc})\n"
104
 
105
  markdown += """
 
 
 
 
 
 
106
  ## Submission Guide
107
 
108
  To submit your model for evaluation:
 
103
  markdown += f" - {display_name} ({metric_desc})\n"
104
 
105
  markdown += """
106
+ ## Important Note on Claude 4.6 Models
107
+
108
+ We observed significant reliability issues with Claude 4.6 during evaluation, which produced inconsistent and non-reproducible results. To preserve the integrity of the benchmark, all Claude 4.6 models have been excluded from the leaderboard.
109
+
110
+ You can learn more about the issue in [this GitHub issue](https://github.com/anthropics/claude-code/issues/17357).
111
+
112
  ## Submission Guide
113
 
114
  To submit your model for evaluation:
model_sizes.csv CHANGED
@@ -23,3 +23,5 @@ openai/gpt-5.4-mini, -
23
  openai/gpt-5.4-nano, -
24
  x-ai/grok-4.20-beta, -
25
  qwen/qwen3.5-27b,27B
 
 
 
23
  openai/gpt-5.4-nano, -
24
  x-ai/grok-4.20-beta, -
25
  qwen/qwen3.5-27b,27B
26
+ anthropic/claude-3.7-sonnet, -
27
+ anthropic/claude-sonnet-4, -