Practical Data Rows or columns
Do you compare values down a column or across a row?
Some benchmark tables show one benchmark in each row and one model in each column. In these tables, each row is one measure.
When a table has two or more number columns, the model tells which direction to compare. Code reads a row-based table one row at a time, as if each row is a column.
- Decision rule
- Read the table by rows when P(rows) is 0.75 or more and the table has 40 rows or fewer.
Results
3 of 3 checks pass, on 3 tables.
Each table shows the shading that the extension applies. The strip under the table shows all judgments on each column. Put the pointer on a result to outline its column. A red outline shows a failure.
Standard Pitching Table
baseball-reference.com · Tarik Skubal: pitching · 36 columns × 9 rows
| Season | Age | Team | Lg | WAR | W | L | W-L% | ERA | G | GS | GF | CG | SHO | SV | IP | H | R | ER | HR | BB | IBB | SO | HBP | BK | WP | BF | ERA+ | FIP | WHIP | H9 | HR9 | BB9 | SO9 | SO/BB | Awards |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2020 | 23 | DET | AL | -0.1 | 1 | 4 | .200 | 5.63 | 8 | 7 | 0 | 0 | 0 | 0 | 32.0 | 28 | 21 | 20 | 9 | 11 | 0 | 37 | 2 | 0 | 0 | 134 | 81 | 5.75 | 1.219 | 7.9 | 2.5 | 3.1 | 10.4 | 3.36 | |
| 2021 | 24 | DET | AL | 1.8 | 8 | 12 | .400 | 4.34 | 31 | 29 | 1 | 0 | 0 | 0 | 149.1 | 141 | 76 | 72 | 35 | 47 | 0 | 164 | 6 | 0 | 4 | 634 | 98 | 5.09 | 1.259 | 8.5 | 2.1 | 2.8 | 9.9 | 3.49 | |
| 2022 | 25 | DET | AL | 1.4 | 7 | 8 | .467 | 3.52 | 21 | 21 | 0 | 0 | 0 | 0 | 117.2 | 104 | 53 | 46 | 9 | 32 | 0 | 117 | 1 | 0 | 1 | 477 | 111 | 2.96 | 1.156 | 8.0 | 0.7 | 2.4 | 8.9 | 3.66 | |
| 2023 | 26 | DET | AL | 2.0 | 7 | 3 | .700 | 2.80 | 15 | 15 | 0 | 0 | 0 | 0 | 80.1 | 58 | 28 | 25 | 4 | 14 | 0 | 102 | 3 | 0 | 1 | 310 | 155 | 2.00 | 0.896 | 6.5 | 0.4 | 1.6 | 11.4 | 7.29 | |
| 2024 | 27 | DET | AL | 6.7 | 18 | 4 | .818 | 2.39 | 31 | 31 | 0 | 0 | 0 | 0 | 192.0 | 142 | 54 | 51 | 15 | 35 | 0 | 228 | 9 | 0 | 2 | 753 | 174 | 2.49 | 0.922 | 6.7 | 0.7 | 1.6 | 10.7 | 6.51 | AS,CYA-1,MVP-7 |
| 2025 | 28 | DET | AL | 6.4 | 13 | 6 | .684 | 2.21 | 31 | 31 | 0 | 1 | 1 | 0 | 195.1 | 141 | 55 | 48 | 18 | 33 | 0 | 241 | 5 | 2 | 3 | 748 | 185 | 2.45 | 0.891 | 6.5 | 0.8 | 1.5 | 11.1 | 7.30 | AS,CYA-1,MVP-5 |
| 2026 | 29 | 2TM | 2LG | 4.6 | 12 | 7 | .632 | 2.72 | 26 | 26 | 0 | 0 | 0 | 0 | 158.2 | 126 | 52 | 48 | 14 | 27 | 0 | 186 | 5 | 0 | 1 | 620 | 155 | 2.51 | 0.964 | 7.1 | 0.8 | 1.5 | 10.6 | 6.89 | |
| 2026 | 29 | DET | AL | 2.7 | 7 | 5 | .583 | 2.79 | 16 | 16 | 0 | 0 | 0 | 0 | 96.2 | 74 | 33 | 30 | 10 | 14 | 0 | 116 | 4 | 0 | 0 | 374 | 151 | 2.60 | 0.910 | 6.9 | 0.9 | 1.3 | 10.8 | 8.29 | |
| 2026 | 29 | LAD | NL | 1.9 | 5 | 2 | .714 | 2.61 | 10 | 10 | 0 | 0 | 0 | 0 | 62.0 | 52 | 19 | 18 | 4 | 13 | 0 | 70 | 1 | 0 | 1 | 246 | 162 | 2.36 | 1.048 | 7.5 | 0.6 | 1.9 | 10.2 | 5.38 | |
| Ignoreda point in time (year, season, age) | Ignoreda point in time (year, season, age) | Ignoredlabels with no natural order | Ignoredlabels with no natural order | Amounthigher is better · linear · shared scale | Amounthigher is better · log scale | Amountlower is better · linear | Amounthigher is better · linear | Amountlower is better · linear | Amountno direction · linear · shared scale | Amounthigher is better · linear · shared scale | Amountno direction · linear | Amounthigher is better · linear | Amounthigher is better · linear | Ignoredevery value is the same | Amounthigher is better · linear · shared scale | Amountlower is better · linear | Amountlower is better · linear · shared scale | Amountlower is better · linear | Amountlower is better · linear | Amountlower is better · linear | Ignoredevery value is the same | Amounthigher is better · linear | Amountlower is better · linear | Amountlower is better · linear | Amountlower is better · linear · shared scale | Amountno direction · linear | Amounthigher is better · linear | Amountlower is better · linear | Amountlower is better · linear | Amountlower is better · linear | Amountlower is better · linear | Amountlower is better · linear | Amounthigher is better · linear | Amounthigher is better · linear | Ignoredtoo few filled cells |
| Column | Hand label | Judgment | Model | Result |
|---|---|---|---|---|
| Whole table | compared down columns | compared down columns | P(rows) 4% | Pass |
72 questions in 3 requests for this table
Copy linkHow is Clef different from other decision models?
blog.cloudflare.com · Clef benchmarks · 7 columns × 10 rows· read by rows
| Benchmark | Clef | Clef-flash | Jev | DiffusionGemma Jev | Kev 9B | Laya |
|---|---|---|---|---|---|---|
| BFCL · case exact | 98.47 | 98.76 | 95.75 | 96.52 | 94.51 | 38.13 |
| ToolRet · nDCG@10 | 69.19 | 66.43 | 65.28 | 61.21 | 64.26 | 12.69 |
| API-Bank · accuracy | 91.93 | 93.11 | 88.19 | 83.66 | 56.30 | 11.41 |
| Home appliances · case exact | 82.95 | 97.73 | 52.27 | 42.05 | 25.00 | 0.00 |
| When2Call · accuracy | 72.37 | 65.58 | 80.97 | 75.44 | 49.62 | 11.94 |
| BANKING77 · macro-F1 | 94.20 | 90.93 | 79.74 | 74.28 | 84.83 | 14.29 |
| CLINC150+OOS · macro-F1 | 97.43 | 66.77 | 89.27 | 83.49 | 79.03 | 3.19 |
| BRIGHT · nDCG@10 | 45.91 | 39.26 | 47.52 | 42.94 | 38.53 | 19.90 |
| Amazon ESCI · macro-F1 | 57.48 | 57.39 | 55.21 | 53.37 | 49.22 | 24.40 |
| PhishNChips · accuracy | 79.60 | 75.05 | 62.55 | 85.35 | 50.75 | 50.15 |
- Benchmark
- Ignored names or versions
- BFCL · case exact
- Amount higher is better · linear
- ToolRet · nDCG@10
- Amount higher is better · linear
- API-Bank · accuracy
- Amount higher is better · linear
- Home appliances · case exact
- Amount higher is better · linear
- When2Call · accuracy
- Amount higher is better · linear
- BANKING77 · macro-F1
- Amount higher is better · linear
- CLINC150+OOS · macro-F1
- Amount higher is better · log scale
- BRIGHT · nDCG@10
- Amount higher is better · linear
- Amazon ESCI · macro-F1
- Amount higher is better · linear
- PhishNChips · accuracy
- Amount higher is better · linear
| Column | Hand label | Judgment | Model | Result |
|---|---|---|---|---|
| Whole table | compared across rows | compared across rows | P(rows) 98% | Pass |
44 questions in 3 requests for this table
OpenAI
docs.github.com · GitHub Copilot pricing · 9 columns × 22 rows
| Model | Release status | Category | Tier | Threshold (input tokens) | Input | Cached input | Cache write | Output |
|---|---|---|---|---|---|---|---|---|
| GPT-5 mini | GA | Lightweight | Default | Not applicable | $0.25 | $0.025 | Not applicable | $2.00 |
| GPT-5.3-Codex | GA | Powerful | Default | Not applicable | $1.75 | $0.175 | Not applicable | $14.00 |
| GPT-5.4 | GA | Versatile | Default | ≤ 272K | $2.50 | $0.25 | Not applicable | $15.00 |
| GPT-5.4 | GA | Versatile | Long context | > 272K | $5.00 | $0.50 | Not applicable | $22.50 |
| GPT-5.4 mini | GA | Lightweight | Default | Not applicable | $0.75 | $0.075 | Not applicable | $4.50 |
| GPT-5.4 nano | GA | Lightweight | Default | Not applicable | $0.20 | $0.02 | Not applicable | $1.25 |
| GPT-5.5 | GA | Powerful | Default | ≤ 272K | $5.00 | $0.50 | Not applicable | $30.00 |
| GPT-5.5 | GA | Powerful | Long context | > 272K | $10.00 | $1.00 | Not applicable | $45.00 |
| GPT-5.6 Luna | GA | Lightweight | Default | ≤ 200K | $0.20 | $0.02 | $0.25 | $1.20 |
| GPT-5.6 Luna | GA | Lightweight | Long context | > 200K | $0.40 | $0.04 | $0.50 | $1.80 |
| GPT-5.6 Sol | GA | Powerful | Default | ≤ 272K | $4.00 | $0.40 | $5.00 | $20.00 |
| GPT-5.6 Sol | GA | Powerful | Long context | > 272K | $8.00 | $0.80 | $10.00 | $30.00 |
| GPT-5.6 Terra | GA | Versatile | Default | ≤ 272K | $2.00 | $0.20 | $2.50 | $12.00 |
| GPT-5.6 Terra | GA | Versatile | Long context | > 272K | $4.00 | $0.40 | $5.00 | $18.00 |
| GPT-6 Astra | GA | Powerful | Default | ≤ 272K | $10.00 | $1.00 | $12.50 | $50.00 |
| GPT-6 Astra | GA | Powerful | Long context | > 272K | $20.00 | $2.00 | $25.00 | $75.00 |
| GPT-6 Luna | GA | Lightweight | Default | ≤ 272K | $0.10 | $0.01 | $0.125 | $0.50 |
| GPT-6 Luna | GA | Lightweight | Long context | > 272K | $0.20 | $0.02 | $0.25 | $0.75 |
| GPT-6 Sol | GA | Powerful | Default | ≤ 272K | $2.00 | $0.20 | $2.50 | $10.00 |
| GPT-6 Sol | GA | Powerful | Long context | > 272K | $4.00 | $0.40 | $5.00 | $15.00 |
| GPT-6.1 Sol | GA | Powerful | Default | ≤ 272K | $2.00 | $0.10 | $2.50 | $10.00 |
| GPT-6.1 Sol | GA | Powerful | Long context | > 272K | $4.00 | $0.20 | $5.00 | $15.00 |
| Ignorednames or free text | Ignoredsingle value | Ordered labels3 levels · shared scale | Ignoredonly two levels | Ignoreda cutoff, not an amount | Amountlower is better · log scale · shared scale | Amountlower is better · log scale · shared scale | Amountlower is better · log scale · shared scale | Amountlower is better · log scale · shared scale |
| Column | Hand label | Judgment | Model | Result |
|---|---|---|---|---|
| Whole table | compared down columns | compared down columns | P(rows) 3% | Pass |