Practical Data / tests

Recorded answers from jev-latest

Shared scales

Do columns in two tables use one scale?

A page can show the prices of two stores in two tables. The same color must then show the same price in the two tables.

Code finds columns in different tables that have the same header and unit. The model tells if they measure the same quantity. If they do, they get one scale that is wide enough for all their values.

Decision rule
Use one scale when P(same quantity) is 0.5 or more.

The questions

The model gets the question text and a short description of the table. It gives a probability for each possible answer. These are real questions from the test pages, with the recorded answers. Here, the question names the column by its header. The question that the extension sends points to the column in the table description.

These columns come from different tables on the same page. Do all of columns grade the same property using labels from one shared scale (a table may use only some of the levels), so that one shared color scale would let a reader compare grades across the tables?
columns
tableheadergrades
OpenAICategoryLightweightVersatilePowerful
AnthropicCategoryVersatilePowerful
Table description sent with this question

Practical Data makes this description from the cells of the table and sends it with the question. examples are values from the column. spread tells how wide the range of values is. notes give hints, such as values that look like years. All fields and their possible values.

page_title
Models and pricing for GitHub Copilot - GitHub Docs
AnswerProbability
yes 56%
no 44%
g0.share · noul
The values of the column "Category" are grades of one property. Which of a and b is the lower grade?
a
lightweight
b
versatile
Table description sent with this question

Practical Data makes this description from the cells of the table and sends it with the question. examples are values from the column. spread tells how wide the range of values is. notes give hints, such as values that look like years. All fields and their possible values.

page_title
Models and pricing for GitHub Copilot - GitHub Docs
AnswerProbability
a 99%
b 1%
g0.lt0_1 · choice

Results

9 of 10 checks pass, on 10 tables. The failure is a known issue. Tables with a failure come first.

Each table shows the shading that the extension applies. The strip under the table shows all judgments on each column. Put the pointer on a result to outline its column. A red outline shows a failure.

By death toll

en.wikipedia.org · Earthquakes of 2023 · 8 columns × 10 rows

RankDeath tollMagnitudeLocationMMIDepth (km)DateEvent
159,488–62,0137.8Turkey, KahramanmaraşXII (Extreme)10.0February 62023 Turkey–Syria earthquakes
159,488–62,0137.5Turkey, KahramanmaraşXI (Extreme)7.4February 62023 Turkey–Syria earthquakes
22,9606.8Morocco, Al HaouzIX (Violent)19.0September 82023 Al Haouz earthquake
31,4826.3Afghanistan, HeratVIII (Severe)14.0October 72023 Herat earthquakes
41535.7Nepal, KarnaliVIII (Severe)16.5November 32023 Nepal earthquake
51515.9China, GansuIX (Violent)10.0December 182023 Jishishan earthquake
6216.5Afghanistan, BadakhshanV (Moderate)187.6March 212023 Badakhshan earthquake
7186.8Ecuador, Guayas offshoreVII (Very strong)65.8March 182023 Guayas earthquake
8116.3Turkey, HatayIX (Violent)16.0February 202023 Turkey–Syria earthquakes
9116.7Philippines, Soccsksargen offshoreVIII (Severe)52.0November 17November 2023 Mindanao earthquake
Ignoreda rank, not an amountAmountlower is better · log scaleAmountno direction · linear · shared scaleIgnoredfree textOrdered labels8 levels · shared scaleAmountno direction · log scale · shared scaleDatesearliest to latestIgnoredfree text
Checks on By death toll: 3 of 4 pass
ColumnHand labelJudgmentResult
Death tollone scaleseparate scalesKnown limitation
Known limitation: A close decision can change from one run to the next. The model's answers are not fully repeatable near a threshold. Two runs of the same question about death tolls in two earthquake tables gave a shared scale once and separate scales once.
Magnitudeone scaleone scalePass
MMIone scaleone scalePass
Depth (km)one scaleone scalePass
47 questions in 3 requests for this table

By magnitude

en.wikipedia.org · Earthquakes of 2023 · 8 columns × 19 rows

RankMagnitudeDeath tollLocationMMIDepth (km)DateEvent
17.859,488–62,013Turkey, KahramanmaraşXII (Extreme)10.0February 62023 Turkey–Syria earthquakes
27.70New Caledonia, Loyalty Islands offshoreIV (Light)18.0May 19-
37.63Philippines, Caraga offshoreVII (Very strong)40.0December 2December 2023 Mindanao earthquake
37.60Indonesia, Maluku offshoreVI (Strong)105.2January 9-
37.60Tonga, Niuatoputapu offshoreVI (Strong)210.1May 10-
67.5-Turkey, KahramanmaraşXI (Extreme)7.4February 62023 Turkey–Syria earthquakes
77.20Tonga, Tongatapu offshoreVI (Strong)167.4June 15-
77.20United States, Alaska offshoreVII (Very strong)25.0July 16-
97.10Indonesia, North Sumatra offshoreVIII (Severe)34.0April 24-
97.10New Zealand, Kermadec Islands offshoreVII (Very strong)46.8April 24-
97.10Indonesia, Banda Sea offshoreVI (Strong)10.0November 8-
97.10Vanuatu, Tafea offshoreVI (Strong)48.4December 7-
97.10New Caledonia, Loyalty Islands offshoreIV (Light)36.0May 20-
97.10Indonesia, East Java offshoreIV (Light)513.5August 28-
157.00Vanuatu, Sanma offshoreVII (Very strong)29.0January 8-
157.08Papua New Guinea, East SepikVII (Very strong)70.0April 22023 Papua New Guinea earthquake
157.00Indonesia, North Maluku offshoreV (Moderate)28.6January 18-
157.01Indonesia, East Java offshoreV (Moderate)597.0April 14-
157.00New Zealand, Kermadec Islands offshoreIV (Light)10.0March 16-
Ignoreda rank, not an amountAmountno direction · linear · shared scaleAmountlower is better · linearIgnorednames or free textOrdered labels8 levels · shared scaleAmountno direction · log scale · shared scaleDatesearliest to latestIgnoredfree text
Checks on By magnitude: 3 of 4 pass
ColumnHand labelJudgmentResult
Magnitudeone scaleone scalePass
Death tollone scaleseparate scalesKnown limitation
Known limitation: A close decision can change from one run to the next. The model's answers are not fully repeatable near a threshold. Two runs of the same question about death tolls in two earthquake tables gave a shared scale once and separate scales once.
MMIone scaleone scalePass
Depth (km)one scaleone scalePass
59 questions in 3 requests for this table

OpenAI

docs.github.com · GitHub Copilot pricing · 9 columns × 22 rows

ModelRelease statusCategoryTierThreshold (input tokens)InputCached inputCache writeOutput
GPT-5 miniGALightweightDefaultNot applicable$0.25$0.025Not applicable$2.00
GPT-5.3-CodexGAPowerfulDefaultNot applicable$1.75$0.175Not applicable$14.00
GPT-5.4GAVersatileDefault≤ 272K$2.50$0.25Not applicable$15.00
GPT-5.4GAVersatileLong context> 272K$5.00$0.50Not applicable$22.50
GPT-5.4 miniGALightweightDefaultNot applicable$0.75$0.075Not applicable$4.50
GPT-5.4 nanoGALightweightDefaultNot applicable$0.20$0.02Not applicable$1.25
GPT-5.5GAPowerfulDefault≤ 272K$5.00$0.50Not applicable$30.00
GPT-5.5GAPowerfulLong context> 272K$10.00$1.00Not applicable$45.00
GPT-5.6 LunaGALightweightDefault≤ 200K$0.20$0.02$0.25$1.20
GPT-5.6 LunaGALightweightLong context> 200K$0.40$0.04$0.50$1.80
GPT-5.6 SolGAPowerfulDefault≤ 272K$4.00$0.40$5.00$20.00
GPT-5.6 SolGAPowerfulLong context> 272K$8.00$0.80$10.00$30.00
GPT-5.6 TerraGAVersatileDefault≤ 272K$2.00$0.20$2.50$12.00
GPT-5.6 TerraGAVersatileLong context> 272K$4.00$0.40$5.00$18.00
GPT-6 AstraGAPowerfulDefault≤ 272K$10.00$1.00$12.50$50.00
GPT-6 AstraGAPowerfulLong context> 272K$20.00$2.00$25.00$75.00
GPT-6 LunaGALightweightDefault≤ 272K$0.10$0.01$0.125$0.50
GPT-6 LunaGALightweightLong context> 272K$0.20$0.02$0.25$0.75
GPT-6 SolGAPowerfulDefault≤ 272K$2.00$0.20$2.50$10.00
GPT-6 SolGAPowerfulLong context> 272K$4.00$0.40$5.00$15.00
GPT-6.1 SolGAPowerfulDefault≤ 272K$2.00$0.10$2.50$10.00
GPT-6.1 SolGAPowerfulLong context> 272K$4.00$0.20$5.00$15.00
Ignorednames or free textIgnoredsingle valueOrdered labels3 levels · shared scaleIgnoredonly two levelsIgnoreda cutoff, not an amountAmountlower is better · log scale · shared scaleAmountlower is better · log scale · shared scaleAmountlower is better · log scale · shared scaleAmountlower is better · log scale · shared scale
Checks on OpenAI: 2 of 2 pass
ColumnHand labelJudgmentResult
Inputone scaleone scalePass
Outputone scaleone scalePass
31 questions in 5 requests for this table

Anthropic

docs.github.com · GitHub Copilot pricing · 7 columns × 11 rows

ModelRelease statusCategoryInputCached inputCache writeOutput
Claude Haiku 4.5GAVersatile$1.00$0.10$1.25$5.00
Claude Sonnet 4GAVersatile$3.00$0.30$3.75$15.00
Claude Sonnet 4.6GAVersatile$3.00$0.30$3.75$15.00
Claude Opus 4.8GAPowerful$5.00$0.50$6.25$25.00
Claude Opus 5GAPowerful$5.00$0.50$6.25$25.00
Claude Opus 5.5GAPowerful$4.00$0.20$5.00$20.00
Claude Sonnet 5GAVersatile$2.00$0.20$2.50$10.00
Claude Sonnet 5.5GAVersatile$2.00$0.20$2.50$10.00
Claude Opus 4.8 (fast mode) (preview)GAPowerful$10.00$1.00$12.50$50.00
Claude Fable 5GAPowerful$10.00$1.00$12.50$50.00
Claude Fable 5.1GAPowerful$10.00$0.25$12.50$50.00
Ignoredfree textIgnoredsingle valueOrdered labels3 levels · shared scaleAmountlower is better · log scale · shared scaleAmountlower is better · log scale · shared scaleAmountlower is better · log scale · shared scaleAmountlower is better · log scale · shared scale
Checks on Anthropic: 2 of 2 pass
ColumnHand labelJudgmentResult
Inputone scaleone scalePass
Outputone scaleone scalePass
21 questions in 3 requests for this table

AL East

en.wikipedia.org · 2024 MLB standings · 7 columns × 5 rows

TeamWLPct.GBHomeRoad
(1) New York Yankees9468.580—44‍–‍3750‍–‍31
(4) Baltimore Orioles9171.562344‍–‍3747‍–‍34
Boston Red Sox8181.5001338‍–‍4343‍–‍38
Tampa Bay Rays8082.4941442‍–‍3938‍–‍43
Toronto Blue Jays7488.4572039‍–‍4235‍–‍46
Ignoredlabels with no natural orderAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleIgnoredlabels look like scores or recordsIgnoredlabels look like scores or records
Checks on AL East: 4 of 4 pass
ColumnHand labelJudgmentResult
Wone scaleone scalePass
Lone scaleone scalePass
Pct.one scaleone scalePass
GBone scaleone scalePass
19 questions in 4 requests for this table

AL Central

en.wikipedia.org · 2024 MLB standings · 7 columns × 5 rows

TeamWLPct.GBHomeRoad
(2) Cleveland Guardians9269.571—50‍–‍3042‍–‍39
(5) Kansas City Royals8676.5316½45‍–‍3641‍–‍40
(6) Detroit Tigers8676.5316½43‍–‍3843‍–‍38
Minnesota Twins8280.50610½43‍–‍3839‍–‍42
Chicago White Sox41121.25351½23‍–‍5818‍–‍63
Ignoredlabels with no natural orderAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleIgnoredlabels look like scores or recordsIgnoredlabels look like scores or records
Checks on AL Central: 4 of 4 pass
ColumnHand labelJudgmentResult
Wone scaleone scalePass
Lone scaleone scalePass
Pct.one scaleone scalePass
GBone scaleone scalePass
19 questions in 4 requests for this table

AL West

en.wikipedia.org · 2024 MLB standings · 7 columns × 5 rows

TeamWLPct.GBHomeRoad
(3) Houston Astros8873.547—46‍–‍3542‍–‍38
Seattle Mariners8577.5253½49‍–‍3236‍–‍45
Texas Rangers7884.48110½44‍–‍3734‍–‍47
Oakland Athletics6993.42619½38‍–‍4331‍–‍50
Los Angeles Angels6399.38925½32‍–‍4931‍–‍50
Ignoredlabels with no natural orderAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleIgnoredlabels look like scores or recordsIgnoredlabels look like scores or records
Checks on AL West: 4 of 4 pass
ColumnHand labelJudgmentResult
Wone scaleone scalePass
Lone scaleone scalePass
Pct.one scaleone scalePass
GBone scaleone scalePass
19 questions in 4 requests for this table

NL East

en.wikipedia.org · 2024 MLB standings · 7 columns × 5 rows

TeamWLPct.GBHomeRoad
(2) Philadelphia Phillies9567.586—54‍–‍2741‍–‍40
(5) Atlanta Braves8973.549646‍–‍3543‍–‍38
(6) New York Mets8973.549646‍–‍3543‍–‍38
Washington Nationals7191.4382438‍–‍4333‍–‍48
Miami Marlins62100.3833330‍–‍5132‍–‍49
Ignoredlabels with no natural orderAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleIgnoredlabels look like scores or recordsIgnoredlabels look like scores or records
Checks on NL East: 4 of 4 pass
ColumnHand labelJudgmentResult
Wone scaleone scalePass
Lone scaleone scalePass
Pct.one scaleone scalePass
GBone scaleone scalePass
19 questions in 4 requests for this table

NL Central

en.wikipedia.org · 2024 MLB standings · 7 columns × 5 rows

TeamWLPct.GBHomeRoad
(3) Milwaukee Brewers9369.574—47‍–‍3446‍–‍35
St. Louis Cardinals8379.5121044‍–‍3739‍–‍42
Chicago Cubs8379.5121044‍–‍3739‍–‍42
Cincinnati Reds7785.4751639‍–‍4238‍–‍43
Pittsburgh Pirates7686.4691739‍–‍4237‍–‍44
Ignoredlabels with no natural orderAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleIgnoredlabels look like scores or recordsIgnoredlabels look like scores or records
Checks on NL Central: 4 of 4 pass
ColumnHand labelJudgmentResult
Wone scaleone scalePass
Lone scaleone scalePass
Pct.one scaleone scalePass
GBone scaleone scalePass
19 questions in 4 requests for this table

NL West

en.wikipedia.org · 2024 MLB standings · 7 columns × 5 rows

TeamWLPct.GBHomeRoad
(1) Los Angeles Dodgers9864.605—52‍–‍2946‍–‍35
(4) San Diego Padres9369.574545‍–‍3648‍–‍33
Arizona Diamondbacks8973.549944‍–‍3745‍–‍36
San Francisco Giants8082.4941842‍–‍3938‍–‍43
Colorado Rockies61101.3773737‍–‍4424‍–‍57
Ignoredlabels with no natural orderAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleAmounthigher is better · linear · shared scaleAmountlower is better · linear · shared scaleIgnoredlabels look like scores or recordsIgnoredlabels look like scores or records
Checks on NL West: 4 of 4 pass
ColumnHand labelJudgmentResult
Wone scaleone scalePass
Lone scaleone scalePass
Pct.one scaleone scalePass
GBone scaleone scalePass
19 questions in 4 requests for this table