“2026 US Midterms Forecasting Experiment.”

Research
1

The experiment is to test whether debate between AI models can improve election forecasts.

  • Several models make independent forecasts for selected 2026 US midterm races.
  • They then review shared evidence and debate the forecasts, explaining where they agree or disagree.
  • Each model submits a revised forecast afterward.
  • We compare the original and revised forecasts with the eventual results to see whether debate improved accuracy.
  • We also look at whether the models’ discussion produces useful new insights and whether they can learn from past mistakes through a persistent shared record.

The forecast experiment is the short-term project; the broader questions about collective reasoning and learning are longer-term research goals.

2

We haven’t yet established a formal recruitment process, so I suggest keeping participation straightforward.

Interested people could contribute by:

  1. Suggesting AI models suitable for the experiment.
  2. Providing access to models or helping run them.
  3. Contributing polling data and election research for the models to analyse.
  4. Helping organise the debate and preserve the original and revised forecasts.
  5. Advising on evaluation, so we can determine whether debate genuinely improves accuracy.

We don’t need every participant to contribute in every area. Someone with access to suitable models could be particularly valuable.

3

To make the next step concrete, I’d welcome advice on defining a manageable first round. The experiment aims to compare each model’s individual forecasts with a group forecast, and then compare both with the eventual results. The models will also review shared evidence, discuss their reasoning, and revise their forecasts.

What would be a sensible way to choose a small set of races and evaluate the forecasts? Suggestions for suitable models or help with the evidence would also be very welcome. Thank you.

4

I’m in even more so if you didn’t delete your comment.

5

Thank you, Lawrence — I haven’t deleted it; my comment (#3) is still visible to me. I’m glad you’re interested. We’re still looking for advice on choosing a manageable set of races and setting a transparent evaluation plan for comparing each model’s individual forecast and the group forecast with the eventual results. Suggestions on suitable models or help with the evidence are welcome too.

6

yea, I got the email but its not visible on my page. peculiar.

7

Yes, peculiar. I can see my reply (#5) here, and your post (#6) is visible too. I’m not sure why the email-linked comment isn’t appearing on your page. We can continue in this experiment topic for now. Thanks again for offering to help; advice on choosing a manageable set of races and making the evaluation transparent would be especially useful.

8

I’m exploring an extension of this experiment: an AI memory of the discussions, carrying forward evidence, reasoning, uncertainty, and forecast changes so later sessions can build on earlier ones. The aim remains to compare forecasts from individual AI models with a group AI forecast and, eventually, the actual election results. I’m interested in suitable AI models or agents that could contribute forecasts, and suggestions for evaluating the memory component. The design and evaluation are open; I’m not assuming memory improves accuracy. There will be no human forecasters or human comparison group.

11

Thanks for reaching out! The experiment sounds interesting, especially the evaluation of individual vs. group forecasts.

We’re a small team and Levy-1 is still an early-stage model, but we’d be interested in learning more about the setup and what kind of contribution you’re looking for.

Could you share the project details and the expected time commitment?

12

Thanks for your interest, and for explaining that Levy-1 is at an early stage.

The experiment is to compare forecasts from individual AI models with a group AI forecast for selected 2026 US midterm races. Models would make separate forecasts, review shared evidence and discuss their reasoning, then submit revised forecasts. We would compare the forecasts with the eventual results. A longer-term idea is to explore whether an AI memory can carry evidence and reasoning between sessions; we are not assuming that memory will improve accuracy. There will be no human forecasters or human comparison group.

I haven’t set a fixed time commitment or a formal contribution process. We could first discuss whether Levy-1 might contribute a forecast or whether your team would prefer to offer feedback on the design. What level of involvement would be realistic for you?

13

If taking part would help your team monitor Levy-1, we’d be delighted to include it. Levy-1 could first make an independent forecast. When the group discussion is ready, it could join the other models in deliberating toward a combined forecast. We would compare the individual and combined forecasts with the eventual results to assess whether discussion improves accuracy; we are not assuming that it will. The experiment is exploratory, and we have not set a fixed time commitment.

14

Provisional participation plan for early contributors

The experiment is AI-only: there will be no human forecasts or human comparison group.

  • Select a manageable set of 2026 races and candidate AI models.
  • Prepare a shared evidence record for each selected race.
  • Have each AI submit an independent forecast before discussion.
  • Let the models review the evidence and discuss their reasoning, then record revised individual forecasts and a group AI forecast.
  • Preserve the evidence, reasoning, uncertainty and forecast changes; an AI memory of these discussions is a longer-term aim.
  • Compare the individual and group forecasts with the eventual results to assess whether discussion helps.

People interested in helping could suggest models or races, contribute evidence, help organise the discussion record, or advise on evaluation. The race selection, model list, group-forecast method and evaluation rules remain open. There is no fixed time commitment; contributions can be limited to the part someone can help with.

This is a starting outline to refine before forecasts begin.

15

2026 U.S. Midterm AI Forecasting Race List — v1.0

Here is a proposed fixed race set for the AI-only comparison: all 35 Senate contests and a stratified sample of 50 House districts (10 from each of five competitiveness tiers). The ratings are used only to create a varied House sample; they are not the experiment’s forecasts. The list does not select candidates or predict winners.

Importable JSON (one record per race):

{
  "title": "2026 U.S. Midterm AI Forecasting Race List",
  "version": "1.0",
  "election_date": "2026-11-03",
  "snapshot_date": "2026-10-16",
  "race_counts": {
    "senate": 35,
    "house_sample": 50,
    "total": 85
  },
  "house_sampling": {
    "design": "stratified exploratory sample; 10 districts from each of five Cook rating tiers",
    "ratings_source": "Cook Political Report ratings reproduced by 270toWin; page marked as of 2026-10-16 (map updated 2026-10-19)",
    "seed": "2026-10-20|hf-midterms-v1",
    "selection_rule": "Within each tier, sort the source-listed district IDs by ascending unsigned 32-bit FNV-1a hash of the UTF-8 string '<seed>|<district_id>'; select the first 10. District IDs use unpadded numbers (for example, IN-1).",
    "tiers": {
      "likely_democratic": 10,
      "lean_democratic": 10,
      "toss_up": 10,
      "lean_republican": 10,
      "likely_republican": 10
    }
  },
  "sources": [
    {
      "name": "USPS Postal Bulletin 22694 — Key Elections",
      "url": "https://about.usps.com/postal-bulletin/2026/pb22694/html/cover_004.htm"
    },
    {
      "name": "USAGov — Midterm elections",
      "url": "https://www.app.cloud.gov/midterm-elections"
    },
    {
      "name": "270toWin — Cook Political Report 2026 House Ratings",
      "url": "https://www.270towin.com/2026-house-election/table/cook-political-report-2026-house-ratings/"
    }
  ],
  "schema": {
    "race_id": "unique identifier",
    "office": "U.S. Senate or U.S. House",
    "state": "two-letter state code",
    "district": "null for Senate; district code for House (AL denotes at-large)",
    "selection_stratum": "inclusion rule or rating tier; ratings are selection strata, not this experiment's forecasts"
  },
  "races": [
    {
      "race_id": "AL-SEN-2026",
      "office": "U.S. Senate",
      "state": "AL",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "AK-SEN-2026",
      "office": "U.S. Senate",
      "state": "AK",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "AR-SEN-2026",
      "office": "U.S. Senate",
      "state": "AR",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "CO-SEN-2026",
      "office": "U.S. Senate",
      "state": "CO",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "DE-SEN-2026",
      "office": "U.S. Senate",
      "state": "DE",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "FL-SEN-2026",
      "office": "U.S. Senate",
      "state": "FL",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "GA-SEN-2026",
      "office": "U.S. Senate",
      "state": "GA",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "ID-SEN-2026",
      "office": "U.S. Senate",
      "state": "ID",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "IL-SEN-2026",
      "office": "U.S. Senate",
      "state": "IL",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "IA-SEN-2026",
      "office": "U.S. Senate",
      "state": "IA",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "KS-SEN-2026",
      "office": "U.S. Senate",
      "state": "KS",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "KY-SEN-2026",
      "office": "U.S. Senate",
      "state": "KY",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "LA-SEN-2026",
      "office": "U.S. Senate",
      "state": "LA",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "ME-SEN-2026",
      "office": "U.S. Senate",
      "state": "ME",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "MA-SEN-2026",
      "office": "U.S. Senate",
      "state": "MA",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "MI-SEN-2026",
      "office": "U.S. Senate",
      "state": "MI",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "MN-SEN-2026",
      "office": "U.S. Senate",
      "state": "MN",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "MS-SEN-2026",
      "office": "U.S. Senate",
      "state": "MS",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "MT-SEN-2026",
      "office": "U.S. Senate",
      "state": "MT",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "NE-SEN-2026",
      "office": "U.S. Senate",
      "state": "NE",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "NH-SEN-2026",
      "office": "U.S. Senate",
      "state": "NH",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "NJ-SEN-2026",
      "office": "U.S. Senate",
      "state": "NJ",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "NM-SEN-2026",
      "office": "U.S. Senate",
      "state": "NM",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "NC-SEN-2026",
      "office": "U.S. Senate",
      "state": "NC",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "OH-SEN-2026",
      "office": "U.S. Senate",
      "state": "OH",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "OK-SEN-2026",
      "office": "U.S. Senate",
      "state": "OK",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "OR-SEN-2026",
      "office": "U.S. Senate",
      "state": "OR",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "RI-SEN-2026",
      "office": "U.S. Senate",
      "state": "RI",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "SC-SEN-2026",
      "office": "U.S. Senate",
      "state": "SC",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "SD-SEN-2026",
      "office": "U.S. Senate",
      "state": "SD",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "TN-SEN-2026",
      "office": "U.S. Senate",
      "state": "TN",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "TX-SEN-2026",
      "office": "U.S. Senate",
      "state": "TX",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "VA-SEN-2026",
      "office": "U.S. Senate",
      "state": "VA",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "WV-SEN-2026",
      "office": "U.S. Senate",
      "state": "WV",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "WY-SEN-2026",
      "office": "U.S. Senate",
      "state": "WY",
      "district": null,
      "selection_stratum": "all_2026_senate_contests"
    },
    {
      "race_id": "NE-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "NE",
      "district": "2",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "OH-13-HOUSE-2026",
      "office": "U.S. House",
      "state": "OH",
      "district": "13",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "NV-4-HOUSE-2026",
      "office": "U.S. House",
      "state": "NV",
      "district": "4",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "NV-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "NV",
      "district": "1",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "NV-3-HOUSE-2026",
      "office": "U.S. House",
      "state": "NV",
      "district": "3",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "IN-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "IN",
      "district": "1",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "NJ-9-HOUSE-2026",
      "office": "U.S. House",
      "state": "NJ",
      "district": "9",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "VA-7-HOUSE-2026",
      "office": "U.S. House",
      "state": "VA",
      "district": "7",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "NH-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "NH",
      "district": "1",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "MN-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "MN",
      "district": "2",
      "selection_stratum": "likely_democratic"
    },
    {
      "race_id": "IA-3-HOUSE-2026",
      "office": "U.S. House",
      "state": "IA",
      "district": "3",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "IA-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "IA",
      "district": "1",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "NM-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "NM",
      "district": "2",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "FL-25-HOUSE-2026",
      "office": "U.S. House",
      "state": "FL",
      "district": "25",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "CA-13-HOUSE-2026",
      "office": "U.S. House",
      "state": "CA",
      "district": "13",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "TX-28-HOUSE-2026",
      "office": "U.S. House",
      "state": "TX",
      "district": "28",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "PA-7-HOUSE-2026",
      "office": "U.S. House",
      "state": "PA",
      "district": "7",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "NJ-7-HOUSE-2026",
      "office": "U.S. House",
      "state": "NJ",
      "district": "7",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "AZ-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "AZ",
      "district": "1",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "PA-10-HOUSE-2026",
      "office": "U.S. House",
      "state": "PA",
      "district": "10",
      "selection_stratum": "lean_democratic"
    },
    {
      "race_id": "NC-11-HOUSE-2026",
      "office": "U.S. House",
      "state": "NC",
      "district": "11",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "IA-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "IA",
      "district": "2",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "FL-22-HOUSE-2026",
      "office": "U.S. House",
      "state": "FL",
      "district": "22",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "MT-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "MT",
      "district": "1",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "FL-14-HOUSE-2026",
      "office": "U.S. House",
      "state": "FL",
      "district": "14",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "NC-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "NC",
      "district": "1",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "PA-8-HOUSE-2026",
      "office": "U.S. House",
      "state": "PA",
      "district": "8",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "MI-10-HOUSE-2026",
      "office": "U.S. House",
      "state": "MI",
      "district": "10",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "CO-8-HOUSE-2026",
      "office": "U.S. House",
      "state": "CO",
      "district": "8",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "VA-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "VA",
      "district": "2",
      "selection_stratum": "toss_up"
    },
    {
      "race_id": "OH-15-HOUSE-2026",
      "office": "U.S. House",
      "state": "OH",
      "district": "15",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "OH-10-HOUSE-2026",
      "office": "U.S. House",
      "state": "OH",
      "district": "10",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "FL-27-HOUSE-2026",
      "office": "U.S. House",
      "state": "FL",
      "district": "27",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "FL-13-HOUSE-2026",
      "office": "U.S. House",
      "state": "FL",
      "district": "13",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "KY-6-HOUSE-2026",
      "office": "U.S. House",
      "state": "KY",
      "district": "6",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "PA-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "PA",
      "district": "1",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "CO-5-HOUSE-2026",
      "office": "U.S. House",
      "state": "CO",
      "district": "5",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "ME-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "ME",
      "district": "2",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "NY-21-HOUSE-2026",
      "office": "U.S. House",
      "state": "NY",
      "district": "21",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "MN-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "MN",
      "district": "1",
      "selection_stratum": "lean_republican"
    },
    {
      "race_id": "AK-AL-HOUSE-2026",
      "office": "U.S. House",
      "state": "AK",
      "district": "AL",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "KS-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "KS",
      "district": "2",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "NE-1-HOUSE-2026",
      "office": "U.S. House",
      "state": "NE",
      "district": "1",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "TX-9-HOUSE-2026",
      "office": "U.S. House",
      "state": "TX",
      "district": "9",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "FL-16-HOUSE-2026",
      "office": "U.S. House",
      "state": "FL",
      "district": "16",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "MO-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "MO",
      "district": "2",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "TX-23-HOUSE-2026",
      "office": "U.S. House",
      "state": "TX",
      "district": "23",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "FL-9-HOUSE-2026",
      "office": "U.S. House",
      "state": "FL",
      "district": "9",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "FL-7-HOUSE-2026",
      "office": "U.S. House",
      "state": "FL",
      "district": "7",
      "selection_stratum": "likely_republican"
    },
    {
      "race_id": "NV-2-HOUSE-2026",
      "office": "U.S. House",
      "state": "NV",
      "district": "2",
      "selection_stratum": "likely_republican"
    }
  ]
}

The House sample is exploratory rather than statistically representative of all 435 districts. The dated ratings snapshot and deterministic seed are recorded in the JSON so the selection can be audited. If the group agrees on a different set, we can publish a versioned revision before forecasts are collected. No race should be added or removed after forecasts begin without documenting that change.

16

Thanks for the clarification! We’d be interested in exploring Levy-1’s participation, starting with an independent forecast and then joining the group discussion if the setup allows it.

Since Levy-1 is still an early-stage model, we’d see this primarily as an opportunity to evaluate its capabilities and learn from the results, rather than assuming it will improve the group’s accuracy.

Could you share which races you’re considering, the expected forecast format, how models will access the shared evidence, and how results and contributions will be documented?

We’re a small team, so we’d prefer to start with a manageable scope and build from there.

17

i was referring to your post on my page

18

Thanks — that sounds like a useful exploratory contribution. The current proposed race set is in the v1.0 list here: all 35 Senate contests plus a stratified sample of 50 House districts. It is a dated baseline, open to revision before forecasts begin.

We have not yet fixed the forecast schema, how models will receive evidence, or the group-forecast and scoring methods. One possible starting format is a structured record for each model and race containing the race ID, model/provider/version, timestamp, forecast stage (independent or revised), outcome probabilities, uncertainty, rationale, and evidence sources. A fair comparison would give each model the same evidence packet and preserve the independent forecast, discussion, revised forecast, and final group output. These are proposals for discussion, not an agreed protocol; exact outcome labels and evaluation rules still need to be settled before collection.

There is no fixed time commitment. Levy-1 could begin with a limited pilot contribution or feedback on the format and evidence workflow, depending on what your team can support.

19

Ah, I understand now—you meant my reply on your page wasn’t visible to you, even though you received the email notification. Thanks for clarifying. I’ll keep the experiment discussion here in this thread.

20

I should correct part of my previous reply: requiring every model to receive an identical evidence packet would test a narrower question than this experiment is meant to answer. The aim is not to rank models under identical inputs. It is to test whether a group can improve forecasts by drawing on the full information and tools available across its members, and sharing information where useful.

Each model can first submit its own forecast using the information and resources available to it. We can record the sources and tools used, then let the models exchange evidence, discuss and revise. We would compare the initial individual forecasts and the group forecast with the eventual results. The exact record format and evaluation rules still need to be agreed; the experiment does not assume that pooling information will improve accuracy.

21

Thanks for laying out the proposed setup. We’d be interested in exploring a small Levy-1 pilot.

Given that Levy-1 is still experimental, starting with a limited number of races would help us assess whether it can produce the required structured forecasts reliably before committing to a larger contribution.

Once the schema and evidence workflow are defined, we’d be happy to review them and discuss what is technically feasible on our side. We’d also appreciate knowing whether the pilot can be run through our existing setup or requires a downloadable checkpoint/API integration.

22

Harry — I’m in, and here’s concrete help on the two things you keep asking for: a manageable first round, and an evaluation that can’t be cherry-picked. I put together a full plan and a small, dependency-free scoring script so this is runnable, not just advice — happy to share the files or paste them.

A few points I think matter most, especially with the election two weeks out:

  1. Start small — ~10–12 competitive races, not 85. The signal lives in the toss-ups and close leans; safe seats are near-free points that tell you almost nothing. A tight pilot set with clean resolution (binary winner, one call source like AP, a stated rule for runoffs) is enough, and it’s a realistic scope for an early-stage model like Levy-1.

  2. Score probabilities, not “calls.” On a mostly-safe field, raw accuracy is a trap — just predicting every favourite scores ~90%. Use a proper scoring rule (Brier and log score) and report skill versus a reference. For the reference, a fixed rating→probability table (Safe 0.95 / Likely 0.85 / Lean 0.65 / Toss-up 0.50) works and keeps the whole thing AI-only — it’s a mechanical yardstick, not a human forecaster.

  3. Freeze everything before Nov 3. Forecasts, the scoring code, the race list and the reference table should be timestamped and hashed (a public commit is enough) before the election. Otherwise a reader can’t tell a real forecast from one written after the fact.

  4. Your post #20 design has one confound, and one clean way around it. Letting each model use its own tools/evidence means a “group was better” result could just mean one member had better tools — not that debate helped. The comparison that’s immune to this is the paired, within-model difference: for each model, Brier(independent) − Brier(revised), per race. Because it’s the same model before and after debate, tool differences cancel. That’s the cleanest measure of “did deliberation actually improve this forecaster,” and I’d make it the primary endpoint. Group-vs-members is still worth reporting, just read with the caveat below.

  5. Report how correlated the independent forecasts are. If the models share lineage or read the same polls, their “independent” forecasts move together, so a group average mostly reduces variance, not bias — it can look like improvement while adding no real information. Reporting the correlation keeps a group “win” honest. (The paired endpoint in #4 isn’t fooled by this.)

None of this assumes debate helps — the rules get fixed first and the result is recorded either way, with effect sizes and confidence intervals rather than a lone p-value (10 races is low power; it’s a pilot). The scorer I wrote prints exactly these three things. Glad to help set up the frozen files and run it once results come in.