Off-Nadir Delta

Does Giving AI Structured Event Data Improve Its Answers? Research and 10 Side-by-Side Trials

Kazushi Motomura13 November 202610 min read
Does Giving AI Structured Event Data Improve Its Answers? Research and 10 Side-by-Side Trials

Quick Answer: Published research supports a narrow claim. In the MIRAI study (2024 preprint), the same LLM agent predicting relations between countries scored a relation F1 of 26.9 when it could query a structured event database, against 5.4 with news articles alone, and did best with both (29.6). ForecastBench (ICLR 2025) found that giving models recent news did not by itself improve their forecasts. None of these studies tested Off-Nadir Delta. In our own ten internal re-enactments with fictional analysts, an AI assistant with web search was compared with the same assistant connected to Delta: Delta changed the decision once, the decision was the same seven times, and the assistant on its own did better twice. Delta's daily outlooks have not yet been scored against what happened.

Giving a language model structured event data, instead of only news articles, made it much better at one forecasting task in published research. It did not make models better forecasters across the board, and none of the studies tested Off-Nadir Delta. We also ran ten small side-by-side trials of our own. Delta changed the decision in one of them, and an AI assistant on its own did better in two.

This article puts the research and those trials in one place, with the parts that did not go our way. The same tables are kept on the structured event data and AI page.

What does published research say about structured event data and AI forecasts?

The clearest result is from MIRAI, a 2024 preprint from UCLA and Caltech. The same LLM agent was asked to predict the type of relation between two countries for 705 questions from November 2023. With a structured event database as its tool, it scored a second-level relation F1 of 26.9. With news articles only, it scored 5.4. With both, 29.6. The authors write that "ReAct agents using News-Only APIs perform much worse than agents with Event-Only APIs."

Other studies set the limits of that result. Figures below are copied from each paper's own tables; lower Brier scores are better.

StudyWhat was testedWhat it foundWhat it does not show
MIRAI (2024, preprint)The same LLM agent (gpt-3.5-turbo-0125, five runs) predicting the type of relation between two countries for 705 questions from November 2023, with different tools.Second-level relation F1 (Table 1): 26.9 with a structured event database only, 5.4 with news articles only, 29.6 with both, and 3.5 with no tools.The answers were built from the same event database (GDELT), which may favour the agent that queries it. It predicts relation types, not probabilities. It is not a study of Off-Nadir Delta.
Approaching Human-Level Forecasting with Language Models (NeurIPS 2024)A retrieval-augmented LLM forecasting system on 914 binary questions from forecasting platforms, all opened after the models’ training cut-off.Brier score (lower is better): 0.206 without retrieval and 0.186 with retrieved news (ablation, Table 5); 0.179 for the full system with fine-tuning, against 0.149 for the human crowd (Table 3).What was retrieved was news articles, not structured event data. The system stayed behind the crowd overall; averaging it with the crowd gave 0.146.
ForecastBench (ICLR 2025)LLMs, the general public and superforecasters on questions generated automatically, including from event datasets such as ACLED.Superforecasters scored a Brier of 0.096, ahead of the best LLM on the same 200 questions (0.122). The authors report that “access to recent topical news related to the questions did not improve performance.”Handing an LLM more news did not, by itself, make it a better forecaster. Results are from the paper’s 2024 runs.
DataGemma (2024, Google technical report)Answering statistical questions with figures retrieved from a public statistics database (Data Commons) instead of from the model’s memory.When answers cited the retrieved statistics, the numbers were right about 99% of the time; the authors also report that inferences drawn from those numbers were wrong 6–20% of the time.About facts, not forecasts. The comparison with the model on its own rests on few claims (28), and only a minority of answers drew on the database at all.
Pitfalls in Evaluating Language Model Forecasters (2025, preprint)How LLM forecasting results are evaluated.Results are hard to trust when questions, retrieved articles or the model itself leak what happened later, and hard to carry over to real forecasting.Any test of “AI with event data” has to fix the questions and record each forecast before the outcome is known. A replay of past events can look better than it is.

Read together, they say three things:

  • Structure helped in the test that measured it. The agent that could query events did far better than the one reading news alone, and best with both.
  • More news alone did not help. ForecastBench found that recent topical news "did not improve performance," and in the Berkeley study the retrieved material was news, which narrowed but did not close the gap to the human crowd.
  • The tests are easy to get wrong. Paleka and colleagues show how later knowledge can leak into a forecasting test. A replay of past events can look better than it is.

What happened when we compared an AI assistant with and without Delta?

These are internal re-enactments with fictional analysts and fictional ledgers, not customer use. The events are real. Each task was run twice: once with a general AI assistant using web search, and once with the same assistant connected to Off-Nadir Delta.

Of the 10 tasks compared this way (12–13 November 2026), Delta changed the decision in 1, the decision was the same in 7, and the assistant on its own did better in 2. In 3 of the 7 with the same decision, Delta’s output led to an error in the work. Two trials with the same decision are not shown: one in a security role, and one whose event we could not match to public reporting. Across all 24 trials so far, the task was completed in full only 2 times. This is a small internal sample, not a measure of forecast accuracy.

Task (fictional analyst)Real eventAI assistant with web searchSame assistant with DeltaResult
Imaging-order desk at a satellite-data reseller (2026-11-12)Toccoa River flood in McCaysville, Georgia and Copperhill, Tennessee (5–6 Nov 2026)Could not name the next chance to image the towns.Combined predicted satellite passes with the cloud forecast and named the next opportunity, with a deadline.Changed the decision: Neither version confirmed whether the image was actually taken.
Grain trader briefing a client on Black Sea loadings (2026-11-12)Strikes around the Odesa ports, including Chornomorsk’s central market (11 Nov 2026), UkraineFound a sharp fall in port throughput and advised getting quotes to switch December loadings.Found no official port figures and advised not switching yet, with conditions for switching.AI alone did better: Delta added one strike but had nothing on port throughput; the assistant’s advice was better grounded.
Claims-intake systems at a property insurer (2026-11-12)Koyoshi River flood, Yurihonjo, Akita, Japan (21 Oct 2026)Found where the river overtopped and narrowed the affected policies to one.Matched policies by municipality from Delta’s location and flagged three candidates.AI alone did better: Delta’s place was coarser than the reporting the assistant found.
LNG procurement analyst at a utility (2026-11-13)Attack on the Qatari LNG carrier Al Mayeda in the Gulf (30 Oct 2026)Did not call it a disruption; checked the two Middle East cargoes first with the seller and the operator.Same decision, adding tanker attacks reported in the strait that week and the recent loading level at Ras Laffan.Same decision: Delta added facts the assistant had not found, but the cargo decisions did not change.
Claims triage lead at a US property insurer (2026-11-12)Toccoa River flood in McCaysville, Georgia and Copperhill, Tennessee (5–6 Nov 2026)Same core decisions; its access and water-service status was newer and correct.Same core decisions, adding that a Sentinel-2 image was taken on 9 Nov and when the next one was likely.Same decision: Delta missed follow-up reports filed under a separate event, and one of its assessments was applied to the wrong question.
Satellite-image desk at a TV broadcaster (2026-11-12)Steel-plant explosion in Aliağa, Türkiye (9 Nov 2026) and Hurricane Isaias on the US Gulf Coast (9 Oct 2026)Reached the same free before-and-after image sets from a public catalogue.Same decision, faster, with a check of which scenes cover the event point.Same decision: A wrong next-image date came from Delta’s values.
Situation unit at a provincial emergency operations centre (2026-11-12)Successive coastal storms in British Columbia, Canada (November 2026)Same decisions; it also checked the official evacuation preparations.Same decisions, adding that no free post-storm image existed yet and when the next pass was.Same decision: The official preparations came from reading the agencies’ pages, not from Delta.
API integration engineer designing event alerts (2026-11-12)M5.7 earthquake in western Shizuoka, Japan (8 Nov 2026), replayed against eight fictional sitesSame design decisions.Same design decisions, with alert volumes, duplicates and cost estimates for the design note.Same decision: For one site, Delta’s verdict disagreed with the assistant’s.

What the table shows:

  • Delta changed the decision once, by naming the next chance to image a flooded town when the assistant on its own could not.
  • Most of the time the decision was the same. Delta's contribution was speed, coverage checks, or a fact the assistant had missed, not a different decision.
  • Twice the assistant did better on its own. In both cases it found more specific reporting (port throughput, the exact place a river overtopped) than Delta's record held.

Why is this not a measure of forecast accuracy?

Because nothing here was a forecast that was scored later. The trials are a handful of tasks, judged by us, on events that had already happened. A proper test would fix the questions in advance, record each forecast before the outcome is known, and compare the same model on its own, with web search, and with Delta, using a score such as Brier over enough questions to see a difference.

Delta's daily report gives an outlook in standard likelihood language, with the signs that would change it. Those outlooks have not yet been checked against what happened, so we do not publish an accuracy figure for them. We will publish one, with the conditions where Delta did not help, only once we have it.

How do you give an AI assistant the event record?

Connect the Off-Nadir Delta MCP server to Claude, ChatGPT or another MCP client, or call the REST API or Python SDK from your own code. The MCP docs cover each client, and Track World Events With ChatGPT or Claude explains what changes once it is connected.

Sources

The trials are internal records. The events in them are real; their places and dates were checked against public reporting on 13 November 2026. Off-Nadir Delta is an independent project and is not affiliated with the authors of these studies, OpenAI or Anthropic.

Kazushi Motomura
Kazushi Motomura

Remote sensing specialist with 10+ years in satellite data processing and AI. Founder of Off-Nadir Lab. Ph.D. in Informatics (Yokohama National University, 2026). Master's in Earth System Science and Technology (Kyushu University). Co-author, Remote Sensing Encyclopedia. More about the author →

From headline to satellite evidence

One connected intelligence workflow across four surfaces — free to start, no GIS software or remote-sensing background required.