FinDendro Finds Date Prompts Do Not Prevent AI Backtest Leakage
Reported by Tech Times1 source ↓
A FinDendro benchmark reports that date prompts sharply reduce GPT-5’s claims of prior knowledge but barely change its accuracy. Its authors say common safeguards cannot reliably certify financial backtests as leak-free.
Key points
- GPT-5’s accuracy changed little when prompted to answer as of a past date
- FinDendro tested 2,226 dated U.S. facts from 1999 to 2026
- The authors say self-reports, date instructions and stated cutoffs cannot reliably rule out leakage
- The benchmark was posted to arXiv on Nov. 4
Details
Tech Times reports that FinDendro tested several methods used to guard against data leakage in AI financial backtests, including restricting tests to dates after a vendor-stated cutoff, instructing a model to answer as of an earlier date, and asking it to report its own cutoff. The 15-author team concluded that none reliably separates what a model learned from what it can predict.
For GPT-5, the share of answers claiming prior knowledge fell from 39% to 1% under a past-date instruction, while accuracy shifted from 76% to 75%. The article also reports that GPT-5 answered 61% of facts correctly even when it said it did not know them.
FinDendro covers 2,226 dated facts from 1999 to 2026, including economic data, stock prices and Federal Reserve rate outcomes, and evaluates six general-purpose models alongside three point-in-time models. The authors acknowledge that their multiple-choice prompts may not reflect how models respond without prompting; the benchmark measures knowledge claims, not data handling or legal exposure.
Written from coverage as of