Your AI's Confidence Score Won't Catch Products You Don't Carry
Reported by Forbes1 source ↓
A study of language models found that their stated confidence did little to filter out recommendations for products missing from a retailer’s catalog. The tests also found that out-of-catalog rates varied widely by catalog type.
Key points
- Stated-confidence filtering reduced out-of-catalog suggestions by a median of 0.02 percentage points across 48 test settings
- Out-of-catalog rates ranged from 0.6% to 61% across the tested models and catalogs
- All four models had out-of-catalog rates below 3% for movies and roughly half or more for toys
- The study tested recommendations generated from user histories without access to the relevant catalog
- The author says the results do not cover systems that choose only from catalog items retrieved in advance
Details
Forbes reports that Srijith Ravikumar, an Amazon principal engineer, tested four large language models on public datasets for movies, local businesses and retail toys. Each model generated 10 recommendations from a user’s history without access to the catalog, and suggestions were checked against the catalog they were meant to come from. The study counted any suggestion absent from that catalog as a miss.
Across the 48 test settings, filtering recommendations by the models’ stated confidence reduced the out-of-catalog share by a median of 0.02 percentage points; in 18 settings, the filtered list was slightly worse. The best reduction was under two percentage points. Out-of-catalog rates ranged from 0.6% to 61% across models and catalogs, with every model below 3% on movies and around half or more on toys.
The models’ stated confidence changed little between movies and toys even as the out-of-catalog rate rose twentyfold or more. Across 12 model-and-catalog combinations, the models were underconfident in seven and overconfident in five; all four were underconfident on movies and overconfident on toys. The author says the tests measured only verbal confidence, not other signals such as token probabilities or agreement across repeated samples.
In a separate test on the business catalog, prompting two models to recommend from a fixed catalog they could not see lowered out-of-catalog rates from 12.9% to 1.5% for one model and from 14.4% to 8.9% for the other. The test changed both the instruction and the confidence question, and covered two models in one catalog. The author says the study did not measure the cost of incorrect recommendations and does not address systems that select only among candidates retrieved from a catalog.
Written from coverage as of
Images
- Forbes
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh