Why vector DB metadata filters fail on agent memory write paths (0% raw leak vs 67% leak) reproducible offline test suite
Reported by r/LocalLLaMA1 source ↓
A community post describes an offline test harness in which agent-written summaries exposed restricted source data despite direct vector searches filtering it out. The author reports leakage in 2 of 3 agent synthesis tasks; the results come from the post’s own test setup.
Key points
- The author reports that 2 of 3 agent synthesis tasks exposed sensitive data
- The test exposed 6 of 18 restricted canaries; a public control passed
- Direct RAG testing retrieved 18 public chunks and exposed no restricted chunks
- The author says strict access intersection would block useful multi-document summaries
Details
A r/LocalLLaMA post describes a risk in systems where background agents can read across folders or teams, create summaries, and save them to a shared vector database. The author says source-document access controls may not carry over to the derived summary, leaving it without group or classification metadata.
In the author’s Chroma and LlamaIndex test setup, direct RAG retrieval exposed no restricted chunks among 18 public chunks. Agent memory exposed 6 of 18 restricted canaries in two of three synthesis tasks, while the public control passed; these are results from the author’s reproduction, not an independently reported evaluation.
The author says restricting summaries to people authorized for every source document would prevent those exposures but make multi-document summaries unavailable to many users. The post also says vector databases cannot evaluate dynamic permission relationships at query time, even when source document IDs are stored in metadata.
Written from coverage as of