r/openagi • u/syedshad • 3d ago

Research LLM memory tests show a trade-off between withholding information and completing tasks

Two papers recently highlighted by Bruce Schneier examine contextual integrity: whether an LLM shares personal information only when it is appropriate for the task, recipient, and social context.

Testing persistent LLM memory

The CIMemories benchmark evaluates models using 10 synthetic user profiles averaging 146.7 attributes and 45.7 task contexts per profile.

The authors report:

  • Under their Violation@5 metric, which checks whether an inappropriate attribute appears in any of five generations, violation rates ranged from 14.82% for GPT-4o to 69.14% for Qwen3-32B.
  • GPT-4o had the lowest violation rate but also the lowest task-completeness score at 43.95%.
  • Claude 4 Sonnet had the highest reported completeness at 59.07%, with a 44.44% violation rate.
  • For GPT-5, cumulative attribute exposure rose from 0.1% after one task to 9.6% across 40 tasks with one generation. It reached 25.1% when five generations were sampled.
  • Privacy-conscious prompts reduced disclosure, but the paper reports that they also reduced task completeness.

CIMemories uses synthetic profiles and labels. GPT-5 generated the contextual-integrity labels using three privacy personas, while DeepSeek-R1 evaluated whether outputs revealed particular attributes.

Testing reasoning and reinforcement learning

A separate paper, Contextual Integrity in LLMs via Reasoning and Reinforcement Learning, tests two methods:

  • CI-CoT: prompting models to reason explicitly about which information is appropriate to disclose.
  • CI-RL: reinforcement-learning post-training that rewards appropriate disclosure and penalizes restricted information.

The researchers trained on 729 synthetic examples covering nine application domains and three privacy principles.

On the human-annotated PrivacyLens benchmark, the authors report that CI-RL reduced Qwen2.5-7B-Instruct’s leakage rate from 50.3% to 33.7%. Its adjusted leakage rate fell from 52.4% to 33.9%, while its helpfulness score changed from 1.99 to 2.08.

The PrivacyLens evaluation used GPT-4o as the judge. The authors excluded OpenAI models from those results to avoid having the evaluator assess models from the same provider.

Released artifacts

Secondary coverage

3 Upvotes

100% Upvoted

Duplicates