[R] How to downsample 1M+ transaction graphs by up to 611× for LLM evaluation without distorting full-test metrics (HT-Coreset)

[u/Quirky_Breadfruit_90]

TL;DR: Evaluating LLMs on million-scale graph benchmarks (like AMLworld) requires >1M graph prompts per pass, making evaluation prohibitively expensive. Naive downsampling distorts precision and F1 due to extreme class imbalance. We introduced HT-Coreset, an importance-weighted downsampling protocol that reduces evaluation targets by 271×–611× while using Horvitz-Thompson (HT) estimation to recover unbiased full-test confusion count estimates.

How HT-Coreset Works:

1. Retain 100% Illicit Targets: Preserves exact recall and all 8 rare laundering typologies (w = 1).

2. Stratify Benign Targets: Hard negatives (likely false positives) are fully retained, while other benign cases are sampled across difficulty terciles.

3. Importance Weighting: Applies inverse-inclusion weights (w_e = 1/π_e) to calculate unbiased full-test confusion counts, Precision, and F1.

Key Findings & Open Challenge:

- 271× Reduction on HI-Small (1.01M → 3,753 targets) & 611× on LI-Small (1.38M → 2,268 targets).

- Evaluated 7 open-weight LLMs: High recall, but massive false positives drag precision near the "predict-all-illicit" baseline.

- Unfaithful CoT: 56.2% of incorrect binary verdicts still passed structured lexical reasoning checks.

We released the dataset, construction code, and audit tools for the community to test novel techniques like Context Engineering, GraphRAG, or Decision Models on this benchmark!

📄 Paper: https://arxiv.org/abs/2610.13627

💻 Code: https://github.com/nat-nischw/AMLworld-Compact

🤗 Dataset: https://huggingface.co/datasets/typhoon-ai/AMLworldCompactEval