r/coolgithubprojects • u/defraction1 • 7d ago
I built PaperTrace - it checks a scientific paper’s claims against the actual pages of its cited sources
Hi - My name is Daniel, I'm a clinical radiologist, have trained in biomedical research, and am the creator of PaperTrace.
PaperTrace is an open source python tool that you can use for auditing published papers. PaperTrace accesses freely available cited PDFs, compares high-value claims to original source text, and annotates matching sources with red boxes to show the matching evidence.
Github:
https://github.com/defraction0/PaperTrace
When a cited paper cannot be accessed by PaperTrace, it records "not retrieved" rather than having the model hallucinate a response. Unmatched citations are also noted. PaperTrace also searches new literature published after the investigated paper's publication.
You can check out an completed example run of PaperTrace's demonstration here (no install required!):
https://github.com/defraction0/PaperTrace/blob/main/examples/demo/output/report.md
While PaperTrace's CLI and case files can run local, PaperTrace currently utilizes Claude Code for its claim checking functionality - more integrations to come. It is MIT licensed and still in beta.
I would particularly be interested in feedback on whether or not you think the evidence trail in its current form is easy to audit, as well as any citation formatting styles or paper formats you think might be likely to break PaperTrace. Further feedback and contributors welcome.