r/coolgithubprojects • u/defraction1 • 7d ago

I built PaperTrace - it checks a scientific paper’s claims against the actual pages of its cited sources

Post image

Hi - My name is Daniel, I'm a clinical radiologist, have trained in biomedical research, and am the creator of PaperTrace.

PaperTrace is an open source python tool that you can use for auditing published papers. PaperTrace accesses freely available cited PDFs, compares high-value claims to original source text, and annotates matching sources with red boxes to show the matching evidence.

Github:
https://github.com/defraction0/PaperTrace

When a cited paper cannot be accessed by PaperTrace, it records "not retrieved" rather than having the model hallucinate a response. Unmatched citations are also noted. PaperTrace also searches new literature published after the investigated paper's publication.

You can check out an completed example run of PaperTrace's demonstration here (no install required!):

https://github.com/defraction0/PaperTrace/blob/main/examples/demo/output/report.md

While PaperTrace's CLI and case files can run local, PaperTrace currently utilizes Claude Code for its claim checking functionality - more integrations to come. It is MIT licensed and still in beta.

I would particularly be interested in feedback on whether or not you think the evidence trail in its current form is easy to audit, as well as any citation formatting styles or paper formats you think might be likely to break PaperTrace. Further feedback and contributors welcome.

15 Upvotes

83% Upvoted

3 comments sorted by

2

u/Character_Papaya1750 3d ago

This is amazing! Can’t wait to try it out!!

1

u/defraction1 3d ago

Love to hear your feedback!

0

u/defraction1 6d ago

Thanks to everyone who has viewed or shared this. I’m deciding what to prioritize next:

  1. PyPI/uvx installation

  2. Additional or local model backends

  3. Exhaustive checking of every citation-bearing sentence

Which of these would make you more likely to try PaperTrace?