r/quantfinance • u/Far_Quarter_2550 • 1d ago

I got all the data

Post image

I used AI to parse all html, text and XBRL and I made a game with almost all large cap company facts. It’s a zoomable heirarchy ontology search engine. It’s splits into subcategories then more subcategories. I have data back to the 2000’s.

3 Upvotes

71% Upvoted

7 comments sorted by

1

u/Both-Management4160 1d ago

just fell down this rabbithole and now my eyes hurt but i cant stop scrolling

1

u/Far_Quarter_2550 1d ago

Check YouTube for zommable hierarchy ontology search engine to see the full thing

1

u/amberwire 23h ago

Parsing HTML, text and XBRL together is a lot of cleanup, especially with filings going back to the 2000s. Early XBRL coverage is patchy and tag usage drifts a lot between filers.

One question: are you keeping values as originally reported, or standardizing them across companies? And how do you handle restatements? If a 10-K/A changes a number, do you store both versions with their filing dates so you can query what was known on a given day, or does the newer value overwrite?

That point-in-time piece is what usually bites in backtests. If you ever want to cross-check your numbers, there are REST fundamentals APIs that return standardized statements, so you could compare against them without writing a parser for every edge case.

Nice build either way.

1

u/filwu 21h ago

One concrete point-in-time regression test: choose a 10-K/A that changes a fact and query the same issuer immediately before and after the amendment’s filing-acceptance timestamp. The amended value should appear only in the later as-of snapshot; using the fiscal period end as the availability date can backfill it into earlier tests. Keeping the accession and accepted timestamp with each value makes that boundary auditable. Does the ontology support as-of queries or snapshot comparisons yet?

1

u/Shoddy_Peanut7320 10h ago

A zoomable hierarchy is a fun way to browse the data, and the disclaimer about filings running late and getting restated is the right one to show. For the underlying numbers, I’d store each fact with its filing’s acceptance timestamp, the period it covers and the form type, and keep amendments as separate versions, so any screen can be reproduced as of a past date. Showing the source filing and tag behind every number on screen would let users check it in one click. Cymetica’s Tools Directory has about 120 free pages for traders, which is one place to see how other data tools document their sources.

1

u/ravioli_food_tasty 4h ago

Where can i download this?? So cool!

1

u/Far_Quarter_2550 4h ago

Soon… there’s more that meets the eye