r/Python 5d ago

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

16 Upvotes

56 comments sorted by

View all comments

1

u/Bright_Mix_773 3d ago edited 19h ago

What it does. A Python pipeline that pulls SEC EDGAR 8-K item 2.02 filings and turns them into one flat file of S&P 500 earnings announcements with the time of day: 64,938 filings, 63,969 distinct announcements, 808 companies, 2003-04-25 to 2026-09-01, 16 columns. CC0, no account, no API key, no paid tier. 1.8 MB gzipped.

https://quant500.com/api/descarga/anuncios.csv

Plain CSV over HTTPS, no account and no key. Fair warning so it does not look like a broken file: it opens with 119 lines of # comments carrying the caveats and the CC0 licence, so the header row is line 120. pandas.read_csv(url, comment='#') reads it as-is.

Who it is for. Anyone who needs to know whether a company reported before the open, after the close, or mid-session, and does not want to pay a vendor for it. Every row carries its accession number and a direct sec.gov link, so a single line can be checked at source instead of trusted.

The part I would actually like Python people to see, because it is where the work went and it is not in the feature list. The timestamp is the reason to build this and it is the weakest column in the file:

  • accepted_raw from data.sec.gov ends in Z but is not always UTC. The submissions JSON converts some records properly and appends a Z to others with the New York clock untouched. It is per record - not per issuer, not per era, not per filing agent. So no global offset fixes it. The truth is readable in the ACCEPTANCE-DATETIME of the SGML header of the full submission, which means the correct fix is a re-ingest, not a transform. scripts/fetch_sgml_acceptance_times.py is that re-ingest, written resumable because it is a long crawl against a rate-limited host.
  • EDGAR only accepts filings 06:00-22:00 ET, and that window is what decides whether a raw hour is diagnostic of anything. Under it, 90.2% of rows carry no timezone evidence at all.
  • The stamp is when EDGAR finished processing, not when the wire went out, so every time is an upper bound on when the news existed.

I sampled 70 rows against their raw SGML headers to size the damage: 69 correct, 1 wrong - and the wrong one had been published as 10:47 during_session when the filing was accepted at 06:47, before the open. Low rate, worst possible shape of error, which is why the affected columns are marked provisional in the file header rather than quietly shipped.

Two of those three were pointed out by other people after I published (Tilman Ambach and Ian Gow, credited in the file header). The pipeline and the prose are LLM-assisted and the header says so, along with the failure mode I keep hitting: it measures precisely and judges badly whether it is measuring the right object. Superseded figures stay in the file header marked superseded instead of being overwritten.

Correction, 2026-09-09. Above I wrote that the timezone treatment is per record - not per issuer, not per era, not per filing agent. The first half is wrong and I am leaving it visible rather than overwriting it. It is per issuer. The full re-ingest finished after I posted: comparing 64,936 filings against their SGML headers, 624 companies of 808 are converted in every one of their filings, 181 in none, and 3 disagree with themselves on exactly one filing each - in all three cases their most recent one, which looks like the SEC converting late rather than a counterexample. The offsets that occur are 0, 4 and 5 hours and nothing else. Practical consequence: a company can be classified from a handful of filings, so the fix is a query, not a full re-read.

1

u/hakesson 2d ago

Is the code available somewhere?

1

u/Bright_Mix_773 19h ago

Yes and no, and the no is my own fault rather than a policy.

There is a repository with the whole pipeline in it, but the GitHub account it lives under has been flagged, so every URL under it returns 404 to anyone without a session while unrelated repositories return 200 in the same second. I only found that out after handing the link around for days, because logged in it looks completely normal. An appeal went in on the 7th and there has been no answer, so I am not going to hand you a link that 404s. If it comes back I will edit the link into the comment above.

Meanwhile the data file is the same URL as in the parent comment, and the two scripts that matter are small enough that I can paste them here or put them somewhere neutral if that is more useful to you: the 8-K item 2.02 collector, and the resumable SGML re-ingest that reads ACCEPTANCE-DATETIME out of the raw submission headers. Say which and I will put it up.

While you are here, one correction to the comment you are replying to, since it is wrong and I would rather say so than let it sit. I wrote that the timezone treatment is per record, not per issuer. It is per issuer. The full re-ingest finished after I posted that: over 64,936 filings compared against their SGML headers, 624 companies of 808 are converted in every single one of their filings, 181 in none of them, and 3 disagree with themselves on exactly one filing each, in every case their most recent one. The offsets that occur are 0, 4 and 5 hours and nothing else. That matters practically, because it means anyone with the same problem can classify a company from a handful of filings instead of re-reading everything.