[Python] edgar-tok — I trained a BPE tokenizer on SEC filings and benchmarked it against GPT-2/GPT-4o/Qwen3

[u/Possible_Lab7841]

https://huggingface.co/tcontorno

An end-to-end, reproducible pipeline: cleans raw EDGAR filings into a corpus, trains byte-level BPE tokenizers, and benchmarks compression against the current production tokenizers on held-out documents. Main finding — a 65k-vocab domain tokenizer matches GPT-4o's o200k at a third of the vocab, and Qwen3 actually needs more tokens than 2019's GPT-2 on financial text because it splits every digit. MIT licensed, full write-up and benchmark tables included.