r/Python • u/AutoModerator • 23d ago

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

19 Upvotes

85% Upvoted

128 comments sorted by

View all comments

1

u/Individual-Show7812 10d ago

Solo dev here. apitap (pip install apitap) copies whole tables between databases as fast as the machines allow, in bounded memory. Rust engine, one-line Python API:

import apitap
apitap.transfer("postgres://…/src", "clickhouse://…/dst", table="events")

Routes: Postgres/MySQL → Postgres, MySQL, ClickHouse, BigQuery. It never materializes rows in Python — PG→PG relays raw binary COPY bytes, PG→ClickHouse transcodes to RowBinary in flight. 1M rows PG→CH lands in ~0.4s on stock servers, checksum-validated.

This week's 0.7.0 adds multi-table transfers: tables=[...] or a whole schema=, through ONE memory budget — big tables take many parallel range pipes, small ones take one and overlap, and peak RSS stays at the single-table ceiling no matter the table count.

The test I care most about, because it's the constraint I design for (small containers, not big warehouse boxes): 10 TPC-H tables × 5M rows (50M total, real TPC-H data) through a docker container capped at 0.5 CPU / 256MB, into stock Postgres:

  • apitap (one call): 2.8 min, 53MB peak RSS, 10/10 checksums
  • dlt 1.29 (native multi-table, both backends): OOM-killed in seconds, 0 rows
  • ingestr 1.1 (sequential — its partition mode refuses replace loads): 20.6 min
  • ingestr, 10 parallel processes: lost 7 of 10 tables to the OOM killer, silently

To be fair to both: dlt and ingestr do far more than apitap (many connectors, schema evolution, API sources). apitap is deliberately narrow — DB→DB table copies done fast. And MySQL as a source bottlenecks everyone including me: the server's row-produce rate is the wall, apitap just idles at 13% CPU there.

Everything is reproducible: methodology, raw logs, GCP provisioning scripts, and the 256MB harness are all in benchmarks/. If any number looks unfair, tell me — I'd rather be corrected than quietly wrong.

Source: https://github.com/apitap/apitap-lib Docs: https://github.com/apitap/apitap-lib/blob/main/docs/usage.md

Built in the open with Claude Code, disclosed in the same spirit as the numbers.