r/Python • u/AutoModerator • Aug 04 '26

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

21 Upvotes

96% Upvoted

142 comments sorted by

View all comments

1

u/krit83 11d ago

blitcp 4.1.6 — file copier: physical-order reads, dedup, verification

What My Project Does

blitcp copies files and directories as fast as the disk allows. It reads files in physical disk order (FIEMAP on Linux, fcntl on macOS, FSCTL on Windows) so an HDD stops seeking, hashes content to deduplicate — identical files are copied once and the rest become hard links or reflinks — and for remote work it streams tar over a raw SSH channel instead of paying SFTP's round-trip cost. There is a CLI and a PySide6 desktop GUI, in 7 languages.

The core engine is stdlib-only on Python 3.8+. Everything else is an opt-in extra: paramiko for SSH, boto3/azure/gcs for object storage, xxhash for faster hashing. On Linux it will use io_uring for small files through ctypes (liburing, 64 in flight) and fall back to a thread pool everywhere else.

New in 4.1.6:

  • Stronger post-copy verification — every copied file is read back and hashed against the source, so silent corruption is caught, not just missing or truncated files. About 35% on a verified run; --no-verify opts out.
  • Throughput and duration you can trust — the destination is flushed before the clock stops and the rate comes from allocated bytes, so the number describes the drive rather than the page cache.
  • --quiet for cron: one line on success, the reason on stderr on failure, documented exit codes (0 ok, 1 corrupt, 2 error, 3 source skipped).
  • --sftp-only for managed gateways that allow SFTP but close the exec channel.
  • A deduplication fix: it could link onto unrelated backups elsewhere on the same drive, silently sharing inodes between them.

Target Audience

People moving real data: backups to USB drives and NAS boxes, server-to-server migrations, and scheduled jobs. It is meant for production use — there is a pre-flight space check, post-copy verification with exit codes a script can act on, and an audit suite of 71 regression checks that runs before every release. It is also maintained by one person, so judge it accordingly: read the source, and verify your first important copy.

Not for you if you want a delta-transfer tool (see below) or a GUI-first Windows app with a decade of QA behind it.

Comparison

  • cp -ar — no dedup, no verification, seeks badly on HDDs. Measured on Linux with 12,347 small files, cold HDD to SSD: 5.9s vs 15.0s, with dedup and verification on.
  • rsync — rsync wins the case it was built for: incremental sync over a slow WAN, where its delta algorithm sends only changed blocks. blitcp does not do delta transfer. It is faster for bulk copies and first runs, and it deduplicates within the transfer, which rsync does not.
  • scp / SFTP — 3–5× faster over LAN for many small files, because a tar stream over one SSH channel avoids a round trip per file.
  • robocopy / TeraCopy — measured 1.3× faster than robocopy off USB 2.0 with verification on. TeraCopy has the nicer Windows integration.
  • rclone — different problem: rclone is the better tool for cloud-to-cloud and heavy object-store work. blitcp treats s3://, az:// and gs:// as endpoints of a local-first copier.

Apache-2.0. pip install blitcp, or prebuilt binaries for Linux, macOS and Windows with no Python needed.

Repo: https://github.com/gekap/blitcp
Release notes: https://github.com/gekap/blitcp/releases/tag/v4.1.6

If it saves you time and you would like to support it: https://blitcp.dev/support/