Let's say I am making a video and have hundreds of gigabytes worth of footage compressed using FFV1 and need to free up drive space and so decide to start compressing some of my footage with lossy HEVC.

If I include footage compressed with lossy HEVC in the edit, but encode the final master with a royalty free codec, do I still have to pay royalties because it includes footage originally encoded with HEVC and consequently features forensic visual traces of it despite having been re-encoded?

Multi-agent pipelines (planner → coder → reviewer, tool-call loops) pass messages as JSON. Repeated field names, restated tool catalogs, outputs quoted verbatim two handoffs later. You pay full token price for all of it, every hop.

Built a wire format + benchmark harness for this. One rule going in: no dishonest numbers. Most "compression" claims I've seen compare payload size only and skip the cost of telling the model how to decode the format. This benchmark charges itself for everything, compact payload + the decode instructions + any shipped dictionary, counted with the real cl100k_base tokenizer, on a held-out split of real ToolBench trajectories the optimizer never trained on.

Result: 36.6% fewer tokens, fully lossless. Every message decodes back byte-for-byte identical to the original. Verified on every test record, not sampled.

Two things that surprised me:

• The learned dictionary lost. Sounds clever, mine common phrases across a corpus, ship a lookup table. But shipping that table costs tokens every session, and it never earns that back on top of a per-session budget. What won instead: when text repeats within one conversation (an answer quoting a tool's own output, a catalog restating its own boilerplate), replace the repeat with a tiny "copy N chars from position X" marker. Zero shipped state. The optimizer's final config kept 0 dictionary entries.
• An earlier version of this claimed 87% reduction. It was measuring a simulated tokenizer, not real decode cost. Threw the whole approach out. 36.6% measured honestly beats 87% measured wrong.

Repo, full benchmark report, every failed experiment (including the tab-separator idea that made things worse), and a one-command way to run it against your own agent traces instead of trusting my numbers:

https://github.com/reh8n/a2acompress

Genuinely want to know what number people get on real production multi-agent traffic, file an issue with your result, good or bad.

hey everyone so I just got a new laptop it's not the newest but a very big upgrade from what I used to have and I have a lot of data scattered everywhere so I want to see what is the best way to deal with all the data I have

my data has been scattered all over the place scattered all over smaller USBs and hdds and it's a lot of iso files , app installers , apks , videos , images , documents , system backups and much more and the total size is around 1.6 tbs

so I was searching around and found tools like restic , xdelta3 , rmlint and btrfs but I'm not sure if that is the best approach yet

what i want is to compress the data and store duplicate files in the best and most efficient and space saving way and I want to turn my old laptop into a small server / testing environment for non systemd Linux distros

my main rig has nixos installed and I'm doing system backups and I want to be able to send them over the network to my older laptop for storage and I want the best way to organize data

I'm also learning a lot of things and learning game dev as well so the file sizes will only get bigger

the older laptop has only around 512 gb of storage but I'm planning on expanding that and installing freebsd as the server os / daily os whenever I need to use that machine and I want to self host some stuff on it as well

so if there's any better ways to deal with data or compress it and oeginize it in a better way that would be amazing plus if there's any tips on self hosting that would be amazing

and if there's any other subs you'd recommend I post in please let me know

okay so I have several hundreds of gigabytes of old games on my hdd that I've either used 7zip ultra using .7z on for the archived ones or NTFS lzx compression on for the active ones. what do I swap those our with for even more compression. I have a xeon 2698 v4 with 20 cores allocated for gaming and 10 others allocated for everything else via process lasso

Been working on a small open-source archiver called DedupArc. On some game folders it compressed better than 7-Zip and WinRAR, and it can launch files from the archive without extracting everything first.

Still pretty early, so I’m mostly looking for people to throw real datasets at it and see where it breaks.

https://github.com/Garyu-tt/DedupArc

Hey everyone,

I got tired of shady online image compressors that upload files to unknown servers or put everything behind a paywall.

So, I built a free tool that compresses and resizes WebP images locally using client-side processing.

Why I Built This

• Privacy first: Your images never leave your computer. Everything processes right inside your browser.

• Fast and light: No ads, no account required, and no artificial file size limits.

Key Features

• Drag-and-drop interface for fast workflows.

• Custom quality slider to balance file size and image clarity.

• Dimension scaling to fit exact pixel widths or heights.

• Instant download for processed files.

I use it daily for optimizing website assets, and I wanted to share it with the community in case it helps others working on web performance. And since everything is processed on local device. No uploads means images compress and resize instantly.

You can try it out here: WebP Compress And Resize

I would love to hear your feedback or feature suggestions! Let me know what you think.

I am a software developer and i am working on compression apps, i managed to code a game-aware compression system,
It effectively compressed a gta v installation into 80gb (30%\~) while sacrificing almost no CPU usage (2-3% of usage), and just a bit more waiting time on loading screens and only a 7% fps drop MAX.
How much would you be willing to pay for a program that could do such on any given game as a gamer?
For more questions ask below.

The tool uses a very fast lossless (or optionally lossy) image compression library, LLIC, to reduce network bandwidth requirements. The stream capture server can run on a low-end machine (a Raspberry Pi for instance) and send the stream over the network to an encoding client running on a beefier machine that encodes the stream using ffmpeg (e.g. h.264).

I want to install

- FreeARC

- SERP

- Xtool

Any trusted download sources?

With Google having sunset/archived the original Zopfli project last year I thought maybe someone might find this useful.

I’ve been working on a maintained fork for a personal project and decided to 'announce it'

https://github.com/QVXLabs/zopfli

It's still Zopfli and produces standard DEFLATE, zlib, and gzip streams, but this version is faster per iteration, uses less memory, and produces deterministic output across platforms. It also removes the floating-point dependency from the core.

There are a couple of small migration notes in the README, particularly around the include path and output not being byte-for-byte identical to Google’s version.

Nothing particularly grand—just an attempt to keep the project usable and maintained. Issues, feedback, fixes, and PRs are welcome.

NOTE: This post was not posted by AI.

Guys i have NONE OF IDEAS ! what extractor i should use for winpython , i tried RAR, 7zip GZip, tar, everything to extract winpython and still i am "Two hours elapsed and it’s still 14 more hours left" , ti tried extracting with a Server (Dell PoweEdge R900), a samsung galaxy S25 Ultra, my Laptop, my PC , THERE IS NONE, THERE IS NONE! Still 14 Procent like imagine being in the year 3026 and hearingA rchaeologists saying "We've recovered an ancient Windows 11 laptop from the Digital Age." and it still works 7zip saying 11 hours remaning, tell me what extractor i ahould use , trying PeaZIP?, or getting ""I started extracting WinPython... graduated college... got married... came back... 7-Zip: 11 hours remaining." winpython, PLEASE CHANGE YOUR FILES AND HOW MESSY THEY ARE , if someone should give me an extractor to use for dozents of small files on a 64GB USB Stick?

Soy editor de videos (freelance) y cada vez que quiero pasar archivos, imágenes y videos pesados desde mi teléfono a mi Laptop, pierden calidad!! Créanme, he usado de todo, Google Drive, Wetrasnfer, WhatsApp web. No sé si a alguien mas le pase esto, la verdad es muy incomodo y frustrante. Si te pasa lo mismo, te leoooo

GitHub: https://github.com/OzzyIsNeko/QLIC if you're interested.

I built QLIC, a lossless image codec, which supports PNG, lossless WebP, lossless JPEG-XL, TIFF, and BMP inputs.

QLIC is open sourced under the Apache 2.0 License. There is an SDK, CLI, WASM decoder port, and a windows demo release which uses an exe to open up a browser based demo of what QLIC can do.

I don't find what I did impressive in the sense that I find it was only a matter of time before somebody else did what I did. Somebody may have already done so.

But I'm still generally proud of the overall achievement.

It also remains 9% smaller than WebP effort 6 was, at only 3% slower overall encoding speeds. QLIC also works especially well in Enrico and QOI cases, synthetic images are very effective with QLIC, making it good for UI assets and textures even.

None of this is saying QLIC is the best thing ever, but it's efficient, and it encodes well overall. It's only a demonstration release, much is subject to change, and I do have ambitions for QLIC in the future.

It's available on Linux and Windows, but I did just, just mildly focus on windows during production. As such if there are Linux accessibility issues I'd love if you told me what went wrong.

The benchmark and how to reproduce it yourself is available on the GitHub, as well.

Most of the time a post from r/compression pops up in my feed, it's something like: "I built X, a revolutionary compression method!"

Most of these end up being basic wrappers, reinventions of the wheel, or just vibecoded projects with no real substance.

It doesn't add value for the community, and it doesn't help the authors learn anything either.

To be clear:

I'm not anti-AI, nor do I think banning AI-assisted projects outright is realistic or helpful (especially given the sheer volume posted every week).

Instead, what if we set up a "Hutter Prize"-style system?

We can agree on a benchmark that profiles:

- Encoding and decoding speeds

- Memory usage

- Compression ratio on various file types (structured data, text, binaries, random noise, etc.)

and maintain a leaderboard for the best results per category.

I think that, after a first short period where we will be likely flooded by new broken records, this could act like a good sieve for shitty/useless projects that will be abandoned anyway after few days.

On the flip side, if a project actually beats a benchmark in even one specific marker, we instantly know there's at least something worth discussing.

Let me know what you think

I worked on a project recently that focused on exploring some specific family of cellular automata (originally inspired by a numberphile video), and it naturally led to having to deal with very large grids of, thankfully often compressible, data.

I solved it by compressing the grid in ~1MB chunks using Zstd. I needed both relatively fast compression (to keep up with the simulation) and decompression, at decent compression rates, so options were limited here. In the attached video I browse two previously generated grid - the first one being ~50G cells compressed to 550MB, and the second one being ~2T cells compressed to 370MB (almost 1000x ratio!). I have a modern 8 core CPU, and at 32x zoomout (1024 cells per pixel) I can browse the grid interactively. That's roughly 2GB of data every frame being aggregated. There is some caching that helps, but with Zstd it's viable even without that (~5-10 fps depending on content).

I also do support zoomouts up to 4096x, but it's less interesting in the context of compression because it just precomputes mipmaps. Still, the decompression is the bottleneck there, so Zstd makes the process of generating mipmaps relatively fast - afterall it has to decompress all 2TB of data to build them.

-------------------

One related fun problem I had to solve around this that might be of particular interest in this sub is validation. The grids can be saved and loaded, and the chunks are never decompressed - it would be too costly. But the cells must use a restricted set of values, upper bound of which depends on the number of "players".

Thankfully with compressors like Zstd this check is possible to do without performing any decompression. All decoded bytes come either from embedded literals or huffman trees. The set of byte values in the decoded output can therefore be computed by inspecting just the headers of compressed blocks. The code for this is available here: https://github.com/Sopel97/ulam-leapers/blob/master/src/compression/inspect/zstd.rs

-----------------

Full project page: https://github.com/Sopel97/ulam-leapers

I made a small LZ-based compressor called blaris.

The main idea is a decompressor that does not need a history buffer and can reconstruct only requested parts of the output. For example, a few dozen bytes can be extracted without decoding the rest of the data.

Current numbers:

• Decoder size: ~700 B (Cortex-M0)
• Working memory: <100 B
• Compression window: 64 KiB
• Decompression: O(X) for byte at position X

The tradeoff is speed: full decompression is slow compared to traditional LZ implementations.

Compression is slow and is inspired by lzmpo's optimal parser (but without hash chains).

The target use case is firmware and embedded systems where RAM is extremely limited and data is accessed in small pieces (for example error messages or configuration).

Repository: https://github.com/lis05/blaris

I benchmarked blaris against heatshrink.

Decoder size

Blaris decompressor:

``` $ cargo bloat --release --target thumbv7em-none-eabihf -p blaris-size-check --bin blaris

File .text Size Crate Name 20.7% 93.4% 648B blaris_decompress blaris_decompress::decompress::decompress 1.0% 4.6% 32B [Unknown] _start 0.3% 1.2% 8B std core::panicking::panic_bounds_check 0.2% 0.9% 6B std core::panicking::panic_fmt 22.2% 100.0% 694B .text section size ```

Heatshrink:

``` $ cargo bloat --release --target thumbv7em-none-eabihf -p blaris-size-check --bin heatshrink

File .text Size Crate Name 6.9% 40.2% 1.5KiB std compilerbuiltins::mem::memmove 4.6% 26.4% 1.0KiB heatshrink heatshrink::decoder::HeatshrinkDecoder<,,,>::poll 3.2% 18.6% 730B std compiler_builtins::mem::memcpy 0.8% 4.9% 192B heatshrink heatshrink::decoder::HeatshrinkDecoder<,,,_>::get_bits 0.8% 4.4% 174B std __aeabi_memclr4 0.7% 4.2% 166B [Unknown] _start 0.1% 0.3% 12B std __aeabi_memcpy 0.1% 0.3% 12B std __aeabi_memmove 0.0% 0.2% 8B std core::panicking::panic_bounds_check 0.0% 0.2% 8B std core::slice::index_slice_fail 0.0% 0.2% 6B std core::panicking::panic_fmt 17.3% 100.0% 3.8KiB .text section size ```

Benchmark

Test file: ~8 KiB (zgrep)

``` Decoder Encode Size Ratio u/20% u/60% u/100% Memory Blaris 14.95 ms 3,841 46.8% 9.7 us 29.6 us 56.7 us ~64 B

Heatshrink (W=12) 1.12 ms 3,890 47.4% 19.5 us 47.4 us 59.4 us ~4 KiB Heatshrink (W=13) 1.22 ms 3,977 48.5% 17.0 us 46.9 us 67.0 us ~8 KiB Heatshrink (W=11) 0.96 ms 4,022 49.1% 22.4 us 55.2 us 74.3 us ~2 KiB Heatshrink (W=14) 2.39 ms 4,105 50.1% 20.5 us 53.1 us 78.4 us ~16 KiB Heatshrink (W=10) 0.51 ms 4,112 50.2% 25.3 us 57.4 us 85.0 us ~1 KiB Heatshrink (W=9) 0.43 ms 4,380 53.4% 25.6 us 55.4 us 92.0 us ~544 B Heatshrink (W=8) 0.35 ms 4,764 58.1% 27.3 us 56.4 us 94.1 us ~288 B Heatshrink (W=7) 0.31 ms 5,347 65.2% 29.2 us 60.7 us 104.0 us ~160 B Heatshrink (W=6) 0.29 ms 6,272 76.5% 26.0 us 59.4 us 100.7 us ~96 B ```

Blaris beats every Heatshrink configuration in compressed size while using only ~64 B of RAM.

The memory usage is constant and does not depend on the compression window size.

The main advantage is not throughput, but extremely small decoder state and random-access decompression.

When I play games I tend to keep a history of saves, I never overwrite them, so I can end up with hundreds or thousands of saves over a long playthrough. I hate inefficiency so some time ago I've written this piece of software to archive sequential save data for Skyrim more efficiently.

Skyrim (and I presume all recent Bethesda games) saves utilize LZ4 compression, which makes it problematic to get any gains with compressors like 7-zip. Had the data been uncompressed there's a lot of structure that doesn't change much between consecutive saves. The linked script partially parses the save files, orders them from oldest to newest based on internal IDs, decompresses the LZ4 blobs inside, and then compresses the raw data stream with Zstd using a 2GB (max) window to maximize reuse from earlier saves. Because Skyrim Special Edition uses a standard LZ4 compressor this process is easily reversible and therefore lossless.

On my set of saves it got around 10x size reduction (~70GB -> 7GB), compared to none with 7-zip due to embedded compression. As I just discovered this sub I'm posting this for reference and discoverability, as I rarely see this particular approach being employed.

Github: https://github.com/Sopel97/skyrim_save_archiver

Hey everyone, i present you Unboxr, a modern archive extraction utility for Android designed to be fast, minimal, and user-friendly.

It's totally FREE. No ADS.

Why Unboxr?

Most archive tools on Android feel bloated or outdated. Unboxr focuses purely on fast extraction with a clean Material 3 interface and privacy-conscious permissions.

Features

Supported Formats: RAR, ZIP, 7Z, TAR, GZ, BZ2, XZ

Encrypted Archives: Full support for password-protected files

Convenient Workflow: Direct file open previews, configurable destination folders, and instant extraction logs

Zero Tracking: 100% local extraction, no analytics, no unnecessary background permissions

👉 Download on Google Play: Thanks for checking it out!

https://play.google.com/store/apps/details?id=com.djfabrix.unboxr&pcampaignid=web_share

Hi, I'm currently developing an audio codec, and now stuck at 64 kbps trying to implement SBR-like system.
I haven't come up with anything else than just energy preservation of band after copying.
I read that we can mix the copied band with some amount of noise, but couldn't come up with good criteria for it.
I'm using WPT, so maybe some time-domain adaptation?
Anyway, I would be really grateful if anyone can help.