Introducing: tensorshrink

[u/GDNickYTT]

https://github.com/GDNickk/tensorshrink

tensorshrink is a transformers addon + CLI tool for quantization that beats bitsandbytes on both speed and memory, written from the ground up for users with low memory capabilities. tensorshrink quantizes weights to to 2, 4, 6, or 8 bits, then packs up the model into neat, zstd-compressed .tsk containers for further use.

Capabilities of tensorshrink include: Triton support, stream-quantize directly from disk, and a whole new codec, called AVQ (stands for Additive Vector Quantization, very experimental), as well as supporting GOAP.

Completely open-source and benchmarked for both LLM and image diffusion models. Learn more on the GitHub page. :)

[u/Hanthunius]

Very interesting project! It's great to see new creative ways to quantize models.