That's exactly what I thought. About ten years everyone is rushing to distribute large downloads in xz format. These days some have started to move away from it just because how slow it is to compress and decompress.
Compression is only mildly faster or on par with xz, but decompression is (at similar ratios) vastly faster. Which really helps consumers of compressed blobs.
Yeah, that's a useful distinction. So it's excellent for ex. packages (hence https://www.archlinux.org/news/now-using-zstandard-instead-o...), but iffy for write-once-read-hopefully-never backups. (Although I've heard it suggested that this might flip again the moment you start test-restoring backups regularly)
Agreed. I think I'd take zstd over xz for WORN backups anyway, just because it's pretty reliable at detecting stream corruption. (Then again, I suggest generating par2 or other FEC against your compressed backups so that's not a problem.)
(I know you know this, as one of the principals; this is for other readers.)
And importantly, that 32-bit checksum is a pretty good checksum; a truncated XXH64(): https://tools.ietf.org/html/rfc8478#section-3 That's about as high-quality as you can expect from 32 bits. It's not, say, Fletcher-32.
Two good options here: Brotli gets within 0.6 % of the density of lzma, zstd within 5 %. Brotli is 5x faster than lzma and zstd 9x. I'm somewhat surprised that some people migrated from lzma to zstd, without considering brotli.
