Years ago I built a slippy map (google maps-style) tile server for non-image data. One of the use cases was to be able to quickly sample elevation data at an arbitrary lat/lng in a few milliseconds.
The data set is so large that you obviously want to delay decompression as long as possible. I turned to 16-bit grayscale PNGs, because PNG is a widely-used a standard. These were fine, but I wasn't close to my target latency.
After some experimentation, I was surprised to discover two things:
1. Deflate, this widely used standard, is just super slow compared to other algorithms (at least, in Go's native PNG decoder)
2. Tool and library support for formats other than ARGB32 is pretty lacking
So I turned to some bespoke integer compression algorithms like Snappy and Simple8b, and got a 20x decompression speedup, with maybe 20% worse compression ratios. This, along with some other tricks, got me where I needed to go.
Maybe there are some niche file formats out there that would've solved this. But in total we're not even talking about that much code, so it was easier to just invent my own.
Ok I'm not an expert here, but it seems they do [0] and in the pr [1] the comment says it's now 13 bytes (I _believe_ previously it was incorrectly stating 17 which it should have been 14)
The data set is so large that you obviously want to delay decompression as long as possible. I turned to 16-bit grayscale PNGs, because PNG is a widely-used a standard. These were fine, but I wasn't close to my target latency.
After some experimentation, I was surprised to discover two things:
1. Deflate, this widely used standard, is just super slow compared to other algorithms (at least, in Go's native PNG decoder)
2. Tool and library support for formats other than ARGB32 is pretty lacking
So I turned to some bespoke integer compression algorithms like Snappy and Simple8b, and got a 20x decompression speedup, with maybe 20% worse compression ratios. This, along with some other tricks, got me where I needed to go.
Maybe there are some niche file formats out there that would've solved this. But in total we're not even talking about that much code, so it was easier to just invent my own.