Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  nodelist  faq  login

IOT trap -- core dumped


computers / comp.compression / Re: Zlib, XZ, Zstd, what to choose?

SubjectAuthor
* Zlib, XZ, Zstd, what to choose?Helm
+- Re: Zlib, XZ, Zstd, what to choose?Elhana
`* Re: Zlib, XZ, Zstd, what to choose?BGB
 `- Re: Zlib, XZ, Zstd, what to choose?Eli the Bearded

1
Subject: Zlib, XZ, Zstd, what to choose?
From: Helm
Newsgroups: comp.compression
Organization: Tula Networks
Date: Thu, 23 Apr 2020 18:41 UTC
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!tulanet.com!esteban.tulanet.com!.POSTED!not-for-mail
From: nos...@please.invalid (Helm)
Newsgroups: comp.compression
Subject: Zlib, XZ, Zstd, what to choose?
Date: Thu, 23 Apr 2020 14:41:33 -0400
Organization: Tula Networks
Message-ID: <r7sngt$305v$1@esteban.tulanet.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: esteban.tulanet.com; logging-data="98495"; posting-host="Mlhx9qKTmA9nlOzvIfGvKg.user.esteban.tulanet.com"; mail-complaints-to="netnewsmaster@gmail.com";
User-Agent: Unison/2.2
X-Notice: Filtered by Postfilter with Rennovations
View all headers
Hello everyone,

I am working on a application and world like to know what you think is the best compression algorithm.

"Best" in terms of decompression time and compression ratio. I don't really care how long it takes to compress. Zstd is good but its owned by Facebook.

Thank you.



Subject: Re: Zlib, XZ, Zstd, what to choose?
From: Elhana
Newsgroups: comp.compression
Date: Sat, 25 Apr 2020 12:00 UTC
References: 1
X-Received: by 2002:a05:620a:120d:: with SMTP id u13mr13525504qkj.157.1587816020936;
Sat, 25 Apr 2020 05:00:20 -0700 (PDT)
X-Received: by 2002:a05:6214:1402:: with SMTP id n2mr13813262qvx.104.1587816020344;
Sat, 25 Apr 2020 05:00:20 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!news.snarked.org!border2.nntp.dca1.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Sat, 25 Apr 2020 05:00:19 -0700 (PDT)
In-Reply-To: <r7sngt$305v$1@esteban.tulanet.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=178.49.154.206; posting-account=y0dFrQoAAADOjqggM6Dv8j29KcFeVnzC
NNTP-Posting-Host: 178.49.154.206
References: <r7sngt$305v$1@esteban.tulanet.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <1139e660-0253-4fe1-b98a-bf4558f53316@googlegroups.com>
Subject: Re: Zlib, XZ, Zstd, what to choose?
From: tanarris...@yahoo.com (Elhana)
Injection-Date: Sat, 25 Apr 2020 12:00:20 +0000
Content-Type: text/plain; charset="UTF-8"
Lines: 5
View all headers
Helm:

I am working on a application and world like to know what you think is
the best compression algorithm.

Zstd, by far.


Subject: Re: Zlib, XZ, Zstd, what to choose?
From: BGB
Newsgroups: comp.compression
Organization: albasani.net
Date: Sat, 25 Apr 2020 18:41 UTC
References: 1
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!news.albasani.net!.POSTED!not-for-mail
From: cr88...@hotmail.com (BGB)
Newsgroups: comp.compression
Subject: Re: Zlib, XZ, Zstd, what to choose?
Date: Sat, 25 Apr 2020 13:41:01 -0500
Organization: albasani.net
Lines: 98
Message-ID: <r8209o$vhf$1@news.albasani.net>
References: <r7sngt$305v$1@esteban.tulanet.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: news.albasani.net LL8bPczPa7IrOJVHklYi90oxjxs5vIByrlqOp7dYaDc0uzBC7pUlKSUkXXC2ujeuJWv38hnP5iSi3l35SOY2lSFoj32FgapyUppF0fVN4+b3DGm9XS45Anl8ED4lVo7c
NNTP-Posting-Date: Sat, 25 Apr 2020 18:42:00 +0000 (UTC)
Injection-Info: news.albasani.net; logging-data="xiIq7qqI8bHKDy7VS8RFACi8nMLLwnIhlfrO7eesnUkhMBaatZideGlhuLzDI6dWVTQNcyDOk+N0vxqv44NkcpS5mJqG+97FoWAH39Z0PnWw9vAK2R44wckxNV6Q5z+M"; mail-complaints-to="abuse@albasani.net"
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
Thunderbird/68.7.0
In-Reply-To: <r7sngt$305v$1@esteban.tulanet.com>
Content-Language: en-US
Cancel-Lock: sha1:1BMocqysnZoLwoQ2HXVu+XZCM7w=
View all headers
On 4/23/2020 1:41 PM, Helm wrote:
Hello everyone,

I am working on a application and world like to know what you think is the best compression algorithm.

"Best" in terms of decompression time and compression ratio. I don't really care how long it takes to compress. Zstd is good but its owned by Facebook.


Tradeoffs are:
Zlib / Deflate: older, works reasonably well, neither the fastest nor best compression, reasonably well supported;
XZ / LZMA: Good compression, but a lot slower than the others.
Zstd: Fast decoding, decent compression (falls between Deflate and LZMA), slightly less open and relies on FSE.

Will add here:
LZ4: not the best compression, but decoding can be really fast, memory overhead for decoding is fairly low, and the decoder can be small/simple. LZ4 is one of the few options here viable for use on small microcontrollers.


My experience with FSE is that it does well on newer PC style hardware, but in past tests (years ago) its speed seems to take a big hit on older or lower end hardware (vs something like Deflate).

The memory overheads of Zstd and FSE are also too large to really be viable for small embedded applications (such as microcontrollers).
In these cases, something like Deflate or LZ4 is preferable because they have much smaller overheads.


Typical ranking in terms of compression (best to worst):
   LZMA
   Zstd
   Deflate
   LZ4

Typical ranking in terms of speed (best to worst, PC-style hardware):
   LZ4
   Zstd
   Deflate
   LZMA

Typical ranking in terms of speed (best to worst, low-end hardware):
   LZ4
   Deflate
   Zstd
   LZMA


Typical ranking in terms of memory overhead (least to most, *):
   LZ4
   Deflate
   LZMA | Zstd

*: LZ4, as noted, has minimal necessary overheads beyond a source and destination buffer. No tables or other structures necessary in this case (and "typical" use is buffer to buffer decoding).

Deflate generally needs lookup tables for Huffman decoding (multiple kB), and (in some implementations) another 32K for a copy of the sliding window (though, it is possible to map this onto a buffer holding the decoded contents). Depending on implementation specifics, likely memory overhead is somewhere between 5kB and 128kB.


Both LZMA and Zstd need some tables for entropy coding, but much of their memory overhead comes from (generally) keeping a full copy of the sliding window for decoding. In these encoders/decoders, this window may be up into the MB range.


Part of the difference here is due to whether the contents are decoded all at one to a destination buffer, vs "streamed". Stream decoding is more common if the uncompressed data may be potentially arbitrarily large, so it is more common to encode things in terms of smaller buffers representing a small piece of the input or output. This is the common case for file compression, and in this case it makes sense to keep the sliding window in its own memory buffer.


If decoding the buffer all at once, it generally makes more sense to use the buffer itself as the sliding window. This generally makes more sense if the compressor is being used as part of something else (such as part of a format like PNG, or part of a loader, ...), as opposed to using it for something like compressing or archiving files.



In some cases, it may also make sense to go "full custom".


Depends mostly on what one wants to do with it.



Subject: Re: Zlib, XZ, Zstd, what to choose?
From: Eli the Bearded
Newsgroups: comp.compression
Organization: Some absurd concept
Date: Sat, 25 Apr 2020 20:32 UTC
References: 1 2
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!news.linkpendium.com!news.linkpendium.com!panix!qz!not-for-mail
From: *...@eli.users.panix.com (Eli the Bearded)
Newsgroups: comp.compression
Subject: Re: Zlib, XZ, Zstd, what to choose?
Date: Sat, 25 Apr 2020 20:32:24 +0000 (UTC)
Organization: Some absurd concept
Lines: 22
Message-ID: <eli$2004251630@qaz.wtf>
References: <r7sngt$305v$1@esteban.tulanet.com> <r8209o$vhf$1@news.albasani.net>
NNTP-Posting-Host: panix5.panix.com
X-Trace: reader2.panix.com 1587846744 4489 166.84.1.5 (25 Apr 2020 20:32:24 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Sat, 25 Apr 2020 20:32:24 +0000 (UTC)
X-Liz: It's actually happened, the entire Internet is a massive game of Redcode
X-Motto: "Erosion of rights never seems to reverse itself." -- kenny@panix
X-US-Congress: Moronic Fucks.
X-Attribution: EtB
XFrom: is a real address
Encrypted: double rot-13
User-Agent: Vectrex rn 2.1 (beta)
View all headers
In comp.compression, BGB  <cr88192@hotmail.com> wrote:
Tradeoffs are:
Zlib / Deflate: older, works reasonably well, neither the fastest nor
best compression, reasonably well supported;
XZ / LZMA: Good compression, but a lot slower than the others.
Zstd: Fast decoding, decent compression (falls between Deflate and
LZMA), slightly less open and relies on FSE.

Will add here:
LZ4: not the best compression, but decoding can be really fast, memory
overhead for decoding is fairly low, and the decoder can be
small/simple. LZ4 is one of the few options here viable for use on small
microcontrollers.

[snip remainder]

I'd like to thank you for the most informative post this group has seen
in a couple of years.

Elijah
------
came here for JPEG discussion some years ago and never stopped reading


1
rocksolid light 0.7.2
clearneti2ptor