Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  nodelist  faq  login

I surely do hope that's a syntax error. -- Larry Wall in <199710011752.KAA21624@wall.org>


computers / comp.compression / Re: Good compression

SubjectAuthor
* Good compressionsundaresh.venugopal
+* Re: Good compressionElhana
|`* Re: Good compressionsundaresh.venugopal
| `* Re: Good compressionEli the Bearded
|  +- Re: Good compressionsundaresh.venugopal
|  +- Re: Good compressionScott
|  `* Re: Good compressionBGB
|   `* Re: Good compressionMatthias Waldhauer
|    `- Re: Good compressionBGB
`* Re: Good compressionHelm
 `* Re: Good compressionsundaresh.venugopal
  `* Re: Good compressionicepic.dz
   `* Re: Good compressionsundaresh.venugopal
    +- Re: Good compressionFöldes László
    `* Re: Good compressionicepic.dz
     `* Re: Good compressionSundaresh Venugopal
      `* Re: Good compressionSundaresh Venugopal
       `* Re: Good compressionicepic.dz
        `* Re: Good compressionSundaresh Venugopal
         `- Re: Good compressionJanne Johansson

1
Subject: Re: Good compression
From: Sundaresh Venugopal
Newsgroups: comp.compression
Date: Wed, 19 Aug 2020 09:45 UTC
References: 1 2 3 4 5 6
X-Received: by 2002:a05:620a:150f:: with SMTP id i15mr20784907qkk.152.1597830323783;
Wed, 19 Aug 2020 02:45:23 -0700 (PDT)
X-Received: by 2002:a5b:2cb:: with SMTP id h11mr31627041ybp.116.1597830323415;
Wed, 19 Aug 2020 02:45:23 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 19 Aug 2020 02:45:23 -0700 (PDT)
In-Reply-To: <4b347e80-5c96-428b-b663-ecc303588c94o@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2401:4900:33b0:5a64:9096:503:d057:d55a;
posting-account=2ecefgoAAACTSQp1lBuI1Iwdq8jeBELE
NNTP-Posting-Host: 2401:4900:33b0:5a64:9096:503:d057:d55a
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
<8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com> <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
<4b347e80-5c96-428b-b663-ecc303588c94o@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <17279bfb-a27e-45e8-b026-8441b9c890e0n@googlegroups.com>
Subject: Re: Good compression
From: sundares...@gmail.com (Sundaresh Venugopal)
Injection-Date: Wed, 19 Aug 2020 09:45:23 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
View all headers
On Friday, July 3, 2020 at 6:58:11 PM UTC+5:30, icep...@gmail.com wrote:
Den fredag 3 juli 2020 kl. 05:38:42 UTC+2 skrev sundaresh...@gmail.com:
On Thursday, July 2, 2020 at 7:08:23 PM UTC+5:30, icep...@gmail.com wrote:
Den torsdag 18 juni 2020 kl. 05:59:27 UTC+2 skrev sundaresh...@gmail.com:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not
allow attachments.

You can always attach it if you use a newsreader instead of the Google
Groups interface (that also puts your IP in the headers by the way).
Don't do it however since all the free newsservers won't be able to
hold your post thus reducing the visibility and reach. Instead, upload
it to a website like WeTransfer and post the link in the newsgroup.

Good research though! Look forward to learning more.

Well thanks ! One encouraging response . Here is the link

https://drive.google.com/file/d/1w5sNV_YVM9XGymz8A0MT0V_cubxMPYCM/view?usp=sharing

If it does not work, let me know.

It doesn't work.

Really, have you tried coding it. If you have, can I see the code ? You can do just what I have done. In the meantime I have made significant improvements to it, and it should be visible in the link and I will be coding this new version.
Please post the code you have then, and the results.

You are the one making a claim that a large number M can be written as a smaller number S by transforming it in one of several ways, which the pigeonhole principle will show anyone will not work for all numbers from 0 to M, since they can't all fit in S.

You can find S numbers out of M that do, but even selecting which set of Ses is out of the set 0->M will cost bits.
I will see what I an do.


Subject: Re: Good compression
From: Sundaresh Venugopal
Newsgroups: comp.compression
Date: Wed, 19 Aug 2020 10:23 UTC
References: 1 2 3 4 5 6 7
X-Received: by 2002:a37:454d:: with SMTP id s74mr21736840qka.373.1597832599032;
Wed, 19 Aug 2020 03:23:19 -0700 (PDT)
X-Received: by 2002:a25:ca0f:: with SMTP id a15mr32628294ybg.256.1597832598722;
Wed, 19 Aug 2020 03:23:18 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 19 Aug 2020 03:23:18 -0700 (PDT)
In-Reply-To: <17279bfb-a27e-45e8-b026-8441b9c890e0n@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2401:4900:33b0:5a64:9096:503:d057:d55a;
posting-account=2ecefgoAAACTSQp1lBuI1Iwdq8jeBELE
NNTP-Posting-Host: 2401:4900:33b0:5a64:9096:503:d057:d55a
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
<8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com> <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
<4b347e80-5c96-428b-b663-ecc303588c94o@googlegroups.com> <17279bfb-a27e-45e8-b026-8441b9c890e0n@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <755a3926-ebd9-431d-b77e-7e45ad36f4d0n@googlegroups.com>
Subject: Re: Good compression
From: sundares...@gmail.com (Sundaresh Venugopal)
Injection-Date: Wed, 19 Aug 2020 10:23:19 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
View all headers
On Wednesday, August 19, 2020 at 3:15:24 PM UTC+5:30, Sundaresh Venugopal wrote:
On Friday, July 3, 2020 at 6:58:11 PM UTC+5:30, icep...@gmail.com wrote:
Den fredag 3 juli 2020 kl. 05:38:42 UTC+2 skrev sundaresh...@gmail.com:
On Thursday, July 2, 2020 at 7:08:23 PM UTC+5:30, icep...@gmail.com wrote:
Den torsdag 18 juni 2020 kl. 05:59:27 UTC+2 skrev sundaresh...@gmail.com:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not
allow attachments.

You can always attach it if you use a newsreader instead of the Google
Groups interface (that also puts your IP in the headers by the way).
Don't do it however since all the free newsservers won't be able to
hold your post thus reducing the visibility and reach. Instead, upload
it to a website like WeTransfer and post the link in the newsgroup.

Good research though! Look forward to learning more.

Well thanks ! One encouraging response . Here is the link

https://drive.google.com/file/d/1w5sNV_YVM9XGymz8A0MT0V_cubxMPYCM/view?usp=sharing

If it does not work, let me know.

It doesn't work.

Really, have you tried coding it. If you have, can I see the code ? You can do just what I have done. In the meantime I have made significant improvements to it, and it should be visible in the link and I will be coding this new version.
Please post the code you have then, and the results.

You are the one making a claim that a large number M can be written as a smaller number S by transforming it in one of several ways, which the pigeonhole principle will show anyone will not work for all numbers from 0 to M, since they can't all fit in S.

You can find S numbers out of M that do, but even selecting which set of Ses is out of the set 0->M will cost bits.
I will see what I an do.
Sorry. Meant, I will see what I can do, but not to convince people like you..


Subject: Re: Good compression
From: icepic...@gmail.com
Newsgroups: comp.compression
Date: Wed, 19 Aug 2020 16:59 UTC
References: 1 2 3 4 5 6 7 8
X-Received: by 2002:a05:620a:1243:: with SMTP id a3mr21982552qkl.423.1597856382964;
Wed, 19 Aug 2020 09:59:42 -0700 (PDT)
X-Received: by 2002:a5b:58f:: with SMTP id l15mr33482731ybp.215.1597856382661;
Wed, 19 Aug 2020 09:59:42 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 19 Aug 2020 09:59:42 -0700 (PDT)
In-Reply-To: <755a3926-ebd9-431d-b77e-7e45ad36f4d0n@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2001:2002:d9d0:9cec:d56b:a8db:8589:c9f5;
posting-account=O3ek6woAAAA3FgCl-C4iYdhpMZfNNRaG
NNTP-Posting-Host: 2001:2002:d9d0:9cec:d56b:a8db:8589:c9f5
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
<8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com> <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
<4b347e80-5c96-428b-b663-ecc303588c94o@googlegroups.com> <17279bfb-a27e-45e8-b026-8441b9c890e0n@googlegroups.com>
<755a3926-ebd9-431d-b77e-7e45ad36f4d0n@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <5d6d7c94-2ff5-4021-88e5-95a7a516fe58o@googlegroups.com>
Subject: Re: Good compression
From: icepic...@gmail.com
Injection-Date: Wed, 19 Aug 2020 16:59:42 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
View all headers
Please post the code you have then, and the results.

You are the one making a claim that a large number M can be written as a smaller number S by transforming it in one of several ways, which the pigeonhole principle will show anyone will not work for all numbers from 0 to M, since they can't all fit in S.

You can find S numbers out of M that do, but even selecting which set of Ses is out of the set 0->M will cost bits.
I will see what I an do.
Sorry. Meant, I will see what I can do, but not to convince people like you.

I am not expecting you to do any work for me, I was suggesting you do the work for *you*, so you see how it does, or does not work for any numbers or all numbers.

It is quite common for people to assume things like "I can find my wanted large number in the decimals of PI and will refer to it by its position" which is true, but is not usable for compression since the pointer will be as long, or longer than the number when you take all possible combinations into account.

If you are only envisioning "I will look for the number 141592653589" then yes, you can find a very short index for it. If you instead look for the far shorter number "93774" then it will not appear in the first 100000 digits..

So what you need to do in order to create fame for yourself with a new novel way of compressing data is 1) understand the pigeon hole principle (everyone in compression research must understand this one) and 2) make some kind of implementation of your idea to know when it will *not* work.

Good compressors work on something like 1% of the inputs, they just chose the 1% which are commonly found in files on computers. For all the 99% other kinds of data which are uncommon, they will expand, but if you don't hand such data to your compressor, noone notices and all is fine.

So by making the implementation, you have a good chance of finding out what your "99%" are.


Subject: Re: Good compression
From: Sundaresh Venugopal
Newsgroups: comp.compression
Date: Thu, 20 Aug 2020 00:08 UTC
References: 1 2 3 4 5 6 7 8 9
X-Received: by 2002:ad4:46ac:: with SMTP id br12mr805071qvb.236.1597882133549;
Wed, 19 Aug 2020 17:08:53 -0700 (PDT)
X-Received: by 2002:a25:cfcc:: with SMTP id f195mr1294057ybg.390.1597882133262;
Wed, 19 Aug 2020 17:08:53 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 19 Aug 2020 17:08:52 -0700 (PDT)
In-Reply-To: <5d6d7c94-2ff5-4021-88e5-95a7a516fe58o@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2401:4900:3308:b24e:f0ef:f578:5c2a:7bed;
posting-account=2ecefgoAAACTSQp1lBuI1Iwdq8jeBELE
NNTP-Posting-Host: 2401:4900:3308:b24e:f0ef:f578:5c2a:7bed
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
<8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com> <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
<4b347e80-5c96-428b-b663-ecc303588c94o@googlegroups.com> <17279bfb-a27e-45e8-b026-8441b9c890e0n@googlegroups.com>
<755a3926-ebd9-431d-b77e-7e45ad36f4d0n@googlegroups.com> <5d6d7c94-2ff5-4021-88e5-95a7a516fe58o@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <5ad265eb-06d3-43ef-ae47-4b99894d2f1an@googlegroups.com>
Subject: Re: Good compression
From: sundares...@gmail.com (Sundaresh Venugopal)
Injection-Date: Thu, 20 Aug 2020 00:08:53 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
View all headers
On Wednesday, August 19, 2020 at 10:29:45 PM UTC+5:30, icep...@gmail.com wrote:
Please post the code you have then, and the results.

You are the one making a claim that a large number M can be written as a smaller number S by transforming it in one of several ways, which the pigeonhole principle will show anyone will not work for all numbers from 0 to M, since they can't all fit in S.

You can find S numbers out of M that do, but even selecting which set of Ses is out of the set 0->M will cost bits.
I will see what I an do.
Sorry. Meant, I will see what I can do, but not to convince people like you.
I am not expecting you to do any work for me, I was suggesting you do the work for *you*, so you see how it does, or does not work for any numbers or all numbers.

It is quite common for people to assume things like "I can find my wanted large number in the decimals of PI and will refer to it by its position" which is true, but is not usable for compression since the pointer will be as long, or longer than the number when you take all possible combinations into account.

If you are only envisioning "I will look for the number 141592653589" then yes, you can find a very short index for it. If you instead look for the far shorter number "93774" then it will not appear in the first 100000 digits.

So what you need to do in order to create fame for yourself with a new novel way of compressing data is 1) understand the pigeon hole principle (everyone in compression research must understand this one) and 2) make some kind of implementation of your idea to know when it will *not* work.

Good compressors work on something like 1% of the inputs, they just chose the 1% which are commonly found in files on computers. For all the 99% other kinds of data which are uncommon, they will expand, but if you don't hand such data to your compressor, noone notices and all is fine.

So by making the implementation, you have a good chance of finding out what your "99%" are.
I am not the one making a false claim here. Do not mislead people.


Subject: Re: Good compression
From: Matthias Waldhauer
Newsgroups: comp.compression
Date: Wed, 26 Aug 2020 07:50 UTC
References: 1 2 3 4 5
X-Received: by 2002:a37:bf03:: with SMTP id p3mr5216907qkf.94.1598428260722;
Wed, 26 Aug 2020 00:51:00 -0700 (PDT)
X-Received: by 2002:a25:244a:: with SMTP id k71mr1006364ybk.504.1598428260393;
Wed, 26 Aug 2020 00:51:00 -0700 (PDT)
Path: i2pn2.org!i2pn.org!aioe.org!peer03.ams4!peer.am4.highwinds-media.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 26 Aug 2020 00:50:59 -0700 (PDT)
In-Reply-To: <rcgfsv$ao7$1@dont-email.me>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=194.114.104.125; posting-account=mZDhIAkAAAA5yZTwP-i7JDS8KSI6Oroh
NNTP-Posting-Host: 194.114.104.125
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com> <d263ee15-ab73-4bc4-b8db-936cb42d99cfo@googlegroups.com>
<eli$2006171800@qaz.wtf> <rcgfsv$ao7$1@dont-email.me>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <6c521d3a-7f3f-4063-8a8f-2245c5899703n@googlegroups.com>
Subject: Re: Good compression
From: matthias...@gmail.com (Matthias Waldhauer)
Injection-Date: Wed, 26 Aug 2020 07:51:00 +0000
Content-Type: text/plain; charset="UTF-8"
X-Received-Bytes: 2654
X-Received-Body-CRC: 3802749190
View all headers
BGB wrote on 18th June 2020 at 21:39:13 UTC+2:
Compression for many files seems to be part-way between LZ4 and Deflate.
Note that, like LZ4, it is byte oriented with no entropy coding.

<--

BtRP2 (Transposed, LE):
* dddddddd-dlllrrr0 (l=3..10, d=0..511, r=0..7)
* dddddddd-dddddlll-lllrrr01 (l=4..67, d=0..8191)
* dddddddd-dddddddd-dlllllll-llrrr011 (l=4..515, d=0..131071)
* rrrr0111 (Raw Bytes, r=(r+1)*8, 8..128)
* * rrr01111 (Long Match)
* rr011111 (r=1..3 bytes, 0=EOB)
* rrrrrrrr-r0111111 (Long Raw, r=(r+1)*8, 8..4096)
** d: Distance
** l: Match Length
** r: Literal Length

Values are encoded in little-endian order, with tag bits located in the
LSB. Bits will be contiguous within the value, with shift-and-mask being
used to extract individual elements.

Interesting tokens! You're using unary coding for the different tokens. In the last one (Long Raw) you might switch the 0 for another r bit, as a limit on code length (6 bits, 111111) would be enough to differentiate the last 2 cases.

I guess with some effort (PhD thesis), one could develop a method to formally describe all possible token variants via a grammar. And based on that, both the efficiency and the decoder complexity could be calculated and optimized.

M.



Subject: Re: Good compression
From: Janne Johansson
Newsgroups: comp.compression
Date: Wed, 26 Aug 2020 10:07 UTC
References: 1 2 3 4 5 6 7 8 9 10
X-Received: by 2002:ac8:19d7:: with SMTP id s23mr7410913qtk.127.1598436429217;
Wed, 26 Aug 2020 03:07:09 -0700 (PDT)
X-Received: by 2002:a5b:2cb:: with SMTP id h11mr19217894ybp.116.1598436429028;
Wed, 26 Aug 2020 03:07:09 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!news.snarked.org!border2.nntp.dca1.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 26 Aug 2020 03:07:08 -0700 (PDT)
In-Reply-To: <5ad265eb-06d3-43ef-ae47-4b99894d2f1an@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2001:2002:d9d0:9cec:b44e:3bac:6213:d473;
posting-account=O3ek6woAAAA3FgCl-C4iYdhpMZfNNRaG
NNTP-Posting-Host: 2001:2002:d9d0:9cec:b44e:3bac:6213:d473
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
<8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com> <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
<4b347e80-5c96-428b-b663-ecc303588c94o@googlegroups.com> <17279bfb-a27e-45e8-b026-8441b9c890e0n@googlegroups.com>
<755a3926-ebd9-431d-b77e-7e45ad36f4d0n@googlegroups.com> <5d6d7c94-2ff5-4021-88e5-95a7a516fe58o@googlegroups.com>
<5ad265eb-06d3-43ef-ae47-4b99894d2f1an@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <675cd8ec-6be4-49c6-b819-4d443ec28a66n@googlegroups.com>
Subject: Re: Good compression
From: icepic...@gmail.com (Janne Johansson)
Injection-Date: Wed, 26 Aug 2020 10:07:09 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Lines: 15
View all headers
torsdag 20 augusti 2020 kl. 02:08:54 UTC+2 skrev Sundaresh Venugopal:
On Wednesday, August 19, 2020 at 10:29:45 PM UTC+5:30, icep...@gmail.com wrote:
Please post the code you have then, and the results.

You are the one making a claim that a large number M can be written as a smaller number S by transforming it in one of several ways, which the pigeonhole principle will show anyone will not work for all numbers from 0 to M, since they can't all fit in S.

So by making the implementation, you have a good chance of finding out what your "99%" are.
I am not the one making a false claim here. Do not mislead people.

False or not remains to be seen. You are making a claim though. We are waiting for some proof of it.


Subject: Re: Good compression
From: BGB
Newsgroups: comp.compression
Organization: A noiseless patient Spider
Date: Wed, 26 Aug 2020 19:29 UTC
References: 1 2 3 4 5 6
Path: i2pn2.org!i2pn.org!eternal-september.org!feeder.eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: cr88...@gmail.com (BGB)
Newsgroups: comp.compression
Subject: Re: Good compression
Date: Wed, 26 Aug 2020 14:29:08 -0500
Organization: A noiseless patient Spider
Lines: 53
Message-ID: <ri6d67$tuv$1@dont-email.me>
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com>
<d263ee15-ab73-4bc4-b8db-936cb42d99cfo@googlegroups.com>
<eli$2006171800@qaz.wtf> <rcgfsv$ao7$1@dont-email.me>
<6c521d3a-7f3f-4063-8a8f-2245c5899703n@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Wed, 26 Aug 2020 19:29:11 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="05709783504ec93fbab28aecc8f06bd3";
logging-data="30687"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19gaq0yR2rWvL2M8PkIMcny"
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
Thunderbird/68.11.0
Cancel-Lock: sha1:4QKlAs0hx/g6YnlhfUn1FT/9U50=
In-Reply-To: <6c521d3a-7f3f-4063-8a8f-2245c5899703n@googlegroups.com>
Content-Language: en-US
View all headers
On 8/26/2020 2:50 AM, Matthias Waldhauer wrote:
BGB wrote on 18th June 2020 at 21:39:13 UTC+2:
Compression for many files seems to be part-way between LZ4 and Deflate.
Note that, like LZ4, it is byte oriented with no entropy coding.

<--

BtRP2 (Transposed, LE):
* dddddddd-dlllrrr0 (l=3..10, d=0..511, r=0..7)
* dddddddd-dddddlll-lllrrr01 (l=4..67, d=0..8191)
* dddddddd-dddddddd-dlllllll-llrrr011 (l=4..515, d=0..131071)
* rrrr0111 (Raw Bytes, r=(r+1)*8, 8..128)
* * rrr01111 (Long Match)
* rr011111 (r=1..3 bytes, 0=EOB)
* rrrrrrrr-r0111111 (Long Raw, r=(r+1)*8, 8..4096)
** d: Distance
** l: Match Length
** r: Literal Length

Values are encoded in little-endian order, with tag bits located in the
LSB. Bits will be contiguous within the value, with shift-and-mask being
used to extract individual elements.

Interesting tokens! You're using unary coding for the different tokens. In the last one (Long Raw) you might switch the 0 for another r bit, as a limit on code length (6 bits, 111111) would be enough to differentiate the last 2 cases.


Leaving it as 0 leaves room for possible further expansion; also the "Long Raw" case is infrequent enough (and big enough) that 1 more bit wont make a big difference (worst case overhead is ~ 0.05%).

I guess with some effort (PhD thesis), one could develop a method to formally describe all possible token variants via a grammar. And based on that, both the efficiency and the decoder complexity could be calculated and optimized.


Dunno.

As noted, compression ratios tend to generally be slightly better than LZ4 in this case (including if encoded using the same match algo and the same match length/distance settings).

This seems to be more so for "general data compression", however for compressing PE/COFF binaries with my ISA, compression ratio seems to be roughly break-even with LZ4.


An older "fast" LZ encoder, which did everything in terms of 32-bit DWORD values, also did reasonably well for binaries, but worse for most other data compression.

A likely factor though is that the ISA is mostly a mix of 16 and 32-bit instructions, and sequences of 32-bit instructions also tend to be 32-bit aligned (though, this is not enforced as doing so in-general has an adverse effect on code-density).



Subject: Good compression
From: sundares...@gmail.com
Newsgroups: comp.compression
Date: Wed, 10 Jun 2020 14:03 UTC
X-Received: by 2002:ac8:2dea:: with SMTP id q39mr3421613qta.2.1591797788971;
Wed, 10 Jun 2020 07:03:08 -0700 (PDT)
X-Received: by 2002:aed:2861:: with SMTP id r88mr3252956qtd.207.1591797788645;
Wed, 10 Jun 2020 07:03:08 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!news.mixmin.net!aioe.org!peer01.ams4!peer.am4.highwinds-media.com!peer03.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 10 Jun 2020 07:03:08 -0700 (PDT)
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2401:4900:330a:64ef:a421:d68f:5e5f:8a3d;
posting-account=2ecefgoAAACTSQp1lBuI1Iwdq8jeBELE
NNTP-Posting-Host: 2401:4900:330a:64ef:a421:d68f:5e5f:8a3d
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
Subject: Good compression
From: sundares...@gmail.com
Injection-Date: Wed, 10 Jun 2020 14:03:08 +0000
Content-Type: text/plain; charset="UTF-8"
X-Received-Bytes: 1241
X-Received-Body-CRC: 2137833831
View all headers
The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not allow attachments.


Subject: Re: Good compression
From: Elhana
Newsgroups: comp.compression
Date: Tue, 16 Jun 2020 13:11 UTC
References: 1
X-Received: by 2002:a05:620a:2290:: with SMTP id o16mr20678489qkh.205.1592313089425;
Tue, 16 Jun 2020 06:11:29 -0700 (PDT)
X-Received: by 2002:ac8:2bc4:: with SMTP id n4mr20943550qtn.222.1592313089157;
Tue, 16 Jun 2020 06:11:29 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Tue, 16 Jun 2020 06:11:28 -0700 (PDT)
In-Reply-To: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=178.49.154.103; posting-account=y0dFrQoAAADOjqggM6Dv8j29KcFeVnzC
NNTP-Posting-Host: 178.49.154.103
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com>
Subject: Re: Good compression
From: tanarris...@yahoo.com (Elhana)
Injection-Date: Tue, 16 Jun 2020 13:11:29 +0000
Content-Type: text/plain; charset="UTF-8"
View all headers
sundaresh...@gmail.com:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not allow attachments.

Show us the code.


Subject: Re: Good compression
From: sundares...@gmail.com
Newsgroups: comp.compression
Date: Tue, 16 Jun 2020 15:37 UTC
References: 1 2
X-Received: by 2002:aed:3169:: with SMTP id 96mr20979175qtg.211.1592321860662; Tue, 16 Jun 2020 08:37:40 -0700 (PDT)
X-Received: by 2002:a05:620a:5e9:: with SMTP id z9mr20012978qkg.459.1592321860395; Tue, 16 Jun 2020 08:37:40 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!feeder1.feed.usenet.farm!feed.usenet.farm!feeder.usenetexpress.com!tr2.iad1.usenetexpress.com!border1.nntp.dca1.giganews.com!nntp.giganews.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Tue, 16 Jun 2020 08:37:40 -0700 (PDT)
In-Reply-To: <b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2401:4900:3305:863a:74b5:2757:8bec:7faf; posting-account=2ecefgoAAACTSQp1lBuI1Iwdq8jeBELE
NNTP-Posting-Host: 2401:4900:3305:863a:74b5:2757:8bec:7faf
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com> <b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <d263ee15-ab73-4bc4-b8db-936cb42d99cfo@googlegroups.com>
Subject: Re: Good compression
From: sundares...@gmail.com
Injection-Date: Tue, 16 Jun 2020 15:37:40 +0000
Content-Type: text/plain; charset="UTF-8"
Lines: 1
View all headers

I started coding but shelved it and put it in the background since l am also simultaneously developing and coding other s/w as well. But why do you ask ? I am quite certain there is nothing wrong with the method.


Subject: Re: Good compression
From: Helm
Newsgroups: comp.compression
Organization: Tula Networks
Date: Wed, 17 Jun 2020 21:58 UTC
References: 1
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!tulanet.com!esteban.tulanet.com!.POSTED!not-for-mail
From: nos...@please.invalid (Helm)
Newsgroups: comp.compression
Subject: Re: Good compression
Date: Wed, 17 Jun 2020 17:58:45 -0400
Organization: Tula Networks
Message-ID: <rce3ml$2taf$1@esteban.tulanet.com>
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: esteban.tulanet.com; logging-data="95567"; posting-host="QyHZHgsw0leReE2ALMhDBg.user.esteban.tulanet.com"; mail-complaints-to="netnewsmaster@gmail.com";
User-Agent: Unison/2.2
X-Notice: Filtered by Postfilter with Rennovations
View all headers
On 2020-06-10 14:03:08 +0000, sundaresh.venugopal@gmail.com said:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not allow attachments.

You can always attach it if you use a newsreader instead of the Google Groups interface (that also puts your IP in the headers by the way). Don't do it however  since all the free newsservers won't be able to hold your post thus reducing the visibility and reach. Instead, upload it to a website like WeTransfer and post the link in the newsgroup.

Good research though! Look forward to learning more.



Subject: Re: Good compression
From: Eli the Bearded
Newsgroups: comp.compression
Organization: Some absurd concept
Date: Wed, 17 Jun 2020 22:02 UTC
References: 1 2 3
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!news.linkpendium.com!news.linkpendium.com!panix!qz!not-for-mail
From: *...@eli.users.panix.com (Eli the Bearded)
Newsgroups: comp.compression
Subject: Re: Good compression
Date: Wed, 17 Jun 2020 22:02:05 +0000 (UTC)
Organization: Some absurd concept
Lines: 12
Message-ID: <eli$2006171800@qaz.wtf>
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com> <b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com> <d263ee15-ab73-4bc4-b8db-936cb42d99cfo@googlegroups.com>
NNTP-Posting-Host: panix5.panix.com
X-Trace: reader1.panix.com 1592431325 26027 166.84.1.5 (17 Jun 2020 22:02:05 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Wed, 17 Jun 2020 22:02:05 +0000 (UTC)
X-Liz: It's actually happened, the entire Internet is a massive game of Redcode
X-Motto: "Erosion of rights never seems to reverse itself." -- kenny@panix
X-US-Congress: Moronic Fucks.
X-Attribution: EtB
XFrom: is a real address
Encrypted: double rot-13
User-Agent: Vectrex rn 2.1 (beta)
View all headers
In comp.compression,  <sundaresh.venugopal@gmail.com> wrote:
I started coding but shelved it and put it in the background since l
am also simultaneously developing and coding other s/w as well. But
why do you ask ? I am quite certain there is nothing wrong with the
method.

This group is littered with the dead claims of people who said they had
a better compression method but could never prove it with code.

Elijah
------
"compression" seems to attrack kooks as much as "perpetual motion"


Subject: Re: Good compression
From: sundares...@gmail.com
Newsgroups: comp.compression
Date: Thu, 18 Jun 2020 03:59 UTC
References: 1 2
X-Received: by 2002:a05:6214:94a:: with SMTP id dn10mr1875389qvb.69.1592452764883;
Wed, 17 Jun 2020 20:59:24 -0700 (PDT)
X-Received: by 2002:a37:b57:: with SMTP id 84mr1876931qkl.364.1592452764230;
Wed, 17 Jun 2020 20:59:24 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 17 Jun 2020 20:59:23 -0700 (PDT)
In-Reply-To: <rce3ml$2taf$1@esteban.tulanet.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=223.186.0.125; posting-account=2ecefgoAAACTSQp1lBuI1Iwdq8jeBELE
NNTP-Posting-Host: 223.186.0.125
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com> <rce3ml$2taf$1@esteban.tulanet.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
Subject: Re: Good compression
From: sundares...@gmail.com
Injection-Date: Thu, 18 Jun 2020 03:59:24 +0000
Content-Type: text/plain; charset="UTF-8"
View all headers
On Thursday, June 18, 2020 at 3:28:47 AM UTC+5:30, Helm wrote:
On 2020-06-10 14:03:08 +0000, sundaresh.venugopal@gmail.com said:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not
allow attachments.

You can always attach it if you use a newsreader instead of the Google
Groups interface (that also puts your IP in the headers by the way).
Don't do it however  since all the free newsservers won't be able to
hold your post thus reducing the visibility and reach. Instead, upload
it to a website like WeTransfer and post the link in the newsgroup.

Good research though! Look forward to learning more.

Well thanks ! One encouraging response . Here is the link

https://drive.google.com/file/d/1w5sNV_YVM9XGymz8A0MT0V_cubxMPYCM/view?usp=sharing

If it does not work, let me know.


Subject: Re: Good compression
From: sundares...@gmail.com
Newsgroups: comp.compression
Date: Thu, 18 Jun 2020 04:58 UTC
References: 1 2 3 4
X-Received: by 2002:ac8:458d:: with SMTP id l13mr2587756qtn.279.1592456318276;
Wed, 17 Jun 2020 21:58:38 -0700 (PDT)
X-Received: by 2002:a05:620a:22ce:: with SMTP id o14mr1651929qki.169.1592456317871;
Wed, 17 Jun 2020 21:58:37 -0700 (PDT)
Path: i2pn2.org!i2pn.org!aioe.org!peer02.ams4!peer.am4.highwinds-media.com!peer03.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 17 Jun 2020 21:58:37 -0700 (PDT)
In-Reply-To: <eli$2006171800@qaz.wtf>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=223.186.0.125; posting-account=2ecefgoAAACTSQp1lBuI1Iwdq8jeBELE
NNTP-Posting-Host: 223.186.0.125
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com> <d263ee15-ab73-4bc4-b8db-936cb42d99cfo@googlegroups.com>
<eli$2006171800@qaz.wtf>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <55db2ece-3c87-4515-9272-301197b1add3o@googlegroups.com>
Subject: Re: Good compression
From: sundares...@gmail.com
Injection-Date: Thu, 18 Jun 2020 04:58:38 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 2523
X-Received-Body-CRC: 248607890
View all headers
On Thursday, June 18, 2020 at 3:32:07 AM UTC+5:30, Eli the Bearded wrote:
In comp.compression,  <sundaresh.venugopal@gmail.com> wrote:
I started coding but shelved it and put it in the background since l
am also simultaneously developing and coding other s/w as well. But
why do you ask ? I am quite certain there is nothing wrong with the
method.

This group is littered with the dead claims of people who said they had
a better compression method but could never prove it with code.

Elijah
------

"compression" seems to attrack kooks as much as "perpetual motion"

The little(of the other stuff)  I have coded works extremely well, just as I expected, in fact in most cases surprisingly better. All I am saying is, proof of the recipe is different from proof of the pudding, a truth which is even more exacting w.r.t recipe's and pudding's in the world of computing or as you may call it the virtual world. Speaking only for myself, and in all candor I will admit, having tried my hand at cooking, I am a much better cook today than I was when I started off cooking, both on my kitchen table and on my computer table.


Subject: Re: Good compression
From: Scott
Newsgroups: comp.compression
Organization: XMission http://xmission.com/
Date: Thu, 18 Jun 2020 16:13 UTC
References: 1 2 3 4
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!xmission!nnrp.xmission!.POSTED.shell.xmission.com!not-for-mail
From: nob...@example.org (Scott)
Newsgroups: comp.compression
Subject: Re: Good compression
Date: Thu, 18 Jun 2020 16:13:42 GMT
Organization: XMission http://xmission.com/
Message-ID: <5eeb9228.760524500@core>
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com> <b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com> <d263ee15-ab73-4bc4-b8db-936cb42d99cfo@googlegroups.com> <eli$2006171800@qaz.wtf>
Injection-Info: news.xmission.com; posting-host="shell.xmission.com:2607:fa18:0:beef::4";
logging-data="28271"; mail-complaints-to="abuse@xmission.com"
X-Newsreader: Forte Free Agent 1.21/32.243
View all headers
On Wed, 17 Jun 2020 22:02:05 +0000 (UTC), Eli the Bearded
<*@eli.users.panix.com> wrote:

In comp.compression,  <sundaresh.venugopal@gmail.com> wrote:
I started coding but shelved it and put it in the background since l
am also simultaneously developing and coding other s/w as well. But
why do you ask ? I am quite certain there is nothing wrong with the
method.

This group is littered with the dead claims of people who said they had
a better compression method but could never prove it with code.

Oh, it's not that bad. Recursive compression is actually pretty easy
to do. The decompressor turns out to be a good deal trickier, though.



Subject: Re: Good compression
From: BGB
Newsgroups: comp.compression
Organization: A noiseless patient Spider
Date: Thu, 18 Jun 2020 19:39 UTC
References: 1 2 3 4
Path: i2pn2.org!i2pn.org!eternal-september.org!feeder.eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: cr88...@gmail.com (BGB)
Newsgroups: comp.compression
Subject: Re: Good compression
Date: Thu, 18 Jun 2020 14:39:10 -0500
Organization: A noiseless patient Spider
Lines: 107
Message-ID: <rcgfsv$ao7$1@dont-email.me>
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<b1403416-49ce-416f-9b1f-25e10bb68e21o@googlegroups.com>
<d263ee15-ab73-4bc4-b8db-936cb42d99cfo@googlegroups.com>
<eli$2006171800@qaz.wtf>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Thu, 18 Jun 2020 19:39:12 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="f6371d327e8300472896de9747942930";
logging-data="11015"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19UEVqSBqcwxBI76ikVCeRp"
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101
Thunderbird/68.9.0
Cancel-Lock: sha1:F4fI+wENmssGs1+ALI13fEgq2TQ=
In-Reply-To: <eli$2006171800@qaz.wtf>
Content-Language: en-US
View all headers
On 6/17/2020 5:02 PM, Eli the Bearded wrote:
In comp.compression,  <sundaresh.venugopal@gmail.com> wrote:
I started coding but shelved it and put it in the background since l
am also simultaneously developing and coding other s/w as well. But
why do you ask ? I am quite certain there is nothing wrong with the
method.

This group is littered with the dead claims of people who said they had
a better compression method but could never prove it with code.

Elijah
------
"compression" seems to attrack kooks as much as "perpetual motion"


Yep.


Luckily at least, data compression is more objectively measurable.


Things like audio and video compression also involve a lot of subjective tradeoffs, like how do the artifacts of one codec look or sound relative to another, and I have noted that it seems my perceptions don't really agree with others.

Eg: I seem to prefer a loss of dynamic range and an increase in "sharpness" a lot more readily than a loss of detail and blurring; whereas many other people would apparently much rather have a blurry mess than sharp high-contrast edges.

Similarly for audio: I prefer codecs which decay into a "metallic" sound over ones which introduce a bunch of warbling and whistling at lower bitrates.

Eg: MP3 below ~ 60 kbps sounds a bit unpleasant with all the whistling and warbling and similar, whereas 11KHz ADPCM still sounds pretty reasonable in comparison.

The result of this has partly been a lot of my own codecs using Color-Cell and VQ style technology, along with ADPCM style audio codecs, which apparently a lot of other people perceive as looking and sounding awful.



But, yeah, as for data compression, don't have all that much notable.

I had a few compressors, which did pretty good (vs, eg, LZ4 and Zstd) at being fast on my past computers (K10 and FX), but after switching to a Ryzen then both LZ4 and Zstd got a pretty significant speed-up.

Getting high compression ratios is, granted, a little harder...


Recently, managed to pull off something which gets slightly better compression at comparable decode speeds to LZ4 (on my Ryzen), and is faster to decode than LZ4 on my own ISA (BJX2).

Decode speeds on a 3.7 GHz Ryzen in both cases is a little over 2GB/sec.
I haven't yet gotten around to running tests on ARM.


Compression for many files seems to be part-way between LZ4 and Deflate.
Note that, like LZ4, it is byte oriented with no entropy coding.

<--

BtRP2 (Transposed, LE):
*                   dddddddd-dlllrrr0 (l=3..10, d=0..511, r=0..7)
*          dddddddd-dddddlll-lllrrr01 (l=4..67, d=0..8191)
* dddddddd-dddddddd-dlllllll-llrrr011 (l=4..515, d=0..131071)
*                            rrrr0111 (Raw Bytes, r=(r+1)*8, 8..128)
*                          * rrr01111 (Long Match)
*                            rr011111 (r=1..3 bytes, 0=EOB)
*                   rrrrrrrr-r0111111 (Long Raw, r=(r+1)*8, 8..4096)
** d: Distance
** l: Match Length
** r: Literal Length

Values are encoded in little-endian order, with tag bits located in the LSB. Bits will be contiguous within the value, with shift-and-mask being used to extract individual elements.

Long Match will encode length and distance using variable-length encodings directly following the initial tag byte.

Length VLN:
           lllllll0,   4..  131
  llllllll-llllll01, 132..16383

Distance VLN:
           dddddddd-ddddddd0, 32K (0..32767)
  dddddddd-dddddddd-dddddd01,  4M

-->


The compression could be improved a little more in the Long Match case via a few more conjoined cases, but the gain in compression would be pretty small relative to the amount of code added (as the first few cases handle the vast majority of LZ matches).

I had also left out adding 1 to the distance, mostly because in my tests this had very little effect on compression, but a somewhat more noticeable effect on decode speed.



Subject: Re: Good compression
From: icepic...@gmail.com
Newsgroups: comp.compression
Date: Thu, 2 Jul 2020 13:38 UTC
References: 1 2 3
X-Received: by 2002:a05:620a:1649:: with SMTP id c9mr7217538qko.330.1593697102621;
Thu, 02 Jul 2020 06:38:22 -0700 (PDT)
X-Received: by 2002:a37:345:: with SMTP id 66mr28640093qkd.272.1593697102291;
Thu, 02 Jul 2020 06:38:22 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Thu, 2 Jul 2020 06:38:21 -0700 (PDT)
In-Reply-To: <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=213.21.126.72; posting-account=O3ek6woAAAA3FgCl-C4iYdhpMZfNNRaG
NNTP-Posting-Host: 213.21.126.72
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com>
Subject: Re: Good compression
From: icepic...@gmail.com
Injection-Date: Thu, 02 Jul 2020 13:38:22 +0000
Content-Type: text/plain; charset="UTF-8"
View all headers
Den torsdag 18 juni 2020 kl. 05:59:27 UTC+2 skrev sundaresh...@gmail.com:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not
allow attachments.

You can always attach it if you use a newsreader instead of the Google
Groups interface (that also puts your IP in the headers by the way).
Don't do it however  since all the free newsservers won't be able to
hold your post thus reducing the visibility and reach. Instead, upload
it to a website like WeTransfer and post the link in the newsgroup.

Good research though! Look forward to learning more.

Well thanks ! One encouraging response . Here is the link

https://drive.google.com/file/d/1w5sNV_YVM9XGymz8A0MT0V_cubxMPYCM/view?usp=sharing

If it does not work, let me know.

It doesn't work.


Subject: Re: Good compression
From: sundares...@gmail.com
Newsgroups: comp.compression
Date: Fri, 3 Jul 2020 03:38 UTC
References: 1 2 3 4
X-Received: by 2002:a37:9ed6:: with SMTP id h205mr32378973qke.384.1593747521851;
Thu, 02 Jul 2020 20:38:41 -0700 (PDT)
X-Received: by 2002:ac8:2601:: with SMTP id u1mr35019859qtu.170.1593747521577;
Thu, 02 Jul 2020 20:38:41 -0700 (PDT)
Path: i2pn2.org!i2pn.org!aioe.org!peer01.ams4!peer.am4.highwinds-media.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Thu, 2 Jul 2020 20:38:41 -0700 (PDT)
In-Reply-To: <8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2401:4900:3307:27f6:c410:6709:13f3:b53;
posting-account=2ecefgoAAACTSQp1lBuI1Iwdq8jeBELE
NNTP-Posting-Host: 2401:4900:3307:27f6:c410:6709:13f3:b53
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
<8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
Subject: Re: Good compression
From: sundares...@gmail.com
Injection-Date: Fri, 03 Jul 2020 03:38:41 +0000
Content-Type: text/plain; charset="UTF-8"
X-Received-Bytes: 2660
X-Received-Body-CRC: 3529542957
View all headers
On Thursday, July 2, 2020 at 7:08:23 PM UTC+5:30, icep...@gmail.com wrote:
Den torsdag 18 juni 2020 kl. 05:59:27 UTC+2 skrev sundaresh...@gmail.com:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not
allow attachments.

You can always attach it if you use a newsreader instead of the Google
Groups interface (that also puts your IP in the headers by the way).
Don't do it however  since all the free newsservers won't be able to
hold your post thus reducing the visibility and reach. Instead, upload
it to a website like WeTransfer and post the link in the newsgroup.

Good research though! Look forward to learning more.

Well thanks ! One encouraging response . Here is the link

https://drive.google.com/file/d/1w5sNV_YVM9XGymz8A0MT0V_cubxMPYCM/view?usp=sharing

If it does not work, let me know.

It doesn't work.

Really, have you tried coding it. If you have, can I see the code ? You can do just what I have done. In the meantime I have made significant improvements to it, and it should be visible in the link and I will be coding this new version.


Subject: Re: Good compression
From: Földes László
Newsgroups: comp.compression
Date: Fri, 3 Jul 2020 10:58 UTC
References: 1 2 3 4 5
X-Received: by 2002:a05:6214:3f3:: with SMTP id cf19mr33943344qvb.94.1593773924932;
Fri, 03 Jul 2020 03:58:44 -0700 (PDT)
X-Received: by 2002:ad4:4732:: with SMTP id l18mr33114582qvz.208.1593773924747;
Fri, 03 Jul 2020 03:58:44 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!3.eu.feeder.erje.net!feeder.erje.net!fdn.fr!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Fri, 3 Jul 2020 03:58:44 -0700 (PDT)
In-Reply-To: <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=80.99.91.197; posting-account=aVFFgAoAAABTsyl0kCVA4QQLS5tdUF3a
NNTP-Posting-Host: 80.99.91.197
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
<8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com> <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <f14d3020-8977-46b7-aaef-c1094797c87fo@googlegroups.com>
Subject: Re: Good compression
From: foldes.l...@gmail.com (Földes László)
Injection-Date: Fri, 03 Jul 2020 10:58:44 +0000
Content-Type: text/plain; charset="UTF-8"
View all headers
On Friday, July 3, 2020 at 5:38:42 AM UTC+2, sundaresh...@gmail.com wrote:
On Thursday, July 2, 2020 at 7:08:23 PM UTC+5:30, icep...@gmail.com wrote:
Den torsdag 18 juni 2020 kl. 05:59:27 UTC+2 skrev sundaresh...@gmail.com:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not
allow attachments.

You can always attach it if you use a newsreader instead of the Google
Groups interface (that also puts your IP in the headers by the way).
Don't do it however  since all the free newsservers won't be able to
hold your post thus reducing the visibility and reach. Instead, upload
it to a website like WeTransfer and post the link in the newsgroup.

Good research though! Look forward to learning more.

Well thanks ! One encouraging response . Here is the link

https://drive.google.com/file/d/1w5sNV_YVM9XGymz8A0MT0V_cubxMPYCM/view?usp=sharing

If it does not work, let me know.

It doesn't work.

Really, have you tried coding it. If you have, can I see the code ? You can do just what I have done. In the meantime I have made significant improvements to it, and it should be visible in the link and I will be coding this new version.

This paper talks about a method with an example applied to a single code but a single code has no entropy (on which compression methods work).


Subject: Re: Good compression
From: icepic...@gmail.com
Newsgroups: comp.compression
Date: Fri, 3 Jul 2020 13:28 UTC
References: 1 2 3 4 5
X-Received: by 2002:a05:6214:a6c:: with SMTP id ef12mr34734015qvb.74.1593782890713;
Fri, 03 Jul 2020 06:28:10 -0700 (PDT)
X-Received: by 2002:a0c:83a5:: with SMTP id k34mr34600474qva.130.1593782890484;
Fri, 03 Jul 2020 06:28:10 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Fri, 3 Jul 2020 06:28:10 -0700 (PDT)
In-Reply-To: <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=213.21.126.72; posting-account=O3ek6woAAAA3FgCl-C4iYdhpMZfNNRaG
NNTP-Posting-Host: 213.21.126.72
References: <1e306461-e8bc-4c34-b833-4c07753dcd28o@googlegroups.com>
<rce3ml$2taf$1@esteban.tulanet.com> <7158ca1b-2e13-425d-8528-379adedf0768o@googlegroups.com>
<8c45e43a-3b74-4490-a804-8fd9670b0f34o@googlegroups.com> <6a34c070-03e9-464f-8ac8-8059909d3642o@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <4b347e80-5c96-428b-b663-ecc303588c94o@googlegroups.com>
Subject: Re: Good compression
From: icepic...@gmail.com
Injection-Date: Fri, 03 Jul 2020 13:28:10 +0000
Content-Type: text/plain; charset="UTF-8"
View all headers
Den fredag 3 juli 2020 kl. 05:38:42 UTC+2 skrev sundaresh...@gmail.com:
On Thursday, July 2, 2020 at 7:08:23 PM UTC+5:30, icep...@gmail.com wrote:
Den torsdag 18 juni 2020 kl. 05:59:27 UTC+2 skrev sundaresh...@gmail.com:

The web site is www.complimentaryware.unaux.com . I have an excellent method on
PDF, an unpublished research of my own but this newsgroup does not
allow attachments.

You can always attach it if you use a newsreader instead of the Google
Groups interface (that also puts your IP in the headers by the way).
Don't do it however  since all the free newsservers won't be able to
hold your post thus reducing the visibility and reach. Instead, upload
it to a website like WeTransfer and post the link in the newsgroup.

Good research though! Look forward to learning more.

Well thanks ! One encouraging response . Here is the link

https://drive.google.com/file/d/1w5sNV_YVM9XGymz8A0MT0V_cubxMPYCM/view?usp=sharing

If it does not work, let me know.

It doesn't work.

Really, have you tried coding it. If you have, can I see the code ? You can do just what I have done. In the meantime I have made significant improvements to it, and it should be visible in the link and I will be coding this new version.

Please post the code you have then, and the results.

You are the one making a claim that a large number M can be written as a smaller number S by transforming it in one of several ways, which the pigeonhole principle will show anyone will not work for all numbers from 0 to M, since they can't all fit in S.

You can find S numbers out of M that do, but even selecting which set of Ses is out of the set 0->M will cost bits.


1
rocksolid light 0.7.2
clearneti2ptor