Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  nodelist  faq  login

If loving linux is wrong, I dont wanna be right. -- Topic for #LinuxGER


computers / comp.compression / Re: i need help in compression algorithm identification

SubjectAuthor
* i need help in compression algorithm identificationtheuswra
`* Re: i need help in compression algorithm identificationHelm
 +- Re: i need help in compression algorithm identificationnews
 `* Re: i need help in compression algorithm identificationwilliamvazmolina1998
  `* Re: i need help in compression algorithm identificationtheuswra
   `- Re: i need help in compression algorithm identificationHelm

1
Subject: i need help in compression algorithm identification
From: theus...@gmail.com
Newsgroups: comp.compression
Date: Tue, 30 Jun 2020 03:51 UTC
X-Received: by 2002:ac8:2f67:: with SMTP id k36mr18700941qta.345.1593489074023;
Mon, 29 Jun 2020 20:51:14 -0700 (PDT)
X-Received: by 2002:ac8:7242:: with SMTP id l2mr18246390qtp.320.1593489073879;
Mon, 29 Jun 2020 20:51:13 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Mon, 29 Jun 2020 20:51:13 -0700 (PDT)
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=138.99.85.10; posting-account=GeslqAoAAABaQFs3YR1SWTzgHoEKDafe
NNTP-Posting-Host: 138.99.85.10
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <583fd5ab-3a52-4983-b071-570f28e86f57o@googlegroups.com>
Subject: i need help in compression algorithm identification
From: theus...@gmail.com
Injection-Date: Tue, 30 Jun 2020 03:51:14 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
View all headers
okay so, i'm modding a game which has compressed files, as far i could see, it has apparently 2 "compression patterns", files starting with "00 05" and "00 07". I was able to get some samples (compressed files) and the respective decompressed ones, (through the in-game memory the decompressed ones are accessible). i'm asking for help because i have no experience in that matter, and i hope someone could help me.

samples: https://drive.google.com/file/d/1slzoYxqr3Egg3Z7GCKlayTNwCR7OZ2Yg/view?usp=sharing


Subject: Re: i need help in compression algorithm identification
From: Helm
Newsgroups: comp.compression
Organization: Tula Networks
Date: Tue, 30 Jun 2020 13:35 UTC
References: 1
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!tulanet.com!esteban.tulanet.com!.POSTED!not-for-mail
From: nos...@please.invalid (Helm)
Newsgroups: comp.compression
Subject: Re: i need help in compression algorithm identification
Date: Tue, 30 Jun 2020 09:35:50 -0400
Organization: Tula Networks
Message-ID: <rdff3m$t78$1@esteban.tulanet.com>
References: <583fd5ab-3a52-4983-b071-570f28e86f57o@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: esteban.tulanet.com; logging-data="29928"; posting-host="cXiXsH0CBSQ/sLT275Qb0g.user.esteban.tulanet.com"; mail-complaints-to="netnewsmaster@gmail.com";
User-Agent: Unison/2.2
X-Notice: Filtered by Postfilter with Rennovations
View all headers
On 2020-06-30 03:51:13 +0000, theuswra@gmail.com said:

okay so, i'm modding a game which has compressed files, as far i could see, it has apparently 2 "compression patterns", files starting with "00 05" and "00 07". I was able to get some samples (compressed files) and the respective decompressed ones, (through the in-game memory the decompressed ones are accessible). i'm asking for help because i have no experience in that matter, and i hope someone could help me.

samples: https://drive.google.com/file/d/1slzoYxqr3Egg3Z7GCKlayTNwCR7OZ2Yg/view?usp=sharing

First off; what game is this? If it runs on a certain engine that can give us clues. I know of certain engines  that have their compression scheme well documented. Otherwise you will have to reverse engineer the decompress function to find out how it does it - probably some DEFLATE/LZW stuff . With the decompressor code (which is obviously present), and knowledege of the compression method used/compression methods in general, you should be able to create a (probably very inefficent) compressor.

Luckily for you, most game developers are not compression experts and looking at the compression ratio its probably something very simple.

I assume you know much about reverse engineering since you are modding a game, but you should take some time to read up on Huffman encoding etc. so you understand the code.

Second of all; when you post on Google Groups, it places your IP address and your email in the headers. Consider downloading a newsreader and using the free newsserver aioe.org (no registration required) to maintain your privacy and prevent your IP from leaking out. In addition, many people on Usenet have Google Groups killfiled, so some wont see this post until someone who doesn't use google groups (like me) reply to it. The only time I use google groups is for reading Usenet on the go since iOS has no good newsreader (even paid newstap is crap) or when I need to send someone a link to a Usenet post. I will always post using my newsreader instead.



Subject: Re: i need help in compression algorithm identification
From: new...@zzo38computer.org.invalid
Newsgroups: comp.compression
Organization: Aioe.org NNTP Server
Date: Tue, 30 Jun 2020 20:14 UTC
References: 1 2
Path: i2pn2.org!i2pn.org!aioe.org!.POSTED.iSqskUzK2KPWtZfZ6bkvfw.user.gioia.aioe.org!not-for-mail
From: new...@zzo38computer.org.invalid
Newsgroups: comp.compression
Subject: Re: i need help in compression algorithm identification
Date: Tue, 30 Jun 2020 13:14:03 -0700
Organization: Aioe.org NNTP Server
Lines: 37
Message-ID: <1593543511.bystand@zzo38computer.org>
References: <583fd5ab-3a52-4983-b071-570f28e86f57o@googlegroups.com> <rdff3m$t78$1@esteban.tulanet.com>
NNTP-Posting-Host: iSqskUzK2KPWtZfZ6bkvfw.user.gioia.aioe.org
Mime-Version: 1.0
X-Complaints-To: abuse@aioe.org
User-Agent: bystand/1.1.2pre1
X-Notice: Filtered by postfilter v. 0.9.2
View all headers
Helm <nospam@please.invalid> wrote:
Luckily for you, most game developers are not compression experts and
looking at the compression ratio its probably something very simple.

What I know is that I tended to make my own compression for the game,
which is sometimes such that unsolvable levels sometimes cannot be
represented in the compressed format at all. Note, however, that in some
games, uncompressing random data will not necessarily result in a solvable
level; in other games, it does. (Do any other games do this? How common is
it? I am curious to know. I think I read that at least Gruniozerca 3 does
something similar; the compression algorithm relies on the solution, so it
is also easily possible to determine the solution from the compressed
level data, too.) (I remember actually that in one computer game I wrote,
uncompressing random data (if it is sufficiently long) always does result
in a solvable level, and that is actually how the "random levels" mode of
the game works; it uncompresses random data (based on a random number seed
entered by the user, so that you can easily find and access the same level
later). The solution is also easily derived from uncompressing it, so it
also has solutions, too.)

[...]                  The only time I use google groups is for reading
Usenet on the go since iOS has no good newsreader (even paid newstap is
crap) or when I need to send someone a link to a Usenet post. I will
always post using my newsreader instead.

My suggestion is when using Google Groups to link to a Usenet post, to
always use the URL based on the message ID. Not everyone uses Google
Groups, and using the URL based on the message ID makes it helpful for
those who do not use Google Groups. (I do not use Google Groups myself.)

(I can also perhaps suggest trying to write a better newsreader for iOS,
if a suitable one does not already exist. Maybe it is possible to use SSH
to access it on another computer, though?)

--
This signature intentionally left blank.
(But if it has these words, then actually it isn't blank, isn't it?)


Subject: Re: i need help in compression algorithm identification
From: williamv...@gmail.com
Newsgroups: comp.compression
Date: Wed, 1 Jul 2020 05:12 UTC
References: 1 2
X-Received: by 2002:a37:4903:: with SMTP id w3mr23058594qka.178.1593580361195;
Tue, 30 Jun 2020 22:12:41 -0700 (PDT)
X-Received: by 2002:a37:345:: with SMTP id 66mr22217556qkd.272.1593580361041;
Tue, 30 Jun 2020 22:12:41 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Tue, 30 Jun 2020 22:12:40 -0700 (PDT)
In-Reply-To: <rdff3m$t78$1@esteban.tulanet.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=179.157.76.2; posting-account=3j9ubQoAAACsABRaLrS8Sds0gm9pQoab
NNTP-Posting-Host: 179.157.76.2
References: <583fd5ab-3a52-4983-b071-570f28e86f57o@googlegroups.com> <rdff3m$t78$1@esteban.tulanet.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <58f35ff5-847d-419c-a228-d96ea5cac6cco@googlegroups.com>
Subject: Re: i need help in compression algorithm identification
From: williamv...@gmail.com
Injection-Date: Wed, 01 Jul 2020 05:12:41 +0000
Content-Type: text/plain; charset="UTF-8"
View all headers
The game is: "Naruto Uzumaki Chronicles 2" (PS2)


Subject: Re: i need help in compression algorithm identification
From: theus...@gmail.com
Newsgroups: comp.compression
Date: Wed, 1 Jul 2020 07:19 UTC
References: 1 2 3
X-Received: by 2002:aed:3048:: with SMTP id 66mr22954081qte.326.1593587995648;
Wed, 01 Jul 2020 00:19:55 -0700 (PDT)
X-Received: by 2002:a0c:ec04:: with SMTP id y4mr20283344qvo.148.1593587995523;
Wed, 01 Jul 2020 00:19:55 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder7.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.compression
Date: Wed, 1 Jul 2020 00:19:55 -0700 (PDT)
In-Reply-To: <58f35ff5-847d-419c-a228-d96ea5cac6cco@googlegroups.com>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=186.250.12.55; posting-account=GeslqAoAAABaQFs3YR1SWTzgHoEKDafe
NNTP-Posting-Host: 186.250.12.55
References: <583fd5ab-3a52-4983-b071-570f28e86f57o@googlegroups.com>
<rdff3m$t78$1@esteban.tulanet.com> <58f35ff5-847d-419c-a228-d96ea5cac6cco@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <b104786d-9518-41c6-9d90-42147e0e35d6o@googlegroups.com>
Subject: Re: i need help in compression algorithm identification
From: theus...@gmail.com
Injection-Date: Wed, 01 Jul 2020 07:19:55 +0000
Content-Type: text/plain; charset="UTF-8"
View all headers
The game is Naruto Uzumaki Chronicles 2 (PS2) by Racjin, btw they use the same compression format for all their games (the games with compressed files, ofc.) and i run it with pcsx2.

Also i dont think that recompress the files is necessary tbh, i just need to mass decompress them without having to do it manually, file by file through memory.

Sorry for any english mistakes, i'm not a native english speaker.


Subject: Re: i need help in compression algorithm identification
From: Helm
Newsgroups: comp.compression
Organization: Tula Networks
Date: Wed, 1 Jul 2020 11:05 UTC
References: 1
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!tulanet.com!esteban.tulanet.com!.POSTED!not-for-mail
From: nos...@please.innvalid (Helm)
Newsgroups: comp.compression
Subject: Re: i need help in compression algorithm identification
Date: Wed, 1 Jul 2020 07:05:04 -0400
Organization: Tula Networks
Message-ID: <rdhql0$1kl3$1@esteban.tulanet.com>
References: <b104786d-9518-41c6-9d90-42147e0e35d6o@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: esteban.tulanet.com; logging-data="53923"; posting-host="jAxNjG8bwRjbl4qCZzv4Pg.user.esteban.tulanet.com"; mail-complaints-to="netnewsmaster@gmail.com";
User-Agent: Unison/2.2
X-Notice: Filtered by Postfilter with Rennovations
View all headers
On 2020-07-01 07:19:55 +0000, theuswra@gmail.com said:

The game is Naruto Uzumaki Chronicles 2 (PS2) by Racjin, btw they use the same compression format for all their games (the games with compressed files, ofc.) and i run it with pcsx2.

Also i dont think that recompress the files is necessary tbh, i just need to mass decompress them without having to do it manually, file by file through memory.

Sorry for any english mistakes, i'm not a native english speaker.

I had a feeling by looking in the files. MIPS is not my cup of tea. Well if you just need to mass decompress them then you still need to figure out the algorithm..... I dont think the ps2 makes use of ASLR so it should be fairly straightforward to find out where it will load the decompressed file, trace memory and go from there (dynamic analysis)... without any function names, it gets harder. Maybe look for all syscalls to open a file and see if you can get the decompression method.


I hear there was a PC version, if they were using the same engine then probably the same compression method was in effect, I don't know if Windows or PS2 reversing is easier for you.




1
rocksolid light 0.7.2
clearneti2ptor