Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

Linux is obsolete (Andrew Tanenbaum)


devel / comp.arch.embedded / Re: Embedding a Checksum in an Image File

SubjectAuthor
* Embedding a Checksum in an Image FileRick C
+* Re: Embedding a Checksum in an Image FileNiklas Holsti
|`- Re: Embedding a Checksum in an Image FileRick C
+- Re: Embedding a Checksum in an Image FilePeter Heitzer
+* Re: Embedding a Checksum in an Image Filedalai lamah
|`- Re: Embedding a Checksum in an Image FileRick C
+- Re: Embedding a Checksum in an Image FileDavid Brown
+* Re: Embedding a Checksum in an Image FileGeorge Neuner
|+* Re: Embedding a Checksum in an Image FileRick C
||+* Re: Embedding a Checksum in an Image FileDavid Brown
|||`* Re: Embedding a Checksum in an Image FileUlf Samuelsson
||| `- Re: Embedding a Checksum in an Image FileUlf Samuelsson
||`* Re: Embedding a Checksum in an Image FileGeorge Neuner
|| +* Re: Embedding a Checksum in an Image FileDon Y
|| |`* Re: Embedding a Checksum in an Image FileDavid Brown
|| | `* Re: Embedding a Checksum in an Image FileDon Y
|| |  `* Re: Embedding a Checksum in an Image FileDavid Brown
|| |   `* Re: Embedding a Checksum in an Image FileDon Y
|| |    `* Re: Embedding a Checksum in an Image FileDavid Brown
|| |     `* Re: Embedding a Checksum in an Image FileDon Y
|| |      `* Re: Embedding a Checksum in an Image FileDavid Brown
|| |       `* Re: Embedding a Checksum in an Image FileDon Y
|| |        `* Re: Embedding a Checksum in an Image FileDavid Brown
|| |         +* Re: Embedding a Checksum in an Image FileUlf Samuelsson
|| |         |`* Re: Embedding a Checksum in an Image FileDavid Brown
|| |         | `* Re: Embedding a Checksum in an Image FileUlf Samuelsson
|| |         |  `- Re: Embedding a Checksum in an Image FileDon Y
|| |         `- Re: Embedding a Checksum in an Image FileDon Y
|| `- Re: Embedding a Checksum in an Image FileStefan Reuther
|`* Re: Embedding a Checksum in an Image FileTauno Voipio
| `- Re: Embedding a Checksum in an Image FileGeorge Neuner
+* Re: Embedding a Checksum in an Image FileRichard Damon
|`* Re: Embedding a Checksum in an Image FileRick C
| `- Re: Embedding a Checksum in an Image FileRichard Damon
+* Re: Embedding a Checksum in an Image FileBrian Cockburn
|`* Re: Embedding a Checksum in an Image FileRick C
| +* Re: Embedding a Checksum in an Image FileDavid Brown
| |+* Re: Embedding a Checksum in an Image FileBrian Cockburn
| ||`- Re: Embedding a Checksum in an Image FileDavid Brown
| |`* Re: Embedding a Checksum in an Image FileRick C
| | +* Re: Embedding a Checksum in an Image FileDavid Brown
| | |`* Re: Embedding a Checksum in an Image FileRick C
| | | `* Re: Embedding a Checksum in an Image FileDavid Brown
| | |  +* Re: Embedding a Checksum in an Image FileGrant Edwards
| | |  |`* Re: Embedding a Checksum in an Image FileDavid Brown
| | |  | `* Re: Embedding a Checksum in an Image FileGrant Edwards
| | |  |  `* Re: Embedding a Checksum in an Image FileDavid Brown
| | |  |   `* Re: Embedding a Checksum in an Image FileRichard Damon
| | |  |    `- Re: Embedding a Checksum in an Image FileDavid Brown
| | |  +- Re: Embedding a Checksum in an Image FileboB
| | |  `* Re: Embedding a Checksum in an Image FileRick C
| | |   `* Re: Embedding a Checksum in an Image FileDavid Brown
| | |    `* Re: Embedding a Checksum in an Image FileRick C
| | |     `* Re: Embedding a Checksum in an Image FileDavid Brown
| | |      `- Re: Embedding a Checksum in an Image FileRick C
| | `* Re: Embedding a Checksum in an Image FileUlf Samuelsson
| |  `- Re: Embedding a Checksum in an Image FileUlf Samuelsson
| `* Re: Embedding a Checksum in an Image FileBrian Cockburn
|  `* Re: Embedding a Checksum in an Image FileRick C
|   `* Re: Embedding a Checksum in an Image FileBrian Cockburn
|    +- Re: Embedding a Checksum in an Image FileRichard Damon
|    `- Re: Embedding a Checksum in an Image FileRick C
+* Re: Embedding a Checksum in an Image FileUlf Samuelsson
|+* Re: Embedding a Checksum in an Image FileRick C
||+* Re: Embedding a Checksum in an Image FileNiklas Holsti
|||+- Re: Embedding a Checksum in an Image FileNiklas Holsti
|||`* Re: Embedding a Checksum in an Image FileUlf Samuelsson
||| `* Re: Embedding a Checksum in an Image FileUlf Samuelsson
|||  `* Re: Embedding a Checksum in an Image FileDavid Brown
|||   `* Re: Embedding a Checksum in an Image FileUlf Samuelsson
|||    `* Re: Embedding a Checksum in an Image FileDavid Brown
|||     `* Re: Embedding a Checksum in an Image FileUlf Samuelsson
|||      `- Re: Embedding a Checksum in an Image FileDavid Brown
||`* Re: Embedding a Checksum in an Image FileUlf Samuelsson
|| `* Re: Embedding a Checksum in an Image FileNiklas Holsti
||  `- Re: Embedding a Checksum in an Image FileUlf Samuelsson
|`* Re: Embedding a Checksum in an Image FileUlf Samuelsson
| `* Re: Embedding a Checksum in an Image FileDavid Brown
|  `- Re: Embedding a Checksum in an Image FileUlf Samuelsson
`- Re: Embedding a Checksum in an Image FileUlf Samuelsson

Pages:1234
Embedding a Checksum in an Image File

<116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1436&group=comp.arch.embedded#1436

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:a05:622a:d2:b0:3ef:309c:521e with SMTP id p18-20020a05622a00d200b003ef309c521emr199936qtw.3.1681956393799;
Wed, 19 Apr 2023 19:06:33 -0700 (PDT)
X-Received: by 2002:ac8:5850:0:b0:3e6:720f:bb02 with SMTP id
h16-20020ac85850000000b003e6720fbb02mr217103qth.8.1681956393570; Wed, 19 Apr
2023 19:06:33 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Wed, 19 Apr 2023 19:06:33 -0700 (PDT)
Injection-Info: google-groups.googlegroups.com; posting-host=198.245.101.90; posting-account=I-_H_woAAAA9zzro6crtEpUAyIvzd19b
NNTP-Posting-Host: 198.245.101.90
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
Subject: Embedding a Checksum in an Image File
From: gnuarm.d...@gmail.com (Rick C)
Injection-Date: Thu, 20 Apr 2023 02:06:33 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
 by: Rick C - Thu, 20 Apr 2023 02:06 UTC

This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.

I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.

I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.

I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.

I keep thinking there is a different way of looking at this to achieve the result I want...

Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact..

This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.

--

Rick C.

- Get 1,000 miles of free Supercharging
- Tesla referral code - https://ts.la/richard11209

Re: Embedding a Checksum in an Image File

<kace4iFke6vU1@mid.individual.net>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1437&group=comp.arch.embedded#1437

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: niklas.h...@tidorum.invalid (Niklas Holsti)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 12:14:58 +0300
Organization: Tidorum Ltd
Lines: 28
Message-ID: <kace4iFke6vU1@mid.individual.net>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: individual.net ZqUEAXP20YhQYPgIDS8TsgLSzYgTQApO8mrhtlJCvq4ueaYJ+a
Cancel-Lock: sha1:QOwEjcFDyssJH9Aklz1DWuFhzV4=
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:102.0)
Gecko/20100101 Thunderbird/102.6.1
Content-Language: en-US
In-Reply-To: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
 by: Niklas Holsti - Thu, 20 Apr 2023 09:14 UTC

On 2023-04-20 5:06, Rick C wrote:
> This is a bit of the chicken and egg thing. If you want a embed a
> checksum in a code module to report the checksum, is there a way of
> doing this? It's a bit like being your own grandfather, I think.
>
> I'm not thinking anything too fancy, like a CRC, but rather a simple
> modulo N addition, maybe N being 2^16.

Some decades ago I was involved with a project for an 8052-based device,
which was required to perform a code-check-sum check at boot.

We decided to use a byte-per-byte xor checksum and make the correct
check-sum be zero. We had a code module (possibly in assembler, I don't
remember) that defined a one-byte "adjustment" constant in code memory.
For each new version of the code, we first set the adjustment constant
to zero, then ran the program, and it usually reported an error at boot
because the check-sum was not zero. We then changed the adjustment
constant to the actual reported checksum, C say, and that zeroed the
check-sum because C xor C = 0. Bingo. You can use this method to make
the checksum anything you like, for example hex 55.

With a more advanced order-sensitive check-sum such as a CRC you could
use the same method if you also ensure (by linker commands) that the
adjustment value is always the last value that enters in the computed
check-sum (assuming that the linking order of the other code modules is
not incidentally changed when the value of the adjustment constant is
changed).

Re: Embedding a Checksum in an Image File

<kacm2qFlk42U1@mid.individual.net>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1438&group=comp.arch.embedded#1438

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.mixmin.net!news2.arglkargh.de!news.karotte.org!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: peter.he...@rz.uni-regensburg.de (Peter Heitzer)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: 20 Apr 2023 11:30:34 GMT
Lines: 15
Message-ID: <kacm2qFlk42U1@mid.individual.net>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
X-Trace: individual.net FIssSCLN5/tAOWuWVcBF/AzdiU0P81IZpmKQKWskL+ggq2PhDcQ1kL0YS5
X-Orig-Path: not-for-mail
Cancel-Lock: sha1:CL5Ge/DQ87BQHNvB0ZWB81NosrA=
User-Agent: tin/2.4.3-20181224 ("Glen Mhor") (UNIX) (Linux/4.19.0-13-amd64 (x86_64))
 by: Peter Heitzer - Thu, 20 Apr 2023 11:30 UTC

Rick C <gnuarm.deletethisbit@gmail.com> wrote:
>This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.

>I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.

What about putting the following structure at a fixed address at the end of
ROM?:
<startaddr><len><checksum>
Your check function then for example does a 16 bit sum of the bytes from
<startaddr>..<startaddr>+<len>-1 and compares with <checksum>
<startaddr>, <len> an <checksum> can be evaluated at compile time.

--
Dipl.-Inform(FH) Peter Heitzer, peter.heitzer@rz.uni-regensburg.de

Re: Embedding a Checksum in an Image File

<5oqjhwwdrcir.1fwtjz4f5xour.dlg@40tude.net>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1439&group=comp.arch.embedded#1439

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.imp.ch!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: antonio1...@hotmail.com (dalai lamah)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 13:47:48 +0200
Lines: 37
Message-ID: <5oqjhwwdrcir.1fwtjz4f5xour.dlg@40tude.net>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: 8bit
X-Trace: individual.net WkfieNFwW57Zm//RPP9AewN+O7ZaMNmtrIjtpmsC4OSKSQfNCs
Cancel-Lock: sha1:v9ZC1mzCYwNQ+1Nawiee2eYoc3s=
User-Agent: 40tude_Dialog/2.0.15.1it
X-Face: #^0cuOKw--#Q:"y/,&C;}2$_tSLc]PEJ^{cm%=omYXmfR"cSLu`<R:LRUq0q8W)T~e20'Ri gN/ct+i_K+F~-u0lXixh"|ga@w^:D?@OdallB_O(Bg<4m39>9oxl2!tVF1/w
 by: dalai lamah - Thu, 20 Apr 2023 11:47 UTC

Un bel giorno Rick C digitò:

> This is a bit of the chicken and egg thing. If you want a embed a
> checksum in a code module to report the checksum, is there a way of
> doing this? It's a bit like being your own grandfather, I think.
>
> I'm not thinking anything too fancy, like a CRC, but rather a simple
> modulo N addition, maybe N being 2^16.
>
> I keep thinking of using a placeholder, but that doesn't seem to work
> out in any useful way. Even if you try to anticipate the impact of
> adding the checksum, that only gives you a different checksum, that you
> then need to anticipate further... ad infinitum.

I'm probably not understanding what you mean, but normally the checksum is
stored in a memory section which is not subjected to the checksum
calculation itself.

The actual implementation depends on the tools you are using. Many linkers
support this directly: you specify the memory section(s) subjected to
checksum calculation, the type of checksum (CRC16, CRC32 etc) and the
memory section that will store the checksum.

Here is a technical note for IAR:
https://www.iar.com/knowledge/support/technical-notes/general/checksum-calculation-with-xlink/

A "poor man" solution is to do it manually:

-In the source code, declare your checksum initializing to a known, fixed
value (e.g. 0xDEADBEEF)
-Run the program with a debugger; set a breakpoint when it calculates the
checksum (and fails), and write down the correct checksum
-Using a binary editor, find the fixed value into the executable binary,
and replace it with the correct value.

--
Fletto i muscoli e sono nel vuoto.

Re: Embedding a Checksum in an Image File

<ab28a285-2a16-4f43-840b-d0b749cdbc6fn@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1440&group=comp.arch.embedded#1440

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:ae9:f809:0:b0:74e:324:d6f0 with SMTP id x9-20020ae9f809000000b0074e0324d6f0mr183522qkh.7.1681995861608;
Thu, 20 Apr 2023 06:04:21 -0700 (PDT)
X-Received: by 2002:a05:6214:b03:b0:5ef:480b:78e2 with SMTP id
u3-20020a0562140b0300b005ef480b78e2mr187250qvj.9.1681995861267; Thu, 20 Apr
2023 06:04:21 -0700 (PDT)
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Thu, 20 Apr 2023 06:04:21 -0700 (PDT)
In-Reply-To: <5oqjhwwdrcir.1fwtjz4f5xour.dlg@40tude.net>
Injection-Info: google-groups.googlegroups.com; posting-host=198.245.101.90; posting-account=I-_H_woAAAA9zzro6crtEpUAyIvzd19b
NNTP-Posting-Host: 198.245.101.90
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com> <5oqjhwwdrcir.1fwtjz4f5xour.dlg@40tude.net>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <ab28a285-2a16-4f43-840b-d0b749cdbc6fn@googlegroups.com>
Subject: Re: Embedding a Checksum in an Image File
From: gnuarm.d...@gmail.com (Rick C)
Injection-Date: Thu, 20 Apr 2023 13:04:21 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 3849
 by: Rick C - Thu, 20 Apr 2023 13:04 UTC

On Thursday, April 20, 2023 at 7:47:54 AM UTC-4, dalai lamah wrote:
> Un bel giorno Rick C digitò:
> > This is a bit of the chicken and egg thing. If you want a embed a
> > checksum in a code module to report the checksum, is there a way of
> > doing this? It's a bit like being your own grandfather, I think.
> >
> > I'm not thinking anything too fancy, like a CRC, but rather a simple
> > modulo N addition, maybe N being 2^16.
> >
> > I keep thinking of using a placeholder, but that doesn't seem to work
> > out in any useful way. Even if you try to anticipate the impact of
> > adding the checksum, that only gives you a different checksum, that you
> > then need to anticipate further... ad infinitum.
> I'm probably not understanding what you mean, but normally the checksum is
> stored in a memory section which is not subjected to the checksum
> calculation itself.

Yes, I didn't explain it clearly. I am not looking for a way to calculate the checksum from a processor. That would be trivial. I want to embed the checksum in the code, so that it can be provided at run time as an ID, a way to validate the version number.

> The actual implementation depends on the tools you are using. Many linkers
> support this directly: you specify the memory section(s) subjected to
> checksum calculation, the type of checksum (CRC16, CRC32 etc) and the
> memory section that will store the checksum.

I wish to perform this checksum on the executable file.

> Here is a technical note for IAR:
> https://www.iar.com/knowledge/support/technical-notes/general/checksum-calculation-with-xlink/
>
> A "poor man" solution is to do it manually:
>
> -In the source code, declare your checksum initializing to a known, fixed
> value (e.g. 0xDEADBEEF)
> -Run the program with a debugger; set a breakpoint when it calculates the
> checksum (and fails), and write down the correct checksum
> -Using a binary editor, find the fixed value into the executable binary,
> and replace it with the correct value.

Yeah, this is not useful, because changing the value stored changes the checksum. It also makes assumptions about the target.

Maybe this was not the best group to ask the question in. I thought this was more of a math problem with I started writing the question and the embedded community had already dealt with it.

--

Rick C.

+ Get 1,000 miles of free Supercharging
+ Tesla referral code - https://ts.la/richard11209

Re: Embedding a Checksum in an Image File

<f4e757c5-31e2-45bd-aaef-34a944a90ebdn@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1441&group=comp.arch.embedded#1441

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:a05:6214:9b3:b0:5ef:181e:2e53 with SMTP id du19-20020a05621409b300b005ef181e2e53mr198013qvb.10.1681996706105;
Thu, 20 Apr 2023 06:18:26 -0700 (PDT)
X-Received: by 2002:ac8:5c8d:0:b0:3e6:71d6:5d51 with SMTP id
r13-20020ac85c8d000000b003e671d65d51mr368271qta.4.1681996705852; Thu, 20 Apr
2023 06:18:25 -0700 (PDT)
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Thu, 20 Apr 2023 06:18:25 -0700 (PDT)
In-Reply-To: <kace4iFke6vU1@mid.individual.net>
Injection-Info: google-groups.googlegroups.com; posting-host=198.245.101.90; posting-account=I-_H_woAAAA9zzro6crtEpUAyIvzd19b
NNTP-Posting-Host: 198.245.101.90
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com> <kace4iFke6vU1@mid.individual.net>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <f4e757c5-31e2-45bd-aaef-34a944a90ebdn@googlegroups.com>
Subject: Re: Embedding a Checksum in an Image File
From: gnuarm.d...@gmail.com (Rick C)
Injection-Date: Thu, 20 Apr 2023 13:18:26 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 4170
 by: Rick C - Thu, 20 Apr 2023 13:18 UTC

On Thursday, April 20, 2023 at 5:15:04 AM UTC-4, Niklas Holsti wrote:
> On 2023-04-20 5:06, Rick C wrote:
> > This is a bit of the chicken and egg thing. If you want a embed a
> > checksum in a code module to report the checksum, is there a way of
> > doing this? It's a bit like being your own grandfather, I think.
> >
> > I'm not thinking anything too fancy, like a CRC, but rather a simple
> > modulo N addition, maybe N being 2^16.
> Some decades ago I was involved with a project for an 8052-based device,
> which was required to perform a code-check-sum check at boot.
>
> We decided to use a byte-per-byte xor checksum and make the correct
> check-sum be zero. We had a code module (possibly in assembler, I don't
> remember) that defined a one-byte "adjustment" constant in code memory.
> For each new version of the code, we first set the adjustment constant
> to zero, then ran the program, and it usually reported an error at boot
> because the check-sum was not zero. We then changed the adjustment
> constant to the actual reported checksum, C say, and that zeroed the
> check-sum because C xor C = 0. Bingo. You can use this method to make
> the checksum anything you like, for example hex 55.
>
> With a more advanced order-sensitive check-sum such as a CRC you could
> use the same method if you also ensure (by linker commands) that the
> adjustment value is always the last value that enters in the computed
> check-sum (assuming that the linking order of the other code modules is
> not incidentally changed when the value of the adjustment constant is
> changed).

Yes, it had occurred to me that a simple checksum could be used with adjustment codes. But I don't want the checksum to be set to some value, in this way. I would like to embed the check sum generated from the file. The way to do this is to embed the checksum in the spot where it can be read for reporting. Then another value can be embedded elsewhere, that complements the checksum, keeping the file checksum constant.

Your mention of the XOR checksum makes me realize that if I use addition, rather than XOR, a 16 bit checksum only has a complement if the data used in the calculation are 16 bit quantities. If the 16 bit checksum is calculated using 8 bit data, there will be a carry out of the lower 8 bits changing the final checksum. The XOR checksum is really the equivalent of 8 separate bit level checksums. This has the short coming of one bit detection, but two bit changes in the same bit of two bytes not being detected. But since I'm not trying to protect against changes, this isn't really a problem. I'm using this as a verification of the version number.

--

Rick C.

-- Get 1,000 miles of free Supercharging
-- Tesla referral code - https://ts.la/richard11209

Re: Embedding a Checksum in an Image File

<u1rj8b$kpek$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1442&group=comp.arch.embedded#1442

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: david.br...@hesbynett.no (David Brown)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 16:46:35 +0200
Organization: A noiseless patient Spider
Lines: 49
Message-ID: <u1rj8b$kpek$1@dont-email.me>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Thu, 20 Apr 2023 14:46:35 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="32ed44ad5f461f1f317cfb44e64d0860";
logging-data="681428"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1//+FOlQPlqp5IggEV/PILJQ5wiE0j+cSY="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.9.0
Cancel-Lock: sha1:qJ8P/TumQGZJosnavrvOzAtky/g=
In-Reply-To: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
Content-Language: en-GB
 by: David Brown - Thu, 20 Apr 2023 14:46 UTC

On 20/04/2023 04:06, Rick C wrote:
> This is a bit of the chicken and egg thing. If you want a embed a
> checksum in a code module to report the checksum, is there a way of
> doing this? It's a bit like being your own grandfather, I think.
>
> I'm not thinking anything too fancy, like a CRC, but rather a simple
> modulo N addition, maybe N being 2^16.
>
> I keep thinking of using a placeholder, but that doesn't seem to work
> out in any useful way. Even if you try to anticipate the impact of
> adding the checksum, that only gives you a different checksum, that
> you then need to anticipate further... ad infinitum.
>
> I'm not thinking of any special checksum generator that excludes the
> checksum data. That would be too messy.
>
> I keep thinking there is a different way of looking at this to
> achieve the result I want...
>
> Maybe I can prove it is impossible. Assume the file checksums to X
> when the checksum data is zero. The goal would then be to include
> the checksum data value Y in the file, that would change X to Y.
> Given the properties of the module N checksum, this would appear to
> be impossible for the general case, unless... Add another data
> value, called, checksum normalizer. This data value checksums with
> the original checksum to give the result zero. Then, when the
> checksum is also added, the resulting checksum is, in fact, the
> checksum. Another way of looking at this is to add a value that
> combines with the added checksum, to be zero, leaving the original
> checksum intact.
>
> This might be inordinately hard for a CRC, but a simple checksum
> would not be an issue, I think. At least, this could work in
> software, where data can be included in an image file as itself. In
> a device like an FPGA, it might not be included in the bit stream
> file so directly... but that might depend on where in the device it
> is inserted. Memory might have data that is stored as itself. I'll
> need to look into that.
>

I am not sure what your intended use-case is here. But it is very
common to add a checksum of some sort to binary image files after
generating them. This is done post-link. You have a struct in your
read-only data that you link at a known fixed point in the binary. Your
post-link patcher can read this struct (for example, to get the program
version number that is then used to rename the final image file). It
can modify the struct (such as inserting the length of the image). Then
it calculates a CRC and appends it to the end of the image.

Re: Embedding a Checksum in an Image File

<pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1443&group=comp.arch.embedded#1443

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: gneun...@comcast.net (George Neuner)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 11:33:19 -0400
Organization: A noiseless patient Spider
Lines: 23
Message-ID: <pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Injection-Info: dont-email.me; posting-host="375d9d92058b7138a9286e61599e7f5f";
logging-data="699160"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18G8lu+5+r8L/Z815odVMe8niZkrmNgcww="
User-Agent: ForteAgent/8.00.32.1272
Cancel-Lock: sha1:ki5NzBLmyDK9Uu5fKyGOeQtLeKU=
 by: George Neuner - Thu, 20 Apr 2023 15:33 UTC

On Wed, 19 Apr 2023 19:06:33 -0700 (PDT), Rick C
<gnuarm.deletethisbit@gmail.com> wrote:

>This is a bit of the chicken and egg thing. If you want a embed a
>checksum in a code module to report the checksum, is there a way of
>doing this? It's a bit like being your own grandfather, I think.

Take a look at the old xmodem/ymodem CRC. It was designed such that
when the CRC was sent immediately following the data, a receiver
computing CRC over the whole incoming packet (data and CRC both) would
get a result of zero.

But AFAIK it doesn't work with CCITT equation(s) - you have to use
xmodem/ymodem.

>I'm not thinking anything too fancy, like a CRC, but rather a simple
>modulo N addition, maybe N being 2^16.

Sorry, I don't know a way to do it with a modular checksum.
YMMV, but I think 16-bit CRC is pretty simple.

George

Re: Embedding a Checksum in an Image File

<f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1444&group=comp.arch.embedded#1444

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:a05:622a:64a:b0:3e6:6502:16b2 with SMTP id a10-20020a05622a064a00b003e6650216b2mr669287qtb.12.1682009160206;
Thu, 20 Apr 2023 09:46:00 -0700 (PDT)
X-Received: by 2002:a05:622a:1001:b0:3ef:2db1:6e63 with SMTP id
d1-20020a05622a100100b003ef2db16e63mr577451qte.10.1682009159985; Thu, 20 Apr
2023 09:45:59 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!proxad.net!feeder1-2.proxad.net!209.85.160.216.MISMATCH!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Thu, 20 Apr 2023 09:45:59 -0700 (PDT)
In-Reply-To: <pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
Injection-Info: google-groups.googlegroups.com; posting-host=198.245.101.90; posting-account=I-_H_woAAAA9zzro6crtEpUAyIvzd19b
NNTP-Posting-Host: 198.245.101.90
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com> <pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
Subject: Re: Embedding a Checksum in an Image File
From: gnuarm.d...@gmail.com (Rick C)
Injection-Date: Thu, 20 Apr 2023 16:46:00 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
 by: Rick C - Thu, 20 Apr 2023 16:45 UTC

On Thursday, April 20, 2023 at 11:33:28 AM UTC-4, George Neuner wrote:
> On Wed, 19 Apr 2023 19:06:33 -0700 (PDT), Rick C
> <gnuarm.del...@gmail.com> wrote:
>
> >This is a bit of the chicken and egg thing. If you want a embed a
> >checksum in a code module to report the checksum, is there a way of
> >doing this? It's a bit like being your own grandfather, I think.
> Take a look at the old xmodem/ymodem CRC. It was designed such that
> when the CRC was sent immediately following the data, a receiver
> computing CRC over the whole incoming packet (data and CRC both) would
> get a result of zero.
>
> But AFAIK it doesn't work with CCITT equation(s) - you have to use
> xmodem/ymodem.
> >I'm not thinking anything too fancy, like a CRC, but rather a simple
> >modulo N addition, maybe N being 2^16.
> Sorry, I don't know a way to do it with a modular checksum.
> YMMV, but I think 16-bit CRC is pretty simple.
>
> George

CRC is not complicated, but I would not know how to calculate an inserted value to force the resulting CRC to zero. How do you do that?

Even so, I'm not trying to validate the file. I'm trying to come up with a substitute for a time stamp or version number. I don't want to have to rely on my consistency in handling the version number correctly. This would be a backup in case there was more than one version released, even only within the "lab", that were different. A checksum that could be read by the controlling software would do the job.

I have run into this before, where the version number was not a 100% indication of the uniqueness of an executable. The checksum would be a second indicator.

I should mention that I'm not looking for a solution that relies on any specific details of the tools.

--

Rick C.

-+ Get 1,000 miles of free Supercharging
-+ Tesla referral code - https://ts.la/richard11209

Re: Embedding a Checksum in an Image File

<u1rs2m$m90s$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1445&group=comp.arch.embedded#1445

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: tauno.vo...@notused.fi.invalid (Tauno Voipio)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 20:17:07 +0300
Organization: A noiseless patient Spider
Lines: 39
Message-ID: <u1rs2m$m90s$1@dont-email.me>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Thu, 20 Apr 2023 17:17:10 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="1a09c6d59e5ffc41ef09b91be3177824";
logging-data="730140"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18eKJyp2EllOFTLUs0RA/pCte8Hw2IBHR4="
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
Gecko/20100101 Thunderbird/102.10.0
Cancel-Lock: sha1:FdUQrm+Hj7areKQtSMuwwkcmbGI=
Content-Language: en-US
In-Reply-To: <pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
 by: Tauno Voipio - Thu, 20 Apr 2023 17:17 UTC

On 20.4.2023 18.33, George Neuner wrote:
> On Wed, 19 Apr 2023 19:06:33 -0700 (PDT), Rick C
> <gnuarm.deletethisbit@gmail.com> wrote:
>
>> This is a bit of the chicken and egg thing. If you want a embed a
>> checksum in a code module to report the checksum, is there a way of
>> doing this? It's a bit like being your own grandfather, I think.
>
> Take a look at the old xmodem/ymodem CRC. It was designed such that
> when the CRC was sent immediately following the data, a receiver
> computing CRC over the whole incoming packet (data and CRC both) would
> get a result of zero.
>
> But AFAIK it doesn't work with CCITT equation(s) - you have to use
> xmodem/ymodem.
>
>
>> I'm not thinking anything too fancy, like a CRC, but rather a simple
>> modulo N addition, maybe N being 2^16.
>
> Sorry, I don't know a way to do it with a modular checksum.
> YMMV, but I think 16-bit CRC is pretty simple.
>
> George

The method to check for a proper constant value after the whole
block and CRC are received and put through the generator works
with the CRC-CCITT (actually ITU-T). The proper final value
depends on the initial CRC and whether the CRC is inverted before
sending. The limitation is that the CRC has to be sent least
significant octet first.

For a reference, see RFC1662, Appendix C.

--

-TV

Re: Embedding a Checksum in an Image File

<u1s76b$o11h$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1446&group=comp.arch.embedded#1446

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: david.br...@hesbynett.no (David Brown)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 22:26:51 +0200
Organization: A noiseless patient Spider
Lines: 58
Message-ID: <u1s76b$o11h$1@dont-email.me>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
<f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 20 Apr 2023 20:26:51 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="da88365e98b108688441a77766b22198";
logging-data="787505"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18zT4FZ/0oC4n5WDi790MHWMixKFQ/cDF0="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.7.1
Cancel-Lock: sha1:ZeE1PxE3nOcc9E2XIbbBhwucDfg=
In-Reply-To: <f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
Content-Language: en-GB
 by: David Brown - Thu, 20 Apr 2023 20:26 UTC

On 20/04/2023 18:45, Rick C wrote:
> On Thursday, April 20, 2023 at 11:33:28 AM UTC-4, George Neuner
> wrote:
>> On Wed, 19 Apr 2023 19:06:33 -0700 (PDT), Rick C
>> <gnuarm.del...@gmail.com> wrote:
>>
>>> This is a bit of the chicken and egg thing. If you want a embed
>>> a checksum in a code module to report the checksum, is there a
>>> way of doing this? It's a bit like being your own grandfather, I
>>> think.
>> Take a look at the old xmodem/ymodem CRC. It was designed such
>> that when the CRC was sent immediately following the data, a
>> receiver computing CRC over the whole incoming packet (data and CRC
>> both) would get a result of zero.
>>
>> But AFAIK it doesn't work with CCITT equation(s) - you have to use
>> xmodem/ymodem.
>>> I'm not thinking anything too fancy, like a CRC, but rather a
>>> simple modulo N addition, maybe N being 2^16.
>> Sorry, I don't know a way to do it with a modular checksum. YMMV,
>> but I think 16-bit CRC is pretty simple.
>>
>> George
>
> CRC is not complicated, but I would not know how to calculate an
> inserted value to force the resulting CRC to zero. How do you do
> that?

You "insert" the value at the end. Anything else is insane.

CRC's are quite good hashes, for suitable sized data. There are perhaps
some special cases, but basically you'd be doing trial-and-error
searches to find an inserted value that gives you a zero CRC overall.
2^16 is not an overwhelming search space, but the whole idea is pointless.

>
> Even so, I'm not trying to validate the file. I'm trying to come up
> with a substitute for a time stamp or version number. I don't want
> to have to rely on my consistency in handling the version number
> correctly. This would be a backup in case there was more than one
> version released, even only within the "lab", that were different. A
> checksum that could be read by the controlling software would do the
> job.

A CRC is fine for that.

>
> I have run into this before, where the version number was not a 100%
> indication of the uniqueness of an executable. The checksum would be
> a second indicator.
>
> I should mention that I'm not looking for a solution that relies on
> any specific details of the tools.
>

A table-based CRC is easy, runs quickly, and can be quickly ported to
pretty much any language (the C and Python code, for example, is almost
the same).

Re: Embedding a Checksum in an Image File

<36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1447&group=comp.arch.embedded#1447

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: gneun...@comcast.net (George Neuner)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 16:44:10 -0400
Organization: A noiseless patient Spider
Lines: 88
Message-ID: <36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com> <pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com> <f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Injection-Info: dont-email.me; posting-host="375d9d92058b7138a9286e61599e7f5f";
logging-data="794869"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+l0A3PLDaY2XldulVFkIUsP9cnY1UFnP4="
User-Agent: ForteAgent/8.00.32.1272
Cancel-Lock: sha1:StZJCxhaPF2oCr90Lk15M3ahWms=
 by: George Neuner - Thu, 20 Apr 2023 20:44 UTC

On Thu, 20 Apr 2023 09:45:59 -0700 (PDT), Rick C
<gnuarm.deletethisbit@gmail.com> wrote:

>On Thursday, April 20, 2023 at 11:33:28?AM UTC-4, George Neuner wrote:
>> On Wed, 19 Apr 2023 19:06:33 -0700 (PDT), Rick C
>> <gnuarm.del...@gmail.com> wrote:
>>
>> >This is a bit of the chicken and egg thing. If you want a embed a
>> >checksum in a code module to report the checksum, is there a way of
>> >doing this? It's a bit like being your own grandfather, I think.
>>
>> Take a look at the old xmodem/ymodem CRC. It was designed such that
>> when the CRC was sent immediately following the data, a receiver
>> computing CRC over the whole incoming packet (data and CRC both) would
>> get a result of zero.
>>
>> But AFAIK it doesn't work with CCITT equation(s) - you have to use
>> xmodem/ymodem.
>> >I'm not thinking anything too fancy, like a CRC, but rather a simple
>> >modulo N addition, maybe N being 2^16.
>> Sorry, I don't know a way to do it with a modular checksum.
>> YMMV, but I think 16-bit CRC is pretty simple.
>>
>> George
>
>CRC is not complicated, but I would not know how to calculate an
>inserted value to force the resulting CRC to zero. How do you do
>that?

It's implicit in the equation they chose. I don't know how it works -
just that it does.

You have some block of data |....data....|

You compute CRC on the data block and then append the resulting value
to the end of the block. xmodem CRC is 16-bit, so it adds 2 bytes to
the data.

So now you have a new extended block |....data....|crc|

Now if you compute a new CRC on the extended block, the resulting
value /should/ come out to zero. If it doesn't, either your data or
the original CRC value appended to it has been changed/corrupted.

>Even so, I'm not trying to validate the file. I'm trying to come up
>with a substitute for a time stamp or version number. I don't want
>to have to rely on my consistency in handling the version number
>correctly. This would be a backup in case there was more than one
>version released, even only within the "lab", that were different. A
>checksum that could be read by the controlling software would do the
>job.

I've actually done this: in the early 90s I designed a system that
used a CRC based scheme to identify load modules and track
inter-module code dependencies.

I computed both 16-bit Xmodem and CCITT CRCs on the modules and
concatenated the two values into a 32-bit identifier. That identifier
then was used to sign the module and to demand load (or unload) it
when needed.

At the time it worked quite well: the system had quite limited memory,
so code modules were small enough that even a 16-bit CRC could
uniquely identify most/all of them. Combining the two different CRCs
into a 32-bit identifier provided more than enough uniqueness, it was
fast and easy to compute, and it saved a lot of space vs using
something with stronger guarantees like a UUID or crypto-strength
signing hash.
[A lot of the hashing functions available today either didn't exist or
just weren't widely known back then. And still most of them that even
have 32-bit variants are weak in guarantees for those variants.]

>I have run into this before, where the version number was not a 100%
>indication of the uniqueness of an executable. The checksum would be
>a second indicator.

I made it the basis of dependency checking. Version numbers were
secondary and for the benefit of the programmer.

>I should mention that I'm not looking for a solution that relies on
>any specific details of the tools.

YMMV.
George

Re: Embedding a Checksum in an Image File

<l2934i9u72lgk252grte75h9q5echomb0h@4ax.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1448&group=comp.arch.embedded#1448

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: gneun...@comcast.net (George Neuner)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 16:49:20 -0400
Organization: A noiseless patient Spider
Lines: 17
Message-ID: <l2934i9u72lgk252grte75h9q5echomb0h@4ax.com>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com> <pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com> <u1rs2m$m90s$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Injection-Info: dont-email.me; posting-host="375d9d92058b7138a9286e61599e7f5f";
logging-data="796256"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+QwWx44keeGn9O734MFpF7168RQQZXAFU="
User-Agent: ForteAgent/8.00.32.1272
Cancel-Lock: sha1:6MEq65QTl5tnXBEls+3XiiMvulQ=
 by: George Neuner - Thu, 20 Apr 2023 20:49 UTC

On Thu, 20 Apr 2023 20:17:07 +0300, Tauno Voipio
<tauno.voipio@notused.fi.invalid> wrote:

>The method to check for a proper constant value after the whole
>block and CRC are received and put through the generator works
>with the CRC-CCITT (actually ITU-T). The proper final value
>depends on the initial CRC and whether the CRC is inverted before
>sending. The limitation is that the CRC has to be sent least
>significant octet first.
>
>For a reference, see RFC1662, Appendix C.

I remember seeing an explanation of it decades ago, but I never would
have been able to find it again.

Thanks,
George

Re: Embedding a Checksum in an Image File

<tfm0M.2629804$GNG9.1468222@fx18.iad>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1449&group=comp.arch.embedded#1449

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.uzoreto.com!peer02.ams4!peer.am4.highwinds-media.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!fx18.iad.POSTED!not-for-mail
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
Gecko/20100101 Thunderbird/102.10.0
Subject: Re: Embedding a Checksum in an Image File
Content-Language: en-US
Newsgroups: comp.arch.embedded
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
From: Rich...@Damon-Family.org (Richard Damon)
In-Reply-To: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 36
Message-ID: <tfm0M.2629804$GNG9.1468222@fx18.iad>
X-Complaints-To: abuse@easynews.com
Organization: Forte - www.forteinc.com
X-Complaints-Info: Please be sure to forward a copy of ALL headers otherwise we will be unable to process your complaint properly.
Date: Thu, 20 Apr 2023 22:09:29 -0400
X-Received-Bytes: 4006
 by: Richard Damon - Fri, 21 Apr 2023 02:09 UTC

On 4/19/23 10:06 PM, Rick C wrote:
> This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
>
> I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.
>
> I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.
>
> I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.
>
> I keep thinking there is a different way of looking at this to achieve the result I want...
>
> Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact.
>
> This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.
>

IF I understand you correctly, what you want is for the file to compute
to some "checksum" that comes from the basic contents of the file, and
then you want to add the "checksum" into the file so the program itself
can print its checksum.

One fact to remember, is that "cryptographic hashes" were invented
because it was too easy to create a faked file that matches a
non-crptographic hash/checksum, so that couldn't be a key to make sure
you really had the right file in the presence of a determined enemy, but
the checksums were good enough to catch "random" errors.

This means that you can add the checksum into the file, and some
additional bytes (likely at the end) and by knowing the propeties of the
checksum algorithm, compute a value for those extra bytes such that the
"undo" the changes caused by adding the checksum bytes to file.

I'm not sure exactly how to computes these, but the key is that you add
something at the end of the file to get the checksum back to what the
original file had before you added the checksum into the file.

Re: Embedding a Checksum in an Image File

<cd494e7f-9952-40ac-98da-4745dff32493n@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1450&group=comp.arch.embedded#1450

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:a05:620a:13db:b0:74a:92e:bcfb with SMTP id g27-20020a05620a13db00b0074a092ebcfbmr742419qkl.3.1682044907995;
Thu, 20 Apr 2023 19:41:47 -0700 (PDT)
X-Received: by 2002:a05:620a:1109:b0:74d:ff4c:45b0 with SMTP id
o9-20020a05620a110900b0074dff4c45b0mr572348qkk.6.1682044907649; Thu, 20 Apr
2023 19:41:47 -0700 (PDT)
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer03.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Thu, 20 Apr 2023 19:41:47 -0700 (PDT)
In-Reply-To: <tfm0M.2629804$GNG9.1468222@fx18.iad>
Injection-Info: google-groups.googlegroups.com; posting-host=198.245.101.90; posting-account=I-_H_woAAAA9zzro6crtEpUAyIvzd19b
NNTP-Posting-Host: 198.245.101.90
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com> <tfm0M.2629804$GNG9.1468222@fx18.iad>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <cd494e7f-9952-40ac-98da-4745dff32493n@googlegroups.com>
Subject: Re: Embedding a Checksum in an Image File
From: gnuarm.d...@gmail.com (Rick C)
Injection-Date: Fri, 21 Apr 2023 02:41:47 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 5127
 by: Rick C - Fri, 21 Apr 2023 02:41 UTC

On Thursday, April 20, 2023 at 10:09:35 PM UTC-4, Richard Damon wrote:
> On 4/19/23 10:06 PM, Rick C wrote:
> > This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
> >
> > I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.
> >
> > I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.
> >
> > I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.
> >
> > I keep thinking there is a different way of looking at this to achieve the result I want...
> >
> > Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact.
> >
> > This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.
> >
> IF I understand you correctly, what you want is for the file to compute
> to some "checksum" that comes from the basic contents of the file, and
> then you want to add the "checksum" into the file so the program itself
> can print its checksum.
>
> One fact to remember, is that "cryptographic hashes" were invented
> because it was too easy to create a faked file that matches a
> non-crptographic hash/checksum, so that couldn't be a key to make sure
> you really had the right file in the presence of a determined enemy, but
> the checksums were good enough to catch "random" errors.
>
> This means that you can add the checksum into the file, and some
> additional bytes (likely at the end) and by knowing the propeties of the
> checksum algorithm, compute a value for those extra bytes such that the
> "undo" the changes caused by adding the checksum bytes to file.
>
> I'm not sure exactly how to computes these, but the key is that you add
> something at the end of the file to get the checksum back to what the
> original file had before you added the checksum into the file.

Yeah, for a simple checksum, I think that would be easy, at least if "checksum" means a bitwise XOR operation. If the checksum and extra bytes are both 16 bits, this would also work for an arithmetic checksum where each 16 bit word were added into the checksum. All the carries would cascade out of the upper 16 bits from adding the inserted checksum and it's 2's complement.

I don't even want to think about using a CRC to try to do this.

--

Rick C.

+- Get 1,000 miles of free Supercharging
+- Tesla referral code - https://ts.la/richard11209

Re: Embedding a Checksum in an Image File

<u1t7eb$10gmu$3@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1451&group=comp.arch.embedded#1451

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: blockedo...@foo.invalid (Don Y)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Thu, 20 Apr 2023 22:37:07 -0700
Organization: A noiseless patient Spider
Lines: 50
Message-ID: <u1t7eb$10gmu$3@dont-email.me>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
<f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
<36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Fri, 21 Apr 2023 05:37:15 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="b915f9468af049b4b701bd32986bb8c3";
logging-data="1065694"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18IeqVzWaiOrlqp0ZS0qVKB"
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.2.2
Cancel-Lock: sha1:tJ0E7++PYStwXnj2QRYy6+eaeNM=
Content-Language: en-US
In-Reply-To: <36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com>
 by: Don Y - Fri, 21 Apr 2023 05:37 UTC

On 4/20/2023 1:44 PM, George Neuner wrote:
> You have some block of data |....data....|
>
> You compute CRC on the data block and then append the resulting value
---------------------------------------------^^^^^^
> to the end of the block. xmodem CRC is 16-bit, so it adds 2 bytes to
> the data.

Exactly. You *don't* drag the "extra bits" into the initial
CRC calculation but *do* into the CRC *verification*. Easy
peasy (since forever).

[Think about it: your performing a division operation
and the residual is the "remainder".]

Note that you want to choose a polynomial that doesn't
give you a "win" result for "obviously" corrupt data.
E.g., if data is all zeros or all 0xFF (as these sorts of
conditions can happen with hardware failures) you probably
wouldn't want a "success" indication!

You can also "salt" the calculation so that the residual
is deliberately nonzero. So, for example, "success" is
indicated by a residual of 0x474E. :>

> So now you have a new extended block |....data....|crc|
>
> Now if you compute a new CRC on the extended block, the resulting
> value /should/ come out to zero. If it doesn't, either your data or
> the original CRC value appended to it has been changed/corrupted.

As there is usually a lack of originality in the algorithms
chosen, you have to consider if you are also hoping to use
this to safeguard the *integrity* of your image (i.e.,
against intentional modification).

I have an old Compaq Portable 386 (lunchbox) that obviously
wasn't designed to support the disk drives that I would
*later* install in it. So, I patched the BIOS ROMs to
add another disk type to the Disk Parameter Table. Then,
made compensating changes to other parts of the ROM
(that I knew would not be referenced) to ensure the original
checksum -- WHEREVER IT MAY HAVE BEEN "STORED" -- would remain
intact.

[I could similarly have altered the boot message to show
a copyright of "Don Y" in place of "Compaq" -- as it would
be pretty easy to locate the "Compaq" string (in plaintext)
in the image.]

Re: Embedding a Checksum in an Image File

<66e9fb2a-b9e6-4597-9afa-6572caaa8ca1n@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1452&group=comp.arch.embedded#1452

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:a37:e307:0:b0:74d:ed60:be11 with SMTP id y7-20020a37e307000000b0074ded60be11mr865376qki.15.1682067195671;
Fri, 21 Apr 2023 01:53:15 -0700 (PDT)
X-Received: by 2002:a05:622a:1194:b0:3de:bafb:82bf with SMTP id
m20-20020a05622a119400b003debafb82bfmr1269593qtk.4.1682067195378; Fri, 21 Apr
2023 01:53:15 -0700 (PDT)
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Fri, 21 Apr 2023 01:53:15 -0700 (PDT)
In-Reply-To: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
Injection-Info: google-groups.googlegroups.com; posting-host=180.150.36.196; posting-account=ZaAWpAoAAABFnmcUGQHISv7vlLrTdUCZ
NNTP-Posting-Host: 180.150.36.196
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <66e9fb2a-b9e6-4597-9afa-6572caaa8ca1n@googlegroups.com>
Subject: Re: Embedding a Checksum in an Image File
From: brian.co...@gmail.com (Brian Cockburn)
Injection-Date: Fri, 21 Apr 2023 08:53:15 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 3957
 by: Brian Cockburn - Fri, 21 Apr 2023 08:53 UTC

On Thursday, April 20, 2023 at 12:06:36 PM UTC+10, Rick C wrote:
> This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
>
> I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.
>
> I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.
>
> I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.
>
> I keep thinking there is a different way of looking at this to achieve the result I want...
>
> Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact.
>
> This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.
>
> --
>
> Rick C.
>
> - Get 1,000 miles of free Supercharging
> - Tesla referral code - https://ts.la/richard11209
Rick, What is the purpose of this? Is it (1) to be able to externally identify a binary, as one might a ROM image by computing a checksum? Is it (2) for a run-able binary to be able to check itself? This would of course only be able to detect corruption, not tampering. Is it (3) for the loader (whatever that might be) to be able to say 'this binary has the correct checksum' and only jump to it if it does? Again this would only be able to detect corruption, not tampering. Are you hoping for more than corruption detection?

Re: Embedding a Checksum in an Image File

<u1tpcr$1377f$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1453&group=comp.arch.embedded#1453

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: david.br...@hesbynett.no (David Brown)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Fri, 21 Apr 2023 12:43:39 +0200
Organization: A noiseless patient Spider
Lines: 97
Message-ID: <u1tpcr$1377f$1@dont-email.me>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
<f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
<36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com> <u1t7eb$10gmu$3@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 21 Apr 2023 10:43:39 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="fe0fb1049f5d07292e4593c1b1226c5a";
logging-data="1154287"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX181n7W2++if6Y8C3RRqEW0dxaps1jcfgZE="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.9.0
Cancel-Lock: sha1:gyd/gR/0i2KnG0W/jjhrZT+OlLY=
In-Reply-To: <u1t7eb$10gmu$3@dont-email.me>
Content-Language: en-GB
 by: David Brown - Fri, 21 Apr 2023 10:43 UTC

On 21/04/2023 07:37, Don Y wrote:
> On 4/20/2023 1:44 PM, George Neuner wrote:
>> You have some block of data   |....data....|
>>
>> You compute CRC on the data block and then append the resulting value
> ---------------------------------------------^^^^^^
>> to the end of the block.  xmodem CRC is 16-bit, so it adds 2 bytes to
>> the data.
>
> Exactly.  You *don't* drag the "extra bits" into the initial
> CRC calculation but *do* into the CRC *verification*.  Easy
> peasy (since forever).
>
> [Think about it:  your performing a division operation
> and the residual is the "remainder".]

George's earlier posts made it look like the algorithm was inserting
("embedding") a value somewhere inside the image, so that the CRC over
the modified image was zero. This is easy to do for simple checksums
such as XOR's or a sum-of-bytes checksum, but infeasible for CRC's.

It is a much easier matter when appending the checksum. Depending
somewhat on the details of the CRC (such as bit/byte reversals,
inversions, starting values, etc.) it is typically the case that for a
binary blob A, crc(A ++ crc(A)) = 0. i.e., if you append the CRC of
your data to the data, the CRC of the whole thing is 0.

Of course, this is pretty much irrelevant - whether you check the
integrity of the final image by running CRC over it all and comparing to
0, or running it over all but the last word and comparing to the last
word is a minor matter.

>
> Note that you want to choose a polynomial that doesn't
> give you a "win" result for "obviously" corrupt data.
> E.g., if data is all zeros or all 0xFF (as these sorts of
> conditions can happen with hardware failures) you probably
> wouldn't want a "success" indication!

No, that is pointless for something like a code image. It just adds
needless complexity to your CRC algorithm.

You should already have checks that would eliminate an all-zero image or
other "obviously corrupt" data. You'll be checking the image for a key
or "magic number" that identifies the image as "program image for board
X, project Y". You'll be checking version numbers. You'll be reading
the length of the image so you know the range for your CRC function, and
where to find the appended CRC check. You might not have all of these
in a given system, but you'll have some kind of check which would fail
on an all-zero image.

>
> You can also "salt" the calculation so that the residual
> is deliberately nonzero.  So, for example, "success" is
> indicated by a residual of 0x474E.  :>
>

Again, pointless.

Salt is important for security-related hashes (like password hashes),
not for integrity checks.

>> So now you have a new extended block   |....data....|crc|
>>
>> Now if you compute a new CRC on the extended block, the resulting
>> value /should/ come out to zero. If it doesn't, either your data or
>> the original CRC value appended to it has been changed/corrupted.
>
> As there is usually a lack of originality in the algorithms
> chosen, you have to consider if you are also hoping to use
> this to safeguard the *integrity* of your image (i.e.,
> against intentional modification).
>

"Integrity" has nothing to do with the motivation for change.
/Security/ is concerned with intentional modifications that deliberately
attempt to defeat /integrity/ checks. Integrity is about detecting any
changes.

If you are concerned about the possibility of intentional malicious
changes, CRC's alone are useless. All the attacker needs to do after
modifying the image is calculate the CRC themselves, and replace the
original checksum with their own.

Using non-standard algorithms for security is a simple way to get things
completely wrong. "Security by obscurity" is very rarely the right
answer. In reality, good security algorithms, and good implementations,
are difficult and specialised tasks, best left to people who know what
they are doing.

To make something secure, you have to ensure that the check algorithms
depend on a key that you know, but that the attacker does not have.
That's the basis of digital signatures (though you use a secure hash
algorithm rather than a simple CRC).

Re: Embedding a Checksum in an Image File

<u1tsm3$13ook$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1454&group=comp.arch.embedded#1454

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: blockedo...@foo.invalid (Don Y)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Fri, 21 Apr 2023 04:39:38 -0700
Organization: A noiseless patient Spider
Lines: 152
Message-ID: <u1tsm3$13ook$1@dont-email.me>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
<f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
<36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com> <u1t7eb$10gmu$3@dont-email.me>
<u1tpcr$1377f$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 21 Apr 2023 11:39:47 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="b915f9468af049b4b701bd32986bb8c3";
logging-data="1172244"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19ZiMCpbAhHxO0k+1jgFdTA"
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:102.0) Gecko/20100101
Thunderbird/102.2.2
Cancel-Lock: sha1:HASwuLnZR5ISD0yMsgARfBZXxKE=
Content-Language: en-US
In-Reply-To: <u1tpcr$1377f$1@dont-email.me>
 by: Don Y - Fri, 21 Apr 2023 11:39 UTC

On 4/21/2023 3:43 AM, David Brown wrote:
>> Note that you want to choose a polynomial that doesn't
>> give you a "win" result for "obviously" corrupt data.
>> E.g., if data is all zeros or all 0xFF (as these sorts of
>> conditions can happen with hardware failures) you probably
>> wouldn't want a "success" indication!
>
> No, that is pointless for something like a code image.  It just adds needless
> complexity to your CRC algorithm.

Perhaps you've forgotten that you don't just use CRCs (secure hashes, etc.)
on "code images"?

> You should already have checks that would eliminate an all-zero image or other
> "obviously corrupt" data.  You'll be checking the image for a key or "magic
> number" that identifies the image as "program image for board X, project Y".
> You'll be checking version numbers.  You'll be reading the length of the image
> so you know the range for your CRC function, and where to find the appended CRC
> check.  You might not have all of these in a given system, but you'll have some
> kind of check which would fail on an all-zero image.

See above.

>> You can also "salt" the calculation so that the residual
>> is deliberately nonzero.  So, for example, "success" is
>> indicated by a residual of 0x474E.  :>
>
> Again, pointless.
>
> Salt is important for security-related hashes (like password hashes), not for
> integrity checks.

You've missed the point. The correct "sum" can be anything.
Why is "0" more special than any other value? As the value is
typically meaningless to anything other than the code that verifies
it, you couldn't look at an image (or the output of the verifier)
and gain anything from seeing that obscure value.

OTOH, if the CRC yields something familiar -- or useful -- then
it can tell you something about the image. E.g., salt the algorithm
with the product code, version number, your initials, 0xDEADBEEF, etc.

>>> So now you have a new extended block   |....data....|crc|
>>>
>>> Now if you compute a new CRC on the extended block, the resulting
>>> value /should/ come out to zero. If it doesn't, either your data or
>>> the original CRC value appended to it has been changed/corrupted.
>>
>> As there is usually a lack of originality in the algorithms
>> chosen, you have to consider if you are also hoping to use
>> this to safeguard the *integrity* of your image (i.e.,
>> against intentional modification).
>
> "Integrity" has nothing to do with the motivation for change. /Security/ is
> concerned with intentional modifications that deliberately attempt to defeat
> /integrity/ checks.  Integrity is about detecting any changes.
>
> If you are concerned about the possibility of intentional malicious changes,

Changes don't have to be malicious. I altered the test procedure for a
piece of military gear we were building simply to skip some lengthy tests that
I *knew* would pass (I don't want to inject an extra 20 minutes of wait time
just to get through a lengthy test I already know works before I can get
to the test of interest to me, now.

I failed to undo the change before the official signoff on the device.

The only evidence of this was the fact that I had also patched the
startup message to say "Go for coffee..." -- which remained on the
screen for the duration of the lengthy (even with the long test
elided) procedure...

...which alerted folks to the fact that this *probably* wasn't the
original image. (The computer running the test suite on the DUT had
no problem accepting my patched binary)

> CRC's alone are useless.  All the attacker needs to do after modifying the
> image is calculate the CRC themselves, and replace the original checksum with
> their own.

That assumes the "alterer" knows how to replace the checksum, how it
is computed, where it is embedded in the image, etc. I modified the Compaq
portable mentioned without ever knowing where the checksum was store
or *if* it was explicitly stored. I had no desire to disassemble the
BIOS ROMs (though could obviously do so as there was no "proprietary
hardware" limiting access to their contents and the instruction set of
the processor is well known!).

Instead, I did this by *guessing* how they would implement such a check
in a bit of kit from that era (ERPOMs aren't easily modified by malware
so it wasn't likely that they would go to great lengths to "protect" the
image). And, if my guess had been incorrect, I could always reinstall
the original EPROMs -- nothing lost, nothing gained.

Had much experience with folks counterfeiting your products and making
"simple" changes to the binaries? Like changing the copyright notice
or splash screen?

Then, bringing the (accused) counterfeit of YOUR product into a courtroom
and revealing the *hidden* checksum that the counterfeiter wasn't aware of?

"Gee, why does YOUR (alleged) device have *my* name in it -- in addition
to behaving exactly like mine??"

[I guess obscurity has its place!]

Use a non-secret approach and you invite folks to alter it, as well.

> Using non-standard algorithms for security is a simple way to get things
> completely wrong.  "Security by obscurity" is very rarely the right answer.  In
> reality, good security algorithms, and good implementations, are difficult and
> specialised tasks, best left to people who know what they are doing.
>
> To make something secure, you have to ensure that the check algorithms depend
> on a key that you know, but that the attacker does not have. That's the basis
> of digital signatures (though you use a secure hash algorithm rather than a
> simple CRC).

If you can remove the check, then what value the key's secrecy? By your
criteria, the adversary KNOWS how you are implementing your security
so he knows exactly what to remove to bypass your checks and allow his
altered image to operate in its place.

Ever notice how manufacturers don't PUBLICLY disclose their security
hooks (without an NDA)? If "security by obscurity" was not important,
they would publish these details INVITING challenges (instead of
trying to limit the knowledge to people with whom they've officially
contracted).

[If it was so good and they were trying to rely on trade secret, why
not just PATENT their approach, also disclosing it in the process?
Surely, the details will "leak" from one of the NDA signers long
before patent protection would expire... And, presumably, these
are "people who know what they are doing"...]

Sign all the binaries and all I have to do is remove the *test* for
those signatures and the images can be as corrupted as I choose.

You need to "secure" the test if you want the image to be securable.
This is why it is so hard to use "open" security protocols on
hardware devices (cuz there are almost always ways to subvert the
verification process/hardware). Having physical access to a device
usually means it can be compromised -- if worth your effort.

[The trick is to make the effort great enough to be on a par with
just copying the *functionality*, from scratch, and not bothering
trying to alter the executable in a way that is not detectable]

[[There are companies who's business models are exactly that -- cloning
other products (e.g., from folks like big blue) at the functional
level -- yet steering clear of any copyright issues.]]

Re: Embedding a Checksum in an Image File

<a4495c87-c68d-4c87-94aa-701c21bdd19cn@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1455&group=comp.arch.embedded#1455

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:a05:6214:1906:b0:5ef:4ecb:b47e with SMTP id er6-20020a056214190600b005ef4ecbb47emr761913qvb.8.1682079166856;
Fri, 21 Apr 2023 05:12:46 -0700 (PDT)
X-Received: by 2002:a05:622a:44:b0:3e3:f70f:fb13 with SMTP id
y4-20020a05622a004400b003e3f70ffb13mr1356904qtw.6.1682079166588; Fri, 21 Apr
2023 05:12:46 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!panix!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer02.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Fri, 21 Apr 2023 05:12:46 -0700 (PDT)
In-Reply-To: <66e9fb2a-b9e6-4597-9afa-6572caaa8ca1n@googlegroups.com>
Injection-Info: google-groups.googlegroups.com; posting-host=198.245.101.90; posting-account=I-_H_woAAAA9zzro6crtEpUAyIvzd19b
NNTP-Posting-Host: 198.245.101.90
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com> <66e9fb2a-b9e6-4597-9afa-6572caaa8ca1n@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <a4495c87-c68d-4c87-94aa-701c21bdd19cn@googlegroups.com>
Subject: Re: Embedding a Checksum in an Image File
From: gnuarm.d...@gmail.com (Rick C)
Injection-Date: Fri, 21 Apr 2023 12:12:46 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 4606
 by: Rick C - Fri, 21 Apr 2023 12:12 UTC

On Friday, April 21, 2023 at 4:53:18 AM UTC-4, Brian Cockburn wrote:
> On Thursday, April 20, 2023 at 12:06:36 PM UTC+10, Rick C wrote:
> > This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
> >
> > I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.
> >
> > I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.
> >
> > I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.
> >
> > I keep thinking there is a different way of looking at this to achieve the result I want...
> >
> > Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact.
> >
> > This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.
> >
> > --
> >
> > Rick C.
> >
> > - Get 1,000 miles of free Supercharging
> > - Tesla referral code - https://ts.la/richard11209
> Rick, What is the purpose of this? Is it (1) to be able to externally identify a binary, as one might a ROM image by computing a checksum? Is it (2) for a run-able binary to be able to check itself? This would of course only be able to detect corruption, not tampering. Is it (3) for the loader (whatever that might be) to be able to say 'this binary has the correct checksum' and only jump to it if it does? Again this would only be able to detect corruption, not tampering. Are you hoping for more than corruption detection?

This is simply to be able to say this version is unique, regardless of what the version number says. Version numbers are set manually and not always done correctly. I'm looking for something as a backup so that if the checksums are different, I can be sure the versions are not the same.

The less work involved, the better.

--

Rick C.

++ Get 1,000 miles of free Supercharging
++ Tesla referral code - https://ts.la/richard11209

Re: Embedding a Checksum in an Image File

<u1u7ro$2poss$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1456&group=comp.arch.embedded#1456

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: david.br...@hesbynett.no (David Brown)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Fri, 21 Apr 2023 16:50:31 +0200
Organization: A noiseless patient Spider
Lines: 252
Message-ID: <u1u7ro$2poss$1@dont-email.me>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
<f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
<36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com> <u1t7eb$10gmu$3@dont-email.me>
<u1tpcr$1377f$1@dont-email.me> <u1tsm3$13ook$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 21 Apr 2023 14:50:32 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="fe0fb1049f5d07292e4593c1b1226c5a";
logging-data="2941852"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19+KD0ksea1GpTLHrd5vURhusvVPr5AT6Q="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.9.0
Cancel-Lock: sha1:B8JTAv9VxgTkyEEgo1ruMWozz6Q=
In-Reply-To: <u1tsm3$13ook$1@dont-email.me>
Content-Language: en-GB
 by: David Brown - Fri, 21 Apr 2023 14:50 UTC

On 21/04/2023 13:39, Don Y wrote:
> On 4/21/2023 3:43 AM, David Brown wrote:
>>> Note that you want to choose a polynomial that doesn't
>>> give you a "win" result for "obviously" corrupt data.
>>> E.g., if data is all zeros or all 0xFF (as these sorts of
>>> conditions can happen with hardware failures) you probably
>>> wouldn't want a "success" indication!
>>
>> No, that is pointless for something like a code image.  It just adds
>> needless complexity to your CRC algorithm.
>
> Perhaps you've forgotten that you don't just use CRCs (secure hashes, etc.)
> on "code images"?

No - but "code images" is the topic here.

However, in almost every case where CRC's might be useful, you have
additional checks of the sanity of the data, and an all-zero or all-one
data block would be rejected. For example, Ethernet packets use CRC for
integrity checking, but an attempt to send a packet type 0 from MAC
address 00:00:00:00:00:00 to address 00:00:00:00:00:00, of length 0,
would be rejected anyway.

I can't think of any use-cases where you would be passing around a block
of "pure" data that could reasonably take absolutely any value, without
any type of "envelope" information, and where you would think a CRC
check is appropriate.

>
>> You should already have checks that would eliminate an all-zero image
>> or other "obviously corrupt" data.  You'll be checking the image for a
>> key or "magic number" that identifies the image as "program image for
>> board X, project Y". You'll be checking version numbers.  You'll be
>> reading the length of the image so you know the range for your CRC
>> function, and where to find the appended CRC check.  You might not
>> have all of these in a given system, but you'll have some kind of
>> check which would fail on an all-zero image.
>
> See above.

See above.

>
>>> You can also "salt" the calculation so that the residual
>>> is deliberately nonzero.  So, for example, "success" is
>>> indicated by a residual of 0x474E.  :>
>>
>> Again, pointless.
>>
>> Salt is important for security-related hashes (like password hashes),
>> not for integrity checks.
>
> You've missed the point.  The correct "sum" can be anything.
> Why is "0" more special than any other value?  As the value is
> typically meaningless to anything other than the code that verifies
> it, you couldn't look at an image (or the output of the verifier)
> and gain anything from seeing that obscure value.

Do you actually know what is meant by "salt" in the context of hashes,
and why it is useful in some circumstances? Do you understand that
"salt" is added (usually prepended, or occasionally mixed in in some
other way) to the data /before/ the hash is calculated?

I have not given the slightest indication to suggest that "0" is a
special value. I fully agree that the value you get from the checking
algorithm does not have to be 0 - I already suggested it could be
compared to the stored value. I.e., your build your image file as "data
++ crc(data)", at check it by re-calculating "crc(data)" on the received
image and comparing the result to the received crc. There is no
necessity or benefit in having a crc run calculated over the received
data plus the received crc being 0.

"Salt" is used in cases where the original data must be kept secret, and
only the hashes are transmitted or accessible - by adding salt to the
original data before hashing it, you avoid a direct correspondence
between the hash and the original data. The prime use-case is to stop
people being able to figure out a password by looking up the hash in a
list of pre-computed hashes of common passwords.

>
> OTOH, if the CRC yields something familiar -- or useful -- then
> it can tell you something about the image.  E.g., salt the algorithm
> with the product code, version number, your initials, 0xDEADBEEF, etc.
>

You are making no sense at all. Are you suggesting that it would be a
good idea to add some value to the start of the image so that the
resulting crc calculation gives a nice recognisable product code? This
"salt" would be different for each program image, and calculated by
trial and error. If you want a product code, version number, etc., in
the program image (and it's a good idea), just put these in the program
image!

>>>> So now you have a new extended block   |....data....|crc|
>>>>
>>>> Now if you compute a new CRC on the extended block, the resulting
>>>> value /should/ come out to zero. If it doesn't, either your data or
>>>> the original CRC value appended to it has been changed/corrupted.
>>>
>>> As there is usually a lack of originality in the algorithms
>>> chosen, you have to consider if you are also hoping to use
>>> this to safeguard the *integrity* of your image (i.e.,
>>> against intentional modification).
>>
>> "Integrity" has nothing to do with the motivation for change.
>> /Security/ is concerned with intentional modifications that
>> deliberately attempt to defeat /integrity/ checks.  Integrity is about
>> detecting any changes.
>>
>> If you are concerned about the possibility of intentional malicious
>> changes,
>
> Changes don't have to be malicious.

Accidental changes (such as human error, noise during data transfer,
memory cell errors, etc.) do not pass integrity tests unnoticed. To be
more accurate, the chances of them passing unnoticed are of the order of
1 in 2^n, for a good n-bit check such as a CRC check. Certain types of
error are always detectable, such as single and double bit errors. That
is the point of using a checksum or hash for integrity checking.

/Intentional/ changes are a different matter. If a hacker changes the
program image, they can change the transmitted hash to their own
calculated hash. Or for a small CRC, they could change a different part
of the image until the original checksum matched - for a 16-bit CRC,
that only takes 65,535 attempts in the worst case.

That is why you need to distinguish between the two possibilities. If
you don't have to worry about malicious attacks, a 32-bit CRC takes a
dozen lines of C code and a 1 KB table, all running extremely
efficiently. If security is an issue, you need digital signatures - an
RSA-based signature system is orders of magnitude more effort in both
development time and in run time.

> I altered the test procedure for a
> piece of military gear we were building simply to skip some lengthy
> tests that I *knew* would pass (I don't want to inject an extra 20
> minutes of wait time
> just to get through a lengthy test I already know works before I can get
> to the test of interest to me, now.
>
> I failed to undo the change before the official signoff on the device.
>
> The only evidence of this was the fact that I had also patched the
> startup message to say "Go for coffee..." -- which remained on the
> screen for the duration of the lengthy (even with the long test
> elided) procedure...
>
> ..which alerted folks to the fact that this *probably* wasn't the
> original image.  (The computer running the test suite on the DUT had
> no problem accepting my patched binary)

And what, exactly, do you think that anecdote tells us about CRC checks
for image files? It reminds us that we are all fallible, but does no
more than that.

>
>> CRC's alone are useless.  All the attacker needs to do after modifying
>> the image is calculate the CRC themselves, and replace the original
>> checksum with their own.
>
> That assumes the "alterer" knows how to replace the checksum, how it
> is computed, where it is embedded in the image, etc.  I modified the Compaq
> portable mentioned without ever knowing where the checksum was store
> or *if* it was explicitly stored.  I had no desire to disassemble the
> BIOS ROMs (though could obviously do so as there was no "proprietary
> hardware" limiting access to their contents and the instruction set of
> the processor is well known!).
>
> Instead, I did this by *guessing* how they would implement such a check
> in a bit of kit from that era (ERPOMs aren't easily modified by malware
> so it wasn't likely that they would go to great lengths to "protect" the
> image).  And, if my guess had been incorrect, I could always reinstall
> the original EPROMs -- nothing lost, nothing gained.
>
> Had much experience with folks counterfeiting your products and making
> "simple" changes to the binaries?  Like changing the copyright notice
> or splash screen?
>
> Then, bringing the (accused) counterfeit of YOUR product into a courtroom
> and revealing the *hidden* checksum that the counterfeiter wasn't aware of?
>
> "Gee, why does YOUR (alleged) device have *my* name in it -- in addition
> to behaving exactly like mine??"
>
> [I guess obscurity has its place!]


Click here to read the complete article
Re: Embedding a Checksum in an Image File

<u1u8hu$2ps79$1@dont-email.me>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1457&group=comp.arch.embedded#1457

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!eternal-september.org!news.eternal-september.org!.POSTED!not-for-mail
From: david.br...@hesbynett.no (David Brown)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Fri, 21 Apr 2023 17:02:21 +0200
Organization: A noiseless patient Spider
Lines: 14
Message-ID: <u1u8hu$2ps79$1@dont-email.me>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<66e9fb2a-b9e6-4597-9afa-6572caaa8ca1n@googlegroups.com>
<a4495c87-c68d-4c87-94aa-701c21bdd19cn@googlegroups.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Fri, 21 Apr 2023 15:02:22 -0000 (UTC)
Injection-Info: dont-email.me; posting-host="fe0fb1049f5d07292e4593c1b1226c5a";
logging-data="2945257"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19lLsz1wmX0B+tOQucQbJWZ2EVLR8DamPQ="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.9.0
Cancel-Lock: sha1:64jSuAxT5IOEsHzMSwVRS85FIhE=
Content-Language: en-GB
In-Reply-To: <a4495c87-c68d-4c87-94aa-701c21bdd19cn@googlegroups.com>
 by: David Brown - Fri, 21 Apr 2023 15:02 UTC

On 21/04/2023 14:12, Rick C wrote:
>
> This is simply to be able to say this version is unique, regardless
> of what the version number says. Version numbers are set manually
> and not always done correctly. I'm looking for something as a backup
> so that if the checksums are different, I can be sure the versions
> are not the same.
>
> The less work involved, the better.
>

Run a simple 32-bit crc over the image. The result is a hash of the
image. Any change in the image will show up as a change in the crc.

Re: Embedding a Checksum in an Image File

<u1uoqs.17k.1@stefan.msgid.phost.de>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1458&group=comp.arch.embedded#1458

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: stefan.n...@arcor.de (Stefan Reuther)
Newsgroups: comp.arch.embedded
Subject: Re: Embedding a Checksum in an Image File
Date: Fri, 21 Apr 2023 19:40:11 +0200
Lines: 26
Message-ID: <u1uoqs.17k.1@stefan.msgid.phost.de>
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<pvl24i57aef4vc7bbdk9mvj7sic9dsh64t@4ax.com>
<f0afa198-e735-4da1-a16a-82764af3de4dn@googlegroups.com>
<36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Trace: individual.net XxHhjbsYxvW/6WUGzQOHogI2DZjZnDt4Q1FkfBQGkdzebSDZw3
Cancel-Lock: sha1:KMi4BHZHmQ0Y2v0Re7unhJeO2JQ=
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101
Thunderbird/68.12.1 Hamster/2.1.0.1538
In-Reply-To: <36534il81ipvnhog6980r9ln9tdqn5cbh6@4ax.com>
 by: Stefan Reuther - Fri, 21 Apr 2023 17:40 UTC

Am 20.04.2023 um 22:44 schrieb George Neuner:
> On Thu, 20 Apr 2023 09:45:59 -0700 (PDT), Rick C
>> CRC is not complicated, but I would not know how to calculate an
>> inserted value to force the resulting CRC to zero. How do you do
>> that?
>
> It's implicit in the equation they chose. I don't know how it works -
> just that it does.

It works for any CRC that starts with zero and does not invert.

CRC is based on polynomial division remainders. Basically, the CRC is a
division remainder of the input interpreted as a polynomial, and if you
add that remainder back into the equation, result is zero.

https://crccalc.com/?crc=12345678&method=CRC-16/AUG-CCITT&datatype=hex&outtype=0
-> result is 0xBA3C

https://crccalc.com/?crc=12345678BA3C&method=CRC-16/AUG-CCITT&datatype=hex&outtype=0
-> result is 0x0000

(Need to be careful with byte orders; for some CRCs on that page, you
need to swap the bytes before appending.)

Stefan

Re: Embedding a Checksum in an Image File

<20F0M.291598$b7Kc.5484@fx39.iad>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1459&group=comp.arch.embedded#1459

 copy link   Newsgroups: comp.arch.embedded
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!panix!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!fx39.iad.POSTED!not-for-mail
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
Gecko/20100101 Thunderbird/102.10.0
Subject: Re: Embedding a Checksum in an Image File
Content-Language: en-US
Newsgroups: comp.arch.embedded
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<tfm0M.2629804$GNG9.1468222@fx18.iad>
<cd494e7f-9952-40ac-98da-4745dff32493n@googlegroups.com>
From: Rich...@Damon-Family.org (Richard Damon)
In-Reply-To: <cd494e7f-9952-40ac-98da-4745dff32493n@googlegroups.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Lines: 48
Message-ID: <20F0M.291598$b7Kc.5484@fx39.iad>
X-Complaints-To: abuse@easynews.com
Organization: Forte - www.forteinc.com
X-Complaints-Info: Please be sure to forward a copy of ALL headers otherwise we will be unable to process your complaint properly.
Date: Fri, 21 Apr 2023 19:30:05 -0400
X-Received-Bytes: 5032
 by: Richard Damon - Fri, 21 Apr 2023 23:30 UTC

On 4/20/23 10:41 PM, Rick C wrote:
> On Thursday, April 20, 2023 at 10:09:35 PM UTC-4, Richard Damon wrote:
>> On 4/19/23 10:06 PM, Rick C wrote:
>>> This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
>>>
>>> I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.
>>>
>>> I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.
>>>
>>> I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.
>>>
>>> I keep thinking there is a different way of looking at this to achieve the result I want...
>>>
>>> Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact.
>>>
>>> This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.
>>>
>> IF I understand you correctly, what you want is for the file to compute
>> to some "checksum" that comes from the basic contents of the file, and
>> then you want to add the "checksum" into the file so the program itself
>> can print its checksum.
>>
>> One fact to remember, is that "cryptographic hashes" were invented
>> because it was too easy to create a faked file that matches a
>> non-crptographic hash/checksum, so that couldn't be a key to make sure
>> you really had the right file in the presence of a determined enemy, but
>> the checksums were good enough to catch "random" errors.
>>
>> This means that you can add the checksum into the file, and some
>> additional bytes (likely at the end) and by knowing the propeties of the
>> checksum algorithm, compute a value for those extra bytes such that the
>> "undo" the changes caused by adding the checksum bytes to file.
>>
>> I'm not sure exactly how to computes these, but the key is that you add
>> something at the end of the file to get the checksum back to what the
>> original file had before you added the checksum into the file.
>
> Yeah, for a simple checksum, I think that would be easy, at least if "checksum" means a bitwise XOR operation. If the checksum and extra bytes are both 16 bits, this would also work for an arithmetic checksum where each 16 bit word were added into the checksum. All the carries would cascade out of the upper 16 bits from adding the inserted checksum and it's 2's complement.
>
> I don't even want to think about using a CRC to try to do this.
>

Its is a bit of work, but even a 32-bit CRC will be solvable to find the
reverse equation. You can do the work once generically, and get a
formula that computes the value you need to put into the final bytes to
get the CRC of the file back to the CRC it was before adding the CRC and
the extra bytes. It wouldn't surprise me if the formula isn't published
somewhere for the common CRCs.

Re: Embedding a Checksum in an Image File

<1f26bbc6-964c-4081-b9f6-f460a799c9b0n@googlegroups.com>

 copy mid

https://www.novabbs.com/devel/article-flat.php?id=1460&group=comp.arch.embedded#1460

 copy link   Newsgroups: comp.arch.embedded
X-Received: by 2002:a05:6214:18cb:b0:5ef:4f8a:2306 with SMTP id cy11-20020a05621418cb00b005ef4f8a2306mr1098713qvb.0.1682121144467;
Fri, 21 Apr 2023 16:52:24 -0700 (PDT)
X-Received: by 2002:a05:622a:14d2:b0:3e4:e8c2:a502 with SMTP id
u18-20020a05622a14d200b003e4e8c2a502mr2050507qtx.7.1682121144108; Fri, 21 Apr
2023 16:52:24 -0700 (PDT)
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!panix!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!peer03.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.arch.embedded
Date: Fri, 21 Apr 2023 16:52:23 -0700 (PDT)
In-Reply-To: <a4495c87-c68d-4c87-94aa-701c21bdd19cn@googlegroups.com>
Injection-Info: google-groups.googlegroups.com; posting-host=180.150.36.196; posting-account=ZaAWpAoAAABFnmcUGQHISv7vlLrTdUCZ
NNTP-Posting-Host: 180.150.36.196
References: <116ff07e-5e25-469a-90a0-9474108aadd3n@googlegroups.com>
<66e9fb2a-b9e6-4597-9afa-6572caaa8ca1n@googlegroups.com> <a4495c87-c68d-4c87-94aa-701c21bdd19cn@googlegroups.com>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <1f26bbc6-964c-4081-b9f6-f460a799c9b0n@googlegroups.com>
Subject: Re: Embedding a Checksum in an Image File
From: brian.co...@gmail.com (Brian Cockburn)
Injection-Date: Fri, 21 Apr 2023 23:52:24 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 5580
 by: Brian Cockburn - Fri, 21 Apr 2023 23:52 UTC

On Friday, April 21, 2023 at 10:12:49 PM UTC+10, Rick C wrote:
> On Friday, April 21, 2023 at 4:53:18 AM UTC-4, Brian Cockburn wrote:
> > On Thursday, April 20, 2023 at 12:06:36 PM UTC+10, Rick C wrote:
> > > This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
> > >
> > > I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.
> > >
> > > I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.
> > >
> > > I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.
> > >
> > > I keep thinking there is a different way of looking at this to achieve the result I want...
> > >
> > > Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact.
> > >
> > > This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.
> > >
> > > --
> > >
> > > Rick C.
> > >
> > > - Get 1,000 miles of free Supercharging
> > > - Tesla referral code - https://ts.la/richard11209
> > Rick, What is the purpose of this? Is it (1) to be able to externally identify a binary, as one might a ROM image by computing a checksum? Is it (2) for a run-able binary to be able to check itself? This would of course only be able to detect corruption, not tampering. Is it (3) for the loader (whatever that might be) to be able to say 'this binary has the correct checksum' and only jump to it if it does? Again this would only be able to detect corruption, not tampering. Are you hoping for more than corruption detection?
> This is simply to be able to say this version is unique, regardless of what the version number says. Version numbers are set manually and not always done correctly. I'm looking for something as a backup so that if the checksums are different, I can be sure the versions are not the same.
>
> The less work involved, the better.
>
> --
>
> Rick C.
>
> ++ Get 1,000 miles of free Supercharging
> ++ Tesla referral code - https://ts.la/richard11209
Rick, so you want the executable to, as part of its execution, print on the console the 'checksum' of itself? Or do you want to be able to inspect the executable with some other tool to calculate its 'checksum'? For the latter there are lots of tools to do that (your OS or PROM programmer for instance), for the former you need to embed the calculation code into the executable (along with the length over which to calculate) and run this when asked. Neither of these involve embedding the 'checksum' value.
And just to be sure I understand what you wrote in a somewhat convoluted way. When you have two binary executables that report the same version number you want to be able to distinguish them with a 'checksum', right?

Pages:1234
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor