Robocopy has lots of options, heaven knows, but it's
missing one I want: don't copy when files are
identical, even if one has a later timestamp.

This arises when I'm copying some particular
subdirectories of %APPDATA% to my USB stick for
synching with my other Windows computer. A number of
programs seem to write out their settings when they
exit, even if there were no changes, and I'd prefer not
to copy a file over a duplicate of itself.

Hopefully there's a way to do this, and I just missed
it in the documentation.

--
Stan Brown, Tehachapi, California, USA
https://BrownMath.com/
Shikata ga nai...

On 22/10/2023 14:27, Stan Brown wrote:
> Robocopy has lots of options, heaven knows, but it's
> missing one I want: don't copy when files are
> identical, even if one has a later timestamp.
>
> This arises when I'm copying some particular
> subdirectories of %APPDATA% to my USB stick for
> synching with my other Windows computer. A number of
> programs seem to write out their settings when they
> exit, even if there were no changes, and I'd prefer not
> to copy a file over a duplicate of itself.
>
> Hopefully there's a way to do this, and I just missed
> it in the documentation.
>
>
"robocopy /xc /xn /xo source destination "

e.g.
robocopy /xc /xn /xo %APPDATA% e:\tmp

Jack wrote:
> On 22/10/2023 14:27, Stan Brown wrote:
>> Robocopy has lots of options, heaven knows, but it's
>> missing one I want: don't copy when files are
>> identical, even if one has a later timestamp.
>>
>> This arises when I'm copying some particular
>> subdirectories of %APPDATA% to my USB stick for
>> synching with my other Windows computer. A number of
>> programs seem to write out their settings when they
>> exit, even if there were no changes, and I'd prefer not
>> to copy a file over a duplicate of itself.
>>
>> Hopefully there's a way to do this, and I just missed
>> it in the documentation.
>>
>>
> "robocopy /xc /xn /xo source destination"
>
> e.g.
> robocopy /xc /xn /xo %APPDATA% e:\tmp
>

But the switch /XC :: eXclude Changed files

The OP wants "eXclude Unchanged files" - which is the default, except
that they might have different timestamps, which then means they get copied.

Perhaps use "/XO : Exclude Older" and "/XN : Exclude Newer" together.

Personally, I would worry about a program's settings always being the
same from one computer to another, so would prefer to identify the
settings file type and exclude it. So perhaps:

\XF *.INI

.... to exclude all files of the form *.INI

I'm not sure how Robocopy identifies whether one file is the same as
another. I think it is matching name, size and datestamp. Clearly it
would be possible to generate two files with these paramaeter matching,
but different content. Not sure any WIndows filesystem stores a
checksum of every file ???

--
Graham J

Jack wrote:
> On 22/10/2023 14:27, Stan Brown wrote:
>> Robocopy has lots of options, heaven knows, but it's
>> missing one I want: don't copy when files are
>> identical, even if one has a later timestamp.
>>
>> This arises when I'm copying some particular
>> subdirectories of %APPDATA% to my USB stick for
>> synching with my other Windows computer. A number of
>> programs seem to write out their settings when they
>> exit, even if there were no changes, and I'd prefer not
>> to copy a file over a duplicate of itself.
>>
>> Hopefully there's a way to do this, and I just missed
>> it in the documentation.
>>
>>
> "robocopy /xc /xn /xo source destination"
>
> e.g.
> robocopy /xc /xn /xo %APPDATA% e:\tmp
>

But the switch /XC :: eXclude Changed files

The OP wants "eXclude Unchanged files" - which is the default, except
that they might have different timestamps, which then means they get copied.

Perhaps use "/XO : Exclude Older" and "/XN : Exclude Newer" together.

Personally, I would worry about a program's settings always being the
same from one computer to another, so would prefer to identify the
settings file type and exclude it. So perhaps:

\XF *.INI

.... to exclude all files of the form *.INI

I'm not sure how Robocopy identifies whether one file is the same as
another. I think it is matching name, size and datestamp. Clearly it
would be possible to generate two files with these paramaeter matching,
but different content. Not sure any WIndows filesystem stores a
checksum of every file ???

--
Graham J

On Sun, 22 Oct 2023 17:37:03 +0100, Graham J wrote:
>
> Jack wrote:
> > On 22/10/2023 14:27, Stan Brown wrote:
> >> Robocopy has lots of options, heaven knows, but it's
> >> missing one I want: don't copy when files are
> >> identical, even if one has a later timestamp.
> >>
> > "robocopy /xc /xn /xo source destination"
> >
> > e.g.
> > robocopy /xc /xn /xo %APPDATA% e:\tmp
> >
>
> But the switch /XC :: eXclude Changed files
>
> The OP wants "eXclude Unchanged files" - which is the default, except
> that they might have different timestamps, which then means they get copied.

"Changed files" in the Robocopy documentation are files
with the same source and destination timestamp but
different file sizes. It's hard for me to see any
process that could cause such files to exist, but in
any case that's not what I'm talking about.

> Perhaps use "/XO : Exclude Older" and "/XN : Exclude Newer" together.

I'm already using /XO, which excludes files where
source timestamp is older than destination timestamp. I
really don't think I want /XN, to exclude all files
where the source is newer than the destination. I want
to exclude a _subset_ of newer, the subset where the
source and destination files are identical even though
the source's timestamp is newer.

> Personally, I would worry about a program's settings always being the
> same from one computer to another,

That's a valid point. I distinguish between subfolders
of %APPDATA% where I do want to keep all settings in
sync and those where I do not. For the latter, I copy
them to a folder on the USB stick whose name has
%COMPUTERNAME% appended.

> I'm not sure how Robocopy identifies whether one file is the same as
> another. I think it is matching name, size and datestamp. Clearly it
> would be possible to generate two files with these paramaeter matching,
> but different content. Not sure any WIndows filesystem stores a
> checksum of every file ???

That's what I was wondering too.

--
Stan Brown, Tehachapi, California, USA
https://BrownMath.com/
Shikata ga nai...

On 10/22/2023 5:21 PM, Stan Brown wrote:
> On Sun, 22 Oct 2023 17:37:03 +0100, Graham J wrote:
>>
>> Jack wrote:
>>> On 22/10/2023 14:27, Stan Brown wrote:
>>>> Robocopy has lots of options, heaven knows, but it's
>>>> missing one I want: don't copy when files are
>>>> identical, even if one has a later timestamp.
>>>>
>>> "robocopy /xc /xn /xo source destination"
>>>
>>> e.g.
>>> robocopy /xc /xn /xo %APPDATA% e:\tmp
>>>
>>
>> But the switch /XC :: eXclude Changed files
>>
>> The OP wants "eXclude Unchanged files" - which is the default, except
>> that they might have different timestamps, which then means they get copied.
>
> "Changed files" in the Robocopy documentation are files
> with the same source and destination timestamp but
> different file sizes. It's hard for me to see any
> process that could cause such files to exist, but in
> any case that's not what I'm talking about.
>
>> Perhaps use "/XO : Exclude Older" and "/XN : Exclude Newer" together.
>
> I'm already using /XO, which excludes files where
> source timestamp is older than destination timestamp. I
> really don't think I want /XN, to exclude all files
> where the source is newer than the destination. I want
> to exclude a _subset_ of newer, the subset where the
> source and destination files are identical even though
> the source's timestamp is newer.
>
>> Personally, I would worry about a program's settings always being the
>> same from one computer to another,
>
> That's a valid point. I distinguish between subfolders
> of %APPDATA% where I do want to keep all settings in
> sync and those where I do not. For the latter, I copy
> them to a folder on the USB stick whose name has
> %COMPUTERNAME% appended.
>
>> I'm not sure how Robocopy identifies whether one file is the same as
>> another. I think it is matching name, size and datestamp. Clearly it
>> would be possible to generate two files with these paramaeter matching,
>> but different content. Not sure any WIndows filesystem stores a
>> checksum of every file ???
>
> That's what I was wondering too.
>

The OS doesn't keep checksums at the logical file layer.

The drive keeps checksums at the sector (physical layer),
and can do error correction in response. On an SSD for example,
there is a 512 byte sector, and there are 51 bytes of syndrome
for usage in computing corrections to the 512 bytes. Which is
very large, compared to the Fire polynomial syndrome the
old hard drives used. The Fire polynominal could only correct
a few random one-bit errors.

Generating checksums at the file level, is a slow process.

The hashdeep/md5deep program, will generate checksums for an
entire disk if you want. Minus the sections of the disk having
permission problems (even for Administrator).

If Robocopy does not have the requires semantics, then a
"sync program" should be used in its place. A "sync program"
doing the extra work, of generating checksums as it goes.

Paul

On Sun, 22 Oct 2023 19:24:03 -0400, Paul wrote:
>
> On 10/22/2023 5:21 PM, Stan Brown wrote:
> > On Sun, 22 Oct 2023 17:37:03 +0100, Graham J wrote:
> >> I'm not sure how Robocopy identifies whether one file is the same as
> >> another. I think it is matching name, size and datestamp. Clearly it
> >> would be possible to generate two files with these paramaeter matching,
> >> but different content. Not sure any WIndows filesystem stores a
> >> checksum of every file ???
> >
> > That's what I was wondering too.
> >
>
> The OS doesn't keep checksums at the logical file layer.
>
> The drive keeps checksums at the sector (physical layer),
> and can do error correction in response. On an SSD for example,
> there is a 512 byte sector, and there are 51 bytes of syndrome
> for usage in computing corrections to the 512 bytes. Which is
> very large, compared to the Fire polynomial syndrome the
> old hard drives used. The Fire polynominal could only correct
> a few random one-bit errors.
>
> Generating checksums at the file level, is a slow process.
>
> The hashdeep/md5deep program, will generate checksums for an
> entire disk if you want. Minus the sections of the disk having
> permission problems (even for Administrator).
>
> If Robocopy does not have the requires semantics, then a
> "sync program" should be used in its place. A "sync program"
> doing the extra work, of generating checksums as it goes.

Well, TCCLE (which I use instead of CMD.EXE) fas a function to
compute CRC32 at the file level. It seems to go pretty fast. I'll
either build that into my synching batch file, or just decide to live
with the unnecessary copies. We're not talking about massive files,
or many of them: maybe a dozen, most around 1 KB or less.

P.S. I googled MD5 versus CRC32 before posting, and an article in
stackoverflow explained that CRC32 was the thing to use if you want
to know if two files are different.

<https://stackoverflow.com/questions/16122067/md5-vs-crc32-which-
ones-better-for-common-use>

"If you want to check if two files are the same, CRC32 checksum is
the way to go because it's faster than MD5. But be careful: CRC only
reliably tells you if the binaries are different; it doesn't tell you
if they're identical."
--
Stan Brown, Tehachapi, California, USA https://BrownMath.com/
Shikata ga nai...

Stan Brown wrote:
> Robocopy has lots of options, heaven knows, but it's
> missing one I want: don't copy when files are
> identical, even if one has a later timestamp.
>
> This arises when I'm copying some particular
> subdirectories of %APPDATA% to my USB stick for
> synching with my other Windows computer. A number of
> programs seem to write out their settings when they
> exit, even if there were no changes, and I'd prefer not
> to copy a file over a duplicate of itself.
>
> Hopefully there's a way to do this, and I just missed
> it in the documentation.

To do so source and target file must be read and compared
and the target written only if a difference was found.
Robocopy has no such feature yet.

My FFC (Fast File Copy) has -u (avoid unnecessary writes)
for that but lacks lots of Robocopy's features.

https://www.uwe-sieber.de/filetools_e.html

Uwe

On 10/22/2023 11:15 PM, Stan Brown wrote:
> On Sun, 22 Oct 2023 19:24:03 -0400, Paul wrote:
>>
>> On 10/22/2023 5:21 PM, Stan Brown wrote:
>>> On Sun, 22 Oct 2023 17:37:03 +0100, Graham J wrote:
>>>> I'm not sure how Robocopy identifies whether one file is the same as
>>>> another. I think it is matching name, size and datestamp. Clearly it
>>>> would be possible to generate two files with these paramaeter matching,
>>>> but different content. Not sure any WIndows filesystem stores a
>>>> checksum of every file ???
>>>
>>> That's what I was wondering too.
>>>
>>
>> The OS doesn't keep checksums at the logical file layer.
>>
>> The drive keeps checksums at the sector (physical layer),
>> and can do error correction in response. On an SSD for example,
>> there is a 512 byte sector, and there are 51 bytes of syndrome
>> for usage in computing corrections to the 512 bytes. Which is
>> very large, compared to the Fire polynomial syndrome the
>> old hard drives used. The Fire polynominal could only correct
>> a few random one-bit errors.
>>
>> Generating checksums at the file level, is a slow process.
>>
>> The hashdeep/md5deep program, will generate checksums for an
>> entire disk if you want. Minus the sections of the disk having
>> permission problems (even for Administrator).
>>
>> If Robocopy does not have the requires semantics, then a
>> "sync program" should be used in its place. A "sync program"
>> doing the extra work, of generating checksums as it goes.
>
> Well, TCCLE (which I use instead of CMD.EXE) fas a function to
> compute CRC32 at the file level. It seems to go pretty fast. I'll
> either build that into my synching batch file, or just decide to live
> with the unnecessary copies. We're not talking about massive files,
> or many of them: maybe a dozen, most around 1 KB or less.
>
> P.S. I googled MD5 versus CRC32 before posting, and an article in
> stackoverflow explained that CRC32 was the thing to use if you want
> to know if two files are different.
>
> <https://stackoverflow.com/questions/16122067/md5-vs-crc32-which-
> ones-better-for-common-use>
>
> "If you want to check if two files are the same, CRC32 checksum is
> the way to go because it's faster than MD5. But be careful: CRC only
> reliably tells you if the binaries are different; it doesn't tell you
> if they're identical."
>

The characteristics of CRC32, do not prevent collisions.

I tried to get the original paper, when I was at work, but
the inter-library loan system would not give it to me. It claimed
the info was "military classified". I think that was just someone
being lazy.

All I could determine at the time I was using CRCs for hardware
design, is above a certain message size, the collision resistance
is a fixed finite amount, and unsuited to every purpose. Maybe it
would buy you three orders of magnitude better performance, on a wire,
with retries as part of the protocol. We used CRC32, for an experimental
device that used "very large packets". I did the CRC32 in hardware
(not with a processor), and it was running at wire-speed as a result.
The implementation would seem a joke today (it's just parallel CRC
with look-up tables for speed, in high speed OTP ROMs).

As you go up the food chain, things improve. Better hash, means
lower probability of collisions.

MD5 is no good for authentication, but it is good for hashing.
It's the fastest one you could reasonably use. You can hack a source
file, to create the same MD5 output value, using a Pentium 4,
in 60-90 seconds or so. If someone wants a file to have the same
MD5, it is easy to hack. But if the project does not involve
"opponents" or "tripwire", MD5 is fine for casual usage. Like
while syncing files, it would be fine for that.

SHA1 doesn't have that problem. At least, that we know of. We have
gone further up the food chain, in an attempt to keep it that way.

You can use whatever hash calc you want, until the calc is slower
than the drive. Since 7ZIP has these in the menu, I can easily test them.

CRC32 2410 MB/sec
CRC64 1627 MB/sec
SHA1 771 MB/sec
SHA256 393 MB/sec (the basis of Bitcoin)

Since a good NVMe can go faster than that, you want to be using
hashdeep, because you can set -j 8 kind of thing, and have
eight threads of execution, and do a better job of soaking up
available bandwidth.

If hashing a HDD, you don't want to smother it in seeks, so -j 1
might be a better policy in that case.

Paul

On Mon, 23 Oct 2023 09:13:59 +0200, Uwe Sieber wrote:
>
> Stan Brown wrote:
> > Robocopy has lots of options, heaven knows, but it's
> > missing one I want: don't copy when files are
> > identical, even if one has a later timestamp.
> >
> > This arises when I'm copying some particular
> > subdirectories of %APPDATA% to my USB stick for
> > synching with my other Windows computer. A number of
> > programs seem to write out their settings when they
> > exit, even if there were no changes, and I'd prefer not
> > to copy a file over a duplicate of itself.
> >
> > Hopefully there's a way to do this, and I just missed
> > it in the documentation.
>
> To do so source and target file must be read and compared
> and the target written only if a difference was found.
> Robocopy has no such feature yet.
>
> My FFC (Fast File Copy) has -u (avoid unnecessary writes)
> for that but lacks lots of Robocopy's features.
>
> https://www.uwe-sieber.de/filetools_e.html

Thanks for posting. I didn't see anything like /XF (exclude files by
name with wildcards) and /XD (exclude directories by name with
wildcards), both of which I need, but I certainly wish Robocopy had
something like your -u option.

You say FFC is optimnized for large fies, so it might not be suitable
anyway for %APPPDATA%, where most of the files are settings files of
about 1 KB or less.

Just a note: the descriptions of -sr and -sw both end in the three
characters \, n, " -- I'm guessing that was not intended?

--
Stan Brown, Tehachapi, California, USA https://BrownMath.com/
Shikata ga nai...

Stan Brown wrote:
> On Mon, 23 Oct 2023 09:13:59 +0200, Uwe Sieber wrote:
>> Stan Brown wrote:
>>> Robocopy has lots of options, heaven knows, but it's
>>> missing one I want: don't copy when files are
>>> identical, even if one has a later timestamp.
>>>
>>> This arises when I'm copying some particular
>>> subdirectories of %APPDATA% to my USB stick for
>>> synching with my other Windows computer. A number of
>>> programs seem to write out their settings when they
>>> exit, even if there were no changes, and I'd prefer not
>>> to copy a file over a duplicate of itself.
>>>
>>> Hopefully there's a way to do this, and I just missed
>>> it in the documentation.
>> To do so source and target file must be read and compared
>> and the target written only if a difference was found.
>> Robocopy has no such feature yet.
>>
>> My FFC (Fast File Copy) has -u (avoid unnecessary writes)
>> for that but lacks lots of Robocopy's features.
>>
>> https://www.uwe-sieber.de/filetools_e.html
>
> Thanks for posting. I didn't see anything like /XF (exclude files by
> name with wildcards) and /XD (exclude directories by name with
> wildcards), both of which I need, but I certainly wish Robocopy had
> something like your -u option.
>
> You say FFC is optimnized for large fies, so it might not be suitable
> anyway for %APPPDATA%, where most of the files are settings files of
> about 1 KB or less.
>
> Just a note: the descriptions of -sr and -sw both end in the three
> characters \, n, " -- I'm guessing that was not intended?
>

For excluding files there is -e, it has effect on file and folder names.
Making separate params for files and folders but would be easy.

It just bypasses the Windows file cache, so on mechanical disks
it causes lots of head movements. But from my experience a flooded
write cache has a similar effect.

The two \n" came into the FFC.txt by copy+paste from the C-source :-)

Uwe

Uwe Sieber wrote:

yaermjy5sks5