On my hard disk, I have 350 files of various sizes (all less than a megabyte).

When I'm supplied with a new file, I want to compare it to all my other 350 files. 9 times out of 10, I will have an exact match to one of my 350 files.

Checking for an identical file among my 350 is the easy part. Also I can stop the search as soon as I find an identical file. I can even compare pre-computed sha256 sums to speed things up further.

The other 10% of the time, the new file will be very very similar to one of the 350 files. In these cases, I want to compare the new file to each of the 350 files, and I want to give each of the 350 files a score from 0 to 100 as to how similar it is to the new file.

Once I have all these scores, I want to pick out the file that is the best match.

I know of two common Linux programs that can help me here.
1) 'diff' is commonly used for text files and source code, and it operates on a line by line basis
2) 'cmp' is used for binary files and just points out individual bytes that don't match

My 350 files contain text however it is XML and there's no new lines in it. However since XML contains a lot of open angle brackets (i.e. < ), all I have to do is put a new line before every <, like this:

cat my_file.xml | sed 's/</\n</g'

After making this alteration, these files will be well-suited to be compared with 'diff'.

The best command I can find so far to get a score of how similar two files area is as follows:

diff -y --suppress-common-lines file1 file2 | wc -l

This command will give me an integer specifying how many lines are different in the two files.

Has anyone got any better idea?

Frederick Gotham <cauldwell.thomas@gmail.com> writes:
>Has anyone got any better idea?

Compress each file individually yielding file sizes s0 and
s1 for the compressed files. Now, compress the concatenation
yielding s. The ratio (s0+s1)/s might measure the similarity
if an appropriate algorithm was chosen for the compression.
Try different compression programs with different settings.

See also:
Normalized Compression Distance (NCD)

ram@zedat.fu-berlin.de (Stefan Ram) writes:
>Normalized Compression Distance (NCD)

Or one can choose n numerical properties p[i] of the files
(like the file size), normalize them to the range [0, 1]
and then calculate the distance between files f and g as

d( f, g )= sqrt( sum over i of:( p[i]( f )-p[i]( g ))^2 )

. The choice of the p[i] can be guided by knowledge of the
purpose of the measurement.

See also: normalized euclidean distance (NED), string
similarity measures, edit distance, bag distance, N-gram
measures, Jaro variant, Smith-Waterman distance, Editex,
and Syllable alignment.

On Monday, 11 October 2021 at 23:52:30 UTC+1, Frederick Gotham wrote:
> On my hard disk, I have 350 files of various sizes (all less than a megabyte).
>
> When I'm supplied with a new file, I want to compare it to all my other 350 files. 9 times out of 10, I will have an exact match to one of my 350 files.
>
> Checking for an identical file among my 350 is the easy part. Also I can stop the search as soon as I find an identical file. I can even compare pre-computed sha256 sums to speed things up further.
>
> The other 10% of the time, the new file will be very very similar to one of the 350 files. In these cases, I want to compare the new file to each of the 350 files, and I want to give each of the 350 files a score from 0 to 100 as to how similar it is to the new file.
>
> Once I have all these scores, I want to pick out the file that is the best match.
>
> I know of two common Linux programs that can help me here.
> 1) 'diff' is commonly used for text files and source code, and it operates on a line by line basis
> 2) 'cmp' is used for binary files and just points out individual bytes that don't match
>
> My 350 files contain text however it is XML and there's no new lines in it. However since XML contains a lot of open angle brackets (i.e. < ), all I have to do is put a new line before every <, like this:
>
> cat my_file.xml | sed 's/</\n</g'
>
> After making this alteration, these files will be well-suited to be compared with 'diff'.
>
> The best command I can find so far to get a score of how similar two files area is as follows:
>
> diff -y --suppress-common-lines file1 file2 | wc -l
>
> This command will give me an integer specifying how many lines are different in the two files.
>
> Has anyone got any better idea?

What is your C question here?

You'd probably be more on-topic at stack exchange's code review site...

On Monday, 11 October 2021 at 23:52:30 UTC+1, Frederick Gotham wrote:
> On my hard disk, I have 350 files of various sizes (all less than a megabyte).
>
> When I'm supplied with a new file, I want to compare it to all my other 350 files. 9 times out of 10, I will have an exact match to one of my 350 files.
>
> Checking for an identical file among my 350 is the easy part. Also I can stop the search as soon as I find an identical file. I can even compare pre-computed sha256 sums to speed things up further.
>
> The other 10% of the time, the new file will be very very similar to one of the 350 files. In these cases, I want to compare the new file to each of the 350 files, and I want to give each of the 350 files a score from 0 to 100 as to how similar it is to the new file.
>
> Once I have all these scores, I want to pick out the file that is the best match.
>
> I know of two common Linux programs that can help me here.
> 1) 'diff' is commonly used for text files and source code, and it operates on a line by line basis
> 2) 'cmp' is used for binary files and just points out individual bytes that don't match
>
> My 350 files contain text however it is XML and there's no new lines in it. However since XML contains a lot of open angle brackets (i.e. < ), all I have to do is put a new line before every <, like this:
>
> cat my_file.xml | sed 's/</\n</g'
>
> After making this alteration, these files will be well-suited to be compared with 'diff'.
>
> The best command I can find so far to get a score of how similar two files area is as follows:
>
> diff -y --suppress-common-lines file1 file2 | wc -l
>
> This command will give me an integer specifying how many lines are different in the two files.
>
> Has anyone got any better idea?
>
You could use an alignment algorithm to align the data in the two files. These are designed for genetic data, but they can be
used on text. Similarity is then the number of perfect matches.

However the real answer depends on what you are doing, how the files differ, and what you consider to be a big change.
For example "Prussia" and "Russia" is only one letter change, but it's as big a difference as "Italy" and "France".
while "The United Kingdom" and "Great Britain" is almost no difference at all, using the same metric. Obviously, an
algorithm would have to be geography-aware to pick this up. It's not always possible to come up with a perfect
measure.

On 10/11/2021 6:35 PM, Stefan Ram wrote:
> ram@zedat.fu-berlin.de (Stefan Ram) writes:
>> Normalized Compression Distance (NCD)
>
> Or one can choose n numerical properties p[i] of the files
> (like the file size), normalize them to the range [0, 1]
> and then calculate the distance between files f and g as
>
> d( f, g )= sqrt( sum over i of:( p[i]( f )-p[i]( g ))^2 )
>
> . The choice of the p[i] can be guided by knowledge of the
> purpose of the measurement.
>
> See also: normalized euclidean distance (NED), string
> similarity measures, edit distance, bag distance, N-gram
> measures, Jaro variant, Smith-Waterman distance, Editex,
> and Syllable alignment.
>
>

God its been ages since did anything close to that. Fwiw, I remember
compressing two files, and adding their sizes together. Then I did what
you said and concatenated the two original files, compressed the result
and compared and contrasted the concatenated compressed size vs the sum
of the two individually compressed file sizes.