Anyone who has more than a few EPUBs will have noticed
that these tend to proliferate, with many bitwise non-identical copies
all having effectively the same content.

Finding bitwise identical files is easy, with a suitable utility,
even if the name and/or extension has been changed.

The problem with EPUBs is that they come in slightly different versions,
which are effectively equal in the way of content,
but which are differing in a few bytes.
The cause is usually that someone has imported the file into Calibre,
and exported it again.
(this may change the .opf file and/or the cover .jpg)

It is of course possible to detect this manually,
for each pair of files.
(convert to .zip, expand, delete duplicate files in the expanded
folders, and see what remains)
This is very labour-intensive, and usually not worth the trouble.
(unless the EPUBs are really huge)

Question: does anyone know of an effective way
to detect -essentially identical- EPUBs?
That is, .epub files with the same readable content,
that differ only in some trivial invisible files?

All 'duplicate finders' that I have seen, including Calibre,
only pretend that they can do this,
but in reality they cannnot,

Jan

J. J. Lodder <nospam@de-ster.demon.nl> wrote:

As expected, no answers.
It would seem that there just isn't an effective way
to detect and eliminate essentially duplicate .epubs,
except labouriously, one by one, and by hand.
The best one can do is just delete similar-sized ones with the same
title, and hope for the best.

This certainly isn't the best of designs,

Jan

PS The .cbr and .cbz formats suffer from the same problem.
Merely using a different compressor on the same folder
will already result in different, but essentially identical files.

> Anyone who has more than a few EPUBs will have noticed
> that these tend to proliferate, with many bitwise non-identical copies
> all having effectively the same content.
>
> Finding bitwise identical files is easy, with a suitable utility,
> even if the name and/or extension has been changed.
>
> The problem with EPUBs is that they come in slightly different versions,
> which are effectively equal in the way of content,
> but which are differing in a few bytes.
> The cause is usually that someone has imported the file into Calibre,
> and exported it again.
> (this may change the .opf file and/or the cover .jpg)
>
> It is of course possible to detect this manually,
> for each pair of files.
> (convert to .zip, expand, delete duplicate files in the expanded
> folders, and see what remains)
> This is very labour-intensive, and usually not worth the trouble.
> (unless the EPUBs are really huge)
>
> Question: does anyone know of an effective way
> to detect -essentially identical- EPUBs?
> That is, .epub files with the same readable content,
> that differ only in some trivial invisible files?
>
> All 'duplicate finders' that I have seen, including Calibre,
> only pretend that they can do this,
> but in reality they cannnot,
>
> Jan

On 19/07/2023 15:04, J. J. Lodder wrote:
> J. J. Lodder <nospam@de-ster.demon.nl> wrote:
>
> As expected, no answers.
> It would seem that there just isn't an effective way
> to detect and eliminate essentially duplicate .epubs,

What makes two epubs "essentially identical"?

What if one used <div>s for each paragraph and the other <p>s? What
about different CSS?

It isn't an easy problem to solve.

That's why things like Calibre and BookFusion exist, to keep your
library in one place and relatively well organized.

--
Chris

Chris Ridd <chrisridd@mac.com> wrote:

> On 19/07/2023 15:04, J. J. Lodder wrote:
> > J. J. Lodder <nospam@de-ster.demon.nl> wrote:
> >
> > As expected, no answers.
> > It would seem that there just isn't an effective way
> > to detect and eliminate essentially duplicate .epubs,
>
> What makes two epubs "essentially identical"?

Having all files that contribute to what you actually see
when you read the book bitwise identical.
(so the actual text files and the jpgs)

The difference that remains are usually only some bits in the .opf file.
(which is merely an invisible html description with metadata)
Or the difference is merely caused by using a different zip utility
on the same folder.

> What if one used <div>s for each paragraph and the other <p>s? What
> about different CSS?
>
> It isn't an easy problem to solve.
>
> That's why things like Calibre and BookFusion exist, to keep your
> library in one place and relatively well organized.

Yes, but organising a library isn't the problem.
The problem is having several, perhaps many, different versions
of essentially the same book.
You would like to eliminate those duplicates,
(without too much work)
just like ordinary duplicate finders do for bitwise identical files.

The problems are in fact -caused- mostly by Calibre.
Importing into and exporting from again usually produces
a slightly different, but essentially identical file.
(depending on the settings)
Calibre may also add invisible files of its own,

Jan

On 2023-07-19 19:26:43 +0000, J. J. Lodder said:

> Chris Ridd <chrisridd@mac.com> wrote:
>
>> On 19/07/2023 15:04, J. J. Lodder wrote:
>>> J. J. Lodder <nospam@de-ster.demon.nl> wrote:
>>>
>>> As expected, no answers.
>>> It would seem that there just isn't an effective way
>>> to detect and eliminate essentially duplicate .epubs,
>>
>> What makes two epubs "essentially identical"?
>
> Having all files that contribute to what you actually see
> when you read the book bitwise identical.
> (so the actual text files and the jpgs)
>
> The difference that remains are usually only some bits in the .opf file.
> (which is merely an invisible html description with metadata)
> Or the difference is merely caused by using a different zip utility
> on the same folder.
>
>> What if one used <div>s for each paragraph and the other <p>s? What
>> about different CSS?
>>
>> It isn't an easy problem to solve.
>>
>> That's why things like Calibre and BookFusion exist, to keep your
>> library in one place and relatively well organized.
>
> Yes, but organising a library isn't the problem.
> The problem is having several, perhaps many, different versions
> of essentially the same book.
> You would like to eliminate those duplicates,
> (without too much work)
> just like ordinary duplicate finders do for bitwise identical files.
>
> The problems are in fact -caused- mostly by Calibre.
> Importing into and exporting from again usually produces
> a slightly different, but essentially identical file.
> (depending on the settings)
> Calibre may also add invisible files of its own,
>
> Jan

You could try here <https://www.mobileread.com/forums/forumdisplay.php?f=110>
--
Cheers ... Mark

On 19/07/2023 20:26, J. J. Lodder wrote:
> Chris Ridd <chrisridd@mac.com> wrote:
>
>> On 19/07/2023 15:04, J. J. Lodder wrote:
>>> J. J. Lodder <nospam@de-ster.demon.nl> wrote:
>>>
>>> As expected, no answers.
>>> It would seem that there just isn't an effective way
>>> to detect and eliminate essentially duplicate .epubs,
>>
>> What makes two epubs "essentially identical"?
>
> Having all files that contribute to what you actually see
> when you read the book bitwise identical.
> (so the actual text files and the jpgs)
>
> The difference that remains are usually only some bits in the .opf file.
> (which is merely an invisible html description with metadata)

No, the OPF is very definitely not HTML.

> Or the difference is merely caused by using a different zip utility
> on the same folder.

Yes, one solution here is to unzip both files to separate directories
and to use something like `diff -r --brief book1 book2`.

> Yes, but organising a library isn't the problem.
> The problem is having several, perhaps many, different versions
> of essentially the same book.
> You would like to eliminate those duplicates,
> (without too much work)
> just like ordinary duplicate finders do for bitwise identical files.

The problem is however much lessened by not having books scattered
around your disk. At least if they're all in the same place, it is
easier to find duplicates.

Personally, duplicate books are not a problem. I'm not obtaining books
in bulk or anything, so that probably makes it easier to avoid dupes.

--
Chris

Chris Ridd <chrisridd@mac.com> wrote:

> On 19/07/2023 20:26, J. J. Lodder wrote:
> > Chris Ridd <chrisridd@mac.com> wrote:
> >
> >> On 19/07/2023 15:04, J. J. Lodder wrote:
> >>> J. J. Lodder <nospam@de-ster.demon.nl> wrote:
> >>>
> >>> As expected, no answers.
> >>> It would seem that there just isn't an effective way
> >>> to detect and eliminate essentially duplicate .epubs,
> >>
> >> What makes two epubs "essentially identical"?
> >
> > Having all files that contribute to what you actually see
> > when you read the book bitwise identical.
> > (so the actual text files and the jpgs)
> >
> > The difference that remains are usually only some bits in the .opf file.
> > (which is merely an invisible html description with metadata)
>
> No, the OPF is very definitely not HTML.

Perhaps, but if this [1] isn't some kind of html, what is it?

> > Or the difference is merely caused by using a different zip utility
> > on the same folder.
>
> Yes, one solution here is to unzip both files to separate directories
> and to use something like `diff -r --brief book1 book2`.

Yes. (supposing that the epub files are not bitwise duplicates)
The results may vary from all files different
to no different files at all.
Usualy there are a few trivial small files that are different.
(often the .opf file)

For example, I have seen machine-generated conversions from .pdf files.
They are excessively ugly inside, and quite different from true epubs
that were created as epubs.

> > Yes, but organising a library isn't the problem.
> > The problem is having several, perhaps many, different versions
> > of essentially the same book.
> > You would like to eliminate those duplicates,
> > (without too much work)
> > just like ordinary duplicate finders do for bitwise identical files.
>
> The problem is however much lessened by not having books scattered
> around your disk. At least if they're all in the same place, it is
> easier to find duplicates.

Yes, but how do you find which really are duplicates?

> Personally, duplicate books are not a problem. I'm not obtaining books
> in bulk or anything, so that probably makes it easier to avoid dupes.

Sure it is not a realy a problem.
Storage cost essentialy nothing.
Epub books are perhaps a megabyte on average,
so storage cost is measured in millicents.
Even very big epubs are rarely more than 100 MB.
It is just that they clutter up directories and lists.

Hence the question is for an utility that will do it automatically,
with just a few mouseclicks for a lot of files.

Jan

--
[1] A typical sample:

<manifest>
<item href="nav.xhtml" id="nav" media-type="application/xhtml+xml"
properties="nav"/>
<item href="toc.ncx" id="toc" media-type="application/x-dtbncx+xml"/>
<item href="xhtml/cover.xhtml" id="html-cover-page"
media-type="application/xhtml+xml"/>
<item href="xhtml/title.xhtml" id="tit"
media-type="application/xhtml+xml"/>
<item href="xhtml/mini_toc.xhtml" id="mini_toc"
media-type="application/xhtml+xml"/>
<item href="xhtml/copyrightnotice.xhtml" id="copyrightnotice"
media-type="application/xhtml+xml"/>
<item href="xhtml/dedication.xhtml" id="dedication"
media-type="application/xhtml+xml"/>
<item href="xhtml/foreword.xhtml" id="foreword"
media-type="application/xhtml+xml"/>
<item href="xhtml/part1.xhtml" id="part1"
media-type="application/xhtml+xml"/>
<item href="xhtml/chapter1.xhtml" id="chapter1"
media-type="application/xhtml+xml"/>
<item href="xhtml/chapter2.xhtml" id="chapter2"
media-type="application/xhtml+xml"/>
<item href="xhtml/part2.xhtml" id="part2"
media-type="application/xhtml+xml"/>
[-]
media-type="application/xhtml+xml"/>
<item href="xhtml/acknowledgments.xhtml" id="acknowledgments"
media-type="application/xhtml+xml"/>
<item href="xhtml/abouttheauthor.xhtml" id="abouttheauthor"
media-type="application/xhtml+xml"/>
<item href="xhtml/newsletter.xhtml" id="newsletter"
media-type="application/xhtml+xml"/>
<item href="xhtml/contents.xhtml" id="contents"
media-type="application/xhtml+xml"/>
<item href="xhtml/copyright.xhtml" id="copyright"
media-type="application/xhtml+xml"/>
<item href="images/9781250876577.jpg" id="cover-image"
media-type="image/jpeg" properties="cover-image"/>
<item href="images/author.jpg" id="image-002" media-type="image/jpeg"/>
<item href="images/title.jpg" id="image-003" media-type="image/jpeg"/>
<item href="styles/stylesheet.css" id="css" media-type="text/css"/>
<item href="images/NewsletterSignup.jpg" id="image-0002"
media-type="image/jpeg"/>
</manifest>

On 20/07/2023 20:10, J. J. Lodder wrote:
> Chris Ridd <chrisridd@mac.com> wrote:
>
>> On 19/07/2023 20:26, J. J. Lodder wrote:
>>> Chris Ridd <chrisridd@mac.com> wrote:
>>>
>>>> On 19/07/2023 15:04, J. J. Lodder wrote:
>>>>> J. J. Lodder <nospam@de-ster.demon.nl> wrote:
>>>>>
>>>>> As expected, no answers.
>>>>> It would seem that there just isn't an effective way
>>>>> to detect and eliminate essentially duplicate .epubs,
>>>>
>>>> What makes two epubs "essentially identical"?
>>>
>>> Having all files that contribute to what you actually see
>>> when you read the book bitwise identical.
>>> (so the actual text files and the jpgs)
>>>
>>> The difference that remains are usually only some bits in the .opf file.
>>> (which is merely an invisible html description with metadata)
>>
>> No, the OPF is very definitely not HTML.
>
> Perhaps, but if this [1] isn't some kind of html, what is it?

XML. See the current specification for it:
https://www.w3.org/TR/epub-33/#sec-package-doc

>> Personally, duplicate books are not a problem. I'm not obtaining books
>> in bulk or anything, so that probably makes it easier to avoid dupes.

BTW, I meant for me they're not a problem as I don't purchase books twice.

> Sure it is not a realy a problem.
> Storage cost essentialy nothing.

Correct, the space required is irrelevant. You just don't want to read a
book twice accidentally - or you want to make sure you have the cleanest
version of the book.

--
Chris