Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> wrote:
> Hi Frank,
>
> > So the questions stands: What do *Thunderbird* users experience?
>
> As a user of Thunderbird, and reading both news.software.readers and
> news.software.nntp, I confirm that when I read (and mark as so) an
> article in this thread either in news.software.readers or
> news.software.nntp first, it still shows up unread in the other newsgroup.
> I'm using the latest version of Thunderbird (115.3.2), and this bug was
> present in previous versions too.
>
> My .rc file is named "newsrc-servername" (newsrc-news.trigofacile.com)
> and seems normal, but not updated when a crossposted article is read in
> one newsgroup.

Hi Julien,

From your response and other recent responses, it indeed seems that
this is a (very, very long standing) bug and - as Russ indicates -
perhaps even an ill-conceived 'design choice' (Jamie (Zawinski) was
known to have ... ahum ... 'special' ideas about how NetNews should
work :-().

But to be sure:

In the above mentioned context, are you reading both groups from the
same server? (Or better yet, perhaps you have configured only one server
in Thunderbird?)

I assume you read both groups from the same server, because you only
mention one .rc file and one servername, but need to be sure.

The reason for my question is obvious: Thunderbird should use article
numbers to mark articles are read in both groups, but (as you are of
course all too aware of), article numbers are server specific, so
marking crossposted articles are read across multiple servers is
(nearly) impossible.

Le 16/10/2023 13:27, Frank Slootweg a écrit :
>
> But to be sure:
>
> In the above mentioned context, are you reading both groups from the
> same server? (Or better yet, perhaps you have configured only one server
> in Thunderbird?)

From my experience in dozens of years (from Netscape 4 to SeaMonkey), yes,
it was always about several groups read from a unique server.

--
Olivier Miakinen

Thus spake Frank Slootweg <this@ddress.is.invalid>

> The reason for my question is obvious: Thunderbird should use article
> numbers to mark articles are read in both groups, but (as you are of
> course all too aware of), article numbers are server specific, so
> marking crossposted articles are read across multiple servers is
> (nearly) impossible.

Obviously, I was referring to a setup where all groups are read from the
same server.
It would have never occurred to me that somebody would expect this to
work across multiple servers ;-)

I don't know any newsreader that is able to do that.

--
Пу́тін — хуйло́
http://www.eternal-september.org

Ray Banana <rayban@raybanana.net> wrote:
> Thus spake Frank Slootweg <this@ddress.is.invalid>
>
>
>
> > The reason for my question is obvious: Thunderbird should use article
> > numbers to mark articles are read in both groups, but (as you are of
> > course all too aware of), article numbers are server specific, so
> > marking crossposted articles are read across multiple servers is
> > (nearly) impossible.
>
> Obviously, I was referring to a setup where all groups are read from the
> same server.
> It would have never occurred to me that somebody would expect this to
> work across multiple servers ;-)

Well, one of the respondents in this thread talked about a multiple
server scenario, so I wanted to be sure that Julien (and you and others)
were talking single server.

> I don't know any newsreader that is able to do that.

Neither do I, but it's just SMOP (Small Matter Of Programming). All
you have to do is to remember all the hundred of thousands / millions of
message-ids of the postings you've ever 'read'. How hard can *that* be!?
:-)

On 10/16/23 10:49, Frank Slootweg wrote:
> Neither do I, but it's just SMOP (Small Matter Of Programming). All
> you have to do is to remember all the hundred of thousands / millions of
> message-ids of the postings you've ever 'read'. How hard can *that* be!?
> :-)

You could just scan for matching ids when you read an article. Or, maybe
you could limit it to the messages read in the last x days, to cut down
on processing time.
--
user <candycane> is generated from /dev/urandom

On 2023-10-16 18:03, candycanearter07 wrote:
> On 10/16/23 10:49, Frank Slootweg wrote:
>>    Neither do I, but it's just SMOP (Small Matter Of Programming). All
>> you have to do is to remember all the hundred of thousands / millions of
>> message-ids of the postings you've ever 'read'. How hard can *that* be!?
>> :-)
>
> You could just scan for matching ids when you read an article. Or, maybe
> you could limit it to the messages read in the last x days, to cut down
> on processing time.

A more practical optimization is to only retain a database of MIDs of
messages that are marked "read" in actively kept message storage. Thus
when a read message is "aged out" of storage, its "read" status is aged
out too. To avoid old messages transiting in and out of view, keep a
count of how many copies are marked read for each MID and count that
down as messages age out. Once in a while, rebuild database from the
stored messages and their "read" status, then do a second pass to mark
duplicates of read messages as read. Bonus: Also index the body
contents hash to catch identical multi-posting.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Jakob Bohm <jb-usenet@wisemo.com.invalid> wrote:
> On 2023-10-16 18:03, candycanearter07 wrote:
> > On 10/16/23 10:49, Frank Slootweg wrote:
> >>    Neither do I, but it's just SMOP (Small Matter Of Programming). All
> >> you have to do is to remember all the hundred of thousands / millions of
> >> message-ids of the postings you've ever 'read'. How hard can *that* be!?
> >> :-)
> >
> > You could just scan for matching ids when you read an article. Or, maybe
> > you could limit it to the messages read in the last x days, to cut down
> > on processing time.
>
> A more practical optimization is to only retain a database of MIDs of
> messages that are marked "read" in actively kept message storage. Thus
> when a read message is "aged out" of storage, its "read" status is aged
> out too. To avoid old messages transiting in and out of view, keep a
> count of how many copies are marked read for each MID and count that
> down as messages age out. Once in a while, rebuild database from the
> stored messages and their "read" status, then do a second pass to mark
> duplicates of read messages as read. Bonus: Also index the body
> contents hash to catch identical multi-posting.

I said "hundred of thousands / millions of message-ids" for a reason.

I have well over a million articles in my "message storage". Keeping
that a amount of messages-ids is already not practical, but *scanning*
(for read/not_read status) them for each an every new (and old) article
is impossible.

But even if you don't keep (that) many articles in your message store,
when you add a new newsserver which has a long retention time, you *do*
have to scan all those articles, just in case there's one which you have
already marked as read.

That's why newsservers and newsreaders use article numbers instead of
message-ids. It makes things much more simple (actually possible instead
of impossible). But yes, it works only for a specific server, not
accross servers and when adding a new group or/and new server you should
only fetch the latest N articles (which is very simple using article
numbers).

Bottom line: Using article numbers is done for a very good reason.

Hi Frank,

>> As a user of Thunderbird, and reading both news.software.readers and
>> news.software.nntp, I confirm that when I read (and mark as so) an
>> article in this thread either in news.software.readers or
>> news.software.nntp first, it still shows up unread in the other newsgroup.
>> I'm using the latest version of Thunderbird (115.3.2), and this bug was
>> present in previous versions too.
>
> In the above mentioned context, are you reading both groups from the
> same server?

Yes.

> (Or better yet, perhaps you have configured only one server in Thunderbird?)

I have configured several servers, but the test case was on the same
server (and newsrc file).

--
Julien ÉLIE

« La vie n'est qu'un tissu de coups de poignard qu'il faut savoir boire
goutte à goutte. »

On 2023-10-16 19:27, Frank Slootweg wrote:
> Jakob Bohm <jb-usenet@wisemo.com.invalid> wrote:
>> On 2023-10-16 18:03, candycanearter07 wrote:
>>> On 10/16/23 10:49, Frank Slootweg wrote:
>>>>    Neither do I, but it's just SMOP (Small Matter Of Programming). All
>>>> you have to do is to remember all the hundred of thousands / millions of
>>>> message-ids of the postings you've ever 'read'. How hard can *that* be!?
>>>> :-)
>>>
>>> You could just scan for matching ids when you read an article. Or, maybe
>>> you could limit it to the messages read in the last x days, to cut down
>>> on processing time.
>>
>> A more practical optimization is to only retain a database of MIDs of
>> messages that are marked "read" in actively kept message storage. Thus
>> when a read message is "aged out" of storage, its "read" status is aged
>> out too. To avoid old messages transiting in and out of view, keep a
>> count of how many copies are marked read for each MID and count that
>> down as messages age out. Once in a while, rebuild database from the
>> stored messages and their "read" status, then do a second pass to mark
>> duplicates of read messages as read. Bonus: Also index the body
>> contents hash to catch identical multi-posting.
>
> I said "hundred of thousands / millions of message-ids" for a reason.
>
> I have well over a million articles in my "message storage". Keeping
> that a amount of messages-ids is already not practical, but *scanning*
> (for read/not_read status) them for each an every new (and old) article
> is impossible.
>
> But even if you don't keep (that) many articles in your message store,
> when you add a new newsserver which has a long retention time, you *do*
> have to scan all those articles, just in case there's one which you have
> already marked as read.
>
> That's why newsservers and newsreaders use article numbers instead of
> message-ids. It makes things much more simple (actually possible instead
> of impossible). But yes, it works only for a specific server, not
> accross servers and when adding a new group or/and new server you should
> only fetch the latest N articles (which is very simple using article
> numbers).
>
> Bottom line: Using article numbers is done for a very good reason.
>

You don't understand what I wrote. Idea would be to use some kind of
available (search optimized) database engine to store the values,
updated incrementally as messages enter and leave the storage or are
marked read. Scanning all messages would be a "once in a while"
database repair in case the database gets out of sync with the message
storage and be done using O(n*log(n)) code.

Database lookups would happen during the initial import of messages (to
see if a newly arrived message was marked read by an earlier user access
to a duplicate) and when displaying a message or its header on screen
(to see if it a duplicate was marked read since its arrival).

Thus common operations change from complexity O(n*n) to O(delta*log(n))
given typically assumed lookup costs in indexed databases.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Frank Slootweg <this@ddress.is.invalid> wrote:

> All you have to do is to remember all the hundred of thousands /
> millions of message-ids of the postings you've ever 'read'. How hard
> can *that* be!?

Thunderbird uses SQLite for the messages databases, one for each folder
(newsgroup). While SQLite is fast for small database sizes, it doesn't
scale well. The bigger the database, the slow queries get, and the
slowdown is not linear. SQLite recommends once you pass into the
terabyte range to find a different solution.

https://www.dbtalks.com/tutorials/learn-sqlite/what-are-the-limitations-of-sqlite
A SQLite database can have maximum 2147483646 pages. Hence the maximum
number of tables in a schema cannot reach more than 2147483646. The
maximum number of rows in a table is 264. The maximum number of columns
is 32767 in a table.

Notice only 264 rows per table. You end up managing thousands of tables
to retain millions of records. Been way too long since I did any work
with SQL, but I thought queries were per table. You'd need a macro that
walked through all the tables to do a query across them all. Maybe for
the table spec you could use a wildcard, like *, to have the query
process across all tables.

Many clients also have an option to either flag or delete articles over
a threshold in age. I have my client purge posts older than 60 days.
Older than that, and the discussion becomes stale. For the expiration,
each table in a database, and for each database (folder/newsgroup),
you'd have to issue a purge and compact.

Overall an SQLite database can hold 140 TB. Is that really big enough?
Consider these are message databases, so it's the entire message that
gets stored, and that includes very long messages that have gads of
quoted content because lots of posters never trim. It would require
even more space to store binaries. There are 20K+ newsgroups, and tons
of messages in each. To be reasonable, some cutoff in expiration would
be needed, but that could result in undoing the read-state tracking, so
the same article in different newsgroups across servers could get out of
sync.

Plus you're talking about caching locally the entire Usenet that a
provider has. It would only be for the newsgroups to which you
subscribe, but some users subscribe to a lot a newsgroups. I was up to
52 at one time, and am now down to 26. Depending on how many newsgroups
to which you subscribe, you could end up having a storage media
requirement equivalent to the Usenet providers. How much free disk
space do you have?

There are some users that run proxy servers for personal use, like using
Hamster to leech from a Usenet provider. That lets them have everything
the Usenet provider has. A lot of work. Very few users operate their
own server.

Hard enough to implement read-state tracking on one server. Massively
more difficult to do across multiple servers. And we're talking about
NNTP clients that could be running on underpowered computers.

I've notice my NNTP client will slow down when its message store gets
large. Right now it's about 700 MB, but it will grow. That's one
reason to have a purge of 60-day old messages to keep quick the
responsiveness of my client. That also means threads that have articles
older than 60 days and articles younger than 60 days end up getting
truncated, so I see a partial thread remaining.

Jakob Bohm <jb-usenet@wisemo.com.invalid> wrote:
> On 2023-10-16 19:27, Frank Slootweg wrote:
> > Jakob Bohm <jb-usenet@wisemo.com.invalid> wrote:
> >> On 2023-10-16 18:03, candycanearter07 wrote:
> >>> On 10/16/23 10:49, Frank Slootweg wrote:
> >>>>    Neither do I, but it's just SMOP (Small Matter Of Programming). All
> >>>> you have to do is to remember all the hundred of thousands / millions of
> >>>> message-ids of the postings you've ever 'read'. How hard can *that* be!?
> >>>> :-)
> >>>
> >>> You could just scan for matching ids when you read an article. Or, maybe
> >>> you could limit it to the messages read in the last x days, to cut down
> >>> on processing time.
> >>
> >> A more practical optimization is to only retain a database of MIDs of
> >> messages that are marked "read" in actively kept message storage. Thus
> >> when a read message is "aged out" of storage, its "read" status is aged
> >> out too. To avoid old messages transiting in and out of view, keep a
> >> count of how many copies are marked read for each MID and count that
> >> down as messages age out. Once in a while, rebuild database from the
> >> stored messages and their "read" status, then do a second pass to mark
> >> duplicates of read messages as read. Bonus: Also index the body
> >> contents hash to catch identical multi-posting.
> >
> > I said "hundred of thousands / millions of message-ids" for a reason.
> >
> > I have well over a million articles in my "message storage". Keeping
> > that a amount of messages-ids is already not practical, but *scanning*
> > (for read/not_read status) them for each an every new (and old) article
> > is impossible.
> >
> > But even if you don't keep (that) many articles in your message store,
> > when you add a new newsserver which has a long retention time, you *do*
> > have to scan all those articles, just in case there's one which you have
> > already marked as read.
> >
> > That's why newsservers and newsreaders use article numbers instead of
> > message-ids. It makes things much more simple (actually possible instead
> > of impossible). But yes, it works only for a specific server, not
> > accross servers and when adding a new group or/and new server you should
> > only fetch the latest N articles (which is very simple using article
> > numbers).
> >
> > Bottom line: Using article numbers is done for a very good reason.
> >
>
> You don't understand what I wrote. Idea would be to use some kind of
> available (search optimized) database engine to store the values,
> updated incrementally as messages enter and leave the storage or are
> marked read. Scanning all messages would be a "once in a while"
> database repair in case the database gets out of sync with the message
> storage and be done using O(n*log(n)) code.
>
> Database lookups would happen during the initial import of messages (to
> see if a newly arrived message was marked read by an earlier user access
> to a duplicate) and when displaying a message or its header on screen
> (to see if it a duplicate was marked read since its arrival).
>
> Thus common operations change from complexity O(n*n) to O(delta*log(n))
> given typically assumed lookup costs in indexed databases.

I understood perfectly what you wrote, but you proposed solution does
not cover the scenarios I described, especially - but not only - the
scenario of adding a new newsserver which has a long retention time.

So I'm afraid "You don't understand what I wrote."

Yes, you can work around that with hacks like not loading old
articles, but that will require use of article numbers instead of
message-ids, so you might as well use article numbers from the start.

Anyway, *if* this issue is a real problem - i.e. not just a minor
inconvience in some corner cases - for some people, they can try to
switch to a newsreader which does aggregrate articles from several
servers (i.e. something which Thunderbird (and Betterbird?) does not do)
or add a 'proxy' which does the aggregration (like Hamster, which I
use).

VanguardLH <V@nguard.lh> wrote:
> Frank Slootweg <this@ddress.is.invalid> wrote:
>
> > All you have to do is to remember all the hundred of thousands /
> > millions of message-ids of the postings you've ever 'read'. How hard
> > can *that* be!?

Sigh!

You post-edited my post by snipping the start and end of my paragraph
and reformatting the rest. By doing that, you've turned my blatantly
obvious joke into something serious. Why?

[...]

> There are some users that run proxy servers for personal use, like using
> Hamster to leech from a Usenet provider. [...]

Actually, Hamster is a solution to this 'problem'.

> Hard enough to implement read-state tracking on one server. Massively
> more difficult to do across multiple servers. And we're talking about
> NNTP clients that could be running on underpowered computers.

Read-state tracking for (not on) one server is trivial. Every
newsreader does it (via article-number range tracking in .newsrc type
files). That Thunderbird et al mess it up, doesn't change that.

Read-state tracking for multiple servers can be done by aggregrating
articles from several servers. I've seen people imply that some
newsreaders can do that, but I have no details (Any takers?). And, as I
said above, Hamster can do this as well. Perhaps Leafnode can do the
same on Linux (the OP uses Linux).

[...]

Frank Slootweg <this@ddress.is.invalid> wrote:

> Read-state tracking for multiple servers can be done by aggregrating
> articles from several servers. I've seen people imply that some
> newsreaders can do that, but I have no details (Any takers?). And, as I
> said above, Hamster can do this as well. Perhaps Leafnode can do the
> same on Linux (the OP uses Linux).

At that point as a cross-server aggregator, using something like Hamster
might suffice. It would collect articles across multiple servers, but
does Hamster then assign its own article numbers (so it could add the
Xref header with its article numbers)?

On 17 Oct 2023 13:29:38 GMT, Frank Slootweg <this@ddress.is.invalid> wrote:
snip
> Read-state tracking for (not on) one server is trivial. Every
>newsreader does it (via article-number range tracking in .newsrc type
>files). That Thunderbird et al mess it up, doesn't change that.
> Read-state tracking for multiple servers can be done by aggregrating
>articles from several servers. I've seen people imply that some
>newsreaders can do that, but I have no details (Any takers?). And, as I
>said above, Hamster can do this as well. Perhaps Leafnode can do the
>same on Linux (the OP uses Linux).

I'm no expert, but 40tude Dialog might do something like what you're
describing, quoting from the Help section on "create virtual groups":
[begin quote]
>How can I... (Procedural Help)
>Almost virtual groups
>There are no virtual groups in Dialog, however you can simply copy or move
>articles from one group to another manually or by using an action rule:
>[a.binary.group]
>!move(a.binary.group;NewsserverOne) bytes %>0
>All previous examples used a folder as the target of the copy/move operation.
>This rule uses a newsgroup as the target. Note that the name of the target
>group and the name of the newsserver where this group is from is separated by
>a semicolon.
>Say, you are using two newsservers named "NewsserverOne" and "NewsserverTwo".
>Both newsservers have the group "a.binary.group", but while most of the
>messages are the same in this group on the two servers, there are some that
>are not available on the other server.
>The first line in the preceding sample makes sure that the following rule is
>only applied to the "a.binary.group" newsgroup.
>The rule itself simply says to move all articles (since the condition "bytes
>%>0" is always met) to the group a.binary.group of NewsserverOne.
>If you retrieve message headers for this group from server NewsserverOne, the
>rule is ignored, since all articles go to the correct group anyway. If you
>retrieve message headers from NewsserverTwo for this group though, they will
>not show up in "a.binary.group (NewsserverTwo)", but in "a.binary.group
>(NewsserverOne)".
>The result is that you joined messages from the same group, but different
>newsservers into one group in Dialog. This is especially useful for binary
>groups to fill missing multipart postings.
[end quote]

while that may not be exactly "aggregrating" articles from multiple servers,
possibly those "action rules" could be customized to make it work after all?

VanguardLH <V@nguard.lh> wrote:
> Frank Slootweg <this@ddress.is.invalid> wrote:
>
> > Read-state tracking for multiple servers can be done by aggregrating
> > articles from several servers. I've seen people imply that some
> > newsreaders can do that, but I have no details (Any takers?). And, as I
> > said above, Hamster can do this as well. Perhaps Leafnode can do the
> > same on Linux (the OP uses Linux).
>
> At that point as a cross-server aggregator, using something like Hamster
> might suffice. It would collect articles across multiple servers, but
> does Hamster then assign its own article numbers (so it could add the
> Xref header with its article numbers)?

Yes, for the user/newsreader side, Hamster works like a news server,
so it indeed assigns its own local article numbers and reports them in a
Xref header (the old Xref header is retained, but renamed to
'X-Old-Xref:').

For the 'real' news server - i.e. News.Individual.Net in my case -,
Hamster works like a newsreader, i.e. it uses NNTP command like
LISTGROUP, XOVER, ARTICLE, etc.. So Hamster is a kind of proxy server.

On 2023-10-17 05:34, VanguardLH wrote:
> Frank Slootweg <this@ddress.is.invalid> wrote:
>
>> All you have to do is to remember all the hundred of thousands /
>> millions of message-ids of the postings you've ever 'read'. How hard
>> can *that* be!?
>
> Thunderbird uses SQLite for the messages databases, one for each folder
> (newsgroup). While SQLite is fast for small database sizes, it doesn't
> scale well. The bigger the database, the slow queries get, and the
> slowdown is not linear. SQLite recommends once you pass into the
> terabyte range to find a different solution.
>
> https://www.dbtalks.com/tutorials/learn-sqlite/what-are-the-limitations-of-sqlite
> A SQLite database can have maximum 2147483646 pages. Hence the maximum
> number of tables in a schema cannot reach more than 2147483646. The
> maximum number of rows in a table is 264. The maximum number of columns
> is 32767 in a table.
>

According to the official docs at https://www.sqlite.org/limits.html
this limit is 2 to the 64th power not twohundredandsixtyfour.

With that and a database table containing only 2 columns: MID and count
of read flags already set, the database would contain only one small row
per read message with only the numeric field changing after adding a
row. Replacing MIDs by fixed-size hashes of MIDs would allow fixed
length records, thus further easing the DB engine burden. Anyways disk
space for known messages would hit physical limits before the database
fills up.

The goal with this proposal is to handle 2 real world situations:

1. Cross-posts to multiple newsgroups from one server.

2. Having multiple newsserver accounts in one xxBird configuration, such
as news.dotsrc.org (the old SunSite) and eternal-september.org .

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

Jakob Bohm <jb-usenet@wisemo.com.invalid> wrote:
[...]

> According to the official docs at https://www.sqlite.org/limits.html
> this limit is 2 to the 64th power not twohundredandsixtyfour.
>
> With that and a database table containing only 2 columns: MID and count
> of read flags already set, the database would contain only one small row
> per read message with only the numeric field changing after adding a
> row. Replacing MIDs by fixed-size hashes of MIDs would allow fixed
> length records, thus further easing the DB engine burden. Anyways disk
> space for known messages would hit physical limits before the database
> fills up.
>
> The goal with this proposal is to handle 2 real world situations:

As I've mentioned before, but you've ignored, your proposal handles
only *part* of the 'problems', only for relatively new not-seen
articles, not for - probably older - other not-yet-seen articles.

Having repeated that, let me comment on the "2 real world situations":

> 1. Cross-posts to multiple newsgroups from one server.

This is already handled by article-number ranges in .newsrc type
files.

That Thunderbird et al inherit a design-flaw from their predecessors
(as far back as Netscape Communicator or even further), does not change
this. Fix Thunderbird et al or live with the flaw.

> 2. Having multiple newsserver accounts in one xxBird configuration, such
> as news.dotsrc.org (the old SunSite) and eternal-september.org .

That's only relevant if one's primary NSP is shaky and you need
another NSP in case as the primary one is down, misses articles, etc.,
etc.. (Best) Solution: Get a proper NSP.

But if you can't get a proper NSP, there's - as I mentioned several
times - Hamster (for Windows) and Leafnode2 (for Linux and Unix-like
OSs). That solves *both* the "2 real world situations".

Bottom line: 'Problems' solved. Next problem.

Frank Slootweg <this@ddress.is.invalid> writes:

> Neither do I, but it's just SMOP (Small Matter Of Programming). All you
> have to do is to remember all the hundred of thousands / millions of
> message-ids of the postings you've ever 'read'. How hard can *that* be!?
> :-)

Well, we know exactly how hard this is, because you're describing a news
server's history database. News servers also have to remember every
message they've seen, and they have to do that by message ID. News
servers limit this to articles within a certain date range and reject all
articles older than that date range, but servers that never expire
essentially keep that data forever.

There are a bunch of techniques for maintaining that database. It mostly
uses dedicated data structures optimized for this specific problem, not a
SQLite database, although I'd be interested to see someone try with modern
SQLite or another SQL database and see if it can be optimized enough,
since those tools have a lot more general optimizations and way more
active developers.

This is certainly *doable*, since it's done all the time, and a modern
server is often smaller (in disk, memory, and CPU) than a typical laptop,
let alone desktop. On my news server, it currently takes up about 660MiB
of disk space, which is smaller than a lot of video games. :) But that's
literally every message the server has on disk, and I can assure you that
I have not read the VAST majority of them, so most history databases for a
personal newsreader would be substantially smaller.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On 10/18/23 12:51, Jakob Bohm wrote:
> 2. Having multiple newsserver accounts in one xxBird configuration, such
> as news.dotsrc.org (the old SunSite) and eternal-september.org .

XXbird? Does that mean there are more clients based on Thunderbird?
--
user <candycane> is generated from /dev/urandom

Russ Allbery <eagle@eyrie.org> wrote:
> Frank Slootweg <this@ddress.is.invalid> writes:
>
> > Neither do I, but it's just SMOP (Small Matter Of Programming). All you
> > have to do is to remember all the hundred of thousands / millions of
> > message-ids of the postings you've ever 'read'. How hard can *that* be!?
> > :-)
>
> Well, we know exactly how hard this is, because you're describing a news
> server's history database. News servers also have to remember every
> message they've seen, and they have to do that by message ID. News
> servers limit this to articles within a certain date range and reject all
> articles older than that date range, but servers that never expire
> essentially keep that data forever.
>
> There are a bunch of techniques for maintaining that database. It mostly
> uses dedicated data structures optimized for this specific problem, not a
> SQLite database, although I'd be interested to see someone try with modern
> SQLite or another SQL database and see if it can be optimized enough,
> since those tools have a lot more general optimizations and way more
> active developers.
>
> This is certainly *doable*, since it's done all the time, and a modern
> server is often smaller (in disk, memory, and CPU) than a typical laptop,
> let alone desktop. On my news server, it currently takes up about 660MiB
> of disk space, which is smaller than a lot of video games. :) But that's
> literally every message the server has on disk, and I can assure you that
> I have not read the VAST majority of them, so most history databases for a
> personal newsreader would be substantially smaller.

Hi Russ,

Just to be sure/clear: I was both joking (hence "SMOP", "How hard can
*that* be!?" and the smiley) and serious in my responses to those who
apparently took my comments seriously.

And indeed, it's not that hard to do, because it's exactly what a
newsserver's history database does. That's why I later talked about
Hamster (which I use) and Leafnode(2).

My response was somewhat in jest, because it was a response to Ray's
likeminded post:

RB> It would have never occurred to me that somebody would expect this to
RB> work across multiple servers ;-)

So neither Ray, nor I took the original 'problem' seriously. But
humour and Usenet often don't go well together.

Hope this clarifies things.

On 2023-10-19 10:11, candycanearter07 wrote:
> On 10/18/23 12:51, Jakob Bohm wrote:
>> 2. Having multiple newsserver accounts in one xxBird configuration,
>> such as news.dotsrc.org (the old SunSite) and eternal-september.org .
>
>
> XXbird? Does that mean there are more clients based on Thunderbird?

Yes, there's at least IceDove, BetterBird (first I heard of it),
Interlink (dead), Epyrus, Hermopolis, SeaMonkey (and other full Mozilla
Communicator implementations).

I'm currently using Epyrus.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

On 2023-10-18 20:35, Frank Slootweg wrote:
> Jakob Bohm <jb-usenet@wisemo.com.invalid> wrote:
> [...]
>
>>
>> The goal with this proposal is to handle 2 real world situations:
>
> As I've mentioned before, but you've ignored, your proposal handles
> only *part* of the 'problems', only for relatively new not-seen
> articles, not for - probably older - other not-yet-seen articles.
>

And you keep refusing to understand any proposals that deal with
imperfect (not necessarily broken) servers.

> Having repeated that, let me comment on the "2 real world situations":
>
>> 1. Cross-posts to multiple newsgroups from one server.
>
> This is already handled by article-number ranges in .newsrc type
> files.
>
This assumes the server has similar but more costly logic to identify
cross-posted messages arriving at different times from different feeds.

> That Thunderbird et al inherit a design-flaw from their predecessors
> (as far back as Netscape Communicator or even further), does not change
> this. Fix Thunderbird et al or live with the flaw.
>

I am not referring to the "each newsgroup for itself" bug.

>> 2. Having multiple newsserver accounts in one xxBird configuration, such
>> as news.dotsrc.org (the old SunSite) and eternal-september.org .
>
> That's only relevant if one's primary NSP is shaky and you need
> another NSP in case as the primary one is down, misses articles, etc.,
> etc.. (Best) Solution: Get a proper NSP.

Given the decline of usenet, the need to use more than one NSP is more
likely to grow than decline. My example specifically mentioned two
volunteer run NSPs with good reputation but different inclusion
policies. Another example would be switching to a different NSP or a
different server at a single NSP while wanting to keep the "read" status
of messages already received from the previous NSP.

>
> But if you can't get a proper NSP, there's - as I mentioned several
> times - Hamster (for Windows) and Leafnode2 (for Linux and Unix-like
> OSs). That solves *both* the "2 real world situations".
>
That's an arrogant assumption, see above.

> Bottom line: 'Problems' solved. Next problem.
>

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

On 10/19/23 09:12, Jakob Bohm wrote:
> On 2023-10-19 10:11, candycanearter07 wrote:
>> XXbird? Does that mean there are more clients based on Thunderbird?
>
> Yes, there's at least IceDove, BetterBird (first I heard of it),
> Interlink (dead), Epyrus, Hermopolis, SeaMonkey (and other full Mozilla
> Communicator implementations).
>
> I'm currently using Epyrus.

Thanks, might check those out.
--
user <candycane> is generated from /dev/urandom

Jakob Bohm <jb-usenet@wisemo.com.invalid> wrote:
> On 2023-10-18 20:35, Frank Slootweg wrote:
> > Jakob Bohm <jb-usenet@wisemo.com.invalid> wrote:
> > [...]
> >
> >> The goal with this proposal is to handle 2 real world situations:
> >
> > As I've mentioned before, but you've ignored, your proposal handles
> > only *part* of the 'problems', only for relatively new not-seen
> > articles, not for - probably older - other not-yet-seen articles.
>
> And you keep refusing to understand any proposals that deal with
> imperfect (not necessarily broken) servers.

No, I don't refuse anything and I fully understand what you're saying.
However what you seem to be unable (unwilling?) to accept, is that some
(even many) people in these groups, especially in news.software.nntp,
have a long and a lot experience in these matters.

> > Having repeated that, let me comment on the "2 real world situations":
> >
> >> 1. Cross-posts to multiple newsgroups from one server.
> >
> > This is already handled by article-number ranges in .newsrc type
> > files.
> >
> This assumes the server has similar but more costly logic to identify
> cross-posted messages arriving at different times from different feeds.

Yes, *every* news server has that logic. It's called the history
database and it's not at all "costly", because it's a special database,
specifically designed for this very purpose (see Russ' post). My
(Hamster) server also has it, because it couldn't be a news server
without it.

> > That Thunderbird et al inherit a design-flaw from their predecessors
> > (as far back as Netscape Communicator or even further), does not change
> > this. Fix Thunderbird et al or live with the flaw.
>
> I am not referring to the "each newsgroup for itself" bug.

OK.

> >> 2. Having multiple newsserver accounts in one xxBird configuration, such
> >> as news.dotsrc.org (the old SunSite) and eternal-september.org .
> >
> > That's only relevant if one's primary NSP is shaky and you need
> > another NSP in case as the primary one is down, misses articles, etc.,
> > etc.. (Best) Solution: Get a proper NSP.
>
> Given the decline of usenet, the need to use more than one NSP is more
> likely to grow than decline. My example specifically mentioned two
> volunteer run NSPs with good reputation but different inclusion
> policies.

What do you mean by "inclusion policies"? Retention policies or which
groups they carry?

Anyway, a proper configuration (for Thunderbird et al) is not to get
the same groups from multiple servers. If you do want to do that anyway,
use Hamster/Leafnode(2).

> Another example would be switching to a different NSP or a
> different server at a single NSP while wanting to keep the "read" status
> of messages already received from the previous NSP.

As said, Hamster/Leafnode(2) solve that problem.

> > But if you can't get a proper NSP, there's - as I mentioned several
> > times - Hamster (for Windows) and Leafnode2 (for Linux and Unix-like
> > OSs). That solves *both* the "2 real world situations".
>
> That's an arrogant assumption, see above.

Why is that "an arrogant assumption"? Because it's a real life,
working solution to some somewhat farfetched problems, while your
proposal is a non-existing, partial solution, which is very unlikely to
be implemented?

> > Bottom line: 'Problems' solved. Next problem.

VanguardLH <V@nguard.LH> wrote:
> https://www.dbtalks.com/tutorials/learn-sqlite/what-are-the-limitations-of-sqlite

I read this URL and immediately didn't understand why dbtalks.com is a
reasonable place to look up information about or refer others to
SQLite's limitations.

> A SQLite database can have maximum 2147483646 pages. Hence the maximum
> number of tables in a schema cannot reach more than 2147483646. The
> maximum number of rows in a table is 264. The maximum number of columns
> is 32767 in a table.
>
> Notice only 264 rows per table.

I noticed that you seem to have taken that seriously. Something should
have immediately caught your eye and made you think 'that is not
reasonable' particularly for a production-ready relational DB as
widely used as SQLite.