Hi Russ,

> It doesn't help that this whole thing grew organically over many years and
> involves multiple programs with multiple somewhat confusing modes of
> operation and one really intrusive configuration option (groupbaseexpiry)
> that basically changes how everything works. This thread makes it clear
> that this is all more complicated than it probably should be, but of
> course making it less complicated without breaking anything is also very
> complicated!

It would perhaps imply to make a new INN 3.0 product with all sort of
oddities removed and less options. Yet, it's still a hard work because
it would also need unifying all the configuration files (in YAML ^^) and
refactoring thigs...

--
Julien ÉLIE

« – D'ailleurs je vais préparer de la potion magique qui ne servira bien
entendu que contre un retour éventuel des Romains !
– Mais, pour l'heure, les Romains ne sont pas encore revenus de leur
mésaventure. » (Astérix)

On Jul 8, 2022 at 12:21:37 PM CDT, "Julien ÉLIE" in
<ta9p31$1f5nu$1@news.trigofacile.com> wrote:

> Hi Jesse and Russ,
>
> Per-group expiry, that is to say when groupbaseexpiry is true, is
> mostly done by expireover(8): news articles are removed from the news
> spool by expireover, and then expire purges the history file.
>
> Regardless the setting of groupbaseexpiry, expireover should be run
> along with expire, usually via news.daily out of cron.
>
> Note that expire never purges articles which do not match any entry in
> expire.ctl.

Those three paragraphs really clear it up for me. My original issue stemmed
from my misunderstanding that expireover and expire functioned the same in
terms of using expire.ctl.

> Now that expire's documentation is fixed and "expire -p" is in fact working
> as it does, let's release INN 2.7.0 :-)
> I'll generate the release this week-end.
>
> Thanks again Jesse for all your questions on your fresh INN installation which
> permitted to make it better, and especially its documentation!

No rc2? I'm still pouring over everything, might find more... ;) (Joking - I
say go for it, I'll gladly update my boxes)

Glad I could contribute something useful.

Regards,

Jesse

Hi Jesse,

>> Now that expire's documentation is fixed and "expire -p" is in fact working
>> as it does, let's release INN 2.7.0 :-)
>> I'll generate the release this week-end.
>
> No rc2? I'm still pouring over everything, might find more... ;) (Joking - I
> say go for it, I'll gladly update my boxes)

I don't plan on releasing rc2 (changes since rc1 are essentially
documentation) and delay a bit more the final release.

And incidentally, if you're looking for what would be rc2, just try the
latest daily snapshot :-)
https://ftp.isc.org/isc/inn/snapshots/

You're of course welcome to go on pouring over everything. Fixes of
possible bugs, as well as other improvements in documentation, will be
in INN 2.7.1.

--
Julien ÉLIE

« I don't know if it's what you want, but it's what you get. » (Larry
Wall)

Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> writes:

> Per-group expiry, that is to say when groupbaseexpiry is true, is
> mostly done by expireover(8): news articles are removed from the
> news spool by expireover, and then expire purges the history
> file.

This is getting at something that if I remember correctly is true, but
which I'm not sure we say anywhere. (But don't take my word for it since
it's been a very long time since I looked at this code!) Specifically, I
believe it's correct to say something like this:

When groupbaseexpiry is true, article expiration is primarily done by
expireover based on the expiration rules that match each newsgroup.
expire then does some additional cleanup to remove old history
database entries.

When groupbaseexpiry is false, article expiration is primarily done by
expire based on the expiration rules that match the storage class of
each article. expire removes the articles and the history entries,
and then expireover does the additional cleanup of removing the
overview database entry.

This is the thing that always confused me about groupbaseexpiry: depending
on what it's set to, which program does the actual work of deleting
articles changes.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On Jul 8, 2022 at 12:53:44 PM CDT, "Russ Allbery" in
<87bktz1oev.fsf@hope.eyrie.org> wrote:

> This is getting at something that if I remember correctly is true, but
> which I'm not sure we say anywhere. (But don't take my word for it since
> it's been a very long time since I looked at this code!) Specifically, I
> believe it's correct to say something like this:
>
> When groupbaseexpiry is true, article expiration is primarily done by
> expireover based on the expiration rules that match each newsgroup.
> expire then does some additional cleanup to remove old history
> database entries.
>
> When groupbaseexpiry is false, article expiration is primarily done by
> expire based on the expiration rules that match the storage class of
> each article. expire removes the articles and the history entries,
> and then expireover does the additional cleanup of removing the
> overview database entry.
>
> This is the thing that always confused me about groupbaseexpiry: depending
> on what it's set to, which program does the actual work of deleting
> articles changes.

That adds additional clarification that is most helpful in understanding the
differences in their operation depending upon the groupbaseexpiry value.

From the content of this thread, I believe it is accurate.

Hi Russ,

> This is getting at something that if I remember correctly is true, but
> which I'm not sure we say anywhere.

I indeed did not see in the documentation the explanation you propose.
I believe it could be inserted in expire.ctl man page, when speaking
about groupbaseexpiry. And also in INSTALL.

> When groupbaseexpiry is true, article expiration is primarily done by
> expireover based on the expiration rules that match each newsgroup.
> expire then does some additional cleanup to remove old history
> database entries.

Yes, that's the behaviour I saw.

> When groupbaseexpiry is false, article expiration is primarily done by
> expire based on the expiration rules that match the storage class of
> each article. expire removes the articles and the history entries,
> and then expireover does the additional cleanup of removing the
> overview database entry.

Yes, agreed too.
With the subtlety that expire won't expire articles in CNFS unless -N is
given!

And you're right that expire doesn't do anything with overview (I'll
update my previous wording, which was wrong).

> This is the thing that always confused me about groupbaseexpiry: depending
> on what it's set to, which program does the actual work of deleting
> articles changes.

Sure! It is really confusing.
And news.daily does magic with the order of all the calls to
expire/expireover depending on groupbaseexpiry.

Hmm, if a news admin ever configures his server with groupbaseexpiry to
true and *without* overview, he won't be able to expire articles from
his spool, will he? (something maybe worth mentioning!)

--
Julien ÉLIE

« C'est une forêt vierge où la main de l'homme n'a jamais mis le pied. »

On Jul 8, 2022 at 1:22:21 PM CDT, "Julien ÉLIE" in
<ta9skt$1fehp$1@news.trigofacile.com> wrote:

> Hmm, if a news admin ever configures his server with groupbaseexpiry to
> true and *without* overview, he won't be able to expire articles from
> his spool, will he? (something maybe worth mentioning!)

This may be helpful, my feeder has no overview and only a CNFS buffer:

$ grep groupbaseexpiry etc/inn.conf
groupbaseexpiry: true

$ expireover
expireover: enableoverview is not true
expireover: can't open overview database

$ expire -v 3
Article lines processed 124432
Articles retained 124428
Entries expired 4

My expire.ctl entries (for this test):

*:A:1:90:never
*.test:AX:1:1:1

Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> writes:

> Hmm, if a news admin ever configures his server with groupbaseexpiry to
> true and *without* overview, he won't be able to expire articles from
> his spool, will he? (something maybe worth mentioning!)

I believe that's correct, although if you're only using self-expiring
storage backends like CNFS, you may not care. The history entries (the
only database outside of the article spool in that scenario) will still
get cleaned up by expire when it detects that the article no longer
exists.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

Hi Jesse,

>> Hmm, if a news admin ever configures his server with groupbaseexpiry to
>> true and *without* overview, he won't be able to expire articles from
>> his spool, will he? (something maybe worth mentioning!)
>
> This may be helpful, my feeder has no overview and only a CNFS buffer:
>
> $ expire -v 3
> Article lines processed 124432
> Articles retained 124428
> Entries expired 4
>
> My expire.ctl entries (for this test):
>
> *:A:1:90:never
> *.test:AX:1:1:1

When groupbaseexpiry is set to true, expire just does a probe whether
the article is still in the spool.

I would tend to think the 4 entries expired are cancelled articles whose
tokens are removed from the history file. (More than 4 entries would
have expired for all test groups...)
Thanks for the test anyway!

If you want to be sure, you may want to run "grephistory <mid>" on a
Message-ID you know was posted to a test group 2 days ago, and see if
the command returns a token.

--
Julien ÉLIE

« C'est une forêt vierge où la main de l'homme n'a jamais mis le pied. »

Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> writes:

> When groupbaseexpiry is set to true, expire just does a probe whether the
> article is still in the spool.

> I would tend to think the 4 entries expired are cancelled articles whose
> tokens are removed from the history file. (More than 4 entries would
> have expired for all test groups...)

The other common case is articles with Expires headers, which is still
sometimes done for the few FAQs that are autoposted.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

On Jul 8, 2022 at 2:59:44 PM CDT, "Russ Allbery" in
<87czefs7db.fsf@hope.eyrie.org> wrote:

> Julien ÉLIE <iulius@nom-de-mon-site.com.invalid> writes:
>
>> When groupbaseexpiry is set to true, expire just does a probe whether the
>> article is still in the spool.
>
>> I would tend to think the 4 entries expired are cancelled articles whose
>> tokens are removed from the history file. (More than 4 entries would
>> have expired for all test groups...)
>
> The other common case is articles with Expires headers, which is still
> sometimes done for the few FAQs that are autoposted.

I ran it again a few minutes later and it expired a few hundred articles.
Though, as mentioned, I can't be certain these were removed from the spool by
expire or they self-expired due to be rolled over.

It only keeps a few days worth of articles, but will let it sit a day and do a
test with a known-existing Message-ID that should expire.

However, I just realized on this box the test with expire may be moot because
I have a minimal active file and everything is going to "junk" (I had
forgotten that earlier).