Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

Space tells matter how to move and matter tells space how to curve. -- Wheeler


computers / news.software.nntp / Re: Question about INN expire

SubjectAuthor
* Question about INN expireJesse Rehmer
`* Re: Question about INN expireJesse Rehmer
 `* Re: Question about INN expireJesse Rehmer
  `* Re: Question about INN expireJesse Rehmer
   `* Re: Question about INN expireJulien ÉLIE
    `* Re: Question about INN expireJesse Rehmer
     `* Re: Question about INN expireJulien ÉLIE
      `* Re: Question about INN expireJesse Rehmer
       +* Re: Question about INN expireJulien ÉLIE
       |`* Re: Question about INN expireJesse Rehmer
       | +* Re: Question about INN expireJesse Rehmer
       | |`- Re: Question about INN expireJulien ÉLIE
       | `- Re: Question about INN expireJulien ÉLIE
       `* Re: Question about INN expireJulien ÉLIE
        +- Re: Question about INN expireJulien ÉLIE
        `* Re: Question about INN expireJesse Rehmer
         `* Re: Question about INN expireJulien ÉLIE
          +* Re: Question about INN expireJesse Rehmer
          |`* Re: Question about INN expireJesse Rehmer
          | `- Re: Question about INN expireJesse Rehmer
          `* Re: Question about INN expireJesse Rehmer
           `* Re: Question about INN expireJulien ÉLIE
            `* Re: Question about INN expireJesse Rehmer
             `* Re: Question about INN expireRuss Allbery
              +* Re: Question about INN expireJulien ÉLIE
              |+* Re: Question about INN expireJesse Rehmer
              ||`- Re: Question about INN expireJulien ÉLIE
              |`* Re: Question about INN expireRuss Allbery
              | +- Re: Question about INN expireJesse Rehmer
              | `* Re: Question about INN expireJulien ÉLIE
              |  +* Re: Question about INN expireJesse Rehmer
              |  |`* Re: Question about INN expireJulien ÉLIE
              |  | `* Re: Question about INN expireRuss Allbery
              |  |  `- Re: Question about INN expireJesse Rehmer
              |  `- Re: Question about INN expireRuss Allbery
              `- Re: Question about INN expireJulien ÉLIE

Pages:12
Question about INN expire

<9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=923&group=news.software.nntp#923

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!spool1.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Question about INN expire
Date: Mon, 4 Jul 2022 01:31:42 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 4 Jul 2022 01:31:42 -0000 (UTC)
Injection-Info: spool1.usenet.blueworldhosting.com;
logging-data="27315"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:15P1/2AcxeVX8tE7aM5x5OO3R7U= sha256:7ekaujoyOBoiaPTPWEyMuEsTGlO51kwG6CcfxyrQ6bU=
sha1:YUJL6Ea975OoVJ48yHaRXWOts68= sha256:ETqLgqyylznKwghzuTDQ5+9N5t/yMTgHftSra7Wrqis=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Mon, 4 Jul 2022 01:31 UTC

I'm trying to expire a bunch of crap that is crossposted to *.test groups. I
have the following in expire.ctl:

*:A:1:never:never
*.jobs*:AX:1:90:90
news.lists.filters:A:1:10:10
*.test:AX:1:100:100

When I run "expire -p" it does expire articles, but articles that have
Newsgroups headers like the one below that are crossposted to other groups
where alt.test is not the primary remain in the history and on the spool. I
thought adding the "X" flag to the expire.ctl line would take care of this,
but these articles still remain.

Date: Sat, 11 Apr 2009 00:22:55 GMT

I assume from the following portion of the expire.ctl man page that these
articles should be expiring, but perhaps I am misinterpreting this:

An expiration policy is applied to every article in a newsgroup it
matches. There is no way to set an expiration policy for articles
crossposted to groups you don't carry that's different than other
articles in the same group. Normally, articles are not completely
deleted until they expire out of every group to which they were posted,
but if an article is expired following a rule where <flag> contains
"X", it is deleted out of all newsgroups to which it was posted
immediately.

I in the example above the article is posted to rec.arts.disney.parks and
crossposted to alt.test, and based on "An expiration policy is applied to
every article in a newsgroup it matches", I assume since the article exists in
alt.test and I have the "X" flag that it be expired but this does not seem to
be the case.

Regards,

Jesse

Re: Question about INN expire

<9879603210.1fd484b5@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=924&group=news.software.nntp#924

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!spool1.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Mon, 4 Jul 2022 01:39:56 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <9879603210.1fd484b5@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 4 Jul 2022 01:39:56 -0000 (UTC)
Injection-Info: spool1.usenet.blueworldhosting.com;
logging-data="7503"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:FPUMQlmOulynlWfBLjP18mbHs2Y= sha256:jfopwosx8EYtlFXNLLkh5/p7ozvyI4i7MU5+5lTsROA=
sha1:cc98O4kgYFQjyIUZEdUhGj2uN3o= sha256:1nb3cpLbVc1h5BDS73Zrz7EwIr21nMc/naXtvTtwMaI=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Mon, 4 Jul 2022 01:39 UTC

Some of the headers that I pasted in my original message didn't make it
through to the posted message. I may be celebrating too hard while computing
this evening. ;)

Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
alt.test:8601

Re: Question about INN expire

<slrntc4ish.2l5u.root@spool1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=925&group=news.software.nntp#925

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!spool1.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Mon, 4 Jul 2022 02:08:17 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <slrntc4ish.2l5u.root@spool1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<9879603210.1fd484b5@freebsd-inject1.usenet.blueworldhosting.com>
Injection-Date: Mon, 4 Jul 2022 02:08:17 -0000 (UTC)
Injection-Info: spool1.usenet.blueworldhosting.com;
logging-data="87857"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: slrn/1.0.3 (FreeBSD)
Cancel-Lock: sha1:V5cpZ9xtSqduxloolO9aeFA0Pf4= sha256:xLpiiXnVbvWf+OHOTrL6KM+BWrajN4Q44Cqz3Py+cAI=
sha1:X9JNv31pUOFjghl495qwNAtp5dU= sha256:1rLv2rxPiMBvqzJqH7om6tM3GgnYES4EWRVX3K/NuJE=
 by: Jesse Rehmer - Mon, 4 Jul 2022 02:08 UTC

On 2022-07-04, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
> Some of the headers that I pasted in my original message didn't make it
> through to the posted message. I may be celebrating too hard while computing
> this evening. ;)
>
> Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
> alt.test:8601

Hmmm or maybe my news client was stripping them, trying from slrn this
time:

Newsgroups: rec.arts.disney.parks,alt.test
Date: Sat, 11 Apr 2009 00:53:23 GMT
Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018 alt.test:8601

Re: Question about INN expire

<0850344569.b37f6807@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=926&group=news.software.nntp#926

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!spool1.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Mon, 4 Jul 2022 04:21:44 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <0850344569.b37f6807@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <9879603210.1fd484b5@freebsd-inject1.usenet.blueworldhosting.com> <slrntc4ish.2l5u.root@spool1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 4 Jul 2022 04:21:44 -0000 (UTC)
Injection-Info: spool1.usenet.blueworldhosting.com;
logging-data="35838"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:x4KWXCITTBQ4Jsg6HAzx7/MRRK4= sha256:/+mKqhCYwmqdUcHKDT35jQJ3umtyOexanYNR+r6MmPE=
sha1:OmTV0dsXa/fzPfYbBGeWQXy+cJg= sha256:WEUSChKrKW1e7V1L+MajVHv9KnbOTrxJDg9iTLHeBFQ=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Mon, 4 Jul 2022 04:21 UTC

On Jul 3, 2022 at 9:08:17 PM CDT, "Jesse Rehmer" in
<slrntc4ish.2l5u.root@spool1.usenet.blueworldhosting.com> wrote:

> On 2022-07-04, Jesse Rehmer <jesse.rehmer@blueworldhosting.com> wrote:
>> Some of the headers that I pasted in my original message didn't make it
>> through to the posted message. I may be celebrating too hard while computing
>> this evening. ;)
>>
>> Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
>> alt.test:8601
>
> Hmmm or maybe my news client was stripping them, trying from slrn this
> time:
>
> Newsgroups: rec.arts.disney.parks,alt.test
> Date: Sat, 11 Apr 2009 00:53:23 GMT
> Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
> alt.test:8601

Also, shouldn't I be getting more output?

[news@spool1 ~]$ expire -p -v 5
Article lines processed 32971875
Articles retained 32966677
Entries expired 5198

Re: Question about INN expire

<t9vlvf$17rri$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=928&group=news.software.nntp#928

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176.143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Mon, 4 Jul 2022 23:27:11 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <t9vlvf$17rri$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<9879603210.1fd484b5@freebsd-inject1.usenet.blueworldhosting.com>
<slrntc4ish.2l5u.root@spool1.usenet.blueworldhosting.com>
<0850344569.b37f6807@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 4 Jul 2022 21:27:11 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176.143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1306482"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:xd2e375dWfTFvVSg9Vyf06A9v/s= sha256:1JHVeziIdXTxKC84SW5jSq5olywY8oqK/f/8epwQJGE=
sha1:2aiLJsYuXr8wvlz/S/x1mkj65qk= sha256:1YJ30NRs8PM1IM2lZu+2L1bZjtYEhKZgWWVc3tPAutQ=
In-Reply-To: <0850344569.b37f6807@freebsd-inject1.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Mon, 4 Jul 2022 21:27 UTC

Hi Jesse,

>> Newsgroups: rec.arts.disney.parks,alt.test
>> Date: Sat, 11 Apr 2009 00:53:23 GMT
>> Xref: spool1.usenet.blueworldhosting.com rec.arts.disney.parks:13018
>> alt.test:8601

What's the posting date INN has for this article?
If you have its Message-ID, look at the last number in the result of
grephistory:

% grephistory -l
'<9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>'
[A6E3D4975ED59BD167F5380850829D76] 1656898303~-~1656898302
@020162C242FF001200000000000000000000@

% convdate -dc 1656898302
Mon, 4 Jul 2022 01:31:42 -0000 (UTC)

> Also, shouldn't I be getting more output?
>
> [news@spool1 ~]$ expire -p -v 5
> Article lines processed 32971875
> Articles retained 32966677
> Entries expired 5198

"-v 5" should definitely be far more verbose, printing something for
each article line processed...

--
Julien ÉLIE

« C'est la goutte qui fait déborder l'amphore ! » (Assurancetourix)

Re: Question about INN expire

<5049358009.85de5044@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=930&group=news.software.nntp#930

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!fx13.iad.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Message-ID: <5049358009.85de5044@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <9879603210.1fd484b5@freebsd-inject1.usenet.blueworldhosting.com> <slrntc4ish.2l5u.root@spool1.usenet.blueworldhosting.com> <0850344569.b37f6807@freebsd-inject1.usenet.blueworldhosting.com> <t9vlvf$17rri$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
User-Agent: Usenapp for MacOS
X-Usenapp: v1.22/d - Full License
Lines: 52
X-Complaints-To: me+support@newsgroup.ninja
NNTP-Posting-Date: Tue, 05 Jul 2022 19:48:14 UTC
Organization: me+support@newsgroup.ninja
Date: Tue, 05 Jul 2022 19:48:14 GMT
X-Received-Bytes: 2660
 by: Jesse Rehmer - Tue, 5 Jul 2022 19:48 UTC

On Jul 4, 2022 at 4:27:11 PM CDT, "Julien ÉLIE" in
<t9vlvf$17rri$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
> What's the posting date INN has for this article?
> If you have its Message-ID, look at the last number in the result of
> grephistory:
>
> % grephistory -l
> '<9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>'
> [A6E3D4975ED59BD167F5380850829D76] 1656898303~-~1656898302
> @020162C242FF001200000000000000000000@
>
> % convdate -dc 1656898302
> Mon, 4 Jul 2022 01:31:42 -0000 (UTC)
>
>
>> Also, shouldn't I be getting more output?
>>
>> [news@spool1 ~]$ expire -p -v 5
>> Article lines processed 32971875
>> Articles retained 32966677
>> Entries expired 5198
>
> "-v 5" should definitely be far more verbose, printing something for
> each article line processed...

I ran expire several times with '-v 2' through '-v 5' and never get more
output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
large tradspool for most articles and a single CNFS buffer for binary
articles.

Seems it is ignoring the '-p' flag or is not acting on the right timestamp:

[news@spool1 ~]$ grephistory -l
'<Xns9BE9D482BF422lkajehoriuasldfjknak@207.115.33.102>'
[AA4B2314FF33940366F138FB4012C2BA] 1655680239~-~1239411203
@0500000091C500000000000032DA00000000@

[news@spool1 ~]$ convdate -dc 1239411203
Sat, 11 Apr 2009 00:53:23 -0000 (UTC)

[news@spool1 ~]$ convdate -dc 1655680239
Sun, 19 Jun 2022 23:10:39 -0000 (UTC)

I'm more inclined to believe it is ignoring the '-p' flag, since it also seems
to be ignoring the '-v X' flag as well. I changed the ordering of the flags as
a test but same results.

Cheers,
Jesse

Re: Question about INN expire

<ta55lp$1bii0$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=933&group=news.software.nntp#933

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.goja.nl.eu.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176.143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 01:25:44 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta55lp$1bii0$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<9879603210.1fd484b5@freebsd-inject1.usenet.blueworldhosting.com>
<slrntc4ish.2l5u.root@spool1.usenet.blueworldhosting.com>
<0850344569.b37f6807@freebsd-inject1.usenet.blueworldhosting.com>
<t9vlvf$17rri$1@news.trigofacile.com>
<5049358009.85de5044@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Wed, 6 Jul 2022 23:25:45 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176.143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1428032"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:RmfpbQ3S4Fu7K1E0mv9+ubmcrKU= sha256:HslQgcvKM++E5gToVbRxUemBr9ELVa55B5f9v6Nz2zs=
sha1:y9TqyY9UjzRMPnbdiwlcMEukLp0= sha256:ETMrwlh4ePIm4Vd7JRW+fZdkA2Kl7Ao9VppD2QyDVXg=
In-Reply-To: <5049358009.85de5044@freebsd-inject1.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Wed, 6 Jul 2022 23:25 UTC

Hi Jesse,

> I ran expire several times with '-v 2' through '-v 5' and never get more
> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
> large tradspool for most articles and a single CNFS buffer for binary
> articles.
>
> I'm more inclined to believe it is ignoring the '-p' flag, since it also seems
> to be ignoring the '-v X' flag as well. I changed the ordering of the flags as
> a test but same results.

I've tested a bit.
In fact, the flags are correctly seen. The -v flag works fine but the
-p flag does not seem to do what is expected, and also maybe without -p
(the part of the code where expire removes articles does not look right
to me, using conditions related to groupbaseexpiry and whether the
storage method self expires).
I have to look at it deeper.

--
Julien ÉLIE

« Inside every large problem is a small problem struggling to get out. »

Re: Question about INN expire

<5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=934&group=news.software.nntp#934

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!feed1.usenet.blueworldhosting.com!usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!spool1.usenet.blueworldhosting.com!news.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Wed, 6 Jul 2022 23:52:22 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <t9vlvf$17rri$1@news.trigofacile.com> <5049358009.85de5044@freebsd-inject1.usenet.blueworldhosting.com> <ta55lp$1bii0$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Wed, 6 Jul 2022 23:52:22 -0000 (UTC)
Injection-Info: news.blueworldhosting.com;
logging-data="89511"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:GEINT/6qqkPybipZ2FuXUcg+fMk= sha256:3KdE2se11usbv3NvU5co+6UpgsZ8Y+DI1p9BQJvd0SM=
sha1:bRm5Ikd00UAQo06PofxMKAGGQng= sha256:WaaTuXK39thor01MYHmrJpKY8+SnSaEMriDMZ3FA2Hc=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Wed, 6 Jul 2022 23:52 UTC

On Jul 6, 2022 at 6:25:44 PM CDT, "Julien ÉLIE" in
<ta55lp$1bii0$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
>> I ran expire several times with '-v 2' through '-v 5' and never get more
>> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
>> large tradspool for most articles and a single CNFS buffer for binary
>> articles.
>>
>> I'm more inclined to believe it is ignoring the '-p' flag, since it also seems
>> to be ignoring the '-v X' flag as well. I changed the ordering of the flags as
>> a test but same results.
>
> I've tested a bit.
> In fact, the flags are correctly seen. The -v flag works fine but the
> -p flag does not seem to do what is expected, and also maybe without -p
> (the part of the code where expire removes articles does not look right
> to me, using conditions related to groupbaseexpiry and whether the
> storage method self expires).
> I have to look at it deeper.

If there is anything I can do to help assist, let me know. I do notice odd
behavior, such as running expire back to back results in a few thousand
articles being expired each time. I wouldn't expect so many articles to expire
in such a short time, but since I cannot get more output I don't know what it
is doing exactly.

What conditions did you run expire where the -v flag was honored? I've ran
expire the following ways and only get three lines of output:

expire -v2 -p
expire -v 2 -p
expire -p -v2
expire -p -v 2
expire -v5 -p
expire -v 5 -p
expire -p -v5
expire -p -v 5
news.daily noexpireover flags="-v5 -p"

Re: Question about INN expire

<ta78l9$1d4m6$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=937&group=news.software.nntp#937

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176-143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 20:28:57 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta78l9$1d4m6$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<t9vlvf$17rri$1@news.trigofacile.com>
<5049358009.85de5044@freebsd-inject1.usenet.blueworldhosting.com>
<ta55lp$1bii0$1@news.trigofacile.com>
<5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 18:28:57 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176-143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1479366"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:HZKv6G5Z2JorE8pVM2oAR60oPUM= sha256:saN6N0dPllRnE9IfKE/bkOVH0WMjgv97hEFZjNyCGC8=
sha1:DPsPKsrtPXULY6a4w3aVGxY8QmY= sha256:VUIzhZu0OZofaeYP7L0haKllBtTtjBazxVoDIT7N5yY=
In-Reply-To: <5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Thu, 7 Jul 2022 18:28 UTC

Hi Jesse,

> If there is anything I can do to help assist, let me know. I do notice odd
> behavior, such as running expire back to back results in a few thousand
> articles being expired each time. I wouldn't expect so many articles to expire
> in such a short time, but since I cannot get more output I don't know what it
> is doing exactly.

Wouldn't it be articles contained in CNFS buffers that are wrapping?
Then each run of expire will mention the overwritten articles as expired.

> What conditions did you run expire where the -v flag was honored?

Simply use "-v 0" and you'll no longer see any output :-)

I've investigated more today, and I believe there's still a bit of
documentation to highlight more.
I discovered a few things (I still had not had a chance to have a closer
look at how expire really works).

I've played a bit with expire, adding a few lines of code and trying a
few things. I obtained errors of class definition missings (which are
the classes numbers in storage.conf).
And then I found out that expire does not do much work when
groupbaseexpiry is true. It does not even parse expire.ctl when using
"<pattern>:<flag>:<min>:<default>:<max>"... and in fact only searches
for "<classnum>:<min>:<default>:<max>" (the syntax when groupbaseexpiry
is false).
And there's a note in the code saying:
/* don't process this line -- per-group expiry is done by expireover */

Ah ah, it becomes clearer now.
From inn.conf man page:

groupbaseexpiry
Whether to enable newsgroup-based expiry. If set to false, article
expiry is done based on storage class of storing method. If set to
true (and overview information is available), expiry is done by
newsgroup name.

Note the "and overview information is available".

So basically, if you want the behaviour you explained in the first
message of this thread:

* either you need to set groupbaseexpiry to false, rewrite expire.ctl
accordingly, and then "expire -p -v 3" will work as expected.
It would mean that you have storage classes definitions suiting your
needs of expiration (as it will be by class), so in fact the classes
should be done with a sorting by newsgroups: one class for "*.jobs*",
another class for "news.lists.filters", another for "*.test", etc.

* either you keep groupbaseexpiry to true, and then it means overview
data is needed.
In your previous article, you mentioned:
news.daily noexpireover flags="-v5 -p"
Does it mean you do not have overview data?

If no, then you need creating it or switching to groupbaseexpiry set to
false...

And if you have overview data, the Key is to run:

expireover -e -N -p

-e: Remove articles from the news spool and all overview databases as
soon as they expire out of any newsgroup to which they are posted

-N: Apply expire.ctl rules to expire articles even from storage methods
that have self-expire functionality

-p: Expiration decisions are based on the article posting date

and update your cron line:
news.daily expireover expireoverflags='-e -N -p' flags='-N -v1'

I agree that the expire man page is clearly misleading about that, and
should clearly explain that. I'll have a look at where to put such
information (several man pages are concerned).

If someone has a different understanding of how expire/expireover work
with regards to withgroupbaseexpiry, do not hesitate to react!
I may as well be misunderstanding something.

--
Julien ÉLIE

« Love is blind but marriage is an eye-opener. »

Re: Question about INN expire

<ta7a19$1d5ke$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=938&group=news.software.nntp#938

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!paganini.bofh.team!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176.143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 20:52:25 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta7a19$1d5ke$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<t9vlvf$17rri$1@news.trigofacile.com>
<5049358009.85de5044@freebsd-inject1.usenet.blueworldhosting.com>
<ta55lp$1bii0$1@news.trigofacile.com>
<5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 18:52:25 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176.143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1480334"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:icv1MEUVNUpst8qFLFcWEfpNy8M= sha256:2+oH6Br8hZldb7ZJK556IW7ujUQsjKn0u4/nx0ua6UI=
sha1:xruEZDzf2enAKC9AMDIkA4kYJt4= sha256:jnQb2au2PyLbFWAQZTTJclrPlyP7mWbNcveAw184Kr8=
In-Reply-To: <5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Thu, 7 Jul 2022 18:52 UTC

Hi Jesse,

>>> I ran expire several times with '-v 2' through '-v 5' and never get more
>>> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
>>> large tradspool for most articles and a single CNFS buffer for binary
>>> articles.
>
> If there is anything I can do to help assist, let me know.

"expireover -e -N -p" should do what you want (see my previous response).

I've a bit improved expire for logging and, above all, with a few comments.

I don't know whether you could try the following patch easily (you will have to rebuild INN and run "make update") but in case you want to test it, feel free :-)

--- a/expire/expire.c
+++ b/expire/expire.c
@@ -372,6 +372,7 @@ EXPremove(const TOKEN *token)

/*
** Do the work of expiring one line.
+** Returns true when the article should be kept for the time being.
*/
static bool
EXPdoline(void *cookie UNUSED, time_t arrived, time_t posted, time_t expires,
@@ -389,12 +390,16 @@ EXPdoline(void *cookie UNUSED, time_t arrived, time_t posted, time_t expires,
HasSelfexpire = true;
Selfexpired = true;
} else {
- /* the article is still alive */
+ /* The article is still alive, free it. */
SMfreearticle(article);
+ /* Per-group expiry is done by expireover, remember!
+ * That's why if groupbaseexpiry is set, expire won't verify
+ * the expiration rules set in expire.ctl. */
if (innconf->groupbaseexpiry || !Ignoreselfexpire)
HasSelfexpire = true;
}
}
+ if (EXPusepost && posted != 0)
when = posted;
else
@@ -403,19 +408,29 @@ EXPdoline(void *cookie UNUSED, time_t arrived, time_t posted, time_t expires,

if (HasSelfexpire) {
if (Selfexpired || token->type == TOKEN_EMPTY) {
+ if (EXPverbose > 3)
+ printf("%s (to remember or forget, not existing)\n",
+ TokenToText(*token));
EXPallgone++;
r = false;
} else {
+ if (EXPverbose > 3)
+ printf("%s (to keep, may self-expire)\n",
+ TokenToText(*token));
EXPstillhere++;
r = true;
}
} else {
kr = EXPkeepit(token, when, expires);
if (kr == Remove) {
+ if (EXPverbose > 3)
+ printf("%s (to remember or forget)\n", TokenToText(*token));
EXPremove(token);
EXPallgone++;
r = false;
} else {
+ if (EXPverbose > 3)
+ printf("%s (to keep)\n", TokenToText(*token));
EXPstillhere++;
r = true;
}
@@ -699,7 +714,7 @@ main(int ac, char *av[])
if (!Bad && NHistory != NULL) {
snprintf(buff, sizeof(buff), "%s.n.done", NHistory);
fclose(EXPfopen(false, buff, "w", true, Server, false));
- CleanupAndExit(Server, false, Bad ? 1 : 0);
+ CleanupAndExit(Server, true, 0);
}

CleanupAndExit(Server, !Bad, Bad ? 1 : 0);

You can use "-x -t" for a dry-run of expire.

expire -x -t -v4 > exp.log

will output in exp.log tons of lines like:

@030346523100000000000000002600000001@ (to keep, may self-expire)

and

grep -v will exp.log

will tell you the tokens expire will remove (or remember depending on the setting of /remember/ in expire.ctl):

@03034652330000000000000F2EBE00000001@ (to remember or forget, not existing)

These tokens no longer correspond to an existing stored article (either the CNFS buffer wrapped or the article was cancelled).

--
Julien ÉLIE

« – À la plage ? Mais il pleut !
– Pas du tout ! Dans le midi de la Gaule, il pleut. Ici, c'est tout
juste un peu humide. Vivifiant. Pas vrai, Astérix ?
– Ce matin, ça devient de plus en plus vivifiant ! » (Astérix)

Re: Question about INN expire

<ta7bjn$1d6cf$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=939&group=news.software.nntp#939

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!aioe.org!news.freedyn.de!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176-143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 21:19:19 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta7bjn$1d6cf$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<t9vlvf$17rri$1@news.trigofacile.com>
<5049358009.85de5044@freebsd-inject1.usenet.blueworldhosting.com>
<ta55lp$1bii0$1@news.trigofacile.com>
<5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
<ta7a19$1d5ke$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 19:19:19 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176-143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1481103"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:GS3SkucPK0LjuDaNTRFp95WRzLc= sha256:Jhz6fYgtcfsSzf7v8lUMe3cEzODreyqP2PllLPcYbu0=
sha1:m50lc+o/vwAvFzlKEG1iAABOiew= sha256:dlm2fbUbxprCabDGnyrpxOO3KWa3+yQns3UiF9ugEQU=
In-Reply-To: <ta7a19$1d5ke$1@news.trigofacile.com>
 by: Julien ÉLIE - Thu, 7 Jul 2022 19:19 UTC

Adding to my previous message:

> You can use "-x -t" for a dry-run of expire.
> expire -x -t -v4 > exp.log
If you run that without the patch I gave (which changes a boolean in a
call to CleanupAndExit), your news server will be paused after the run.
So you should add "-n" to the command (which prevents the pause).

Otherwise, run:

ctlinnd mode

and you'll see the reason to use in the first line "Expiring process
XXXX". Then:

ctlinnd go 'Expiring process XXXX'

and your news server will be running again.

--
Julien ÉLIE

« – À la plage ? Mais il pleut !
– Pas du tout ! Dans le midi de la Gaule, il pleut. Ici, c'est tout
juste un peu humide. Vivifiant. Pas vrai, Astérix ?
– Ce matin, ça devient de plus en plus vivifiant ! » (Astérix)

Re: Question about INN expire

<2243230851.cb2dfbc4@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=940&group=news.software.nntp#940

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 19:33:53 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <2243230851.cb2dfbc4@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <ta55lp$1bii0$1@news.trigofacile.com> <5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com> <ta78l9$1d4m6$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 19:33:53 -0000 (UTC)
Injection-Info: news.usenet.blueworldhosting.com;
logging-data="36000"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:Mse12C0BbxlcVPTUYEvKaAAmHVc= sha256:gEyXXrs8bfQnzmeg9majBKRWJ/QH+JxTuCzffYFRSRk=
sha1:UM2LmD3GbcLcsWWZc2/XQCX8sUc= sha256:VMZVH7V16bKBTEf4NqwRhUKruHa3CSqQKxBBxk6Wl60=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Thu, 7 Jul 2022 19:33 UTC

On Jul 7, 2022 at 1:28:57 PM CDT, "Julien ÉLIE" in
<ta78l9$1d4m6$1@news.trigofacile.com> wrote:

> Simply use "-v 0" and you'll no longer see any output :-)

I want MORE output, not less. With any -v flag above '1' I only get three
lines of output, the same three lines from '-v 1'.

>
> I've investigated more today, and I believe there's still a bit of
> documentation to highlight more.
> I discovered a few things (I still had not had a chance to have a closer
> look at how expire really works).
>
> I've played a bit with expire, adding a few lines of code and trying a
> few things. I obtained errors of class definition missings (which are
> the classes numbers in storage.conf).
> And then I found out that expire does not do much work when
> groupbaseexpiry is true. It does not even parse expire.ctl when using
> "<pattern>:<flag>:<min>:<default>:<max>"... and in fact only searches
> for "<classnum>:<min>:<default>:<max>" (the syntax when groupbaseexpiry
> is false).
> And there's a note in the code saying:
> /* don't process this line -- per-group expiry is done by expireover */
>
>
> Ah ah, it becomes clearer now.
> From inn.conf man page:
>
> groupbaseexpiry
> Whether to enable newsgroup-based expiry. If set to false, article
> expiry is done based on storage class of storing method. If set to
> true (and overview information is available), expiry is done by
> newsgroup name.
>
> Note the "and overview information is available".
>
>
> So basically, if you want the behaviour you explained in the first
> message of this thread:
>
> * either you need to set groupbaseexpiry to false, rewrite expire.ctl
> accordingly, and then "expire -p -v 3" will work as expected.
> It would mean that you have storage classes definitions suiting your
> needs of expiration (as it will be by class), so in fact the classes
> should be done with a sorting by newsgroups: one class for "*.jobs*",
> another class for "news.lists.filters", another for "*.test", etc.
>
> * either you keep groupbaseexpiry to true, and then it means overview
> data is needed.
> In your previous article, you mentioned:
> news.daily noexpireover flags="-v5 -p"
> Does it mean you do not have overview data?
>
> If no, then you need creating it or switching to groupbaseexpiry set to
> false...
>
> And if you have overview data, the Key is to run:
>
> expireover -e -N -p
>
> -e: Remove articles from the news spool and all overview databases as
> soon as they expire out of any newsgroup to which they are posted
>
> -N: Apply expire.ctl rules to expire articles even from storage methods
> that have self-expire functionality
>
> -p: Expiration decisions are based on the article posting date
>
>
> and update your cron line:
> news.daily expireover expireoverflags='-e -N -p' flags='-N -v1'
>
>
> I agree that the expire man page is clearly misleading about that, and
> should clearly explain that. I'll have a look at where to put such
> information (several man pages are concerned).
>
> If someone has a different understanding of how expire/expireover work
> with regards to withgroupbaseexpiry, do not hesitate to react!
> I may as well be misunderstanding something.

I'm going to have to take more time later to digest this further. I do have
overview data, but I don't run expireover very often because on my spool it
takes more than a day and causes the locking problems I brought up recently,
but I was attempting to see how much space I could reclaim by getting rid of
stuff cross posted to *.test using expire.

Re: Question about INN expire

<2305212026.55527169@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=941&group=news.software.nntp#941

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 19:44:12 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <2305212026.55527169@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <ta55lp$1bii0$1@news.trigofacile.com> <5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com> <ta78l9$1d4m6$1@news.trigofacile.com> <2243230851.cb2dfbc4@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 19:44:12 -0000 (UTC)
Injection-Info: news.usenet.blueworldhosting.com;
logging-data="10321"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:+QCgqyTeu6V0li8Q5T1AA82eCII= sha256:ejkxczdsFbi8XzL7k0t9jpjNrmbfElMwUxNEfU6MmSA=
sha1:eQXWjAYzeO9MU7Tv4yq7A9z39JI= sha256:EeUlLcTrU42Kwg6/cVId4Q1cNGutpVcMFVD23Wzk/Ow=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Thu, 7 Jul 2022 19:44 UTC

On Jul 7, 2022 at 2:33:53 PM CDT, "Jesse Rehmer" in
<2243230851.cb2dfbc4@freebsd-inject1.usenet.blueworldhosting.com> wrote:

> I want MORE output, not less. With any -v flag above '1' I only get three
> lines of output, the same three lines from '-v 1'.

Perhaps I am not articulating this well, but I expect more output from these
commands:

$ expire -v 3 -n -t
Expire tracing: history files not updated.
Article lines processed 206532
Articles retained 111477
Entries expired 95055

$ expire -v 5 -n -t
Expire tracing: history files not updated.
Article lines processed 206534
Articles retained 111479
Entries expired 95055

The above is from a different INN 2.7.0rc1 box than I've been having issues
with, just to test that it wasn't a single server issue.

Re: Question about INN expire

<ta7da7$1d757$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=942&group=news.software.nntp#942

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176-143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 21:48:22 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta7da7$1d757$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<ta55lp$1bii0$1@news.trigofacile.com>
<5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
<ta78l9$1d4m6$1@news.trigofacile.com>
<2243230851.cb2dfbc4@freebsd-inject1.usenet.blueworldhosting.com>
<2305212026.55527169@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 19:48:23 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176-143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1481895"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:X1aa2G6BVv3vCWbP7l9XhjlAkFU= sha256:gU/5A8GFW27+AWAvKctkpSRuNZBR6bV1gCQrE+FOJ6A=
sha1:pnPeANzCLQ4ZhfJwsuvtsJjtvao= sha256:apVX9W4E1cfd9hTR6RGBC/1H547uvpEM1gvxTw0Dguw=
In-Reply-To: <2305212026.55527169@freebsd-inject1.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Thu, 7 Jul 2022 19:48 UTC

Hi Jesse,

>> I want MORE output, not less. With any -v flag above '1' I only get three
>> lines of output, the same three lines from '-v 1'.
>
> Perhaps I am not articulating this well, but I expect more output from these
> commands:
>
> $ expire -v 3 -n -t
> Expire tracing: history files not updated.
> Article lines processed 206532
> Articles retained 111477
> Entries expired 95055

Yes I have well understood what you want with "-v 3".

When I said to try "-v 0", it was in response to your question of how I
was sure the arguments were read.

As for MORE verbose logs, unfortunately, as I wrote a few minutes ago,
they are available only when groupbaseexpiry is false.
I sent a patch to add more logs (which will now appear with
groupbaseexpiry set to true).

> The above is from a different INN 2.7.0rc1 box than I've been having issues
> with, just to test that it wasn't a single server issue.

No, that's not a single server issue.
I have the same behaviour on mine.

--
Julien ÉLIE

« – À la plage ? Mais il pleut !
– Pas du tout ! Dans le midi de la Gaule, il pleut. Ici, c'est tout
juste un peu humide. Vivifiant. Pas vrai, Astérix ?
– Ce matin, ça devient de plus en plus vivifiant ! » (Astérix)

Re: Question about INN expire

<ta7dpi$1d7ca$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=943&group=news.software.nntp#943

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176-143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 21:56:34 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta7dpi$1d7ca$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<ta55lp$1bii0$1@news.trigofacile.com>
<5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
<ta78l9$1d4m6$1@news.trigofacile.com>
<2243230851.cb2dfbc4@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 19:56:34 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176-143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1482122"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:xo9jyf+1yj/bSZlOs2sx4BX3O5U= sha256:xZqsrggfA7xmaJ2BkVmwGywcAfA6ij2Omta1q3Zc1VE=
sha1:YDwZ5TNvbkeC5eXTurbwBnq3QA8= sha256:jCbCxstZtiELZgogdOW1+0q7mE9+3NySywjiuGuHCpg=
In-Reply-To: <2243230851.cb2dfbc4@freebsd-inject1.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Thu, 7 Jul 2022 19:56 UTC

Hi Jesse,

> I'm going to have to take more time later to digest this further.

Yes, take your time!

> I do have
> overview data, but I don't run expireover very often because on my spool it
> takes more than a day and causes the locking problems I brought up recently

Ah yes, I remember.
Did you try to run the command I suggested to expire the overview of
only a subset of groups?

For your tests groups for instance:

grep -E '\.test ' <pathdb>/active | expireover -f - -Z <pathtmp>/lowmark

Or several groups:

grep -E 'news.lists.filters |\.jobs|\.test ' <pathdb>/active \
| expireover -f - -Z <pathtmp>/lowmark

And then:

ctlinnd lowmark <pathtmp>/lowmark
rm -f <pathtmp>/lowmark

How long does it take with only 1 populated group?
Do for instance:

echo "rec.arts.tv" | expireover -f - -Z <pathtmp>/lowmark

I'm still surprised that it takes so long for tradindexed.

> but I was attempting to see how much space I could reclaim by getting rid of
> stuff cross posted to *.test using expire.

Unfortunately, expire won't do that when groupbaseexpiry is set to true.

--
Julien ÉLIE

« Dites, ne vendez tout de même pas la peau du sanglier avant de l'avoir
tué ! » (Détritus)

Re: Question about INN expire

<2514663264.b5c06a4a@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=945&group=news.software.nntp#945

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 20:19:07 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <2514663264.b5c06a4a@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <ta55lp$1bii0$1@news.trigofacile.com> <5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com> <ta7a19$1d5ke$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 20:19:07 -0000 (UTC)
Injection-Info: news.usenet.blueworldhosting.com;
logging-data="17101"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:9xDnBFb2wTO+IFwwwlb037KDZ3s= sha256:mbZWuih+8y1v6ct0ilwmMnh3TEzkbHUYJERlCT7dmUc=
sha1:7+LKlKsBLE3NTHeK1KW3Arhut2o= sha256:u/hTgnEe7FCgNquFoR7MUMQRPPUhkxezFSu58zAoRdY=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Thu, 7 Jul 2022 20:19 UTC

On Jul 7, 2022 at 1:52:25 PM CDT, "Julien ÉLIE" in
<ta7a19$1d5ke$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>>> I ran expire several times with '-v 2' through '-v 5' and never get more
>>>> output than the standard three lines. I'm running INN 2.7.0rc1 with a rather
>>>> large tradspool for most articles and a single CNFS buffer for binary
>>>> articles.
>>
>> If there is anything I can do to help assist, let me know.
>
> "expireover -e -N -p" should do what you want (see my previous response).
>
> I've a bit improved expire for logging and, above all, with a few comments.
>
> I don't know whether you could try the following patch easily (you will have
> to rebuild INN and run "make update") but in case you want to test it, feel
> free :-)

That expireover command should do what I'm after without having to change
groupbaseexpiry to false? Or do I still need to set groupbaseexpiry to false,
setup the various storage classes, and change expire.ctl accordingly?

Re: Question about INN expire

<ta7fgr$1d8cj$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=946&group=news.software.nntp#946

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176.143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 22:26:03 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta7fgr$1d8cj$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<ta55lp$1bii0$1@news.trigofacile.com>
<5154189543.d51f7a32@freebsd-inject1.usenet.blueworldhosting.com>
<ta7a19$1d5ke$1@news.trigofacile.com>
<2514663264.b5c06a4a@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 20:26:03 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176.143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1483155"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:qotvP8MTGrXmdQN6zSyNaXz+ck8= sha256:5TgmUXvVopzJvGr0q350vgUItlp2gySg8ydA1+BHbYk=
sha1:IE94AHf+Vyvd8OGhXS/vlMWsRQU= sha256:ez/d/l9MPZAxujHnq6sSJuRF97+uaihhLRY9yh7Oii4=
In-Reply-To: <2514663264.b5c06a4a@freebsd-inject1.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Thu, 7 Jul 2022 20:26 UTC

Hi Jesse,

>> "expireover -e -N -p" should do what you want (see my previous response).
>
> That expireover command should do what I'm after without having to change
> groupbaseexpiry to false?

Exactly!
And you can even try it on 1 newsgroup (or more) with the example of
echo or grep active piped to it.

> Or do I still need to set groupbaseexpiry to false,
> setup the various storage classes, and change expire.ctl accordingly?

No need to do change anything; groupbaseexpiry set to false was to make
expire work as you wanted (without expireover).

--
Julien ÉLIE

« Dites, ne vendez tout de même pas la peau du sanglier avant de l'avoir
tué ! » (Détritus)

Re: Question about INN expire

<2592318811.31a25522@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=947&group=news.software.nntp#947

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 20:32:03 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <2592318811.31a25522@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <ta7a19$1d5ke$1@news.trigofacile.com> <2514663264.b5c06a4a@freebsd-inject1.usenet.blueworldhosting.com> <ta7fgr$1d8cj$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 20:32:03 -0000 (UTC)
Injection-Info: news.usenet.blueworldhosting.com;
logging-data="98938"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:y+52xXF/MugVorW+CWogdQaSrEc= sha256:Oq4ogM3/3q94bMGGGWuOmrqkZ0xaL3i5lfevpL7Y1Qw=
sha1:0Nah223x+e7TK1a3Xggofg7sXJw= sha256:DJxXB1NgXoKbYcx3405Wl9Lv59W05AD+F9ZDT6SNzH0=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Thu, 7 Jul 2022 20:32 UTC

On Jul 7, 2022 at 3:26:03 PM CDT, "Julien ÉLIE" in
<ta7fgr$1d8cj$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>> "expireover -e -N -p" should do what you want (see my previous response).
>>
>> That expireover command should do what I'm after without having to change
>> groupbaseexpiry to false?
>
> Exactly!
> And you can even try it on 1 newsgroup (or more) with the example of
> echo or grep active piped to it.
>
>
>> Or do I still need to set groupbaseexpiry to false,
>> setup the various storage classes, and change expire.ctl accordingly?
>
> No need to do change anything; groupbaseexpiry set to false was to make
> expire work as you wanted (without expireover).

I'm going to do testing with a few groups as you outlined in a previous
response and report back my findings. The expireover issues I'm struggling
with on this box are perplexing me. I've used these tools on larger spools in
the past with faster results, so it feels like I am hitting a constraint, but
can't point to where. The IOPS and throughput during expireover don't even
come close to what the box does when I inject an entire spool's worth of
articles into it from another server so it doesn't feel like
hardware/throughput is the issue, but still trying to figure it out.

It's also possible my memory is absolutely trashed and I am not recalling my
previous experience or usage correctly. :)

Re: Question about INN expire

<2706264316.c2f1d3a4@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=948&group=news.software.nntp#948

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Thu, 7 Jul 2022 20:51:03 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <2706264316.c2f1d3a4@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <ta7a19$1d5ke$1@news.trigofacile.com> <2514663264.b5c06a4a@freebsd-inject1.usenet.blueworldhosting.com> <ta7fgr$1d8cj$1@news.trigofacile.com> <2592318811.31a25522@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 7 Jul 2022 20:51:03 -0000 (UTC)
Injection-Info: news.usenet.blueworldhosting.com;
logging-data="18685"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:dv2+9rdIan0JOrcdh4sRZRk3c6I= sha256:Jf0LpFP5jbES3ffeFK6VECmygHCJSzxorbbzyh5ihvk=
sha1:jJ1Y6LaecI3xoJY2Y8FKnFTv9Po= sha256:ZCSRMj59fOrRBhqmw07eOcOYILGiBgCyJj56kduMQ2Q=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Thu, 7 Jul 2022 20:51 UTC

On Jul 7, 2022 at 3:32:03 PM CDT, "Jesse Rehmer" in
<2592318811.31a25522@freebsd-inject1.usenet.blueworldhosting.com> wrote:

> I'm going to do testing with a few groups as you outlined in a previous
> response and report back my findings.

This is interesting:

$ time echo "rec.arts.tv" | expireover -f - -Z /usr/local/news/tmp/lowmark
Article lines processed 1371021
Articles dropped 0
Overview index dropped 0

real 8m4.764s
user 0m22.424s
sys 1m3.622s

So for the largest group I currently have, it only took 8 minutes. I wonder
why it is taking so long on the entire spool when that group represents
something like 2-3% of total articles. It did cause the locking issue for a
few minutes during, but I think I can live with that if I can get things
staggered a bit.

Re: Question about INN expire

<4185399788.df02eb50@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=949&group=news.software.nntp#949

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Fri, 8 Jul 2022 00:57:34 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <4185399788.df02eb50@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <ta7a19$1d5ke$1@news.trigofacile.com> <2514663264.b5c06a4a@freebsd-inject1.usenet.blueworldhosting.com> <ta7fgr$1d8cj$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 8 Jul 2022 00:57:34 -0000 (UTC)
Injection-Info: news.usenet.blueworldhosting.com;
logging-data="27213"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:u6Npz7kJX4GQDBTXQnAaEk80V4Q= sha256:dJ13tXHNUii31cjfFi8liOIHQDRWo6dJT9iK9DG/Hag=
sha1:pFS9T6fw3iXewjwXF3HM4PPfsGU= sha256:3qPWgEenRX7thCzeUFY2Bq/NmTwPsmc/WpDtAmbhMIs=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Fri, 8 Jul 2022 00:57 UTC

On Jul 7, 2022 at 3:26:03 PM CDT, "Julien ÉLIE" in
<ta7fgr$1d8cj$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>> "expireover -e -N -p" should do what you want (see my previous response).
>>
>> That expireover command should do what I'm after without having to change
>> groupbaseexpiry to false?
>
> Exactly!
> And you can even try it on 1 newsgroup (or more) with the example of
> echo or grep active piped to it.

Can I inquire the purpose of the "X" flag in expire.ctl, specifically if
expireover also needs to be called with '-e' to get the same behavior as the
"X" flag as described in the manpage for expire.ctl:

> An expiration policy is applied to every article in a newsgroup it
> matches. There is no way to set an expiration policy for articles
> crossposted to groups you don't carry that's different than other
> articles in the same group. Normally, articles are not completely
> deleted until they expire out of every group to which they were posted,
> but if an article is expired following a rule where <flag> contains
> "X", it is deleted out of all newsgroups to which it was posted
> immediately.

My assumption from reading the expire.ctl manpage was that my "AX" flag in
expire.ctl would accomplish the same thing as what 'expireover -e' does.

It would seem based on the observation you made (quoted below) that the X flag
in expire.ctl has no purpose if expire ignores expire.ctl and expireover wants
the '-e' flag. Is that correct, or am I still missing something?

> And then I found out that expire does not do much work when
> groupbaseexpiry is true. It does not even parse expire.ctl when using
> "<pattern>:<flag>:<min>:<default>:<max>"... and in fact only searches
> for "<classnum>:<min>:<default>:<max>" (the syntax when groupbaseexpiry
> is false).

Re: Question about INN expire

<5493892904.c133526c@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=950&group=news.software.nntp#950

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Fri, 8 Jul 2022 04:35:39 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <5493892904.c133526c@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <ta7fgr$1d8cj$1@news.trigofacile.com> <2592318811.31a25522@freebsd-inject1.usenet.blueworldhosting.com> <2706264316.c2f1d3a4@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 8 Jul 2022 04:35:39 -0000 (UTC)
Injection-Info: news.usenet.blueworldhosting.com;
logging-data="66476"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:PK1vyaTAtQWfFpeGarw3KR1QUi4= sha256:knF6fhoUZZjLX8HQZ+CRtovf9RDYekGSpmdD5f+/gLk=
sha1:t4G+s1uWTbwqD29bKhDtqPzTJyc= sha256:0cieNNC+m6pgUpfTiSgAe6JbNZVDtL5Fv8B1PzXwi1A=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Fri, 8 Jul 2022 04:35 UTC

On Jul 7, 2022 at 3:51:03 PM CDT, "Jesse Rehmer" in
<2706264316.c2f1d3a4@freebsd-inject1.usenet.blueworldhosting.com> wrote:

> On Jul 7, 2022 at 3:32:03 PM CDT, "Jesse Rehmer" in
> <2592318811.31a25522@freebsd-inject1.usenet.blueworldhosting.com> wrote:
>
>> I'm going to do testing with a few groups as you outlined in a previous
>> response and report back my findings.
>
> This is interesting:
>
> $ time echo "rec.arts.tv" | expireover -f - -Z /usr/local/news/tmp/lowmark
> Article lines processed 1371021
> Articles dropped 0
> Overview index dropped 0
>
> real 8m4.764s
> user 0m22.424s
> sys 1m3.622s
>
> So for the largest group I currently have, it only took 8 minutes. I wonder
> why it is taking so long on the entire spool when that group represents
> something like 2-3% of total articles. It did cause the locking issue for a
> few minutes during, but I think I can live with that if I can get things
> staggered a bit.

Using these flags seems to significantly increases performance of expireover:

$ time expireover -e -N -p
Article lines processed 53656755
Articles dropped 8039
Overview index dropped 39271

real 349m6.183s
user 16m33.126s
sys 44m39.500s

The last time I ran expireover without any flags it ran for many more hours.

Re: Question about INN expire

<ta9im0$1eoqq$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=957&group=news.software.nntp#957

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!aioe.org!news.freedyn.de!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.san13-h02-176-143-2-105.dsl.sta.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Fri, 8 Jul 2022 17:32:16 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta9im0$1eoqq$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<ta7a19$1d5ke$1@news.trigofacile.com>
<2514663264.b5c06a4a@freebsd-inject1.usenet.blueworldhosting.com>
<ta7fgr$1d8cj$1@news.trigofacile.com>
<4185399788.df02eb50@freebsd-inject1.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 8 Jul 2022 15:32:16 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="san13-h02-176-143-2-105.dsl.sta.abo.bbox.fr:176.143.2.105";
logging-data="1532762"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:5+1MN8TKw40gdICSpJGQRGrjT9s= sha256:muw+hQZZKqLdll2IA007sdjxyb838UwIiG8gq3mXlhE=
sha1:J2Mg9lswO84oZAkYpaOrO0ENSzY= sha256:zLmjfaXb/0a67rMLLHR7yqbfhZkDutujQ/K9T7/ql0E=
In-Reply-To: <4185399788.df02eb50@freebsd-inject1.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Fri, 8 Jul 2022 15:32 UTC

Hi Jesse,

>> "expireover -e -N -p" should do what you want (see my previous response).
>
> Can I inquire the purpose of the "X" flag in expire.ctl, specifically if
> expireover also needs to be called with '-e' to get the same behavior as the
> "X" flag as described in the manpage for expire.ctl

Running "expireover -e" is tantamount to adding "X" to every expiry rule
(and in that case, you are not obliged to add these "X").

Oh, I see that you did not add "X" to news.lists.filters so in fact you
should not use the "-e" flag.

"expireover -N -p" is the command to run in your case.
I don't think you lost useful articles with your run with "-e",
according to your expire.ctl rules.

> My assumption from reading the expire.ctl manpage was that my "AX" flag in
> expire.ctl would accomplish the same thing as what 'expireover -e' does.

Yes, and "-e" will force "X" in all rules.

> It would seem based on the observation you made (quoted below) that the X flag
> in expire.ctl has no purpose if expire ignores expire.ctl and expireover wants
> the '-e' flag. Is that correct, or am I still missing something?

You're not obliged to add "-e" to expireover.

--
Julien ÉLIE

« Avez-vous remarqué qu'à table les mets que l'on vous sert vous mettent
les mots à la bouche ? » (Raymond Devos)

Re: Question about INN expire

<9587585379.2acfce4f@freebsd-inject1.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=961&group=news.software.nntp#961

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!news.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Fri, 8 Jul 2022 15:57:56 -0000 (UTC)
Organization: BlueWorld Hosting Usenet (https://usenet.blueworldhosting.com)
Message-ID: <9587585379.2acfce4f@freebsd-inject1.usenet.blueworldhosting.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com> <ta7fgr$1d8cj$1@news.trigofacile.com> <4185399788.df02eb50@freebsd-inject1.usenet.blueworldhosting.com> <ta9im0$1eoqq$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 8 Jul 2022 15:57:56 -0000 (UTC)
Injection-Info: news.usenet.blueworldhosting.com;
logging-data="99974"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:ecl6yma6JPBB6QfTZ2T3xu/Opl4= sha256:STAnaT3Srylz26HU7LTdYXOeX4YFWsXictFLQHjoWwo=
sha1:KOPphygrx92zxHxDc63um3k60wo= sha256:oQyTvWRx98Hf8FQ/9+wE73g2B/K0UUQkhDwV9g6ZoUE=
X-Usenapp: v1.22/d - Full License
 by: Jesse Rehmer - Fri, 8 Jul 2022 15:57 UTC

On Jul 8, 2022 at 10:32:16 AM CDT, "Julien ÉLIE" in
<ta9im0$1eoqq$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
>>> "expireover -e -N -p" should do what you want (see my previous response).
>>
>> Can I inquire the purpose of the "X" flag in expire.ctl, specifically if
>> expireover also needs to be called with '-e' to get the same behavior as the
>> "X" flag as described in the manpage for expire.ctl
>
> Running "expireover -e" is tantamount to adding "X" to every expiry rule
> (and in that case, you are not obliged to add these "X").
>
> Oh, I see that you did not add "X" to news.lists.filters so in fact you
> should not use the "-e" flag.
>
> "expireover -N -p" is the command to run in your case.
> I don't think you lost useful articles with your run with "-e",
> according to your expire.ctl rules.
>
>
>
>> My assumption from reading the expire.ctl manpage was that my "AX" flag in
>> expire.ctl would accomplish the same thing as what 'expireover -e' does.
>
> Yes, and "-e" will force "X" in all rules.
>
>
>> It would seem based on the observation you made (quoted below) that the X flag
>> in expire.ctl has no purpose if expire ignores expire.ctl and expireover wants
>> the '-e' flag. Is that correct, or am I still missing something?
>
> You're not obliged to add "-e" to expireover.

Got it, thank you very much for the clear explanation. I think I am still
mixing/confusing the operations of expireover and expire, where when running
expire along it wasn't doing what I expected (which now I also understand
thanks to your help).

Cheers,
Jesse

Re: Question about INN expire

<87fsjb1ti0.fsf@hope.eyrie.org>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=963&group=news.software.nntp#963

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!paganini.bofh.team!news.killfile.org!news.eyrie.org!.POSTED!not-for-mail
From: eag...@eyrie.org (Russ Allbery)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Fri, 08 Jul 2022 09:03:51 -0700
Organization: The Eyrie
Message-ID: <87fsjb1ti0.fsf@hope.eyrie.org>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<ta7fgr$1d8cj$1@news.trigofacile.com>
<4185399788.df02eb50@freebsd-inject1.usenet.blueworldhosting.com>
<ta9im0$1eoqq$1@news.trigofacile.com>
<9587585379.2acfce4f@freebsd-inject1.usenet.blueworldhosting.com>
Mime-Version: 1.0
Content-Type: text/plain
Injection-Info: hope.eyrie.org;
logging-data="22194"; mail-complaints-to="news@eyrie.org"
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
Cancel-Lock: sha1:0WRyqcz5DWxbqJ7sd9CyrWiEJqY=
 by: Russ Allbery - Fri, 8 Jul 2022 16:03 UTC

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:

> Got it, thank you very much for the clear explanation. I think I am
> still mixing/confusing the operations of expireover and expire, where
> when running expire along it wasn't doing what I expected (which now I
> also understand thanks to your help).

It doesn't help that this whole thing grew organically over many years and
involves multiple programs with multiple somewhat confusing modes of
operation and one really intrusive configuration option (groupbaseexpiry)
that basically changes how everything works. This thread makes it clear
that this is all more complicated than it probably should be, but of
course making it less complicated without breaking anything is also very
complicated!

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

Re: Question about INN expire

<ta9p31$1f5nu$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=967&group=news.software.nntp#967

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176-143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: Question about INN expire
Date: Fri, 8 Jul 2022 19:21:37 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ta9p31$1f5nu$1@news.trigofacile.com>
References: <9830233015.44666439@freebsd-inject1.usenet.blueworldhosting.com>
<ta7fgr$1d8cj$1@news.trigofacile.com>
<4185399788.df02eb50@freebsd-inject1.usenet.blueworldhosting.com>
<ta9im0$1eoqq$1@news.trigofacile.com>
<9587585379.2acfce4f@freebsd-inject1.usenet.blueworldhosting.com>
<87fsjb1ti0.fsf@hope.eyrie.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 8 Jul 2022 17:21:37 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176-143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="1545982"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
Gecko/20100101 Thunderbird/91.11.0
Cancel-Lock: sha1:hZH6QbA7Y/mKJA7BZDO0FOzSsFM= sha256:v1V5r6gZOnbhBibJTTQhYjsckFXFIlq7AlCQWpyrsWg=
sha1:5EhxyiJ4DXqKXei2NKa5s9bgg1I= sha256:8xhdq2TVE785uZDCYtPLnonncWJASvGKQqK7CE51bcs=
In-Reply-To: <87fsjb1ti0.fsf@hope.eyrie.org>
 by: Julien ÉLIE - Fri, 8 Jul 2022 17:21 UTC

Hi Jesse and Russ,

>> Got it, thank you very much for the clear explanation. I think I am
>> still mixing/confusing the operations of expireover and expire, where
>> when running expire along it wasn't doing what I expected (which now I
>> also understand thanks to your help).

You're welcome!
I also learnt a few things thanks to your questions.

> It doesn't help that this whole thing grew organically over many years and
> involves multiple programs with multiple somewhat confusing modes of
> operation and one really intrusive configuration option (groupbaseexpiry)
> that basically changes how everything works. This thread makes it clear
> that this is all more complicated than it probably should be, but of
> course making it less complicated without breaking anything is also very
> complicated!

Indeed, groupbaseexpiry complicates the whole thing...

Moreover, the expire(8) man page was cryptic and not totally right. Here is
a new wording for the description of expire(8). I hope it will now be clear
enough.

expire scans the history(5)-format text file pathdb/history and uses
the information recorded in it to purge itself of old news articles.
Its behaviour depends on the setting of groupbaseexpiry in inn.conf.

If groupbaseexpiry is false, expiry is done according storage class
numbers in storage.conf and related rules in expire.ctl. Articles that
should be expired are removed from the news spool and overview (if
any); their history records are purged if they are older than the
number of days specified in the "/remember/" line in expire.ctl.
Articles stored using a storage method that has self-expire
functionality like CNFS are by default not affected by expire's primary
behaviour (but see the -N flag to disable this).

For articles in self-expiring storage methods when groupbaseexpiry is
set to false in inn.conf and the -N flag is not given, or for *all*
articles when groupbaseexpiry is set to true, expire.ctl is ignored
except the "/remember/" line; expire only probes to see if the article
still exists, and purges the relevant history and overview entries if
appropriate.

Per-group expiry, that is to say when groupbaseexpiry is true, is
mostly done by expireover(8): news articles are removed from the news
spool by expireover, and then expire purges the history file.

Regardless the setting of groupbaseexpiry, expireover should be run
along with expire, usually via news.daily out of cron.

Note that expire never purges articles which do not match any entry in
expire.ctl.

Now that expire's documentation is fixed and "expire -p" is in fact working
as it does, let's release INN 2.7.0 :-)
I'll generate the release this week-end.

Thanks again Jesse for all your questions on your fresh INN installation which
permitted to make it better, and especially its documentation!

--
Julien ÉLIE

« – D'ailleurs je vais préparer de la potion magique qui ne servira bien
entendu que contre un retour éventuel des Romains !
– Mais, pour l'heure, les Romains ne sont pas encore revenus de leur
mésaventure. » (Astérix)

Pages:12
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor