On 7 Apr 2023 at 09:30:02 CEST, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Eli,
>
>> It was the 'server' setting indeed.
>>
>> I'm getting close, but now inews is trying to connect using the
>> IPv6 address of the server, instead of IPv4.
>
> Did you try to just put the IPv4 address of your server in the "server"
> setting? (instead of its hostname)
> It will maybe work (I have not tested).

Hello Julien,

Yes, that was the trick.
It works now :)

Thank you.

I have another question about the settings in expiration.ctl.

What settings should I use if I want to delete all posts older than 90 days
for 1 specific newsgroup (e.g. linux.debian.bugs.dist).

As far as I can read in the documentation, this mainly concerns the middle
field (the default value) and the first and last fields (keep and purge) are
only important for messages that have an expiration header. But if I set the
<purge> field lower than the <default> field, inncheck still throws a warning.
So it seems that the <keep and purge> fields still affect the <default> value.

linux.debian.bugs.dist:AX:0:90:11
inncheck returns: purge `11' younger than default `90'

linux.debian.bugs.dist:AX:0:90:90
seems good.

What I would like is to have all messages older than 90 days deleted
immediately and messages with the expiration header deleted immediately after
the expiration date.

Which setting do you recommend for this in expire.ctl?

On 22 Mar 2023 at 23:49:55 CET, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Eli,
>
>> Another question, is it possible to limit the maximum number of connections
>> per authenticated user? I know this is possible for peers, but can this also
>> be set up for authenticated users? Maybe a setting in readers.conf or nnrpd
>> that I'm overlooking?
>
> Unfortunately, the response is no. There's no native way of limiting
> users' connections.
> You may want to write a custom authentication hook (perl_auth or
> python_auth in readers.conf) that would do the job by accounting how
> many connections are open by a given user, and deny access if it exceeds
> the limit. I am not aware of existing scripts to do that :-(
>
> It could be worthwhile having though, as you're not the first one to ask
> (but nobody wrote or shared what he came up with).

Hi Julies,

The nnrpd manual states:

"As each command is received, nnrpd tries to change its "argv" array so that
ps(1) will print out the command being executed."

This will then look like this:
nnrpd: <xxx.xxx.xxx.xxx> GROUP
nnrpd: <xxx.xxx.xxx.xxx> XOVER

Is it perhaps also possible to add the authenticated user to this?

Something like:
nnrpd: <xxx.xxx.xxx.xxx> Eli GROUP
nnrpd: <xxx.xxx.xxx.xxx> Eli XOVER

This would make it possible to limit the number of connections per user via a
perl script.

Hi Eli,

> I have another question about the settings in expire.ctl.
>
> What settings should I use if I want to delete all posts older than 90 days
> for 1 specific newsgroup (e.g. linux.debian.bugs.dist).
>
> As far as I can read in the documentation, this mainly concerns the middle
> field (the default value) and the first and last fields (keep and purge) are
> only important for messages that have an expiration header. But if I set the
> <purge> field lower than the <default> field, inncheck still throws a warning.
> So it seems that the <keep and purge> fields still affect the <default> value.
>
> linux.debian.bugs.dist:AX:0:90:11
> inncheck returns: purge `11' younger than default `90'

"""
<pattern>:<flag>:<min>:<default>:<max>

The middle field, <default>, will be used as the expiration period for
most articles. The other two fields, <min> and <max>, only come into
play if the article requests a particular expiration date with an
Expires header field. Articles with an Expires header field will be
expired at the date given in that header field, subject to the
constraints that they will be retained at least <min> days and no
longer than <max> days.

One should think of the fields as a lower bound, the default, and an
upper bound. Since most articles do not have an Expires header field,
the second field is the most important and most commonly applied.

It is often useful to honor the Expires header field in articles,
especially those in moderated groups. To do this, set <min> to zero,
<default> to whatever normal expiration you wish, and <max> to "never"
or some large number, like 365 days for a maximum article life of a
year.
"""

> linux.debian.bugs.dist:AX:0:90:90
> seems good.

It is indeed what I would parameter for this newsgroup.

> What I would like is to have all messages older than 90 days deleted
> immediately and messages with the expiration header deleted immediately after
> the expiration date.

That's what the 0:90:90 setting does.

With 0:90:11, suppose that you have an article with an Expires header
field corresponding to 30 days, it would be deleted after 11 days which
is not what you were expecting. That's the reason of the warning from
inncheck; it looks unusual to force the deletion of articles with an
Expires header field sooner than other articles. (The date in the
Expires header field will still be respected.)

--
Julien ÉLIE

« A killfile on Usenet can get you peace and quiet. A killfile in the
real world can get you twenty years to life. » (Nils Nieuwjaar)

On 7 Apr 2023 at 17:23:46 CEST, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> With 0:90:11, suppose that you have an article with an Expires header
> field corresponding to 30 days, it would be deleted after 11 days which
> is not what you were expecting. That's the reason of the warning from
> inncheck; it looks unusual to force the deletion of articles with an
> Expires header field sooner than other articles. (The date in the
> Expires header field will still be respected.)

Another stunning explanation.
Thanks again, Julien!

Hi Eli,

> The nnrpd manual states:
>
> "As each command is received, nnrpd tries to change its "argv" array so that
> ps(1) will print out the command being executed."
>
> This will then look like this:
> nnrpd: <xxx.xxx.xxx.xxx> GROUP
> nnrpd: <xxx.xxx.xxx.xxx> XOVER
>
> Is it perhaps also possible to add the authenticated user to this?
>
> Something like:
> nnrpd: <xxx.xxx.xxx.xxx> Eli GROUP
> nnrpd: <xxx.xxx.xxx.xxx> Eli XOVER
>
> This would make it possible to limit the number of connections per user via a
> perl script.

It is indeed possible to use that "feature".

If you can rebuild INN from sources, just change the following command
in nnrpd/nnrpd.c:

- setproctitle("%s %s", Client.host, av[0]);
+ + setproctitle("%s %s %s", Client.host,
+ PERMuser[0] != '\0' ? PERMuser : "-", av[0]);

I then have lines like that in a ps output:

nnrpd: accepting connections
nnrpd: 176-143-2-105.abo.bbox.fr julien GROUP
nnrpd: 5.14.145.68 <all> LIST

I am unsure if this would be worth having in an official release; there
may be privacy concerns. Maybe it should be configurable with a
readers.conf option (like addprocesstitleuser which would enable that
behaviour when set to true in an access group).

--
Julien ÉLIE

« Et maintenant, la balle est dans le camp des slalomeurs. »

On 11 Apr 2023 at 12:12:12 CEST, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Eli,
>
>> The nnrpd manual states:
>>
>> "As each command is received, nnrpd tries to change its "argv" array so that
>> ps(1) will print out the command being executed."
>>
>> This will then look like this:
>> nnrpd: <xxx.xxx.xxx.xxx> GROUP
>> nnrpd: <xxx.xxx.xxx.xxx> XOVER
>>
>> Is it perhaps also possible to add the authenticated user to this?
>>
>> Something like:
>> nnrpd: <xxx.xxx.xxx.xxx> Eli GROUP
>> nnrpd: <xxx.xxx.xxx.xxx> Eli XOVER
>>
>> This would make it possible to limit the number of connections per user via a
>> perl script.
>
> It is indeed possible to use that "feature".
>
> If you can rebuild INN from sources, just change the following command
> in nnrpd/nnrpd.c:
>
> - setproctitle("%s %s", Client.host, av[0]);
> +
> + setproctitle("%s %s %s", Client.host,
> + PERMuser[0] != '\0' ? PERMuser : "-", av[0]);

That is great news.
Thanks !

> I am unsure if this would be worth having in an official release; there
> may be privacy concerns. Maybe it should be configurable with a
> readers.conf option (like addprocesstitleuser which would enable that
> behaviour when set to true in an access group).

I don't know if many people will use this feature, but it is nice if INN
supports it. Making it configurable is a good idea.

About pullnews:

In pullnews I use the '-w -1000000' option to download the 1 million most
recent articles per newsgroup.

This works fine, but when pullnews is restarted (for example after a server
timeout), pullnews will redownload all already downloaded articles.

It does come with the message that the articles already exist, but when there
are almost a million per newsgroup, it is not very pleasant.

It would be nice if pullnews continued downloading where it left off. For
example, only if the high water mark >0.

Could you improve this on pullnews?
Thanks.

Hi Eli,

>> I am unsure if this would be worth having in an official release; there
>> may be privacy concerns. Maybe it should be configurable with a
>> readers.conf option (like addprocesstitleuser which would enable that
>> behaviour when set to true in an access group).
>
> I don't know if many people will use this feature, but it is nice if INN
> supports it. Making it configurable is a good idea.

OK, I'll have a look for a future version.

> About pullnews:
>
> In pullnews I use the '-w -1000000' option to download the 1 million most
> recent articles per newsgroup.
>
> This works fine, but when pullnews is restarted (for example after a server
> timeout), pullnews will redownload all already downloaded articles.
>
> It does come with the message that the articles already exist, but when there
> are almost a million per newsgroup, it is not very pleasant.
>
> It would be nice if pullnews continued downloading where it left off. For
> example, only if the high water mark >0.
>
> Could you improve this on pullnews?

Ah, I see the point.
You may want to use "-O" right now to prevent the download of already
existing articles.

When pullnews automatically restart after a timeout, -w is indeed
applied again. After a quick look at all the options, only -w should be
improved in that case, for not being applied on the newsgroup it was
treating when the timeout occurred, but directly restart from where it
was. I'll provide a patch (not today however but probably tomorrow, I
hope that's fine for you).

I assume you do not use '-w -1000000' for another run of pullnews
(manual or out of cron); it would otherwise be normal for it to download
again a million of articles.

--
Julien ÉLIE

« Ils ont refusé une offre de Normand ?!? » (Astérix)

On 12 Apr 2023 at 10:33:39 CEST, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

>> About pullnews:
>>
>> In pullnews I use the '-w -1000000' option to download the 1 million most
>> recent articles per newsgroup.
>>
>> This works fine, but when pullnews is restarted (for example after a server
>> timeout), pullnews will redownload all already downloaded articles.
>>
>> It does come with the message that the articles already exist, but when there
>> are almost a million per newsgroup, it is not very pleasant.
>>
>> It would be nice if pullnews continued downloading where it left off. For
>> example, only if the high water mark >0.
>>
>> Could you improve this on pullnews?
>
> I assume you do not use '-w -1000000' for another run of pullnews
> (manual or out of cron); it would otherwise be normal for it to download
> again a million of articles.

Yes, I do because there are more newsgroups in the pullnews.mark file that may
not have been downloaded yet.

But I modified pullnews with the line below and this seems to work fine:

if (defined $watermark) {
printf LOG "\tOur previous highest: %d\n", $prevHigh if not $quiet;
$high = $watermark;
$high = $last + $watermark if substr($watermark, 0, 1) eq '-';
+ $high = $prevHigh if $prevHigh > 0;
$high = 0 if $high < 0;$shash->{$group} = [time, $high];
}

Hi Eli,

> But I modified pullnews with the line below and this seems to work fine:
>
> if (defined $watermark) {
> printf LOG "\tOur previous highest: %d\n", $prevHigh if not $quiet;
> $high = $watermark;
> $high = $last + $watermark if substr($watermark, 0, 1) eq '-';
> + $high = $prevHigh if $prevHigh > 0;
> $high = 0 if $high < 0;
> $shash->{$group} = [time, $high];
> }

I am unsure this change does what you expected.
On *each* run, including the first one before any timeout occurs, $high will be forced to $prevHigh, and therefore -w won't be taken into account.
$prevHigh is the highest article number previously downloaded (as recorded in pullnews.marks). In case it is "0" or unset in pullnews.marks, -w will work, but in case pullnews.marks has for instance recorded 500, and you use "-w -100", nothing will happen ($high won't be set to 400 to force the re-downloading of the latest 100 articles).

I would suggest the following patch instead:

--- a/frontends/pullnews
+++ b/frontends/pullnews
@@ -560,6 +560,7 @@ if (not $quiet and not $quietness) {
}

my $connectionAttempts = 0;
+my %groupsStarted = ();

UPSTREAM:
foreach my $server (@servers) {
@@ -689,6 +690,7 @@ foreach my $server (@servers) {
} continue {
# Reinitialize the counter for the next server.
$connectionAttempts = 0;
+ %groupsStarted = ();
}

saveConfig();
@@ -852,12 +854,17 @@ sub crossFeedGroup {
printf LOG "\t%d article%s available (first %d, last %d)\n",
$narticles, $narticles != 1 ? "s" : "", $first, $last;
}
- if (defined $watermark) {
+ + # Do not set several times the water mark to another value. Just go on
+ # downloading articles from the last retrieved one when the connection
+ # timed out.
+ if (defined($watermark) and !exists($groupsStarted{$group})) {
printf LOG "\tOur previous highest: %d\n", $prevHigh if not $quiet;
$high = $watermark;
$high = $last + $watermark if substr($watermark, 0, 1) eq '-';
$high = 0 if $high < 0;
$shash->{$group} = [time, $high];
+ $groupsStarted{$group} = 1;
}
printf LOG "\tOur current highest: %d", $high if not $quiet;

Incidentally, I see that "pullnews -w" just sets the watermark without downloading articles in the same run.
pullnews -w -100 modifies 500 in pullnews.marks to 400, and exits. I have to run pullnews again to actually download the articles.
It should not be this way I think. Suggestion of another fix:

@@ -876,7 +883,7 @@ sub crossFeedGroup {
$high = $first + floor(($last - $first + 1) * (1 - $opt_b));
$toget = $last - $high;
print LOG " (reset highwater mark to $high)" unless $quiet;
- } elsif ($prevHigh == -1 || $last <= $prevHigh) {
+ } elsif ($prevHigh == -1 || $last <= $high) {
# We connected OK but there's nothing there, or we just want
# to reset our highwater mark.
$shash->{$group} = [time, $high];

--
Julien ÉLIE

« – Par Thor !
– Par Odin !
– Par exemple ! » (Astérix)

Hi Jesse,

>>> [news@spool1 ~]$ cat pullnews4.marks
>>> [news@spool1 ~]$
>>
>> Could you try to add an explicit message error ?
>>
>> close(FILE) or die "can't close $groupFile: $!\n";;
>
> Sure, may be a few days before I reply back with results!

Is the problem of an empty pullnews.marks file still happening?

--
Julien ÉLIE

« Le caramel est un invité du palais qui menace la couronne. » (Tristan
Bernard)