Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

/earth: file system full.


devel / rocksolid.programming / rslight inventory syncronization and $maxfirstrequest slowdown

SubjectAuthor
* rslight inventory syncronization and $maxfirstrequest slowdownSyber Shock
`- rslight inventory syncronization and $maxfirstrequest slowdownRetro Guy

1
rslight inventory syncronization and $maxfirstrequest slowdown

<ba97d34baed57a9e26482adc6f9990e0$1@sybershock.com>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=751&group=rocksolid.programming#751

  copy link   Newsgroups: rocksolid.programming
Path: i2pn2.org!rocksolid2!.POSTED!not-for-mail
From: adm...@sybershock.com (Syber Shock)
Newsgroups: rocksolid.programming
Subject: rslight inventory syncronization and $maxfirstrequest slowdown
Date: Mon, 25 Sep 2023 03:36:26 -0500
Organization: sybershock.com
Message-ID: <ba97d34baed57a9e26482adc6f9990e0$1@sybershock.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Info: novabbs.org;
logging-data="3701985"; mail-complaints-to="usenet@novabbs.org";
posting-account="TzG3Hl99Aa0Fgb506WreKRgRTO2mG9+aGjVDifyfNqo";
 by: Syber Shock - Mon, 25 Sep 2023 08:36 UTC

Something with inventory comparison is making peer syncronization very
slow. It was taking almost 15 minutes to complete every run of
cron.php. My php is not very savvy but here's what I've gathered so far.

spoolnews.php had these variables:

if(!isset($maxarticles_per_run)) {
$maxarticles_per_run = 30;
} if(!isset($maxfirstrequest)) {
$maxfirstrequest = 3000;
}

It seems that every time cron.php runs the $maxfirstrequest value
is read and stat compared that many messages backward on the remote
peer. So for every newsgroup synced, 1000 stats and message-id
comparisons being performed, not just the first time, but every time
cron.php runs. This was happening regardless of the size of
$maxarticles_per_run. At least that is how it looks in the logs.

So if 1000 newsgroups are already syncronized then 1 million message
lookups will be performed even though there may be no new messages on
either peer. At least that's how it seems.

It also has the effect of making sync extremely slow even when there
are no new messages on the remote peer, and all messages have been
synchronized for months.

I reduced $maxfirstrequest to 1 and now there is a 25x speedup each
time cron.php runs.

Is there a workaround to get it just to check the head count value and
work back from newest and stop when the first inventory collision is
found? Or maybe should I try to create a function for it?

--
3883@sugar.bug | web: sybershock.com | news: alt.sources.crypto

Re: rslight inventory syncronization and $maxfirstrequest slowdown

<ff393398fbb6037b08b17221f2df5345@news.novabbs.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=752&group=rocksolid.programming#752

  copy link   Newsgroups: rocksolid.programming
Path: i2pn2.org!.POSTED!not-for-mail
From: retro....@rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.programming
Subject: Re: rslight inventory syncronization and $maxfirstrequest slowdown
Date: Mon, 25 Sep 2023 09:16:16 +0000
Organization: Rocksolid Light
Message-ID: <ff393398fbb6037b08b17221f2df5345@news.novabbs.org>
References: <ba97d34baed57a9e26482adc6f9990e0$1@sybershock.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: i2pn2.org;
logging-data="3228133"; mail-complaints-to="usenet@i2pn2.org";
posting-account="PGd4t4cXnWwgUWG9VtTiCsm47oOWbHLcTr4rYoM0Edo";
User-Agent: Rocksolid Light 0.9.1
X-Rslight-Posting-User: 91053d4a47d51b416144568e5a1040f05e31ed1b
X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-13) on novalink.us
X-Rslight-Site: $2y$10$Tx6BcXV/kcXstbtJMgEojOyqC.uVf2j.6PXCT.eIrW3i77xbtbOra
X-Face: .&YR-G(w(DZ$$,}%k=]*5*!p'=(anr"IT`wZG'2VWdfl\r)l[42u7JH`n(JUQ*e5*A|XCDf
?&\X&uwkl38"CYX3O8m}C8E4p'%N$2#kSTVzx{Ly|DjLT\Vk7NE}NQ(VC$Yq]i:7|z[.9iv^g>*8_B
H0=hZt'[%)4kG|
 by: Retro Guy - Mon, 25 Sep 2023 09:16 UTC

Syber Shock wrote:

> Something with inventory comparison is making peer syncronization very
> slow. It was taking almost 15 minutes to complete every run of
> cron.php. My php is not very savvy but here's what I've gathered so far.

> spoolnews.php had these variables:

> if(!isset($maxarticles_per_run)) {
> $maxarticles_per_run = 30;
> }
> if(!isset($maxfirstrequest)) {
> $maxfirstrequest = 3000;
> }

This is what's in the file. It looks modified (just the numbers) but that
is what's in spoolnews.php.

> It seems that every time cron.php runs the $maxfirstrequest value
> is read and stat compared that many messages backward on the remote
> peer. So for every newsgroup synced, 1000 stats and message-id

I believe that cron.php is timing out. I think 15 minutes was the value
hardcoded to timeout. If that's the case, the last article retrieved
is not being recorded, so it's starting over every time.

> comparisons being performed, not just the first time, but every time
> cron.php runs. This was happening regardless of the size of
> $maxarticles_per_run. At least that is how it looks in the logs.

> So if 1000 newsgroups are already syncronized then 1 million message
> lookups will be performed even though there may be no new messages on
> either peer. At least that's how it seems.

> It also has the effect of making sync extremely slow even when there
> are no new messages on the remote peer, and all messages have been
> synchronized for months.

> I reduced $maxfirstrequest to 1 and now there is a 25x speedup each
> time cron.php runs.

Maybe it's not timing out?

Are you running through a slow network, or clearnet? If it's I2P, I'm
not terribly surprised if all the STAT requests are slowing it down to
an extreme level.

> Is there a workaround to get it just to check the head count value and
> work back from newest and stop when the first inventory collision is
> found? Or maybe should I try to create a function for it?

spoolnews.php has not used STAT for quite a while now. This has been
switched to XOVER. So it's one request for all messages it wants, not
one per request (for header info).

Then, duplicate checking is done from the overview received.

I do sync one rslight over I2P, and it works fine, but I'm running the
latest code. There have been a lot of changes since php7 code.

--
Retro Guy


devel / rocksolid.programming / rslight inventory syncronization and $maxfirstrequest slowdown

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor