Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  nodelist  faq  login

echo "Your stdio isn't very std." -- Larry Wall in Configure from the perl distribution


rocksolid / Rocksolid Nodes / rate limiting in rslight

SubjectAuthor
* rate limiting in rslightRetro Guy
`* Re: rate limiting in rslightAnonymous
 `* Re: rate limiting in rslightRetro Guy
  `* Re: rate limiting in rslighttrw
   `- Re: rate limiting in rslightRetro Guy

1
Subject: rate limiting in rslight
From: Retro Guy
Newsgroups: rocksolid.nodes
Organization: Rocksolid Light
Date: Wed, 6 Jan 2021 10:06 UTC
Path: i2pn2.org!rocksolid2!.POSTED.localhost!not-for-mail
From: retro....@rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.nodes
Subject: rate limiting in rslight
Date: Wed, 6 Jan 2021 10:06:19 +0000
Organization: Rocksolid Light
Message-ID: <cca9cd3838da484861b2129d7d32bb89$1@news.novabbs.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: novabbs.org; posting-account="retrobbs1"; posting-host="localhost:127.0.0.1";
logging-data="26704"; mail-complaints-to="usenet@novabbs.org"
User-Agent: Rocksolid Light (news.novabbs.com/getrslight)
X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on novabbs.org
X-Rslight-Site: $2y$10$YxEGNIzfuNIiwhxjk84LYOX.htRsaQn6pOGb45NZ6XvQxx9hxAtX6
View all headers
I've finally set up rate limiting in rslight so that it's not necessary to do so with web server settings.

The server works great for this as long as each client has it's own ip address, which is fine in clearnet and i2p, but does not work well in tor. tor users all have the same ip address, so it ends up blocking all tor users if one is abusing the server.

I'm now limiting per session, and so far it seems to be working fine. It should be fine for clearnet and tor also, but it's really for tor. Most of my tor traffic is badly configured crawlers that just hit the server as fast as they possibly can, so you get many hits per second. Now the logs show the crawlers getting a '429' and regular users are still able to load the site.

I'll keep testing this and see how it goes.

Retro Guy

--
Posted on Rocksolid Light
news.novabbs.org


Subject: Re: rate limiting in rslight
From: Anonymous
Newsgroups: rocksolid.nodes
Organization: def5
Date: Wed, 6 Jan 2021 12:27 UTC
References: 1
Path: i2pn2.org!rocksolid2!def5!.POSTED.bogusentry!not-for-mail
From: pos...@anon.com (Anonymous)
Newsgroups: rocksolid.nodes
Subject: Re: rate limiting in rslight
Date: Wed, 06 Jan 2021 04:27:53 -0800
Organization: def5
Message-ID: <n.1283.3r3v9b@anon.com>
References: <cca9cd3838da484861b2129d7d32bb89$1@news.novabbs.org>
Content-Type: text/plain; charset=UTF-8
Injection-Info: def5.org; posting-host="bogusentry:192.168.1.189";
logging-data="27087"; mail-complaints-to="usenet@def5.org"
View all headers
I'm now limiting per session,

I was not aware that rslight uses sessions. How did you do that, cookies ?

Good that it works against the crawlers, though. I just realized that cookies will not work against crawlers that do not take them, so probably you have found another way (really curious here).

--
Posted on def2


Subject: Re: rate limiting in rslight
From: Retro Guy
Newsgroups: rocksolid.nodes
Organization: RetroBBS
Date: Wed, 6 Jan 2021 22:59 UTC
References: 1 2
Path: i2pn2.org!rocksolid2!.POSTED.localhost!not-for-mail
From: retro_...@retrobbs.rocksolidbbs.com (Retro Guy)
Newsgroups: rocksolid.nodes
Subject: Re: rate limiting in rslight
Date: Wed, 6 Jan 2021 15:59:13 -0700
Organization: RetroBBS
Message-ID: <20210106155913.279b35b5@desktop14.dt>
References: <cca9cd3838da484861b2129d7d32bb89$1@news.novabbs.org>
<n.1283.3r3v9b@anon.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Info: novabbs.org; posting-account="retrobbs1"; posting-host="localhost:127.0.0.1";
logging-data="18418"; mail-complaints-to="usenet@novabbs.org"
X-Newsreader: Claws Mail 3.11.1 (GTK+ 2.24.25; x86_64-pc-linux-gnu)
View all headers
On Wed, 06 Jan 2021 04:27:53 -0800
Anonymous <poster@anon.com> wrote:

I'm now limiting per session,

I was not aware that rslight uses sessions. How did you do that,
cookies ?

Good that it works against the crawlers, though. I just realized that
cookies will not work against crawlers that do not take them, so
probably you have found another way (really curious here).


What I do is connect to the session in specific files, not every file:
session_start();

Then verify a start time or create a start time (of the session), count
hits (views), then I can monitor hit rates:
$rate = ($_SESSION['views'] / (time() - $_SESSION['starttime']));

If there is a big gap since start time, it is reset to avoid spoofing
rate. Then, reading $rate it's decided whether it's too fast or not and
if it is, it sends a 429 "Too Many Requests" and exits the script
(sending no content).

In my logs, all the crawlers are receiving 429, and the non crawlers
(users) are able to load the page with no issues.

Cookies are not involved, just sessions.

Retro Guy



Subject: Re: rate limiting in rslight
From: trw
Newsgroups: rocksolid.nodes
Organization: def5
Date: Sat, 9 Jan 2021 18:17 UTC
References: 1
Path: i2pn2.org!rocksolid2!.POSTED.127.117.190.215!POSTED.localhost!not-for-mail
From: trw...@anon.org (trw)
Newsgroups: rocksolid.nodes
Subject: Re: rate limiting in rslight
Date: Sat, 09 Jan 2021 18:17:04+0000
Organization: def5
Message-ID: <bde362c847086d7194b65713c57ce9cc@def4>
References: <20210106155913.279b35b5@desktop14.dt>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: novabbs.org; posting-account="def.i2p"; posting-host="127.117.190.215";
logging-data="31908"; mail-complaints-to="usenet@novabbs.org"
In-Reply-To: <20210106155913.279b35b5@desktop14.dt>
View all headers

thx for the explanation. I was just wondering how a specific request is mapped to a session, but I can read it up in the php documentation then.
if there are no cookies involved it is probably done with a session token ? I will check it out.
I have not used sessions in any of my own php code, but fudforum does of course (in the adapted version the session token is embedded in the links, so that the resulting GET string contains the token).

--
Posted on def4


Subject: Re: rate limiting in rslight
From: Retro Guy
Newsgroups: rocksolid.nodes
Organization: novaBBS
Date: Fri, 5 Feb 2021 02:00 UTC
References: 1 2
Path: i2pn2.org!.POSTED!not-for-mail
From: retro_...@novabbs.com (Retro Guy)
Newsgroups: rocksolid.nodes
Subject: Re: rate limiting in rslight
Date: Thu, 4 Feb 2021 19:00:49 -0700
Organization: novaBBS
Message-ID: <20210204190049.0f3eff6a@desktop14.dt>
References: <20210106155913.279b35b5@desktop14.dt>
<bde362c847086d7194b65713c57ce9cc@def4>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Injection-Info: i2pn2.org; posting-account="retrobbs1";
logging-data="20961"; mail-complaints-to="usenet@i2pn2.org"
X-Newsreader: Claws Mail 3.11.1 (GTK+ 2.24.25; x86_64-pc-linux-gnu)
View all headers
On Sat, 09 Jan 2021 18:17:04+0000
trw <trw@anon.org> wrote:


thx for the explanation. I was just wondering how a specific request
is mapped to a session, but I can read it up in the php documentation
then. if there are no cookies involved it is probably done with a
session token ? I will check it out. I have not used sessions in any
of my own php code, but fudforum does of course (in the adapted
version the session token is embedded in the links, so that the
resulting GET string contains the token).


That's interesting. I was wondering how you resolved the sessions in
tor issue :)

My reasons in rslight is that tor users all have the same ip address
from the web server's view, so fail2ban, throttling in the web server,
etc. won't work properly without just blocking all tor users.

I can see in the logs clients being throttled, while other tor users
are not affected. It's much better than when a spider could just shut
the site down by getting itself blocked (so all tor users blocked). The
spiders seem to hit around 5 request per second and get blocked very
quickly.

I wish the operators of the crawlers would check their logs and see all
they are getting is 429 and nothing useful, but they'll probably just
keep it up forever. To get actual stuff to use in your search engine,
or whatever, slow the f down and don't get blocked.



1
rocksolid light 0.7.2
clearneti2ptor