Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

Steal my cash, car and TV - but leave the computer! -- Soenke Lange <soenke@escher.north.de>


computers / alt.os.linux.slackware / Re: High system load on NFS snafu

SubjectAuthor
* High system load on NFS snafuS.K.R. de Jong
+* Re: High system load on NFS snafuLew Pitcher
|`- Re: High system load on NFS snafuS.K.R. de Jong
`* Re: High system load on NFS snafuHenrik Carlqvist
 `- Re: High system load on NFS snafuS.K.R. de Jong

1
High system load on NFS snafu

<t7l52e$1nu8$1@gioia.aioe.org>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=1272&group=alt.os.linux.slackware#1272

 copy link   Newsgroups: alt.os.linux.slackware
Path: i2pn2.org!i2pn.org!aioe.org!fs4vz7lwhQCwq5L3H1slGg.user.46.165.242.75.POSTED!not-for-mail
From: SKR...@nowhere.net (S.K.R. de Jong)
Newsgroups: alt.os.linux.slackware
Subject: High system load on NFS snafu
Date: Mon, 6 Jun 2022 15:04:46 -0000 (UTC)
Organization: Aioe.org NNTP Server
Message-ID: <t7l52e$1nu8$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Info: gioia.aioe.org; logging-data="57288"; posting-host="fs4vz7lwhQCwq5L3H1slGg.user.gioia.aioe.org"; mail-complaints-to="abuse@aioe.org";
User-Agent: Pan/0.149 (Bellevue; 4c157ba git@gitlab.gnome.org:GNOME/pan.git)
X-Notice: Filtered by postfilter v. 0.9.2
 by: S.K.R. de Jong - Mon, 6 Jun 2022 15:04 UTC

I have a Slackware64 15.0 system on which I had several
directories mounted by NFS from a remote system. That remote system was
actually rebooted a few times - for maintenance purposes - but I was
stupid enough not to unmount those directories in my system. In fact, I
had at least one terminal emulator where I was in one of the NFS-mounted
directories. I foolishly tried to list the contents of that directory,
and the shell just froze up on me. I had to kill the terminal emulator.

The system load has shot up to at least 4.00 ever since, even
when, according to top, nothing much is going on in the system. I mean, I
have a few things running, but nothing to justify that load: all the
cores are at least 95% idle at any given time.

I was able to unmount those NFS directories - forcefully, on
occasion - and I was able to stop the RPC and NFSD daemons. However, the
high load issue did not disappear.

Anybody got any suggestions as to how to diagnose and solve this
problem, without rebooting? top is not helping, and I see nothing
relevant in dmesg, or any of the /var/log files. More precisely, there
are relevant entries, but they are all old and not being updated - but
the high load stubbornly remains.

Re: High system load on NFS snafu

<t7l8ko$k3r$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=1273&group=alt.os.linux.slackware#1273

 copy link   Newsgroups: alt.os.linux.slackware
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: lew.pitc...@digitalfreehold.ca (Lew Pitcher)
Newsgroups: alt.os.linux.slackware
Subject: Re: High system load on NFS snafu
Date: Mon, 6 Jun 2022 16:05:44 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 46
Message-ID: <t7l8ko$k3r$1@dont-email.me>
References: <t7l52e$1nu8$1@gioia.aioe.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 6 Jun 2022 16:05:44 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="d52157fe5539e0980f53a11c9372a871";
logging-data="20603"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+kmCnooMdDjGJPSRbkTnEjzcbpslkoZf0="
User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
git://git.gnome.org/pan2)
Cancel-Lock: sha1:+BcIWqLvB7NmWbZUGlsijGvpE70=
 by: Lew Pitcher - Mon, 6 Jun 2022 16:05 UTC

On Mon, 06 Jun 2022 15:04:46 +0000, S.K.R. de Jong wrote:

> I have a Slackware64 15.0 system on which I had several directories
> mounted by NFS from a remote system. That remote system was actually
> rebooted a few times - for maintenance purposes - but I was stupid
> enough not to unmount those directories in my system.

[snip]

> The system load has shot up to at least 4.00 ever since, even
> when, according to top, nothing much is going on in the system. I mean,
> I have a few things running, but nothing to justify that load: all the
> cores are at least 95% idle at any given time.
>
> I was able to unmount those NFS directories - forcefully, on
> occasion - and I was able to stop the RPC and NFSD daemons. However, the
> high load issue did not disappear.
>
> Anybody got any suggestions as to how to diagnose and solve this
> problem, without rebooting? top is not helping, and I see nothing
> relevant in dmesg, or any of the /var/log files. More precisely, there
> are relevant entries, but they are all old and not being updated - but
> the high load stubbornly remains.

I have no insights into your high loadavg problem. However, I do note a
suspicious co-incidence:
a) you have high loadavg, and
b) your system logs "are not being updated".

Perhaps you should investigate /why/ the system logs are not being
updated; this phenomenon might be related to the cause of your high
loadavg.

BTW, the getloadavg(3) manpage refers to the proc(5) manpage, and
according to proc(5), the loadavg figures represent "the number of
jobs in the run queue (state R) or waiting for disk I/O (state D)
averaged over 1, 5, and 15 minutes".

I note that you looked at top(1), presumably to see the runnable
processes; did you look to see the processes "waiting for disk I/O"?
If so, was there anything suspicious there? Perhaps a kernel module
or logging daemon that shouldn't have been waiting?

--
Lew Pitcher
"In Skills, We Trust"

Re: High system load on NFS snafu

<t7lf5r$ge9$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=1274&group=alt.os.linux.slackware#1274

 copy link   Newsgroups: alt.os.linux.slackware
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: Henrik.C...@deadspam.com (Henrik Carlqvist)
Newsgroups: alt.os.linux.slackware
Subject: Re: High system load on NFS snafu
Date: Mon, 6 Jun 2022 17:57:15 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 69
Message-ID: <t7lf5r$ge9$1@dont-email.me>
References: <t7l52e$1nu8$1@gioia.aioe.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 6 Jun 2022 17:57:15 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="b629e8b971e30cb50f6289a50bf945f9";
logging-data="16841"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19fPORTYJwu8wWcDz1mlL4G"
User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
git://git.gnome.org/pan2)
Cancel-Lock: sha1:yN5bFhrZKvi/I56zcldl9VfCR7k=
 by: Henrik Carlqvist - Mon, 6 Jun 2022 17:57 UTC

On Mon, 06 Jun 2022 15:04:46 +0000, S.K.R. de Jong wrote:
> I have a Slackware64 15.0 system on which I had several directories
> mounted by NFS from a remote system. That remote system was actually
> rebooted a few times - for maintenance purposes - but I was stupid
> enough not to unmount those directories in my system.

Usually that is not needed when an NFS server is rebooted, once it is
back up again everything is supposed to be fine again.

> In fact, I had at least one terminal emulator where I was in one of the
> NFS-mounted directories. I foolishly tried to list the contents of that
> directory, and the shell just froze up on me. I had to kill the
> terminal emulator.

Most likely, somehow, the NFS server has not come back as it should.

> The system load has shot up to at least 4.00 ever since, even when,
> according to top, nothing much is going on in the system. I mean,
> I have a few things running, but nothing to justify that load: all the
> cores are at least 95% idle at any given time.

Even if you killed the terminal, your ls process is probably still there
in a "D" state (waiting for disk) and your system load is the sum of all
processes wayting for CPU and all processes waiting for disk.
> I was able to unmount those NFS directories - forcefully, on
> occasion - and I was able to stop the RPC and NFSD daemons. However, the
> high load issue did not disappear.

To get rid of the high load you will need to kill the processes in "D"
state. This is probably only possible if you mounted the NFS directories
with the "intr" option.

Stopping the rpc and nfsd daemons on the NFS server will from the NFS
clients point of view be just as bad as shutting down the NFS server
completely. Any processes being hung in "D" state will be so until the
NFS service is restored. Instead of stopping the NFS service you should
do something like "/etc/rc.d/rc.nfsd restart".

> Anybody got any suggestions as to how to diagnose and solve this
> problem, without rebooting? top is not helping, and I see nothing
> relevant in dmesg, or any of the /var/log files.

In both dmesg and your log files you should see something like this:

nfs: server foo.example.com not responding, still trying

When this is the latest you see about that NFS server you will get
processes stuck in "D" state. Once the NFS server is rebooted and up
again you should see:

nfs: server foo.example.com OK

and all your processes in "D" state should get back to normal again.

> More precisely, there are relevant entries, but they are all old and
> not being updated - but the high load stubbornly remains.

If you do:

ps aux | grep D

and look for processes with a "D" in the STAT column those processes
might explain your high load. There are other tools like lsof and fuser
to find out which processes are in an NFS mounted directory (or any other
directory), but you should focus on bringing that NFS server back instead
of killing unfinished processes.

regards Henrik

Re: High system load on NFS snafu

<t7llsl$i95$1@gioia.aioe.org>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=1275&group=alt.os.linux.slackware#1275

 copy link   Newsgroups: alt.os.linux.slackware
Path: i2pn2.org!i2pn.org!aioe.org!fs4vz7lwhQCwq5L3H1slGg.user.46.165.242.75.POSTED!not-for-mail
From: SKR...@nowhere.net (S.K.R. de Jong)
Newsgroups: alt.os.linux.slackware
Subject: Re: High system load on NFS snafu
Date: Mon, 6 Jun 2022 19:51:50 -0000 (UTC)
Organization: Aioe.org NNTP Server
Message-ID: <t7llsl$i95$1@gioia.aioe.org>
References: <t7l52e$1nu8$1@gioia.aioe.org> <t7lf5r$ge9$1@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Info: gioia.aioe.org; logging-data="18725"; posting-host="fs4vz7lwhQCwq5L3H1slGg.user.gioia.aioe.org"; mail-complaints-to="abuse@aioe.org";
User-Agent: Pan/0.149 (Bellevue; 4c157ba git@gitlab.gnome.org:GNOME/pan.git)
X-Notice: Filtered by postfilter v. 0.9.2
 by: S.K.R. de Jong - Mon, 6 Jun 2022 19:51 UTC

On Mon, 6 Jun 2022 17:57:15 -0000 (UTC), Henrik Carlqvist wrote:

> Even if you killed the terminal, your ls process is probably still there
> in a "D" state (waiting for disk) and your system load is the sum of all
> processes wayting for CPU and all processes waiting for disk.

Thanks. That did the trick: I had three processes in a "D" state
(actually, D and something else) - one of them being indeed the shell
where I tried to do the ls. After killing them the system load is back to
the levels that I would expect from the ordinary system activity.

Re: High system load on NFS snafu

<t7lmop$i95$2@gioia.aioe.org>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=1276&group=alt.os.linux.slackware#1276

 copy link   Newsgroups: alt.os.linux.slackware
Path: i2pn2.org!i2pn.org!aioe.org!fs4vz7lwhQCwq5L3H1slGg.user.46.165.242.75.POSTED!not-for-mail
From: SKR...@nowhere.net (S.K.R. de Jong)
Newsgroups: alt.os.linux.slackware
Subject: Re: High system load on NFS snafu
Date: Mon, 6 Jun 2022 20:06:49 -0000 (UTC)
Organization: Aioe.org NNTP Server
Message-ID: <t7lmop$i95$2@gioia.aioe.org>
References: <t7l52e$1nu8$1@gioia.aioe.org> <t7l8ko$k3r$1@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Info: gioia.aioe.org; logging-data="18725"; posting-host="fs4vz7lwhQCwq5L3H1slGg.user.gioia.aioe.org"; mail-complaints-to="abuse@aioe.org";
User-Agent: Pan/0.149 (Bellevue; 4c157ba git@gitlab.gnome.org:GNOME/pan.git)
X-Notice: Filtered by postfilter v. 0.9.2
 by: S.K.R. de Jong - Mon, 6 Jun 2022 20:06 UTC

On Mon, 6 Jun 2022 16:05:44 -0000 (UTC), Lew Pitcher wrote:

> I have no insights into your high loadavg problem. However, I do note a
> suspicious co-incidence:
> a) you have high loadavg, and b) your system logs "are not being
> updated".

I'm sorry - I meant they were not being updated with NFS-related
diagnostics. They were updated all along with diagnostics associated with
other events - like e.g. when ssh clients closed a connection.

1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor