novaBBS - uk.comp.os.linux - Process Duplication Within for Loop

Process Duplication Within for Loop

<sdh57p$1une$1@gioia.aioe.org>

https://www.novabbs.com/aus+uk/article-flat.php?id=273&group=uk.comp.os.linux#273

copy link Newsgroups: alt.os.linux uk.comp.os.linux

Path: i2pn2.org!i2pn.org!aioe.org!st4z+icvTwfUsqGJRr80Xw.user.46.165.242.75.POSTED!not-for-mail
From: jav...@evij.com.invalid (Java Jive)
Newsgroups: alt.os.linux,uk.comp.os.linux
Subject: Process Duplication Within for Loop
Date: Sat, 24 Jul 2021 14:40:08 +0100
Organization: Aioe.org NNTP Server
Message-ID: <sdh57p$1une$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Info: gioia.aioe.org; logging-data="64238"; posting-host="st4z+icvTwfUsqGJRr80Xw.user.gioia.aioe.org"; mail-complaints-to="abuse@aioe.org";
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:68.0) Gecko/20100101
Thunderbird/68.4.2
X-Notice: Filtered by postfilter v. 0.9.2
X-Mozilla-News-Host: news://nntp.aioe.org:119
Content-Language: en-GB

by: Java Jive - Sat, 24 Jul 2021 13:40 UTC

I have the following piece of code creating an rsync process launched in
the early hours of every morning to sync a NAS from another upstream. I
have only included the business part of the file, the beginning of it is
to do with setting up the variable values used in these lines. As you
will note, there are two debugging statements trapping the output of ps ...

# Debug Line 1:
ps | grep -v grep | grep tutsync.sh > $HD/rsync.log
echo "Syncing the following shares with rsync: ${SHARES}" >> $HD/rsync.log
for SHARE in ${SHARES}
do
# Debug Line 2:
ps | grep -v grep | grep tutsync.sh >> $HD/rsync.log
echo "nice -n ${NICE} rsync ${OPTIONS}
${SOURCE}/${SHARE}/ ${BU}/${SHARE}/"
nice -n ${NICE} rsync ${OPTIONS} ${SOURCE}/${SHARE}/
${BU}/${SHARE}/
done | grep -v '/$' >> $HD/rsync.log 2>&1

.... because, when run, the process appears to duplicate itself within
the for loop - not only does the tutsync.sh process itself get
duplicated, but also the rsync process it launches as well, as can be
proven by launching with a trailing & and running a similar command from
the shell:

~ # cat $HD/rsync.log
32453 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
Syncing the following shares with rsync: A List Of Shares
32453 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
32458 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
receiving incremental file list
etc

~ # pf rsync
9680 root 1972 S /opt/bin/rsync --ipv4 --daemon
--config=/etc/rsyncd.conf
32467 root 2908 R N rsync -Drltugpv --exclude=*.flv
--exclude=*.partial.* --exclude=*.part --exclude=**/.streams
32468 root 49772 S N rsync -Drltugpv --exclude=*.flv
--exclude=*.partial.* --exclude=*.part --exclude=**/.streams

What is going on, why are the processes duplicating themselves?

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

Re: Process Duplication Within for Loop

<sdhbj8$t3j$2@dont-email.me>

copy mid

https://www.novabbs.com/aus+uk/article-flat.php?id=274&group=uk.comp.os.linux#274

copy link Newsgroups: alt.os.linux uk.comp.os.linux

Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: mar...@mydomain.invalid (Martin Gregorie)
Newsgroups: alt.os.linux,uk.comp.os.linux
Subject: Re: Process Duplication Within for Loop
Date: Sat, 24 Jul 2021 15:28:40 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 66
Message-ID: <sdhbj8$t3j$2@dont-email.me>
References: <sdh57p$1une$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Date: Sat, 24 Jul 2021 15:28:40 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="ba015f1880194200f9533cc2b9ab10c0";
logging-data="29811"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+493J+lvB/zv0FObG8S3g0dIo0suaxdwI="
User-Agent: Pan/0.146 (Hic habitat felicitas; 8107378
git@gitlab.gnome.org:GNOME/pan.git)
Cancel-Lock: sha1:M0BcRTosI2IZTekKKK1tUN+UUMY=

by: Martin Gregorie - Sat, 24 Jul 2021 15:28 UTC

On Sat, 24 Jul 2021 14:40:08 +0100, Java Jive wrote:

> I have the following piece of code creating an rsync process launched in
> the early hours of every morning to sync a NAS from another upstream. I
> have only included the business part of the file, the beginning of it is
> to do with setting up the variable values used in these lines. As you
> will note, there are two debugging statements trapping the output of ps
> ...
>
> # Debug Line 1:
> ps | grep -v grep | grep tutsync.sh > $HD/rsync.log echo "Syncing the
> following shares with rsync: ${SHARES}" >> $HD/rsync.log for SHARE in
> ${SHARES}
> do
> # Debug Line 2:
> ps | grep -v grep | grep tutsync.sh >> $HD/rsync.log
> echo "nice -n ${NICE} rsync ${OPTIONS}
> ${SOURCE}/${SHARE}/ ${BU}/${SHARE}/"
> nice -n ${NICE} rsync ${OPTIONS} ${SOURCE}/${SHARE}/
> ${BU}/${SHARE}/
> done | grep -v '/$' >> $HD/rsync.log 2>&1
>
>
> ... because, when run, the process appears to duplicate itself within
> the for loop - not only does the tutsync.sh process itself get
> duplicated, but also the rsync process it launches as well, as can be
> proven by launching with a trailing & and running a similar command from
> the shell:
>
> ~ # cat $HD/rsync.log 32453 root 2720 S /bin/sh
> /opt/share/bin/tutsync.sh Syncing the following shares with rsync: A
> List Of Shares 32453 root 2720 S /bin/sh
> /opt/share/bin/tutsync.sh 32458 root 2720 S /bin/sh
> /opt/share/bin/tutsync.sh receiving incremental file list etc
>
> ~ # pf rsync
> 9680 root 1972 S /opt/bin/rsync --ipv4 --daemon
> --config=/etc/rsyncd.conf 32467 root 2908 R N rsync -Drltugpv
> --exclude=*.flv --exclude=*.partial.* --exclude=*.part
> --exclude=**/.streams 32468 root 49772 S N rsync -Drltugpv
> --exclude=*.flv --exclude=*.partial.* --exclude=*.part
> --exclude=**/.streams
>
> What is going on, why are the processes duplicating themselves?

FWIW my weekly backup process is run from the host with the backup disk
mounted on it and uses rsync to back up itself as well as other hosts on
my LAN. Backups are run from a menu with each backup being triggered
manually. I've watched this run using 'top' and seen duplicate copies of
rsync get spawned from time to time: the duplicated rsync processes are
not present in all backup runs and and can come and go during a backup.

I have no idea why the duplicate processes are created, but the backups
always complete without errors reported and seem to not duplicate,
corrupt or omit any files, so I just accept that this is what rsync does.

For an explanation, look at this (which I've just found):
https://serverfault.com/questions/547165/why-does-rsync-spawn-multiple-
processes-for-me#547194

--
Martin | martin at
Gregorie | gregorie dot org

Re: Process Duplication Within for Loop

<87tukjsm8u.fsf@LkoBDZeT.terraraq.uk>

copy mid

https://www.novabbs.com/aus+uk/article-flat.php?id=275&group=uk.comp.os.linux#275

copy link Newsgroups: alt.os.linux uk.comp.os.linux

Path: i2pn2.org!i2pn.org!aioe.org!nntp.terraraq.uk!.POSTED.nntp.terraraq.uk!not-for-mail
From: inva...@invalid.invalid (Richard Kettlewell)
Newsgroups: alt.os.linux,uk.comp.os.linux
Subject: Re: Process Duplication Within for Loop
Date: Sat, 24 Jul 2021 17:35:45 +0100
Organization: terraraq NNTP server
Message-ID: <87tukjsm8u.fsf@LkoBDZeT.terraraq.uk>
References: <sdh57p$1une$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain
Injection-Info: mantic.terraraq.uk; posting-host="nntp.terraraq.uk:2a00:1098:0:86:1000:3f:0:2";
logging-data="17296"; mail-complaints-to="usenet@mantic.terraraq.uk"
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux)
Cancel-Lock: sha1:CgVKya8pGltjdBuVVA6NpwqAlu8=
X-Face: h[Hh-7npe<<b4/eW[]sat,I3O`t8A`(ej.H!F4\8|;ih)`7{@:A~/j1}gTt4e7-n*F?.Rl^
F<\{jehn7.KrO{!7=:(@J~]<.[{>v9!1<qZY,{EJxg6?Er4Y7Ng2\Ft>Z&W?r\c.!4DXH5PWpga"ha
+r0NzP?vnz:e/knOY)PI-
X-Boydie: NO

by: Richard Kettlewell - Sat, 24 Jul 2021 16:35 UTC

Java Jive <java@evij.com.invalid> writes:
> I have the following piece of code creating an rsync process launched
> in the early hours of every morning to sync a NAS from another
> upstream. I have only included the business part of the file, the
> beginning of it is to do with setting up the variable values used in
> these lines. As you will note, there are two debugging statements
> trapping the output of ps ...
>
> # Debug Line 1:
> ps | grep -v grep | grep tutsync.sh > $HD/rsync.log
> echo "Syncing the following shares with rsync: ${SHARES}" >> $HD/rsync.log
> for SHARE in ${SHARES}
> do
> # Debug Line 2:
> ps | grep -v grep | grep tutsync.sh >> $HD/rsync.log
> echo "nice -n ${NICE} rsync ${OPTIONS}
> ${SOURCE}/${SHARE}/ ${BU}/${SHARE}/"
> nice -n ${NICE} rsync ${OPTIONS} ${SOURCE}/${SHARE}/
> ${BU}/${SHARE}/
> done | grep -v '/$' >> $HD/rsync.log 2>&1
>
>
> ... because, when run, the process appears to duplicate itself within
> the for loop - not only does the tutsync.sh process itself get
> duplicated, but also the rsync process it launches as well, as can be
> proven by launching with a trailing & and running a similar command
> from the shell:
>
> ~ # cat $HD/rsync.log
> 32453 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
> Syncing the following shares with rsync: A List Of Shares
> 32453 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
> 32458 root 2720 S /bin/sh /opt/share/bin/tutsync.sh
> receiving incremental file list
> etc
>
> ~ # pf rsync
> 9680 root 1972 S /opt/bin/rsync --ipv4 --daemon
> --config=/etc/rsyncd.conf
> 32467 root 2908 R N rsync -Drltugpv --exclude=*.flv
> --exclude=*.partial.* --exclude=*.part --exclude=**/.streams
> 32468 root 49772 S N rsync -Drltugpv --exclude=*.flv
> --exclude=*.partial.* --exclude=*.part --exclude=**/.streams
>
> What is going on, why are the processes duplicating themselves?

The first half of the main pipeline (for ... done) runs in a separate
process.

rsync uses multiple processes as part of its normal operation.

--
https://www.greenend.org.uk/rjk/

Re: Process Duplication Within for Loop

<sdi091$1qm1$1@gioia.aioe.org>

copy mid

https://www.novabbs.com/aus+uk/article-flat.php?id=279&group=uk.comp.os.linux#279

copy link Newsgroups: alt.os.linux uk.comp.os.linux

Path: i2pn2.org!i2pn.org!aioe.org!st4z+icvTwfUsqGJRr80Xw.user.46.165.242.75.POSTED!not-for-mail
From: jav...@evij.com.invalid (Java Jive)
Newsgroups: alt.os.linux,uk.comp.os.linux
Subject: Re: Process Duplication Within for Loop
Date: Sat, 24 Jul 2021 22:21:36 +0100
Organization: Aioe.org NNTP Server
Message-ID: <sdi091$1qm1$1@gioia.aioe.org>
References: <sdh57p$1une$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: gioia.aioe.org; logging-data="60097"; posting-host="st4z+icvTwfUsqGJRr80Xw.user.gioia.aioe.org"; mail-complaints-to="abuse@aioe.org";
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:68.0) Gecko/20100101
Thunderbird/68.4.2
X-Notice: Filtered by postfilter v. 0.9.2
Content-Language: en-GB

by: Java Jive - Sat, 24 Jul 2021 21:21 UTC

On 24/07/2021 14:40, Java Jive wrote:
> I have the following piece of code creating an rsync process launched in
> the early hours of every morning to sync a NAS from another upstream. I
> have only included the business part of the file, the beginning of it is
> to do with setting up the variable values used in these lines. As you
> will note, there are two debugging statements trapping the output of ps ...
>
> # Debug Line 1:
> ps | grep -v grep | grep tutsync.sh > $HD/rsync.log
> echo "Syncing the following shares with rsync: ${SHARES}" >> $HD/rsync.log
> for SHARE in ${SHARES}
>         do
>         # Debug Line 2:
>                 ps | grep -v grep | grep tutsync.sh >> $HD/rsync.log
>                 echo "nice -n ${NICE} rsync ${OPTIONS}
> ${SOURCE}/${SHARE}/ ${BU}/${SHARE}/"
>                 nice -n ${NICE} rsync ${OPTIONS} ${SOURCE}/${SHARE}/
> ${BU}/${SHARE}/
>         done | grep -v '/$' >> $HD/rsync.log 2>&1
>
>
> ... because, when run, the process appears to duplicate itself within
> the for loop - not only does the tutsync.sh process itself get
> duplicated, but also the rsync process it launches as well, as can be
> proven by launching with a trailing & and running a similar command from
> the shell:
>
> ~ # cat $HD/rsync.log
> 32453 root      2720 S    /bin/sh /opt/share/bin/tutsync.sh
> Syncing the following shares with rsync: A List Of Shares
> 32453 root      2720 S    /bin/sh /opt/share/bin/tutsync.sh
> 32458 root      2720 S    /bin/sh /opt/share/bin/tutsync.sh
> receiving incremental file list
> etc
>
> ~ # pf rsync
> 9680 root      1972 S    /opt/bin/rsync --ipv4 --daemon
> --config=/etc/rsyncd.conf
> 32467 root      2908 R N rsync -Drltugpv --exclude=*.flv
> --exclude=*.partial.* --exclude=*.part --exclude=**/.streams
> 32468 root     49772 S N rsync -Drltugpv --exclude=*.flv
> --exclude=*.partial.* --exclude=*.part --exclude=**/.streams
>
> What is going on, why are the processes duplicating themselves?

Thanks for both the replies. It seems there's nothing to worry about,
and the script seems to have been working aright anyhow.

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

Re: Process Duplication Within for Loop

<im4nvoFn7m3U1@mid.individual.net>

copy mid

https://www.novabbs.com/aus+uk/article-flat.php?id=281&group=uk.comp.os.linux#281

copy link Newsgroups: alt.os.linux uk.comp.os.linux

Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!3.eu.feeder.erje.net!feeder.erje.net!news2.arglkargh.de!news.karotte.org!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: use...@example.net (J.O. Aho)
Newsgroups: alt.os.linux,uk.comp.os.linux
Subject: Re: Process Duplication Within for Loop
Date: Sun, 25 Jul 2021 11:08:07 +0200
Lines: 61
Message-ID: <im4nvoFn7m3U1@mid.individual.net>
References: <sdh57p$1une$1@gioia.aioe.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Trace: individual.net UwmPHmqGh4J6b6XsEYxOkA1i2rSyGpFLG37lFIAMZpy+oMJ2RD
Cancel-Lock: sha1:AjN3AgeUWNq+N2FAFqn9tXtZG5s=
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
Thunderbird/78.12.0
In-Reply-To: <sdh57p$1une$1@gioia.aioe.org>
Content-Language: en-US-large

by: J.O. Aho - Sun, 25 Jul 2021 09:08 UTC

On 24/07/2021 15.40, Java Jive wrote:
> I have the following piece of code creating an rsync process launched in
> the early hours of every morning to sync a NAS from another upstream. I
> have only included the business part of the file, the beginning of it is
> to do with setting up the variable values used in these lines. As you
> will note, there are two debugging statements trapping the output of ps ...
>
> # Debug Line 1:
> ps | grep -v grep | grep tutsync.sh > $HD/rsync.log
> echo "Syncing the following shares with rsync: ${SHARES}" >> $HD/rsync.log
> for SHARE in ${SHARES}
>         do
>         # Debug Line 2:
>                 ps | grep -v grep | grep tutsync.sh >> $HD/rsync.log
>                 echo "nice -n ${NICE} rsync ${OPTIONS}
> ${SOURCE}/${SHARE}/ ${BU}/${SHARE}/"
>                 nice -n ${NICE} rsync ${OPTIONS} ${SOURCE}/${SHARE}/
> ${BU}/${SHARE}/
>         done | grep -v '/$' >> $HD/rsync.log 2>&1
>
>
> ... because, when run, the process appears to duplicate itself within
> the for loop - not only does the tutsync.sh process itself get
> duplicated, but also the rsync process it launches as well, as can be
> proven by launching with a trailing & and running a similar command from
> the shell:
>
> ~ # cat $HD/rsync.log
> 32453 root      2720 S    /bin/sh /opt/share/bin/tutsync.sh
> Syncing the following shares with rsync: A List Of Shares
> 32453 root      2720 S    /bin/sh /opt/share/bin/tutsync.sh
> 32458 root      2720 S    /bin/sh /opt/share/bin/tutsync.sh
> receiving incremental file list
> etc
>
> ~ # pf rsync
> 9680 root      1972 S    /opt/bin/rsync --ipv4 --daemon
> --config=/etc/rsyncd.conf
> 32467 root      2908 R N rsync -Drltugpv --exclude=*.flv
> --exclude=*.partial.* --exclude=*.part --exclude=**/.streams
> 32468 root     49772 S N rsync -Drltugpv --exclude=*.flv
> --exclude=*.partial.* --exclude=*.part --exclude=**/.streams
>
> What is going on, why are the processes duplicating themselves?
>

Running ps with the f option will give you a clearer picture of process
relations

5467 ? Ss 0:02 sshd: /usr/sbin/sshd -f /etc/ssh/sshd_config
13958 ? Ss 0:00 \_ sshd: user1 [priv]
13960 ? S 0:01 \_ sshd: user1@pts/4
13962 pts/4 Ss 0:00 \_ -bash
18569 pts/4 S 0:00 \_ su -
18571 pts/4 S 0:00 \_ -su
20700 pts/4 R+ 0:00 \_ ps axf

//Aho

Re: Process Duplication Within for Loop

<sdjf0t$u04$1@gioia.aioe.org>

copy mid

https://www.novabbs.com/aus+uk/article-flat.php?id=282&group=uk.comp.os.linux#282

copy link Newsgroups: alt.os.linux uk.comp.os.linux

Path: i2pn2.org!i2pn.org!usenet.goja.nl.eu.org!aioe.org!st4z+icvTwfUsqGJRr80Xw.user.46.165.242.75.POSTED!not-for-mail
From: jav...@evij.com.invalid (Java Jive)
Newsgroups: alt.os.linux,uk.comp.os.linux
Subject: Re: Process Duplication Within for Loop
Date: Sun, 25 Jul 2021 11:39:24 +0100
Organization: Aioe.org NNTP Server
Message-ID: <sdjf0t$u04$1@gioia.aioe.org>
References: <sdh57p$1une$1@gioia.aioe.org> <im4nvoFn7m3U1@mid.individual.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Info: gioia.aioe.org; logging-data="30724"; posting-host="st4z+icvTwfUsqGJRr80Xw.user.gioia.aioe.org"; mail-complaints-to="abuse@aioe.org";
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:68.0) Gecko/20100101
Thunderbird/68.4.2
Content-Language: en-GB
X-Notice: Filtered by postfilter v. 0.9.2

by: Java Jive - Sun, 25 Jul 2021 10:39 UTC

On 25/07/2021 10:08, J.O. Aho wrote:
>
> Running ps with the f option will give you a clearer picture of process
> relations
>
> 5467 ?        Ss     0:02 sshd: /usr/sbin/sshd -f /etc/ssh/sshd_config
> 13958 ?        Ss     0:00 \_ sshd: user1 [priv]
> 13960 ?        S      0:01      \_ sshd: user1@pts/4
> 13962 pts/4    Ss     0:00          \_ -bash
> 18569 pts/4    S      0:00              \_ su -
> 18571 pts/4    S      0:00                  \_ -su
> 20700 pts/4    R+     0:00                      \_ ps axf

That's a useful general tip which I didn't know about before, thanks.
However, in this particular instance, this is an embedded device, and
the busybox version of ps doesn't support that option, more's the pity.

Fake news kills!

I may be contacted via the contact address given on my website:
www.macfh.co.uk

I stick my neck out for nobody. -- Humphrey Bogart, "Casablanca"

aus+uk / uk.comp.os.linux / Process Duplication Within for Loop

Subject	Author
Process Duplication Within for Loop	Java Jive
Re: Process Duplication Within for Loop	Martin Gregorie
Re: Process Duplication Within for Loop	Richard Kettlewell
Re: Process Duplication Within for Loop	Java Jive
Re: Process Duplication Within for Loop	J.O. Aho
Re: Process Duplication Within for Loop	Java Jive