Message-ID:

But you'll notice Perl has a goto. -- Larry Wall in <199710211624.JAA17833@wall.org>

devel / comp.lang.python / Re: Python child process in while True loop blocks parent

Re: Python child process in while True loop blocks parent

<mailman.26.1638826392.15287.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=16321&group=comp.lang.python#16321

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: bar...@barrys-emacs.org (Barry Scott)
Newsgroups: comp.lang.python
Subject: Re: Python child process in while True loop blocks parent
Date: Mon, 6 Dec 2021 21:33:09 +0000
Lines: 120
Message-ID: <mailman.26.1638826392.15287.python-list@python.org>
References: <MphoqOf--3-2@tutanota.com>
<7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
<MpqihHm--3-2@tutanota.com>
<27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
<MqAitZV--3-2@tutanota.com>
<0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
<MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
<MqFiCj---3-2@tutanota.com>
<3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>
<MqGZH61--3-2@tutanota.com>
<6E104B3E-1FD8-476F-A560-87CF47B183CC@barrys-emacs.org>
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\))
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de fQABfX5dbB4EFDL09GWMDQEVjNoziuAR1PTiG5b0bZDg==
Return-Path: <barry@barrys-emacs.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.000
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'generated': 0.03;
'thread': 0.05; 'parent': 0.07; 'suggestion': 0.07; 'url-
ip:162.159/16': 0.07; 'cc:addr:python-list': 0.09; 'child': 0.09;
'flags': 0.09; 'fork': 0.09; 'from:addr:barry': 0.09; 'linux':
0.09; 'overhead': 0.09; 'processor': 0.09; 'received:217.70':
0.09; 'received:gandi.net': 0.09; 'received:mail.gandi.net': 0.09;
'threads': 0.09; 'ubuntu': 0.09; 'unix,': 0.09; 'url-
ip:162.159.152.4/32': 0.09; 'url-ip:162.159.152/24': 0.09; 'url-
ip:162.159.153.4/32': 0.09; 'url-ip:162.159.153/24': 0.09;
'url:medium': 0.09; 'url:post': 0.09; 'subject:Python': 0.12;
'url:mailman': 0.15; 'memory': 0.15; '"creative': 0.16; '(may':
0.16; '__/': 0.16; 'arguments': 0.16; 'barry': 0.16;
'cc:name:python list': 0.16; 'challenge!"': 0.16; 'confused':
0.16; 'confused.': 0.16; 'creation,': 0.16; 'frames': 0.16;
'from:addr:barrys-emacs.org': 0.16; 'from:name:barry scott': 0.16;
'hjp@hjp.at': 0.16; 'holzer': 0.16; 'instance': 0.16; 'message-id
:@barrys-emacs.org': 0.16; 'processes.': 0.16; 'reality.': 0.16;
'segment': 0.16; 'stross,': 0.16; 'subject:True': 0.16;
'subject:child': 0.16; 'subject:loop': 0.16; 'subject:parent':
0.16; 'subject:while': 0.16; 'thread.': 0.16; 'tuning': 0.16;
'url-ip:212.17.106.137/32': 0.16; 'url-ip:212.17.106/24': 0.16;
'url-ip:212.17/16': 0.16; 'url:hjp': 0.16; 'x-mailer:apple mail
(2.3654.120.0.1.13)': 0.16; '|_|_)': 0.16; 'wrote:': 0.16;
'python': 0.16; 'code.': 0.17; "can't": 0.17; 'calls': 0.19;
'hardware': 0.19; 'cc:addr:python.org': 0.20; 'issue': 0.21;
'creates': 0.22; 'maybe': 0.22; 'run': 0.23; 'url-
ip:188.166.95.178/32': 0.25; 'url-ip:188.166.95/24': 0.25;
'url:listinfo': 0.25; 'cc:2**0': 0.25; 'programming': 0.25; 'url-
ip:188.166/16': 0.25; 'library': 0.26; 'creating': 0.27;
'function': 0.27; 'chris': 0.28; 'fact': 0.28; 'sense': 0.28;
'email addr:python.org>': 0.28; 'it,': 0.29; 'comment': 0.31;
'dec': 0.31; 'program,': 0.31; 'url-ip:188/8': 0.31; 'program':
0.31; 'think': 0.32; "doesn't": 0.32; 'guess': 0.32; 'logical':
0.32; 'python-list': 0.32; 'but': 0.32; "i'm": 0.33; 'same': 0.34;
'appreciated.': 0.34; 'core': 0.34; 'header:In-Reply-To:1': 0.34;
'running': 0.34; 'book': 0.35; '"the': 0.35; 'runs': 0.35;
'processes': 0.36; 'those': 0.36; 'main': 0.37; 'using': 0.37;
'way': 0.38; 'means': 0.38; 'two': 0.39; 'single': 0.39; 'this,':
0.39; 'program.': 0.40; 'want': 0.40; 'try': 0.40; 'url-
ip:104.21/16': 0.61; 'here': 0.62; 'ever': 0.63; 'between': 0.63;
'complete': 0.64; 'your': 0.64; 'process.': 0.65; 'url-ip:45/8':
0.65; 'digital': 0.65; 'let': 0.66; 'back': 0.67; 'forget': 0.67;
'received:217': 0.67; 'matter': 0.68; 'know.': 0.68; 'further':
0.69; 'ocean': 0.69; 'url-ip:212/8': 0.69; 'relationship': 0.71;
'performance': 0.71; 'man': 0.74; 'yes': 0.76; 'reply': 0.77;
'comments,': 0.84; 'distinction': 0.84; 'eliminate': 0.84; 'email
name:<python-list': 0.84; 'implies': 0.84; 'doing.': 0.91;
'said.': 0.91; 'consists': 0.93
In-Reply-To: <MqGZH61--3-2@tutanota.com>
X-Mailer: Apple Mail (2.3654.120.0.1.13)
X-Content-Filtered-By: Mailman/MimeDel 2.1.38
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.38
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <6E104B3E-1FD8-476F-A560-87CF47B183CC@barrys-emacs.org>
X-Mailman-Original-References: <MphoqOf--3-2@tutanota.com>
<7A3CE5EB-C53A-404D-BF32-77CF0243BF3D@barrys-emacs.org>
<MpqihHm--3-2@tutanota.com>
<27742189-51F0-49A8-ACA7-344EFCD2B1BD@barrys-emacs.org>
<MqAitZV--3-2@tutanota.com>
<0633A53B-A722-4CEE-8F20-E2D0AEBB8D01@barrys-emacs.org>
<MqC-YXO--B-2@tutanota.com> <20211206020831.ytkglawzaeoueyhh@hjp.at>
<MqFiCj---3-2@tutanota.com>
<3065988D-E2CF-4C36-B319-1DEC87B864E3@barrys-emacs.org>
<MqGZH61--3-2@tutanota.com>

by: Barry Scott - Mon, 6 Dec 2021 21:33 UTC

> On 6 Dec 2021, at 21:05, Jen Kris <jenkris@tutanota.com> wrote:
>
> Here is what I don't understand from what you said. "The child process is created with a single thread—the one that called fork()." To me that implies that the thread that called fork() is the same thread as the child process. I guess you're talking about the distinction between logical threads and physical threads.

The thread that called fork is cloned into a new thread in the new process.
It has a clone of the processor registers of the thread, stack memory segment of that thread,
which means that it has all the stack variables and stack frames from the parent process's thread.

> But the main issue is your suggestion that I should call fork-execv from the thread that runs the main C program, not from a separate physical pthread. That would certainly eliminate the overhead of creating a new pthread.

Forget about physical threads, they only matter when you are performance tuning your code.

> I am working now to finish this, and I will try your suggestion of calling fork-execv from the "main" thread. When I reply back next I can give you a complete picture of what I'm doing.
>
> Your comments, and those of Peter Holzer and Chris Angelico, are most appreciated.

Barry

>
>
> Dec 6, 2021, 10:37 by barry@barrys-emacs.org:
>
> On 6 Dec 2021, at 17:09, Jen Kris via Python-list <python-list@python.org> wrote:
>
> I can't find any support for your comment that "Fork creates a new
> process and therefore also a new thread." From the Linux man pages https://www.man7.org/linux/man-pages/man2/fork.2.html, "The child process is created with a single thread—the one that called fork()."
>
> You just quoted the evidence!
>
> All new processes on unix (may all OS) only ever have one thread when they start.
> The thread-id of the first thread is the same as the process-id and referred to as the main thread.
>
> I have a one-core one-thread instance at Digital Ocean available running Ubuntu 18.04. I can fork and create a new process on it, but it doesn't create a new thread because it doesn't have one available.
>
>
> By that logic it can only run one process...
>
> It has one hardware core that support one hardware thread.
> Linux can create as many software threads as it likes.
> You may also want to see "Forking vs Threading" (https://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel), "Fork vs Thread" (https://medium.com/obscure-system/fork-vs-thread-38e09ec099e2), and "Linux process and thread" (https://zliu.org/post/linux-process-and-thread) ("This means that to create a normal process fork() is used that further calls clone() with appropriate arguments while to create a thread or LWP, a function from pthread library calls clone() with relvant flags. So, the main difference is generated by using different flags that can be passed to clone() funciton(to be exact, it is a system call").
>
> You may be confused by the fact that threads are called light-weight processes.
>
> No Peter and I are not confused.
>
> Or maybe I'm confused :)
>
> Yes you are confused.
>
> If you have other information, please let me know. Thanks.
>
> Please get the book I recommended, or another that covers systems programming on unix, and have a read.
>
> Barry
>
> Jen
>
>
> Dec 5, 2021, 18:08 by hjp-python@hjp.at:
> On 2021-12-06 00:51:13 +0100, Jen Kris via Python-list wrote:
> The C program creates two threads (using pthreads), one for itself and
> one for the child process. On creation, the second pthread is pointed
> to a C program that calls fork-execv to run the Python program. That
> way Python runs on a separate thread.
>
> I think you have the relationship between processes and threads
> backwards. A process consists of one or more threads. Fork creates a new
> process and therefore also a new thread.
>
> hp
>
> --
> _ | Peter J. Holzer | Story must make more sense than reality.
> |_|_) | |
> | | | hjp@hjp.at | -- Charles Stross, "Creative writing
> __/ | http://www.hjp.at/ | challenge!"
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Subject	Author
Re: Python child process in while True loop blocks parent	Barry Scott