Message-ID:

You canna change the laws of physics, Captain; I've got to have thirty minutes!

devel / comp.lang.python / Re: Python child process in while True loop blocks parent

Re: Python child process in while True loop blocks parent

<mailman.50.1639001473.15287.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=16359&group=comp.lang.python#16359

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: cs...@cskk.id.au (Cameron Simpson)
Newsgroups: comp.lang.python
Subject: Re: Python child process in while True loop blocks parent
Date: Thu, 9 Dec 2021 08:44:52 +1100
Lines: 33
Message-ID: <mailman.50.1639001473.15287.python-list@python.org>
References: <MqQ0tig--3-2@tutanota.com>
<YbEnVHyeqrNr3sBx@cskk.homeip.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Trace: news.uni-berlin.de 5fkmuyE8CrcqlGdHERHXugvOhMwq1Mp5kPmT00zBY1LQ==
Return-Path: <cameron@cskk.id.au>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.006
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'this:': 0.03; 'files)':
0.07; 'hopefully': 0.07; '(python': 0.09; 'components:': 0.09;
'shift': 0.09; 'cheers,': 0.11; 'subject:Python': 0.12; '(eg':
0.16; 'cameron': 0.16; 'from:addr:cs': 0.16;
'from:addr:cskk.id.au': 0.16; 'from:name:cameron simpson': 0.16;
'message-id:@cskk.homeip.net': 0.16; 'newline': 0.16; 'pipe':
0.16; 'received:10.10': 0.16; 'simpson': 0.16; 'subject:True':
0.16; 'subject:child': 0.16; 'subject:loop': 0.16;
'subject:parent': 0.16; 'subject:while': 0.16; 'wrote:': 0.16;
'python': 0.16; 'to:addr:python-list': 0.20; 'issue': 0.21;
'written': 0.22; 'lines': 0.23; 'contents.': 0.26; 'normally':
0.26; 'object': 0.26; 'output': 0.28; 'header:User-Agent:1': 0.30;
'whole': 0.30; 'flow': 0.31; 'encountered': 0.32; 'unless': 0.32;
'but': 0.32; 'there': 0.33; 'header:In-Reply-To:1': 0.34; 'using':
0.37; 'file': 0.38; 'read': 0.38; 'two': 0.39; 'use': 0.39;
'block': 0.39; 'data.': 0.40; 'something': 0.40; 'should': 0.40;
'complete': 0.64; 'your': 0.64; 'let': 0.66; 'received:userid':
0.66; 'header:Received:6': 0.67; 'clear.': 0.69; 'receive': 0.71;
'waiting': 0.73; 'records': 0.75; '(c)': 0.81; 'points': 0.84;
'lines,': 0.84; 'record.': 0.84; 'records.': 0.84; 'strings':
0.84; 'aspects': 0.93
X-RG-Spam: Unknown
X-RazorGate-Vade: gggruggvucftvghtrhhoucdtuddrgedvuddrjeekgdduheduucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuuffpveftpgfvgffnuffvtfetpdfqfgfvnecuuegrihhlohhuthemucegtddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpeffhffvuffkgggtuggjfghfsehttdertddtredvnecuhfhrohhmpeevrghmvghrohhnucfuihhmphhsohhnuceotghssegtshhkkhdrihgurdgruheqnecuggftrfgrthhtvghrnhepleevfeeghfeggfehuddvgfehhfffjeegvdeuiefhgeelgeekfeffiefguddtveehnecukfhppedurddugeehrdeigedrkedunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehhvghlohepsghorhhgrdgtshhkkhdrhhhomhgvihhprdhnvghtpdhinhgvthepuddrudeghedrieegrdekuddpmhgrihhlfhhrohhmpegtrghmvghrohhnsegtshhkkhdrihgurdgruhdprhgtphhtthhopegtshestghskhhkrdhiugdrrghupdhrtghpthhtohepphihthhhohhnqdhlihhsthesphihthhhohhnrdhorhhg
X-RazorGate-Vade-Verdict: clean 0
X-RazorGate-Vade-Classification: clean
X-RG-VS-CLASS: clean
Mail-Followup-To: Python List <python-list@python.org>
Content-Disposition: inline
In-Reply-To: <MqQ0tig--3-2@tutanota.com>
User-Agent: Mutt/2.1.3 (2021-09-10)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.38
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <YbEnVHyeqrNr3sBx@cskk.homeip.net>
X-Mailman-Original-References: <MqQ0tig--3-2@tutanota.com>

by: Cameron Simpson - Wed, 8 Dec 2021 21:44 UTC

On 08Dec2021 18:11, Jen Kris <jenkris@tutanota.com> wrote:
>Python must terminate its write strings with \n, or read will block in
>C waiting for something that never comes.

There are two aspects to this:
- if upstream is rding "lines of text" then you need a newline to
terminate the lines
- you (probably) should flush the output pipe (Python to C) after the
newline

I see you're using file descriptors and os.write() to sent data. This is
unbuffered, so there is nothing to flush, so you have not encountered
the second point above.

But if you shift to using a Python file object for your output (eg
f=os.fdopen(pw)), which would let you use print() or any number of other
things which do things with Python files) your file object would have a
buffer and normally that would not be sent to the pipe unless it was
full.

So your deadlock issue has 2 components:
- you need to terminate your records for upstream (C) to see complete
records. Your records are lines, so you need a newline character.
- you need to ensure the whole record has been sent upstream (been
written to the pipe). If you use a buffered Python file object for
your output, you need to flush it at your synchronisation points or
upstream will not receive the buffer contents. That synchronisation
point for you is the end of the record.

Hopefully this makes the flow considerations more clear.

Cheers,
Cameron Simpson <cs@cskk.id.au>

Subject	Author
Re: Python child process in while True loop blocks parent	Cameron Simpson