novaBBS - comp.lang.python - Re: Execute in a multiprocessing child dynamic code loaded by the parent process

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

<mailman.214.1646590353.2329.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=17403&group=comp.lang.python#17403

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: die...@handshake.de (Dieter Maurer)
Newsgroups: comp.lang.python
Subject: Re: Execute in a multiprocessing child dynamic code loaded by the
parent process
Date: Sun, 6 Mar 2022 19:12:25 +0100
Lines: 38
Message-ID: <mailman.214.1646590353.2329.python-list@python.org>
References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
<25124.63881.205740.249595@ixdm.fritz.box>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Trace: news.uni-berlin.de r72HZscj7FXZwwqEZ3LEdwCK4mgPgB1up2jQipbq/cYw==
Return-Path: <dieter@handshake.de>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.006
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'parallel': 0.05;
'modules': 0.07; 'parent': 0.07; '(python': 0.09; 'cc:addr:python-
list': 0.09; 'child': 0.09; 'everyone.': 0.09; 'fails': 0.09;
'macos': 0.09; 'macos,': 0.09; '"start': 0.16; '(instead': 0.16;
'initialize': 0.16; 'pickle': 0.16; 'skip:> 10': 0.16;
'subject:child': 0.16; 'subject:code': 0.16; 'subject:dynamic':
0.16; 'subject:parent': 0.16; 'subject:skip:m 10': 0.16; 'python':
0.16; 'cc:addr:python.org': 0.20; 'code': 0.23; 'run': 0.23;
'received:de': 0.23; 'cannot': 0.25; 'cc:2**0': 0.25; "wasn't":
0.26; 'it,': 0.29; 'module': 0.31; 'program': 0.31; 'question':
0.32; 'but': 0.32; 'server': 0.33; 'skip:" 20': 0.34; 'header:In-
Reply-To:1': 0.34; 'target': 0.36; 'really': 0.37; 'using': 0.37;
'enough': 0.39; 'ago': 0.39; 'use': 0.39; 'wrote': 0.39; 'martin':
0.40; 'wants': 0.40; 'try': 0.40; 'method': 0.61; 'load': 0.62;
'days': 0.62; 'skip:m 20': 0.63; 'independent': 0.65; 'process.':
0.65; 'well': 0.65; 'named': 0.65; 'worked': 0.67;
'header:Received:6': 0.67; 'piece': 0.69; 'implemented': 0.76;
'paola': 0.84; 'received:88': 0.84; 'subject: \n ': 0.84
In-Reply-To: <20220306124208.orqbcezhqqeysdpe@gmail.com>
X-Mailer: VM 8.0.12-devo-585 under 21.4 (patch 24) "Standard C" XEmacs Lucid
(x86_64-linux-gnu)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <25124.63881.205740.249595@ixdm.fritz.box>
X-Mailman-Original-References: <20220306124208.orqbcezhqqeysdpe@gmail.com>

by: Dieter Maurer - Sun, 6 Mar 2022 18:12 UTC

Martin Di Paola wrote at 2022-3-6 12:42 +0000:
>Hi everyone. I implemented time ago a small plugin engine to load code
>dynamically.
>
>So far it worked well but a few days ago an user told me that he wasn't
>able to run in parallel a piece of code in MacOS.
>
>He was using multiprocessing.Process to run the code and in MacOS, the
>default start method for such process is using "spawn". My understanding
>is that Python spawns an independent Python server (the child) which
>receives what to execute (the target function) from the parent process.
>
>In pseudo code this would be like:
>
>modules = loader() # load the plugins (Python modules at the end)
>objs = init(modules) # initialize the plugins
>
># One of the plugins wants to execute part of its code in parallel
># In MacOS this fails
>ch = multiprocessing.Process(target=objs[0].sayhi)
>ch.start()
>
>The code fails with "ModuleNotFoundError: No module named 'foo'" (where
>'foo' is the name of the loaded plugin).
>
>This is because the parent program sends to the serve (the child) what
>needs to execute (objs[0].sayhi) using pickle as the serialization
>mechanism.
>
>Because Python does not really serialize code but only enough
>information to reload it, the serialization of "objs[0].sayhi" just
>points to its module, "foo".
>
>Module which it cannot be imported by the child process.
>
>So the question is, what would be the alternatives and workarounds?

Try to use `fork` as "start method" (instead of "spawn").

Subject	Author
Re: Execute in a multiprocessing child dynamic code loaded by the	Dieter Maurer

"The medium is the message." -- Marshall McLuhan

devel / comp.lang.python / Re: Execute in a multiprocessing child dynamic code loaded by the parent process