Message-ID:

Just go with the flow control, roll with the crunches, and, when you get a prompt, type like hell.

devel / comp.lang.python / Execute in a multiprocessing child dynamic code loaded by the parent process

Execute in a multiprocessing child dynamic code loaded by the parent process

<mailman.204.1646570536.2329.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=17393&group=comp.lang.python#17393

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: martinp....@gmail.com (Martin Di Paola)
Newsgroups: comp.lang.python
Subject: Execute in a multiprocessing child dynamic code loaded by the parent
process
Date: Sun, 6 Mar 2022 12:42:08 +0000
Lines: 66
Message-ID: <mailman.204.1646570536.2329.python-list@python.org>
References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
X-Trace: news.uni-berlin.de hvevjnCVVb2SdvuLe/uKPwaaQxxNVSPniQ0Aex6TSyNQ==
Return-Path: <martinp.dipaola@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=BCVGMpuq;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.002
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'url-ip:140.82/16': 0.03;
'def': 0.04; 'parallel': 0.05; 'modules': 0.07; 'parent': 0.07;
'(python': 0.09; 'child': 0.09; 'everyone.': 0.09; 'fails': 0.09;
'macos': 0.09; 'macos,': 0.09; 'url:github': 0.14; 'url-ip:140/8':
0.15; 'accessed': 0.16; 'hack:': 0.16; 'initialize': 0.16;
'mechanism.': 0.16; 'pickle': 0.16; 'received:209.85.210.45':
0.16; 'received:mail-ot1-f45.google.com': 0.16; 'subject:child':
0.16; 'subject:code': 0.16; 'subject:dynamic': 0.16;
'subject:parent': 0.16; 'subject:skip:m 10': 0.16; 'python': 0.16;
'message-id:@gmail.com': 0.18; 'it?': 0.19; 'to:addr:python-list':
0.20; 'issue': 0.21; 'code': 0.23; 'run': 0.23; 'thanks!': 0.24;
'to:name:python-list@python.org': 0.24; 'cannot': 0.25; "wasn't":
0.26; 'function': 0.27; 'received:209.85.210': 0.29; 'it,': 0.29;
'default': 0.31; 'module': 0.31; 'program': 0.31; 'question':
0.32; 'but': 0.32; 'server': 0.33; 'able': 0.34; 'skip:" 20':
0.34; 'received:google.com': 0.34; 'following': 0.35;
'from:addr:gmail.com': 0.35; 'target': 0.36; 'really': 0.37;
'using': 0.37; 'received:209.85': 0.37; 'way': 0.38;
'received:209': 0.39; 'enough': 0.39; 'ago': 0.39; 'use': 0.39;
'wants': 0.40; 'method': 0.61; 'load': 0.62; 'days': 0.62; 'skip:m
20': 0.63; 'skip:r 20': 0.64; 'came': 0.65; 'independent': 0.65;
'process.': 0.65; 'well': 0.65; 'named': 0.65; 'skip:t 20': 0.66;
'worked': 0.67; 'piece': 0.69; 'implemented': 0.76; 'points':
0.84; 'executing': 0.84; 'martin.': 0.84; 'subject: \n ': 0.84;
'receives': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=date:from:to:subject:message-id:mail-followup-to:'user-agent
:mime-version:content-disposition;
bh=KlzIOLDgNplxjhr3zvtUrNta0frh160u+AGVe2I6myc=;
b=BCVGMpuqzyWGpb07XMi/R2JNEN3kbo+H7tFbC56E4s5G036+V8UDmwA+fu2RxvSh/b
2b0twOKkR+aK5SOs+cc+GCAQkz0OAw++aHEXinatKIZ5dRGI9dlpthmpcsQxmzi4ltIa
CqhAOWUzxQgZgraC2y9BGOjh1w5iZiE2tsu0dAeyzsgZS2TkJ2kFoNRjhoLvbVHDewnD
4zghzYmSMC/wWCeEI6jaW9Wip4I8+hGt74JN86TIlCPoayl/sOYlX4LCRZsWkRNri22E
UzZGOGgvBsez5ESbrXBkoYnx0WQBcgRO3qwjn46TneaacOGgvQPca2DBJcU0WVAQ0umZ
/mZQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:date:from:to:subject:message-id:mail-followup-to
:'user-agent:mime-version:content-disposition;
bh=KlzIOLDgNplxjhr3zvtUrNta0frh160u+AGVe2I6myc=;
b=LyTBOPQEpahjJueKyz2kpl2bETe3fde9F9oclBCdhD3mG2v1YiTjisrqh1Vd/1OTBb
ejjoQjta83kyCaARoYWmhLoPCJ4E3Elxv4HIEQOBDgxC7YGmyLyBo7+9H0+gnkhPS2bF
WvQkYDcr6esxSMDQJGUHSeH5Xk53M2VKe7BlSpuQKqcB7Q/OZywo07AI1woOBP7dmSxb
US5AB4ZeqraLpkm6nC5qKK1lFnYAoLCmU9ZZJMx2KOzaNGb9pesqhGCkzg479Onq65Lh
jZUZcMP7QFJeMOMoE6r1jh3TLVvyX1fLQ3muX4g/IrA5WnOMiFz3WZRcjmizSFeH483M
VRzw==
X-Gm-Message-State: AOAM530TH62M/tP+B6fPwwrK4MoDgTHGHQQCsWSs3gkVHBXkogDZMXOF
0Px+OF9qx+6V5+MmhX/czczOJ0ErAWw=
X-Google-Smtp-Source: ABdhPJyZKtSviz4vRaXG5c9OIgic9c9k/M84nvJA3ysdXRPgBH2tsgJPZewlX6HiZRU7U19YAyQfgA==
X-Received: by 2002:a9d:135:0:b0:5b2:2c31:8e2a with SMTP id
50-20020a9d0135000000b005b22c318e2amr1324314otu.233.1646570533757;
Sun, 06 Mar 2022 04:42:13 -0800 (PST)
Mail-Followup-To: "python-list@python.org" <python-list@python.org>
'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
Thunderbird/52.5.2 Lightning/5.4.5.2'
Content-Disposition: inline
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <20220306124208.orqbcezhqqeysdpe@gmail.com>

by: Martin Di Paola - Sun, 6 Mar 2022 12:42 UTC

Hi everyone. I implemented time ago a small plugin engine to load code
dynamically.

So far it worked well but a few days ago an user told me that he wasn't
able to run in parallel a piece of code in MacOS.

He was using multiprocessing.Process to run the code and in MacOS, the
default start method for such process is using "spawn". My understanding
is that Python spawns an independent Python server (the child) which
receives what to execute (the target function) from the parent process.

In pseudo code this would be like:

modules = loader() # load the plugins (Python modules at the end)
objs = init(modules) # initialize the plugins

# One of the plugins wants to execute part of its code in parallel
# In MacOS this fails
ch = multiprocessing.Process(target=objs[0].sayhi)
ch.start()

The code fails with "ModuleNotFoundError: No module named 'foo'" (where
'foo' is the name of the loaded plugin).

This is because the parent program sends to the serve (the child) what
needs to execute (objs[0].sayhi) using pickle as the serialization
mechanism.

Because Python does not really serialize code but only enough
information to reload it, the serialization of "objs[0].sayhi" just
points to its module, "foo".

Module which it cannot be imported by the child process.

So the question is, what would be the alternatives and workarounds?

I came with a hack: use a trampoline() function to load the plugins
in the child before executing the target function.

In pseudo code it is:

modules = loader() # load the plugins (Python modules at the end)
objs = init(modules) # initialize the plugins

def trampoline(target_str):
loader() # load the plugins now that we are in the child process

# deserialize the target and call it
target = reduction.loads(target_str)
target()

# Serialize the real target function, but call in the child
# trampoline(). Because it can be accessed by the child it will
# not fail
target_str = reduction.dumps(objs[0].sayhi)
ch = multiprocessing.Process(target=trampoline, args=(target_str,))
ch.start()

The hack works but is this the correct way to do it?

The following gist has the minimal example code that triggers the issue
and its workaround:
https://gist.github.com/eldipa/d9b02875a13537e72fbce4cdb8e3f282

Thanks!
Martin.

Subject	Author
Execute in a multiprocessing child dynamic code loaded by the parent	Martin Di Paola