novaBBS - comp.lang.python - Re: Execute in a multiprocessing child dynamic code loaded by the parent process

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

<mailman.216.1646598968.2329.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=17405&group=comp.lang.python#17405

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: martinp....@gmail.com (Martin Di Paola)
Newsgroups: comp.lang.python
Subject: Re: Execute in a multiprocessing child dynamic code loaded by the
parent process
Date: Sun, 6 Mar 2022 20:36:01 +0000
Lines: 44
Message-ID: <mailman.216.1646598968.2329.python-list@python.org>
References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
<CAPTjJmpJT4zf6nGgF5j-J89uPALwzs-Gf4aXWYAS65oYBfcE1Q@mail.gmail.com>
<20220306203601.eqv7geya3l7joxrj@gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
X-Trace: news.uni-berlin.de 6GcWdgzbwN6rdoyyxF2/xwU94Jm4P08iQ/BAvGqz0bmA==
Return-Path: <martinp.dipaola@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=FyuYx+sd;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.042
X-Spam-Evidence: '*H*': 0.92; '*S*': 0.00; 'fails': 0.09; 'fault':
0.09; 'lookup': 0.09; 'module.': 0.09; 'that.': 0.15; '"the"':
0.16; 'assuming': 0.16; 'idea.': 0.16; 'sounds': 0.16; 'static':
0.16; 'subject:child': 0.16; 'subject:code': 0.16;
'subject:dynamic': 0.16; 'subject:parent': 0.16; 'subject:skip:m
10': 0.16; 'developer': 0.16; 'message-id:@gmail.com': 0.18;
'to:addr:python-list': 0.20; 'code': 0.23; 'thanks!': 0.24;
'to:name:python-list@python.org': 0.24; 'function': 0.27; 'wrong':
0.28; 'received:209.85.210': 0.29; 'it,': 0.29; 'module': 0.31;
'but': 0.32; "i'm": 0.33; 'there': 0.33; 'particular': 0.33;
'header:In-Reply-To:1': 0.34; 'received:google.com': 0.34;
'from:addr:gmail.com': 0.35; 'target': 0.36; 'using': 0.37;
"it's": 0.37; 'received:209.85': 0.37; 'class': 0.37; 'way': 0.38;
'could': 0.38; 'received:209': 0.39; 'enough': 0.39; 'valid':
0.39; 'use': 0.39; 'hand': 0.40; 'martin': 0.40; 'wants': 0.40;
'something': 0.40; 'method': 0.61; "there's": 0.61; 'load': 0.62;
'skip:m 20': 0.63; 'required': 0.65; 'smart': 0.67; 'decision':
0.68; 'choice': 0.76; 'need,': 0.76; 'practical': 0.84; 'catch':
0.84; 'subject: \n ': 0.84; 'received:168': 0.93
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=date:from:to:subject:message-id:mail-followup-to:'user-agent
:references:mime-version:content-disposition:in-reply-to;
bh=JhukLIFJGbRkT2b+bjdOxreglZ1QGFxkqHngHjuUisc=;
b=FyuYx+sdMP9hGCU+UbxDFWkDpBjJ+DjUxx9WuXX9J9+TO7EVNtf0uMWaUvDfjALBV0
PoAoauvQFgZJP16cT2FSaz6qfzyhsiwC3EWQWfynbF5RXmaMsuEQamie44kQonnaDu5B
8RTcXy2ctAVdUjGKvI68cctcoOyQXu4m1j9vyi6q4ZeFyC0XyxddmgD+zyM7XREUk4nw
uUQz+t8cqWMgnLd7yGJMjETMGorRJsBch0LEAjWn9sMzD+SgELG0vLjg1+FoziOIDbzl
+3gU4y3ddBjRXXhdEJ8hM5EesVDwYAHUaYs61DXZXwIhxNPdhM2QMeWgIIyR6DoXvj4m
pHgg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:date:from:to:subject:message-id:mail-followup-to
:'user-agent:references:mime-version:content-disposition:in-reply-to;
bh=JhukLIFJGbRkT2b+bjdOxreglZ1QGFxkqHngHjuUisc=;
b=BuynPxIywRrltujXdBQqk79wRQGTvz3SuVswaTJHtT4OqDUm1suiiS3AlhLl+WfX3G
eOA4sDOuk60iuKSUCt2A0cyCgbFl2qNXzwESKeoxGwfKj9ZGLC/oQRCxubfbs0RPWNUX
64dIX6LB1EmEflK4Y5Tc8b4cGLdV03bvWFe9s9tB3csbHYl3JAi8Yg5VLJhQVumQfhAo
/NbMHrb6um84DuVywvapmEdps3cefrjw1ujnZQ3oXr48xydKYUf5rvi25OXD/T5o9eMY
7DNLkDZNhC0jdbcVgM8cLuLk645LLg6DPnb18Ay2i32vl7s/q6PzNt/OdTOLhsXBMlmX
7CYw==
X-Gm-Message-State: AOAM530GSMCQF606fLFmNFOKdFe2oTZZjlPkTScFBsbPhENv1iMx61RA
qDQTE+Ar83sOzBGQKiRVCNHoE3xvn9c=
X-Google-Smtp-Source: ABdhPJwvLzTMIq9ocSvKRHIW7zJtyzNqnV9ISvuIWa1+638vL6dpXVhkXn5iIvobOIfb09eCeqe2OA==
X-Received: by 2002:a9d:6a44:0:b0:5af:1886:86ec with SMTP id
h4-20020a9d6a44000000b005af188686ecmr4327691otn.333.1646598965316;
Sun, 06 Mar 2022 12:36:05 -0800 (PST)
Mail-Followup-To: "python-list@python.org" <python-list@python.org>
'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
Thunderbird/52.5.2 Lightning/5.4.5.2'
Content-Disposition: inline
In-Reply-To: <CAPTjJmpJT4zf6nGgF5j-J89uPALwzs-Gf4aXWYAS65oYBfcE1Q@mail.gmail.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <20220306203601.eqv7geya3l7joxrj@gmail.com>
X-Mailman-Original-References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
<CAPTjJmpJT4zf6nGgF5j-J89uPALwzs-Gf4aXWYAS65oYBfcE1Q@mail.gmail.com>

by: Martin Di Paola - Sun, 6 Mar 2022 20:36 UTC

>
>The way you've described it, it's a hack. Allow me to slightly redescribe it.
>
>modules = loader()
>objs = init(modules)
>
>def invoke(mod, func):
> # I'm assuming that the loader is smart enough to not load
> # a module that's already loaded. Alternatively, load just the
> # module you need, if that's a possibility.
> loader()
> target = getattr(modules[mod], func)
> target()
>
>ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi"))
>ch.start()
>

Yeup, that would be my first choice but the catch is that "sayhi" may
not be a function of the given module. It could be a static method of
some class or any other callable.

And doing the lookup by hand sounds complex.

The thing is that the use of multiprocessing is not something required by me
(by my plugin-engine), it was a decision of the developer of a particular
plugin so I don't have any control on that.

Using multiprocessing.reduction was a practical decision: if the user
wants to call something non-pickleable, it is not my fault, it is
multiprocessing's fault.

It *would* be my fault if multiprocessing.Process fails only because I'm
loading the code dynamically.

> [...] I won't say "the" correct way, as there are other valid
>ways, but there's certainly nothing wrong with this idea.

Do you have some in mind? Or may be a project that I could read?

Thanks!
Martin

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

<j8kr3rFba1nU1@mid.individual.net>

copy mid

https://www.novabbs.com/devel/article-flat.php?id=17410&group=comp.lang.python#17410

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!individual.net!not-for-mail
From: greg.ew...@canterbury.ac.nz (Greg Ewing)
Newsgroups: comp.lang.python
Subject: Re: Execute in a multiprocessing child dynamic code loaded by the
parent process
Date: Mon, 7 Mar 2022 11:32:27 +1300
Lines: 12
Message-ID: <j8kr3rFba1nU1@mid.individual.net>
References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
<CAPTjJmpJT4zf6nGgF5j-J89uPALwzs-Gf4aXWYAS65oYBfcE1Q@mail.gmail.com>
<20220306203601.eqv7geya3l7joxrj@gmail.com>
<mailman.216.1646598968.2329.python-list@python.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Trace: individual.net D4eGqNbIPjhdFtxkcZa5mg69DFq66ZuCXrS1qQYM58tsy5Vf+O
Cancel-Lock: sha1:AeFkyrAPn6ivHIzoP0o0h2D6VMM=
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:91.0)
Gecko/20100101 Thunderbird/91.3.2
Content-Language: en-US
In-Reply-To: <mailman.216.1646598968.2329.python-list@python.org>

by: Greg Ewing - Sun, 6 Mar 2022 22:32 UTC

Re: Execute in a multiprocessing child dynamic code loaded by the parent process

<mailman.226.1646619204.2329.python-list@python.org>

copy mid

https://www.novabbs.com/devel/article-flat.php?id=17416&group=comp.lang.python#17416

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: martinp....@gmail.com (Martin Di Paola)
Newsgroups: comp.lang.python
Subject: Re: Execute in a multiprocessing child dynamic code loaded by the
parent process
Date: Mon, 7 Mar 2022 02:13:17 +0000
Lines: 48
Message-ID: <mailman.226.1646619204.2329.python-list@python.org>
References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
<CAPTjJmpJT4zf6nGgF5j-J89uPALwzs-Gf4aXWYAS65oYBfcE1Q@mail.gmail.com>
<20220306203601.eqv7geya3l7joxrj@gmail.com>
<mailman.216.1646598968.2329.python-list@python.org>
<j8kr3rFba1nU1@mid.individual.net>
<20220307021317.yncapixpaxlxbzb5@gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
X-Trace: news.uni-berlin.de L0R5a7zSr6Xs1y5ae2uYmwzv7Q5MMvSillRTSw7uxJhA==
Return-Path: <martinp.dipaola@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=OB804K0L;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.032
X-Spam-Evidence: '*H*': 0.94; '*S*': 0.00; 'knows': 0.04; 'thread':
0.05; 'variable': 0.05; 'developer.': 0.07; 'reference:': 0.09;
'threads': 0.09; 'worker': 0.09; 'url:github': 0.14; 'that.':
0.15; 'behaviour': 0.16; 'instance': 0.16;
'received:209.85.210.45': 0.16; 'received:mail-
ot1-f45.google.com': 0.16; 'rough': 0.16; 'side-effect': 0.16;
'subject:child': 0.16; 'subject:code': 0.16; 'subject:dynamic':
0.16; 'subject:parent': 0.16; 'subject:skip:m 10': 0.16;
'thread.': 0.16; 'times:': 0.16; 'usable': 0.16; 'problem': 0.16;
'developer': 0.16; 'message-id:@gmail.com': 0.18; 'uses': 0.19;
'to:addr:python-list': 0.20; 'code': 0.23; 'object': 0.26;
'example,': 0.28; 'wrong': 0.28; 'received:209.85.210': 0.29;
'whole': 0.30; 'default': 0.31; 'module': 0.31; 'think': 0.32;
'unexpected': 0.32; 'but': 0.32; "i'm": 0.33; 'there': 0.33;
'requires': 0.34; 'header:In-Reply-To:1': 0.34;
'received:google.com': 0.34; 'trying': 0.35; 'complex': 0.35;
'definitely': 0.35; 'from:addr:gmail.com': 0.35; 'main': 0.37;
'received:209.85': 0.37; 'author': 0.37; 'class': 0.37; 'could':
0.38; 'put': 0.38; 'received:209': 0.39; 'quite': 0.39; 'single':
0.39; 'this,': 0.39; 'developers': 0.39; 'use': 0.39; 'shared':
0.39; 'something': 0.40; 'try': 0.40; 'sharing': 0.60; 'provide':
0.60; 'load': 0.62; 'once': 0.63; 'lock': 0.64; 'required': 0.65;
'less': 0.65; 'back': 0.67; 'forget': 0.67; 'guaranteed': 0.67;
'per': 0.68; 'decision': 0.68; 'account.': 0.68; 'safer': 0.69;
'within': 0.69; 'conditions': 0.70; 'care': 0.71; 'risk': 0.71;
'yes': 0.76; 'attribute': 0.84; 'corner,': 0.84; 'locking': 0.84;
'subject: \n ': 0.84; 'suffer': 0.91; 'word.': 0.91; 'race': 0.93
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=date:from:to:subject:message-id:mail-followup-to:'user-agent
:references:mime-version:content-disposition:in-reply-to;
bh=gh2Rn/naqXiv9btUt7ap7uR5LiUd1CvG7UtjsovCp+A=;
b=OB804K0LXIDslTbcta55YEjEmsHgevr7Ja5JVcFT9J6YDe8dAQVMO8/sTeOIaGT4AD
bsEif2AGeItWCv39QpCkMVTbuNGeEW/vGPHvehG9Z1bD+Nl7wwf+kXNGtOjOK3tNCa57
9DK3gZI1MjQh+e8Hhj2Wfh3UoYQe8ypW/exHbDsKDHlerKWsJpLDIojjFVuvhN80v6ab
RSIUjp3UyKSdq49eI+MAFNuU45KzLaP1LJhcgUrcUPoJaWP/IRIYryvjSOz/YUBwEoT8
8gXPuMjZ7+BWRjv9XCSFRqEf8+qcLycGbCx4Ly9qScFDJp3vqp0F47IyO2mmyMQL5iI5
4Gig==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:date:from:to:subject:message-id:mail-followup-to
:'user-agent:references:mime-version:content-disposition:in-reply-to;
bh=gh2Rn/naqXiv9btUt7ap7uR5LiUd1CvG7UtjsovCp+A=;
b=A2G6SkuLksH9nYZbQbDTU1X9grAwYzY479cC64n9NmKTtiXVcYXF5RYVfQUDhft1j7
1vHvKyPuxsYxGxJWvL6ZbnujTUcNhRqIupQn5CaAvuboWLDvYfAYCELKLuT5KaCmv67B
Ke2SHK93KJxFZaERfj+bwPoAFYxT+JyqtqVbplKoDNssF34zDl0+gn+6t3evPIUDYYG6
dKO8YMA2W+yIli0SwzmUj/1YzC+0OuE0uZs+f0R6OkW5bC0i1lxHxrQmFbcRm7L6vJDB
hCRkCnkWYkF5hcsOYEusChEmhmVRyhCOY3o8N02HXhP7PPTciJnr2kR8Td+fV2Bu7hKE
9GPQ==
X-Gm-Message-State: AOAM530tNOp5UjR91xPbPlrWNalHuiIMgmtNmBGynZackxjUKs+YO8Wi
oyNrQLPQMdtM2GtKJNLxPN5LbF9xapU=
X-Google-Smtp-Source: ABdhPJzXqZWIj1NMgJT+WGDM8IjAKYxhtNmmVxwgdHVzcjjPZsiTPU54O6FrRoEKdlWsAlFwmikVrw==
X-Received: by 2002:a05:6830:1af2:b0:5b2:2566:59f9 with SMTP id
c18-20020a0568301af200b005b2256659f9mr3697941otd.195.1646619201758;
Sun, 06 Mar 2022 18:13:21 -0800 (PST)
Mail-Followup-To: python-list@python.org
'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
Thunderbird/52.5.2 Lightning/5.4.5.2'
Content-Disposition: inline
In-Reply-To: <j8kr3rFba1nU1@mid.individual.net>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <20220307021317.yncapixpaxlxbzb5@gmail.com>
X-Mailman-Original-References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
<CAPTjJmpJT4zf6nGgF5j-J89uPALwzs-Gf4aXWYAS65oYBfcE1Q@mail.gmail.com>
<20220306203601.eqv7geya3l7joxrj@gmail.com>
<mailman.216.1646598968.2329.python-list@python.org>
<j8kr3rFba1nU1@mid.individual.net>

by: Martin Di Paola - Mon, 7 Mar 2022 02:13 UTC

>I'm not so sure about that. The author of the plugin knows they're
>writing code that will be dynamically loaded, and can therefore
>expect the kind of problem they're having. It could be argued that
>it's their responsibility to ensure that all the needed code is loaded
>into the subprocess.

Yes but I try to always make my libs/programs as much as usable as
possible. "Ergonomic" would be the word.

In the case of the plugin-engine I'm trying to hide any side-effect or
unexpected behaviour of the engine so the developer of the plugin
does not have take that into account.

I agree that if the developer uses multiprocessing he/she needs to know
its implications. But if I can "smooth" any rough corner, I will try to
do it.

For example, the main project (developed by me) uses threads for
concurrency. It would be simpler to load the plugins and instantiate
them *once* and ask the plugins developers to take care of any
race condition (RC) within their implementation.

Because the plugins were instantiated *once*, it is almost guaranteed
that the plugins will suffer from race conditions and they will require
some sort of locking.

This is quite risky: you may forget to protect something and you will
end up with a RC and/or you may put the lock in the wrong place and the
whole thing will not work concurrently.

My decision back then was to instantiate each plugin N+1 times: once in
the main thread and then once per worker thread.

With this, no single plugin instance will be shared so there is no risk
of RC and no need for locking. (Yes, I know, the developer just needs to
use a module variable or a class attribute and it will get a RC and
these are shared but it is definitely not the default scenario).

If sharing is required I provide an object that minimizes the locking
needed.

It was much complex for me at the design and at the implementation level
but I think that it is safer and requires less from the plugin
developer.

Reference: https://byexamples.github.io/byexample/contrib/concurrency-model

Sorry. I just realized this sentance makes no sense :) -- Ian Main

devel / comp.lang.python / Re: Execute in a multiprocessing child dynamic code loaded by the parent process

Subject	Author
Re: Execute in a multiprocessing child dynamic code loaded by the	Martin Di Paola
Re: Execute in a multiprocessing child dynamic code loaded by the	Greg Ewing
Re: Execute in a multiprocessing child dynamic code loaded by the	Martin Di Paola