Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

C'est magnifique, mais ce n'est pas l'Informatique. -- Bosquet [on seeing the IBM 4341]

devel / comp.lang.python / Re: Execute in a multiprocessing child dynamic code loaded by the parent process

SubjectAuthor
o Re: Execute in a multiprocessing child dynamic code loaded by theChris Angelico

1
Re: Execute in a multiprocessing child dynamic code loaded by the parent process

<mailman.218.1646601198.2329.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=17407&group=comp.lang.python#17407

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: ros...@gmail.com (Chris Angelico)
Newsgroups: comp.lang.python
Subject: Re: Execute in a multiprocessing child dynamic code loaded by the
parent process
Date: Mon, 7 Mar 2022 08:13:05 +1100
Lines: 54
Message-ID: <mailman.218.1646601198.2329.python-list@python.org>
References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
<CAPTjJmpJT4zf6nGgF5j-J89uPALwzs-Gf4aXWYAS65oYBfcE1Q@mail.gmail.com>
<20220306203601.eqv7geya3l7joxrj@gmail.com>
<CAPTjJmrf0o4tFhEMLEw6A=sGwc-d2t7nu=y6s=oU_S3h9-0bQA@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
X-Trace: news.uni-berlin.de imUcXLESWVxb0bY6DPYqPguqIPyRODUyqQcDwxGfV34Q==
Return-Path: <rosuav@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=RgbZgBpd;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.006
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; '2022': 0.05; 'have,':
0.07; 'mar': 0.07; 'fails': 0.09; 'fault': 0.09; 'module.': 0.09;
'arbitrary': 0.16; 'assuming': 0.16; 'chrisa': 0.16;
'from:addr:rosuav': 0.16; 'from:name:chris angelico': 0.16;
'modules,': 0.16; 'received:209.85.128.42': 0.16; 'received:mail-
wm1-f42.google.com': 0.16; 'static': 0.16; 'subject:child': 0.16;
'subject:code': 0.16; 'subject:dynamic': 0.16; 'subject:parent':
0.16; 'subject:skip:m 10': 0.16; 'target.': 0.16; 'wrote:': 0.16;
'instead': 0.17; 'calls': 0.19; 'to:addr:python-list': 0.20;
'basically': 0.22; 'version': 0.23; 'code': 0.23; 'to:name:python-
list@python.org': 0.24; 'depends': 0.25; 'saying': 0.25;
'register': 0.25; 'function': 0.27; 'it,': 0.29; 'module': 0.31;
'message-id:@mail.gmail.com': 0.32; 'but': 0.32; "i'm": 0.33;
'there': 0.33; 'path': 0.33; 'able': 0.34; 'header:In-Reply-To:1':
0.34; 'received:google.com': 0.34; 'from:addr:gmail.com': 0.35;
'mon,': 0.36; 'target': 0.36; 'using': 0.37; "it's": 0.37;
'received:209.85': 0.37; 'class': 0.37; 'way': 0.38; 'could':
0.38; 'received:209': 0.39; 'enough': 0.39; 'rest': 0.39;
'define': 0.40; 'martin': 0.40; 'wants': 0.40; 'something': 0.40;
'best': 0.61; 'method': 0.61; 'load': 0.62; 'skip:m 20': 0.63;
'your': 0.64; 'smart': 0.67; 'choice': 0.76; 'need,': 0.76;
'practical': 0.84; 'catch': 0.84; 'paola': 0.84; 'subject: \n ':
0.84; 'doing.': 0.91
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to;
bh=MZAzF9+xaPz6yp+2WeebXL1THRpl2WaL1WxUU2X4xHc=;
b=RgbZgBpdJmw8QTQyER12JbFI49Ex3Py7/2zvXsiJQctXWN2yCl8xLfD8OV3ua5PLvy
38PUDBG+PyQMmvhxXy9sZ9UzdeRtjySigA3Yjm8zTIGAxIhObXMr+pqErgZxvN1BTmf+
YRnyfELhrjdc/so2XCDX5b4Ki/+f046I97s8Z88KD3QFTjv87Z5e34O0+qcOehhuJzep
Q6perMcAubOhxqnR0T3V5ZKvB7a/WoHMOmGkGmCjW+6Q6tML336Q7AeYmah53Oa9z5Js
OMpAMwkVTF2/bLPdxehcu8qISjaYTretGrPZYZEMeq6DkwJleX1UAl6WuDu0hdQAcfJ3
cncQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to;
bh=MZAzF9+xaPz6yp+2WeebXL1THRpl2WaL1WxUU2X4xHc=;
b=kpYFl5iRbdENSt3EPZK6XuvGxAHol9iUjiT9qj4udsm/M9cdZckzcCJdfXUUnOFOwG
mFykC2y5/mKkMWCj3FshJvh9CCuzrw3hI11IhPTzYFrdOfmRmcMPWZjBIfgDPUY4PhRz
Y89D/t3S2tQCqHYL40JQQr2EO5TGbgEQ1nngj2dFUIozRazbO8cklB4IX2obSP+7Ef2I
l2N9DV224JhkBS+C5cl6hna5c7qNbhuEXydW62M3ZrzaMi2j0Fvj7ltzHkVEzHDaExvd
Pv4WLavDDxewraYV8uNxBVjcv+vycp8bbBZXUa3qrXakUIsk3epWQlvNCdHyJkCmUGsV
XdLg==
X-Gm-Message-State: AOAM530gQMuY7Yy1Se3rORF2/+gLgQOCbOgZZHIX4PBZqO1R7xkhY633
79vq9tAtrieVy3wyBN9dsrxauStYwmOYJlMrRa1NCD1mnUY=
X-Google-Smtp-Source: ABdhPJyvOBqeOIOu6u/PlK3wDpRRiaE3ZvCtNUpluZcETaPfuM0Mw7umn8vhVJsd5WxedQy+ZjqbuhEVFcNQfN5T30A=
X-Received: by 2002:a7b:c19a:0:b0:381:8495:9dd with SMTP id
y26-20020a7bc19a000000b00381849509ddmr15978897wmi.33.1646601196425; Sun, 06
Mar 2022 13:13:16 -0800 (PST)
In-Reply-To: <20220306203601.eqv7geya3l7joxrj@gmail.com>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAPTjJmrf0o4tFhEMLEw6A=sGwc-d2t7nu=y6s=oU_S3h9-0bQA@mail.gmail.com>
X-Mailman-Original-References: <20220306124208.orqbcezhqqeysdpe@gmail.com>
<CAPTjJmpJT4zf6nGgF5j-J89uPALwzs-Gf4aXWYAS65oYBfcE1Q@mail.gmail.com>
<20220306203601.eqv7geya3l7joxrj@gmail.com>
 by: Chris Angelico - Sun, 6 Mar 2022 21:13 UTC

On Mon, 7 Mar 2022 at 07:37, Martin Di Paola <martinp.dipaola@gmail.com> wrote:
>
>
>
> >
> >The way you've described it, it's a hack. Allow me to slightly redescribe it.
> >
> >modules = loader()
> >objs = init(modules)
> >
> >def invoke(mod, func):
> > # I'm assuming that the loader is smart enough to not load
> > # a module that's already loaded. Alternatively, load just the
> > # module you need, if that's a possibility.
> > loader()
> > target = getattr(modules[mod], func)
> > target()
> >
> >ch = multiprocessing.Process(target=invoke, args=("some_module", "sayhi"))
> >ch.start()
> >
>
> Yeup, that would be my first choice but the catch is that "sayhi" may
> not be a function of the given module. It could be a static method of
> some class or any other callable.

Ah, fair. Are you able to define it by a "path", where each step in
the path is a getattr() call?

The trouble is, arbitrary callables might not be available in a
reconstructed version of the module.

> Using multiprocessing.reduction was a practical decision: if the user
> wants to call something non-pickleable, it is not my fault, it is
> multiprocessing's fault.
>
> It *would* be my fault if multiprocessing.Process fails only because I'm
> loading the code dynamically.

Fair. I guess, then, that the best thing to do is to preload the
modules, then unpickle. So, basically what you already have, but with
more caveats.

> Do you have some in mind? Or may be a project that I could read?

Not handy, but there are always many different ways to do things. For
instance, instead of saying "spawn a subprocess and call this
function", you could invert it, and have the function register itself
as the target. Then it's just "spawn a subprocess and load this
module", and that calls the registered invocation. It all depends on
what the rest of your project is doing. Mainly, though, I'm just not
ruling out the possibility of other options :)

ChrisA

1
server_pubkey.txt

rocksolid light 0.9.8
clearnet tor