Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

"It's the best thing since professional golfers on 'ludes." -- Rick Obidiah

devel / comp.lang.python / Re: Parallel(?) programming with python

SubjectAuthor
o Re: Parallel(?) programming with pythonCameron Simpson

1
Re: Parallel(?) programming with python

<mailman.233.1660012258.20444.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=19169&group=comp.lang.python#19169

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: cs...@cskk.id.au (Cameron Simpson)
Newsgroups: comp.lang.python
Subject: Re: Parallel(?) programming with python
Date: Tue, 9 Aug 2022 12:30:53 +1000
Lines: 47
Message-ID: <mailman.233.1660012258.20444.python-list@python.org>
References: <CAHVvXxTbmaLZnRg1O25nXb+TKvAg24HsV9YjYBra50q14Gr_pQ@mail.gmail.com>
<YvHG3S3tqX+sEL6i@cskk.homeip.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de gENN1RPbR96vvTVez3z/fQYK4SmC6Gojs3WRQPzEpbCw==
Return-Path: <cameron@cskk.id.au>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.004
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; '2022': 0.05;
'subject:python': 0.06; 'areas.': 0.07; 'aug': 0.07; 'anyway,':
0.09; 'coordinate': 0.09; 'difficulty': 0.09; 'driven': 0.09;
'etc).': 0.09; 'mechanism': 0.09; 'cheers,': 0.11; '"in': 0.16;
'(because': 0.16; '(eg': 0.16; 'cameron': 0.16; 'from:addr:cs':
0.16; 'from:addr:cskk.id.au': 0.16; 'from:name:cameron simpson':
0.16; 'message-id:@cskk.homeip.net': 0.16; 'program"': 0.16;
'received:13.237': 0.16; 'received:13.237.201': 0.16;
'received:13.237.201.189': 0.16; 'received:cskk.id.au': 0.16;
'received:id.au': 0.16; 'received:l': 0.16;
'received:mail.cskk.id.au': 0.16; 'removes': 0.16; 'separately':
0.16; 'simpson': 0.16; 'subject:programming': 0.16; 'writes':
0.16; 'wrote:': 0.16; 'probably': 0.17; 'to:addr:python-list':
0.20; 'basically': 0.22; 'run': 0.23; 'else': 0.27; 'example,':
0.28; 'keeping': 0.28; 'header:User-Agent:1': 0.30; 'approach':
0.31; 'program': 0.31; 'official': 0.32; 'question': 0.32; 'same':
0.34; 'header:In-Reply-To:1': 0.34; 'running': 0.34; 'programme':
0.35; 'received:au': 0.35; 'files': 0.36; 'mon,': 0.36;
'processes': 0.36; 'change': 0.36; 'those': 0.36; 'directory':
0.37; 'others': 0.37; 'this.': 0.37; 'file': 0.38; 'way': 0.38;
'two': 0.39; 'changes': 0.39; 'least': 0.39; 'single': 0.39;
'use': 0.39; 'still': 0.40; 'files.': 0.40; 'match': 0.40;
'something': 0.40; 'want': 0.40; 'sharing': 0.60; 'best': 0.61;
'data,': 0.63; 'simply': 0.63; 'down': 0.64; 'about.': 0.64;
'needs.': 0.64; 'received:13': 0.64; 'your': 0.64; 'parts': 0.65;
'less': 0.65; 'received:userid': 0.66; 'depending': 0.70;
'ignore': 0.71; '"to': 0.84; 'feasible,': 0.84; 'modes': 0.84;
'oscar': 0.84; 'programmes': 0.84; 'protection.': 0.84; 'doing.':
0.91; 'largely': 0.91
Mail-Followup-To: Python List <python-list@python.org>
Content-Disposition: inline
In-Reply-To: <CAHVvXxTbmaLZnRg1O25nXb+TKvAg24HsV9YjYBra50q14Gr_pQ@mail.gmail.com>
User-Agent: Mutt/2.2.3 (2022-04-12)
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <YvHG3S3tqX+sEL6i@cskk.homeip.net>
X-Mailman-Original-References: <CAHVvXxTbmaLZnRg1O25nXb+TKvAg24HsV9YjYBra50q14Gr_pQ@mail.gmail.com>
 by: Cameron Simpson - Tue, 9 Aug 2022 02:30 UTC

On 09Aug2022 00:22, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
>On Mon, 8 Aug 2022 at 19:01, Andreas Croci <andrea.croci@gmx.de> wrote:
>> Basically the question boils down to wether it is possible to have
>> parts
>> of a program (could be functions) that keep doing their job while other
>> parts do something else on the same data, and what is the best way to do
>> this.

Which is of course feasible, as others have outlined.

>Why do these "parts of a program" need to be part of the *same*
>program. I would write this as just two separate programs. One
>collects the data and writes it to a file. The other periodically
>reads the file and computes the DFT.

I would also write these as separate programmes, or at least as distinct
modes of the same programme (eg "myprog poll" and "myprog archive" etc).
Largely because you might run the "poll" regularly and briefly, and the
processes phase separately and less frequently. You don't need to keep a
single programme lurking around forever - fire it up as required.

However, I want to point out that this _in no way_ removes the need for
access contol and mutexes. It will change the mechanism (because your
two programmes are now operating separately) and makes it more concrete
in your mind what _actually and precisely_ needs protection.

For example, you probably want to avoid _processing_ a data file at the
same time as _updating_ that file. Depending on what you're doing this
can be as simple as keeping "to be updated" files with distinct names
from "available to be processed/archived" files. This is a standard
difficulty with "hot folder" upload areas.

A common approach might be to write a file with a "temp" style name (eg
".tmp*") until completed, then rename it to its official name (eg
"datafile*"). And then your processing/archiving side can simply ignore
the "in progress" files because they do not match the names it cares
about.

Anyway, those are specifics, which will be driven by what you're
actually doing. The point is that you still need to coordinate use of
the files suitably for your needs. Doing this in one long running
programme with Threads/mutexes or separate programmes sharing a data
directory just changes the mechanisms.

Cheers,
Cameron Simpson <cs@cskk.id.au>

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor