Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

The Macintosh is Xerox technology at its best.

devel / comp.lang.python / Re: Accuracy of multiprocessing.Queue.qsize before any Queue.get invocations?

SubjectAuthor
o Re: Accuracy of multiprocessing.Queue.qsize before any Queue.getMartin Di Paola

1
Re: Accuracy of multiprocessing.Queue.qsize before any Queue.get invocations?

<mailman.400.1652444684.20749.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=18261&group=comp.lang.python#18261

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: martinp....@gmail.com (Martin Di Paola)
Newsgroups: comp.lang.python
Subject: Re: Accuracy of multiprocessing.Queue.qsize before any Queue.get
invocations?
Date: Fri, 13 May 2022 09:24:37 -0300
Lines: 65
Message-ID: <mailman.400.1652444684.20749.python-list@python.org>
References: <20220512180702.41656ce3@bigbox.attlocal.net>
<20220513122437.hev25jfdccbrnznf@gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
X-Trace: news.uni-berlin.de B0WZH+TZefDImp14IsNjowrtcvvMLaj1WzMGevNByC8A==
Return-Path: <martinp.dipaola@gmail.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=VCU4WHPY;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.069
X-Spam-Evidence: '*H*': 0.86; '*S*': 0.00; '2022': 0.05;
'approximate': 0.05; 'bunch': 0.05; 'queue': 0.07; 'consistency':
0.09; "hasn't": 0.09; 'url:mailman': 0.15; 'queue?': 0.16;
'semantics,': 0.16; 'skip:> 10': 0.16; 'subject:Accuracy': 0.16;
'wrote:': 0.16; 'developer': 0.16; 'probably': 0.17; 'message-
id:@gmail.com': 0.18; 'thu,': 0.19; 'to:addr:python-list': 0.20;
"i've": 0.22; 'code': 0.23; "i'd": 0.24; '(and': 0.25; 'anything':
0.25; 'url-ip:188.166.95.178/32': 0.25; 'url-ip:188.166.95/24':
0.25; 'url:listinfo': 0.25; 'url-ip:188.166/16': 0.25; 'tim':
0.26; 'thinking': 0.28; 'it,': 0.29; 'url-ip:188/8': 0.31;
'"this': 0.32; 'guess': 0.32; 'but': 0.32; 'there': 0.33;
"didn't": 0.34; 'header:In-Reply-To:1': 0.34;
'received:google.com': 0.34; 'reflect': 0.35;
'from:addr:gmail.com': 0.35; 'count': 0.36; 'processes': 0.36;
'thanks,': 0.36; 'those': 0.36; 'special': 0.37; 'currently':
0.37; 'received:209.85': 0.37; 'could': 0.38; 'received:209':
0.39; 'added': 0.39; 'shared': 0.39; 'case.': 0.40; 'skip:o 10':
0.61; 'reasonable': 0.62; 'once': 0.63; 'skip:m 20': 0.63;
'between': 0.63; 'share': 0.63; 'skip:r 20': 0.64; 'process.':
0.65; 'well': 0.65; 'items': 0.68; 'accurately': 0.69; 'counter':
0.69; 'care': 0.71; 'chance': 0.71; 'accurate': 0.74; 'process,':
0.75; 'chase': 0.76; 'implemented': 0.76; 'tracking': 0.76;
'martin.': 0.84; 'subject: \n ': 0.84; 'reliable.': 0.91;
'subject:before': 0.93
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=date:from:to:subject:message-id:mail-followup-to:'user-agent
:references:mime-version:content-disposition:in-reply-to;
bh=9z3ZUSlOF19kOYSb6gmNLQ4UUdBYYJ9ZRMJ6DPZX65E=;
b=VCU4WHPY8JD7Golp/x6dYrloou1XfMHFpGpLqTPWOZkUn0QuKdwuGG8ISu6kSPffEB
smuo8oP0WiFRVnXR3hoe60UiMGV9mX1sDn0cziEJV3gjcekEKZ70MFscmrmO65kE+Xq3
QZQDBvDNGcxklIVPn2/N7AVXFUY1savxYP1bAQYiuVvpmsqLTxW2Zdld9UIYSH7QCYLj
mU0o7ZpUYRb33B4bE/21JaxEvrrVaiXGoltCtEAWY83tDVwdPt+YvvhI8YuwE9ksEOU4
gSf611QShy0y17pe1HJg0SS5I5PIYjmo9le1M7BGe0C8fnymmxcxFeImDMZoIOhZXCWx
Qq4Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=x-gm-message-state:date:from:to:subject:message-id:mail-followup-to
:'user-agent:references:mime-version:content-disposition:in-reply-to;
bh=9z3ZUSlOF19kOYSb6gmNLQ4UUdBYYJ9ZRMJ6DPZX65E=;
b=OPQBvspFRHdCgCDJhTL8fFuAFJWxNCFQs63K/Mv6W0gkAaTdQLk8RDE1RfQfdNF/Az
1BiDzBJ5G3ZSvX6jVXqL6oXI4cPuGQ/G8rQgsuq3XcUt/NUgKv4VEDdnsb6WbD0T8XaU
dnk+kvfLrcA3QaXmw/1MHRfnF+yxvGodySQtUy4LPaKlZHoARgQvv+7sbuoFnt/+kKnv
5+Z9K9AfQXw6DwdZSZHyWLJhWmRUcdMtixpWXGRaO2W4bJYIs1RyjY3bScJ6bg+7Gmg/
Po355M2rQgqj2pPrtOTWoANEkBULSuPfOJgfkT0y7qu+8EsM1y0QmgGZhLJvpTeOX2RW
G2MA==
X-Gm-Message-State: AOAM533v/n5JPh8z0TqnGazOo6l/k/voG3+vmrv0lBy4w2ukSv8CUsmj
7kNR/CkG4yyVHpA7wF71h601n9eLjVo=
X-Google-Smtp-Source: ABdhPJzCEfhnhrOb0doHo1phSRZjl6b0mJe00V5451SvEyaf+yJY3Dc4/L0ENSHDpjUct3jkzkP1jQ==
X-Received: by 2002:a54:4688:0:b0:325:9a36:ecfe with SMTP id
k8-20020a544688000000b003259a36ecfemr2381968oic.96.1652444680781;
Fri, 13 May 2022 05:24:40 -0700 (PDT)
Mail-Followup-To: python-list@python.org
'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
Thunderbird/52.5.2 Lightning/5.4.5.2'
Content-Disposition: inline
In-Reply-To: <20220512180702.41656ce3@bigbox.attlocal.net>
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <20220513122437.hev25jfdccbrnznf@gmail.com>
X-Mailman-Original-References: <20220512180702.41656ce3@bigbox.attlocal.net>
 by: Martin Di Paola - Fri, 13 May 2022 12:24 UTC

If the queue was not shared to any other process, I would guess that its
size is reliable.

However, a plain counter could be much simpler/safer.

The developer of multiprocessing.Queue, implemented
size() thinking in how to share the size and maintain a reasonable
consistency between process.
He/she probably didn't care how well works in a single-process scenario
as this is a very special case.

Thanks,
Martin.

On Thu, May 12, 2022 at 06:07:02PM -0500, Tim Chase wrote:
>The documentation says[1]
>
>> Return the approximate size of the queue. Because of
>> multithreading/multiprocessing semantics, this number is not
>> reliable.
>
>Are there any circumstances under which it *is* reliable? Most
>germane, if I've added a bunch of items to the Queue, but not yet
>launched any processes removing those items from the Queue, does
>Queue.qsize accurately (and reliably) reflect the number of items in
>the queue?
>
> q = Queue()
> for fname in os.listdir():
> q.put(fname)
> file_count = q.qsize() # is this reliable?
> # since this hasn't yet started fiddling with it
> for _ in range(os.cpu_count()):
> Process(target=myfunc, args=(q, arg2, arg3)).start()
>
>I'm currently tracking the count as I add them to my Queue,
>
> file_count = 0
> for fname in os.listdir():
> q.put(fname)
> file_count += 1
>
>but if .qsize is reliably accurate before anything has a chance to
>.get data from it, I'd prefer to tidy the code by removing the
>redunant counting code if I can.
>
>I'm just not sure what circumstances the "this number is not
>reliable" holds. I get that things might be asynchronously
>added/removed once processes are running, but is there anything that
>would cause unreliability *before* other processes/consumers run?
>
>Thanks,
>
>-tkc
>
>[1]
>https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue.qsize
>
>
>
>
>
>--
>https://mail.python.org/mailman/listinfo/python-list

1
server_pubkey.txt

rocksolid light 0.9.8
clearnet tor