Message-ID:

We come to bury DOS, not to praise it. -- Paul Vojta, vojta@math.berkeley.edu

devel / comp.lang.python / Accuracy of multiprocessing.Queue.qsize before any Queue.get invocations?

Accuracy of multiprocessing.Queue.qsize before any Queue.get invocations?

<mailman.390.1652398083.20749.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=18252&group=comp.lang.python#18252

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: python.l...@tim.thechases.com (Tim Chase)
Newsgroups: comp.lang.python
Subject: Accuracy of multiprocessing.Queue.qsize before any Queue.get
invocations?
Date: Thu, 12 May 2022 18:07:02 -0500
Lines: 47
Message-ID: <mailman.390.1652398083.20749.python-list@python.org>
References: <20220512180702.41656ce3@bigbox.attlocal.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-Trace: news.uni-berlin.de SMJ8JW7DktoNgmonDL2Q7wkXba0PaagOOvkj2kYDRd3A==
Return-Path: <python.list@tim.thechases.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.003
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'approximate': 0.05;
'bunch': 0.05; "hasn't": 0.09; 'received:108': 0.09;
'received:sbcglobal.net': 0.09; '-tkc': 0.16;
'from:addr:python.list': 0.16; 'from:addr:tim.thechases.com':
0.16; 'from:name:tim chase': 0.16; 'queue?': 0.16;
'received:108.167': 0.16; 'received:108.167.139': 0.16;
'received:108.167.139.23': 0.16; 'received:174.136.13': 0.16;
'received:174.136.13.174': 0.16; 'received:74.220': 0.16;
'received:76': 0.16; 'received:accountservergroup.com': 0.16;
'received:cm3.websitewelcome.com': 0.16;
'received:lightspeed.rcsntx.sbcglobal.net': 0.16;
'received:rcsntx.sbcglobal.net': 0.16;
'received:unifiedlayer.com': 0.16;
'received:uscentral455.accountservergroup.com': 0.16;
'semantics,': 0.16; 'subject:Accuracy': 0.16; 'to:addr:python-
list': 0.20; "i've": 0.22; 'code': 0.23; "i'd": 0.24; '(and':
0.25; 'anything': 0.25; 'it,': 0.29; '"this': 0.32; 'launched':
0.32; 'but': 0.32; "i'm": 0.33; 'there': 0.33; 'reflect': 0.35;
'count': 0.36; 'processes': 0.36; 'thanks,': 0.36; 'those': 0.36;
'currently': 0.37; 'added': 0.39; 'skip:o 10': 0.61; 'once': 0.63;
'skip:r 20': 0.64; '[1]': 0.67; 'items': 0.68; 'accurately': 0.69;
'chance': 0.71; 'accurate': 0.74; 'tracking': 0.76; 'subject: \n
': 0.84; 'reliable.': 0.91; 'subject:before': 0.93
X-Authority-Reason: nr=8
X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.0)
X-AntiAbuse: This header was added to track abuse,
please include it with any abuse report
X-AntiAbuse: Primary Hostname - uscentral455.accountservergroup.com
X-AntiAbuse: Original Domain - python.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - tim.thechases.com
X-BWhitelist: no
X-Source-IP: 76.222.220.222
X-Source-L: No
X-Exim-ID: 1npHtf-002czm-Sl
X-Source:
X-Source-Args:
X-Source-Dir:
X-Source-Sender: 76-222-220-222.lightspeed.rcsntx.sbcglobal.net
(bigbox.attlocal.net) [76.222.220.222]:22983
X-Source-Auth: tim@thechases.com
X-Email-Count: 1
X-Source-Cap: dGhlY2hhc2U7dGhlY2hhc2U7dXNjZW50cmFsNDU1LmFjY291bnRzZXJ2ZXJncm91cC5jb20=
X-Local-Domain: yes
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <20220512180702.41656ce3@bigbox.attlocal.net>

by: Tim Chase - Thu, 12 May 2022 23:07 UTC

The documentation says[1]

> Return the approximate size of the queue. Because of
> multithreading/multiprocessing semantics, this number is not
> reliable.

Are there any circumstances under which it *is* reliable? Most
germane, if I've added a bunch of items to the Queue, but not yet
launched any processes removing those items from the Queue, does
Queue.qsize accurately (and reliably) reflect the number of items in
the queue?

q = Queue()
for fname in os.listdir():
q.put(fname)
file_count = q.qsize() # is this reliable?
# since this hasn't yet started fiddling with it
for _ in range(os.cpu_count()):
Process(target=myfunc, args=(q, arg2, arg3)).start()

I'm currently tracking the count as I add them to my Queue,

file_count = 0
for fname in os.listdir():
q.put(fname)
file_count += 1

but if .qsize is reliably accurate before anything has a chance to
..get data from it, I'd prefer to tidy the code by removing the
redunant counting code if I can.

I'm just not sure what circumstances the "this number is not
reliable" holds. I get that things might be asynchronously
added/removed once processes are running, but is there anything that
would cause unreliability *before* other processes/consumers run?

Thanks,

-tkc

[1]
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue.qsize

Subject	Author
Accuracy of multiprocessing.Queue.qsize before any Queue.get	Tim Chase