novaBBS - comp.lang.python - Re: PyObject_CallFunctionObjArgs segfaults

Re: PyObject_CallFunctionObjArgs segfaults

<mailman.506.1664553755.20444.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=19616&group=comp.lang.python#19616

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!usenet.goja.nl.eu.org!3.eu.feeder.erje.net!feeder.erje.net!news2.arglkargh.de!news.karotte.org!fu-berlin.de!uni-berlin.de!not-for-mail
From: jenk...@tutanota.com (Jen Kris)
Newsgroups: comp.lang.python
Subject: Re: PyObject_CallFunctionObjArgs segfaults
Date: Fri, 30 Sep 2022 18:02:27 +0200 (CEST)
Lines: 172
Message-ID: <mailman.506.1664553755.20444.python-list@python.org>
References: <ND8xHA---3-2@tutanota.com>
<fbd7fcc1-6095-e381-4721-bc5c09975202@mrabarnett.plus.com>
<ND9prTQ--3-2@tutanota.com-ND9qgBV----2> <ND9zJoA--7-2@tutanota.com>
<b60d1d3b-1381-8646-fd43-15496bf993dd@mrabarnett.plus.com>
<NDAPOvd--3-2@tutanota.com>
<aadf0f47-468b-158e-a0fc-82bb72fa2b98@mrabarnett.plus.com>
<4e8fa90c-6f59-1999-93bc-2a31ba14c12e@mrabarnett.plus.com>
<NDE7c5l--3-2@tutanota.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de mxfbVdXZHnyfycO3uCYfiAZr6chc0B0CNL2Zvw+r58qQ==
Return-Path: <jenkris@tutanota.com>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=pass
reason="2048-bit key; unprotected key"
header.d=tutanota.com header.i=@tutanota.com header.b=Lg8dWzxZ;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.005
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'comments': 0.03; 'cc:addr
:python-list': 0.09; "hasn't": 0.09; 'help.\xc2\xa0': 0.09;
'something,': 0.09; '>': 0.14; 'cc:no real name:2**0': 0.14;
'questions.': 0.14; 'url:mailman': 0.15; '(because': 0.16;
'2022,': 0.16; '[snip]': 0.16; 'assuming': 0.16; 'comment.': 0.16;
'confused': 0.16; 'followup': 0.16; 'not.\xc2\xa0': 0.16;
'object,': 0.16; 'operations,': 0.16; 'reference,': 0.16;
'solved': 0.16; 'wrote:': 0.16; 'api': 0.17; 'says': 0.17;
'values': 0.17; 'probably': 0.17; 'it?': 0.19;
'cc:addr:python.org': 0.20; 'input': 0.21; "i've": 0.22;
'basically': 0.22; 'problem,': 0.22; 'skip:p 30': 0.23;
'received:de': 0.23; 'url-ip:188.166.95.178/32': 0.25; 'url-
ip:188.166.95/24': 0.25; 'help.': 0.25; 'url:listinfo': 0.25;
'cc:2**0': 0.25; 'url-ip:188.166/16': 0.25; 'anyone': 0.25;
'object': 0.26; 'leave': 0.27; 'else': 0.27; 'function': 0.27;
'>>>': 0.28; 'example,': 0.28; 'it,': 0.29; 'effect': 0.31; 'url-
ip:188/8': 0.31; 'official': 0.32; 'guess': 0.32; 'needed,': 0.32;
'said,': 0.32; 'sep': 0.32; 'received:192.168.1': 0.32; 'but':
0.32; 'there': 0.33; 'hold': 0.33; 'url:blog': 0.33; 'mean': 0.34;
'header:In-Reply-To:1': 0.34; 'question.': 0.35; 'yes,': 0.35;
'count': 0.36; 'functions': 0.36; 'member': 0.37; "it's": 0.37;
'others': 0.37; 'received:192.168': 0.37; 'could': 0.38; 'means':
0.38; 'thanks': 0.38; 'list': 0.39; 'should': 0.40; 'helps': 0.60;
'reference': 0.60; 'here': 0.62; 'once': 0.63; 'finished': 0.64;
'your': 0.64; 'also:': 0.69; 'performance': 0.71; 'longer': 0.71;
'free': 0.72; '8bit%:33': 0.74; 'returned': 0.81; 'increments':
0.84; 'out.\xe2\x80\x9d': 0.84; 'pointer': 0.84; 'required)':
0.84; 'said:': 0.84; 'steal': 0.84
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1664553747;
s=s1; d=tutanota.com;
h=From:From:To:To:Subject:Subject:Content-Description:Content-ID:Content-Type:Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:In-Reply-To:In-Reply-To:MIME-Version:MIME-Version:Message-ID:Message-ID:Reply-To:References:References:Sender;
bh=l7aqzxSNR0XZW5wTQqfkSmqVq4y6JXQGEU5e7TPmLdw=;
b=Lg8dWzxZscpj1OSKUypPvaRWux4KtVFIao4GM6ZmGJw66dVfWtYiGBm3Re7u8ukG
352fSI6oZiKc29wXWeWnMwWjs5gfk/mtZoQKB6NgWMSyshhxV+RxXtpxZMSeZBZPraD
0PY/2f0SRJhbG50Nuo+an44hQtUDXx+gs2ZFxJz79u8CoNQ2qc2WRk6W5g01uK9raEN
80GwsPSbgLJbAD9Mn1daJL4COrGtkGHNFa6BxUV0dQ0kWUL8o8RaDAYqc3vVvU7pGmY
eYPIxs1PypYUBh6PkZziDOX1eA3pROAA25LJwvbA7CyeRcOYPB7siA/WwsFgRu9ks5A
NdxfldUXfA==
In-Reply-To: <4e8fa90c-6f59-1999-93bc-2a31ba14c12e@mrabarnett.plus.com>
X-Content-Filtered-By: Mailman/MimeDel 2.1.39
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <NDE7c5l--3-2@tutanota.com>
X-Mailman-Original-References: <ND8xHA---3-2@tutanota.com>
<fbd7fcc1-6095-e381-4721-bc5c09975202@mrabarnett.plus.com>
<ND9prTQ--3-2@tutanota.com-ND9qgBV----2> <ND9zJoA--7-2@tutanota.com>
<b60d1d3b-1381-8646-fd43-15496bf993dd@mrabarnett.plus.com>
<NDAPOvd--3-2@tutanota.com>
<aadf0f47-468b-158e-a0fc-82bb72fa2b98@mrabarnett.plus.com>
<4e8fa90c-6f59-1999-93bc-2a31ba14c12e@mrabarnett.plus.com>

by: Jen Kris - Fri, 30 Sep 2022 16:02 UTC

Thanks very much for your detailed reply. I have a few followup questions.

You said, “Some functions return an object that has already been incref'ed ("new reference"). This occurs when it has either created a new object (the refcount will be 1) or has returned a pointer to an existing object (the refcount will be > 1 because it has been incref'ed). Other functions return an object that hasn't been incref'ed. This occurs when you're looking up something, for example, looking at a member of a list or the value of an attribute.”

In the official docs some functions show “Return value: New reference” and others do not. Is there any reason why I should not just INCREF on every new object, regardless of whether it’s a new reference or not, and DECREF when I am finished with it? The answer at https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to says “With out-of-order execution, the INCREF/DECREF are basically free operations, so performance is no reason to leave them out.” Doing so means I don’t have to check each object to see if it needs to be INCREF’d or not, and that is a big help.

Also:

What is a borrowed reference, and how does it effect reference counting? According to https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use Py_INCREF on a borrowed PyObject pointer you already have. This increments the reference count on the object, and obligates you to dispose of it properly.” So I guess it’s yes, but I’m confused by “pointer you already have.”

What does it mean to steal a reference? If a function steals a reference does it have to decref it without incref (because it’s stolen)?

Finally, you said:

if (pMod_random == 0x0){
PyErr_Print();
Leaks here because of the refcount

Assuming pMod_random is not null, why would this leak?

Thanks again for your input on this question.

Jen

Sep 29, 2022, 17:33 by python@mrabarnett.plus.com:

> On 2022-09-30 01:02, MRAB wrote:
>
>> On 2022-09-29 23:41, Jen Kris wrote:
>>
>>>
>>> I just solved this C API problem, and I’m posting the answer to help anyone else who might need it.
>>>
> [snip]
>
> What I like to do is write comments that state which variables hold a reference, followed by '+' if it's a new reference (incref'ed) and '?' if it could be null. '+?' means that it's probably a new reference but could be null. Once I know that it's not null, I can remove the '?', and once I've decref'ed it (if required) and no longer need it, I remobe it from the comment..
>
> Clearing up references, as soon as they're not needed, helps to keep the number of current references more manageable.
>
>
> int64_t Get_LibModules(int64_t * return_array) {
> PyObject * pName_random = PyUnicode_FromString("random");
> //> pName_random+?
> if (!pName_random) {
> PyErr_Print();
> return 1;
> }
>
> //> pName_random+
> PyObject * pMod_random = PyImport_Import(pName_random);
> //> pName_random+ pMod_random+?
> Py_DECREF(pName_random);
> //> pMod_random+?
> if (!pMod_random) {
> PyErr_Print();
> return 1;
> }
>
> //> pMod_random+
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> //> pMod_random+ pAttr_seed?
> if (!pAttr_seed) {
> Py_DECREF(pMod_random);
> PyErr_Print();
> return 1;
> }
>
> //> pMod_random+ pAttr_seed
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
> //> pMod_random+ pAttr_seed pAttr_randrange?
> Py_DECREF(pMod_random);
> //> pAttr_seed pAttr_randrange?
> if (!pAttr_randrange) {
> PyErr_Print();
> return 1;
> }
>
> //> pAttr_seed pAttr_randrange
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> //> value_ptr+?
> if (!!value_ptr) {
> PyErr_Print();
> return 1;
> }
>
> //> value_ptr+
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);
> //> value_ptr+ p_seed_calc+?
> Py_DECREF(value_ptr);
> //> p_seed_calc+?
> if (!p_seed_calc) {
> PyErr_Print();
> return 1;
> }
>
> //> p_seed_calc+
> Py_DECREF(p_seed_calc);
> return 0;
> }
>
> int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> //> value_ptr+?
> if (!value_ptr) {
> PyErr_Print();
> return 1;
> }
>
> //> value_ptr+
> PyObject * p_randrange_calc = PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
> //> value_ptr+ p_randrange_calc+?
> Py_DECREF(value_ptr);
> //> p_randrange_calc+?
> if (!p_randrange_calc) {
> PyErr_Print();
> return 1;
> }
>
> //Prepare return values
> //> p_randrange_calc+
> return_val = PyLong_AsLong(p_randrange_calc);
> Py_DECREF(p_randrange_calc);
>
> return return_val;
> }
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

I just thought of something funny...your mother. -- Cheech Marin

devel / comp.lang.python / Re: PyObject_CallFunctionObjArgs segfaults

Subject	Author
Re: PyObject_CallFunctionObjArgs segfaults	Jen Kris