Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

Why are there always boycotts? Shouldn't there be girlcotts too? -- argon on #Linux

devel / comp.lang.python / Re: Embedding Python crash on PyTuple_New

SubjectAuthor
o Re: Embedding Python crash on PyTuple_NewArnaud Loonstra

1
Re: Embedding Python crash on PyTuple_New

<mailman.0.1637740758.9869.python-list@python.org>

  copy mid

https://www.novabbs.com/devel/article-flat.php?id=16185&group=comp.lang.python#16185

  copy link   Newsgroups: comp.lang.python
Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: arn...@sphaero.org (Arnaud Loonstra)
Newsgroups: comp.lang.python
Subject: Re: Embedding Python crash on PyTuple_New
Date: Wed, 24 Nov 2021 08:59:15 +0100
Lines: 101
Message-ID: <mailman.0.1637740758.9869.python-list@python.org>
References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
<4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
<c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
<d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
<f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
<8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
<1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
<b575c5da-7354-57e5-039b-3b1c05861ffa@sphaero.org>
<d341afa3-b3ae-ee2e-25e9-ed76a0273704@mrabarnett.plus.com>
<19876e9f-3b51-851e-9339-e36fb35028c1@sphaero.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Trace: news.uni-berlin.de t3+Xw1f1Rh817KwpWTcd4wsARPAYH6p6u9i788Uk+ZNw==
Return-Path: <arnaud@sphaero.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.004
X-Spam-Evidence: '*H*': 0.99; '*S*': 0.00; 'url-ip:140.82/16': 0.03;
'debug': 0.07; 'debugging': 0.07; 'acquired': 0.09; 'allocation':
0.09; 'skip:_ 20': 0.09; 'skip:z 20': 0.09; 'subject:Python':
0.12; 'url:github': 0.14; 'url-ip:140/8': 0.15; '"after': 0.16;
'1240': 0.16; '1262': 0.16; '1977': 0.16; '1987': 0.16;
'8bit%:55': 0.16; '>>>>': 0.16; '>>>>>': 0.16; '>>>>>>': 0.16;
'>>>>>>>': 0.16; '>>>>>>>>': 0.16; '>>>>>>>>>': 0.16; '[snip]':
0.16; 'arguments': 0.16; 'char': 0.16; 'compiled': 0.16;
'cpython': 0.16; 'cpython.': 0.16; 'intuition': 0.16;
'received:194.109': 0.16; 'received:194.109.24': 0.16;
'received:xs4all.net': 0.16; 'right.': 0.16; 'sole': 0.16;
'static': 0.16; 'tuple': 0.16; 'wrote:': 0.16; 'python': 0.16;
'to:addr:python-list': 0.20; "i've": 0.22; 'returns': 0.22;
'skip:p 30': 0.23; 'skip:- 10': 0.25; 'stuff': 0.25; 'practices':
0.26; 'creating': 0.27; 'else': 0.27; 'function': 0.27; '>>>':
0.28; 'purpose': 0.28; 'sense': 0.28; 'wrong': 0.28; 'this?':
0.29; 'header:User-Agent:1': 0.30; 'program': 0.31; 'think': 0.32;
"doesn't": 0.32; 'guess': 0.32; 'unless': 0.32; 'but': 0.32;
"i'm": 0.33; 'appreciated.': 0.34; 'header:In-Reply-To:1': 0.34;
'running': 0.34; 'trying': 0.35; 'yes,': 0.35; 'following': 0.35;
'those': 0.36; 'currently': 0.37; 'really': 0.37; "it's": 0.37;
'hard': 0.37; 'skip:_ 30': 0.37; 'this.': 0.37;
'received:192.168': 0.37; 'could': 0.38; 'thanks': 0.38;
'finding': 0.39; '8bit%:71': 0.40; 'something': 0.40; '8bit%:32':
0.60; 'best': 0.61; 'here.': 0.61; 'skip:z 10': 0.62; 'here':
0.62; 'skip:o 20': 0.63; '8bit%:9': 0.64; 'skip:r 20': 0.64;
'skip:v 30': 0.64; 'your': 0.64; 'came': 0.65; 'bad': 0.67;
'skip:t 30': 0.67; '8bit%:63': 0.69; 'times': 0.69; '2016': 0.70;
'received:194': 0.70; 'skip:f 30': 0.71; '8bit%:31': 0.73;
'limits': 0.76; '8bit%:67': 0.81; 'reference.': 0.81; 'from.':
0.84; 'pointer': 0.84; 'surprised': 0.84; 'weird': 0.84;
'\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0\xc2\xa0': 0.84; 'caused': 0.86
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
Thunderbird/78.14.0
In-Reply-To: <d341afa3-b3ae-ee2e-25e9-ed76a0273704@mrabarnett.plus.com>
Content-Language: en-US
X-CMAE-Envelope: MS4xfG2cbTqI7kYQN+wr2LyiW8p6eWLUrs0E9Z48ljUcdMJgCkm3flf3oZgxHP7BgAxPW0hsT9U0xLS5oR+1HWY4ID7sYoSbx0/lvrV4P803I2l+kE17dFsp
Ecmuk9KsSe393The7LjX1t2oR1utJxWhipLYX/7PshqRzx0T52YDsaiWmv3JMkNhwN8r/6mVl8FE+zC69Hy6aw6h/6di4rh1Q5jeKTUQbKACFvDE0aeAATvJ
mOYaP6sAQZH5/9khNO/iFQ==
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.37
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <19876e9f-3b51-851e-9339-e36fb35028c1@sphaero.org>
X-Mailman-Original-References: <d0b57569-ef0d-3ab4-9918-9cd498f1e658@sphaero.org>
<4eb693c6-fd5f-7c54-246d-0b081a0488d5@mrabarnett.plus.com>
<c827faf9-0871-fde3-83a0-fa19576b3c62@sphaero.org>
<d6c7a17a-b24c-9a45-a94e-7bbc3d8fb1c1@mrabarnett.plus.com>
<f7711e63-a9a7-eb7a-b2b1-0c7ebd9829af@mrabarnett.plus.com>
<8788bcb1-5e02-b241-3533-43c40d363906@sphaero.org>
<1d070d88-8b19-c1ea-c700-4a47876fe871@mrabarnett.plus.com>
<b575c5da-7354-57e5-039b-3b1c05861ffa@sphaero.org>
<d341afa3-b3ae-ee2e-25e9-ed76a0273704@mrabarnett.plus.com>
 by: Arnaud Loonstra - Wed, 24 Nov 2021 07:59 UTC

On 24-11-2021 01:46, MRAB wrote:
> On 2021-11-23 20:25, Arnaud Loonstra wrote:
>> On 23-11-2021 18:31, MRAB wrote:
>>> On 2021-11-23 16:04, Arnaud Loonstra wrote:
>>>> On 23-11-2021 16:37, MRAB wrote:
>>>>> On 2021-11-23 15:17, MRAB wrote:
>>>>>> On 2021-11-23 14:44, Arnaud Loonstra wrote:
>>>>>>> On 23-11-2021 15:34, MRAB wrote:
>>>>>>>> On 2021-11-23 12:07, Arnaud Loonstra wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've got Python embedded successfully in a program up until now
>>>>>>>>> as I'm
>>>>>>>>> now running into weird GC related segfaults. I'm currently
>>>>>>>>> trying to
>>>>>>>>> debug this but my understanding of CPython limits me here.
>>>>>>>>>
>>>>>>>>> I'm creating a Tuple in C but it crashes on creating it after a
>>>>>>>>> while.
>>>>>>>>> It doesn't make sense which makes me wonder something else must be
>>>>>>>>> happening? Could be it just crashes here because the GC is
>>>>>>>>> cleaning up
>>>>>>>>> stuff completely unrelated to the allocation of the new tuple?
>>>>>>>>> How can I
>>>>>>>>> troubleshoot this?
>>>>>>>>>
>>>>>>>>> I've got CPython compiled with  --with-valgrind --without-pymalloc
>>>>>>>>> --with-pydebug
>>>>>>>>>
>>>>>>>>> In C I'm creating a tuple with the following method:
>>>>>>>>>
>>>>>>>>> static PyObject *
>>>>>>>>> s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
>>>>>>>>> {
>>>>>>>>>       assert(self);
>>>>>>>>>       assert(oscmsg);
>>>>>>>>>       char *format = zosc_format(oscmsg);
>>>>>>>>>
>>>>>>>>>       PyObject *rettuple = PyTuple_New((Py_ssize_t)
>>>>>>>>> strlen(format) );
>>>>>>>>>
>>>>>>>>> It segfaults here (frame 16) after 320 times (consistently)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1   __GI_raise             raise.c          49   0x7ffff72c4e71
>>>>>>>>> 2   __GI_abort             abort.c          79   0x7ffff72ae536
>>>>>>>>> 3   fatal_error            pylifecycle.c    2183 0x7ffff7d84b4f
>>>>>>>>> 4   Py_FatalError          pylifecycle.c    2193 0x7ffff7d878b2
>>>>>>>>> 5   _PyObject_AssertFailed object.c         2200 0x7ffff7c93cf2
>>>>>>>>> 6   visit_decref           gcmodule.c       378  0x7ffff7dadfd5
>>>>>>>>> 7   tupletraverse          tupleobject.c    623  0x7ffff7ca3e81
>>>>>>>>> 8   subtract_refs          gcmodule.c       406  0x7ffff7dad340
>>>>>>>>> 9   collect                gcmodule.c       1054 0x7ffff7dae838
>>>>>>>>> 10  collect_with_callback  gcmodule.c       1240 0x7ffff7daf17b
>>>>>>>>> 11  collect_generations    gcmodule.c       1262 0x7ffff7daf3f6
>>>>>>>>> 12  _PyObject_GC_Alloc     gcmodule.c       1977 0x7ffff7daf4f2
>>>>>>>>> 13  _PyObject_GC_Malloc    gcmodule.c       1987 0x7ffff7dafebc
>>>>>>>>> 14  _PyObject_GC_NewVar    gcmodule.c       2016 0x7ffff7daffa5
>>>>>>>>> 15  PyTuple_New            tupleobject.c    118  0x7ffff7ca4da7
>>>>>>>>> 16  s_py_zosc_tuple        pythonactor.c    366  0x55555568cc82
>>>>>>>>> 17  pythonactor_socket     pythonactor.c    664  0x55555568dac7
>>>>>>>>> 18  pythonactor_handle_msg pythonactor.c    862  0x55555568e472
>>>>>>>>> 19  pythonactor_handler    pythonactor.c    828  0x55555568e2e2
>>>>>>>>> 20  sphactor_actor_run     sphactor_actor.c 855  0x5555558cb268
>>>>>>>>> ... <More>
>>>>>>>>>
>>>>>>>>> Any pointer really appreciated.

[snip]

>>>>
>>> Basically, yes, but I won't be surprised if it was due to too few
>>> INCREFs or too many DECREFs somewhere.
>>>
>>>> https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286
>>>>
>>>>
>>> Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);"
>>> after "after zosc_pop_float" or "zosc_pop_double".
>>
>> Thanks for those pointers! I think your intuition is right. I might have
>> found the bugger. In s_py_zosc I call Py_DECREF on pAddress and pData.
>> However they are acquired by PyTuple_GetItem which returns a borrowed
>> reference. I think pAddress and pData are then also 'decrefed' when the
>> pReturn tuple which contains pAddress and pData is 'decrefed'?
>>
> Yes, members of a container are DECREFed when the container is destroyed.
>
> It's bad practice for a function to DECREF its arguments unless the
> function's sole purpose is cleanup because the function won't know where
> the arguments came from.
>

I'm finding it out now. What strikes me was how hard it was to debug
this. I think it was caused because I INCREFed the return object. I
guess I did that to workaround the wrong DECREF data in the return
object. However that caused a hell to debug. I'm really curious what the
best practices are for debugging embedded CPython.

Thanks big time for your feedback!

1
server_pubkey.txt

rocksolid light 0.9.81
clearnet tor