Message-ID:

Delta: We never make the same mistake three times. -- David Letterman

devel / comp.lang.python / Re: Data unchanged when passing data to Python in multiprocessing shared memory

Re: Data unchanged when passing data to Python in multiprocessing shared memory

<mailman.12.1643806350.2976.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=16846&group=comp.lang.python#16846

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: bar...@barrys-emacs.org (Barry Scott)
Newsgroups: comp.lang.python
Subject: Re: Data unchanged when passing data to Python in multiprocessing
shared memory
Date: Wed, 2 Feb 2022 12:52:22 +0000
Lines: 105
Message-ID: <mailman.12.1643806350.2976.python-list@python.org>
References: <Muqv9er--3-2@tutanota.com>
<35BA674C-1AAF-43B3-9203-BBF7472E6AE4@barrys-emacs.org>
<MureHAB----2@tutanota.com>
<B1B7170F-7A66-45F9-8B4A-E39078D3BE10@barrys-emacs.org>
Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.60.0.1.1\))
Content-Type: text/plain;
charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Trace: news.uni-berlin.de d0/KxWCyQpqPaCTCxMaHMwstxqtQrEAMQx385NY3Uf1Q==
Return-Path: <barry@barrys-emacs.org>
X-Original-To: python-list@python.org
Delivered-To: python-list@mail.python.org
Authentication-Results: mail.python.org; dkim=none reason="no signature";
dkim-adsp=none (unprotected policy); dkim-atps=neutral
X-Spam-Status: OK 0.001
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'python?': 0.03; 'knows':
0.04; 'python:': 0.05; 'subject:when': 0.07; 'python.': 0.08;
'cc:addr:python-list': 0.09; 'from:addr:barry': 0.09; 'memory.':
0.09; 'received:217.70': 0.09; 'received:217.70.178': 0.09;
'received:217.70.178.230': 0.09; 'received:gandi.net': 0.09;
'received:mail.gandi.net': 0.09;
'received:relay10.mail.gandi.net': 0.09; 'theory': 0.09;
'subject:Python': 0.12; 'url:mailman': 0.15; 'memory': 0.15;
'2022,': 0.16; 'barry': 0.16; 'cc:name:python list': 0.16; 'char':
0.16; 'from:addr:barrys-emacs.org': 0.16; 'from:name:barry scott':
0.16; 'integer': 0.16; 'language?': 0.16; 'message-id:@barrys-
emacs.org': 0.16; 'pipe': 0.16; 'protocol.': 0.16;
'subject:shared': 0.16; 'subject:skip:m 10': 0.16; 'sync': 0.16;
'unsigned': 0.16; 'want,': 0.16; 'wrote:': 0.16; 'problem': 0.16;
'python': 0.16; 'feb': 0.17; "can't": 0.17; 'uses': 0.19;
'cc:addr:python.org': 0.20; 'language': 0.21; 'maybe': 0.22;
'code': 0.23; '(and': 0.25; 'url-ip:188.166.95.178/32': 0.25;
'url-ip:188.166.95/24': 0.25; 'url:listinfo': 0.25; 'cc:2**0':
0.25; 'url-ip:188.166/16': 0.25; 'anyone': 0.25; 'binary': 0.26;
'email addr:python.org>': 0.28; 'module': 0.31; 'url-ip:188/8':
0.31; 'program': 0.31; 'question': 0.32; 'attaching': 0.32; 'do.':
0.32; 'python-list': 0.32; 'but': 0.32; 'header:In-Reply-To:1':
0.34; 'following': 0.35; 'thanks,': 0.36; 'people': 0.36; 'using':
0.37; "it's": 0.37; 'this.': 0.37; 'read': 0.38; 'thanks': 0.38;
'text': 0.39; 'list': 0.39; 'use': 0.39; 'shared': 0.39; 'data.':
0.40; 'both': 0.40; 'four': 0.60; 'seen': 0.62; 'skip:i 20': 0.62;
'send': 0.63; 'skip:m 20': 0.63; 'between': 0.63; 'pass': 0.64;
'verify': 0.64; 'clear': 0.64; 'lock': 0.64; 'your': 0.64;
'&': 0.65; 'let': 0.66; 'back': 0.67; 'received:217': 0.67;
'exactly': 0.68; 'know.': 0.68; 'refers': 0.69; 'subject:Data':
0.71; 'little': 0.73; 'points': 0.84; 'email name:<python-
list': 0.84; 'naturally': 0.84; 'pointer': 0.84; 'subject: \n ':
0.84; 'subject:memory': 0.84; 'want.': 0.84
In-Reply-To: <MureHAB----2@tutanota.com>
X-Mailer: Apple Mail (2.3693.60.0.1.1)
X-Content-Filtered-By: Mailman/MimeDel 2.1.39
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <B1B7170F-7A66-45F9-8B4A-E39078D3BE10@barrys-emacs.org>
X-Mailman-Original-References: <Muqv9er--3-2@tutanota.com>
<35BA674C-1AAF-43B3-9203-BBF7472E6AE4@barrys-emacs.org>
<MureHAB----2@tutanota.com>

by: Barry Scott - Wed, 2 Feb 2022 12:52 UTC

> On 1 Feb 2022, at 23:40, Jen Kris <jenkris@tutanota.com> wrote:
>
> Barry, thanks for your reply.
>
> On the theory that it is not yet possible to pass data from a non-Python language to Python with multiprocessing.shared_memory, I bypassed the problem by attaching 4 bytes to my FIFO pipe message from NASM to Python:
>
> byte_val = v[10:14]
>
> where v is the message read from the FIFO. Then:
>
> breakup = int.from_bytes(byte_val, "big")
> print("this is breakup " + str(breakup))
>
> Python prints: this is breakup 32894
>
> Note that I had to switch from little endian to big endian. Python is little endian by default, but in this case it's big endian.
>
> However, if anyone on this list knows how to pass data from a non-Python language to Python in multiprocessing.shared_memory please let me (and the list) know.

By using the struct module you can control for endian of the data.

Barry

>
> Thanks.
>
>
> Feb 1, 2022, 14:20 by barry@barrys-emacs.org:
>
> On 1 Feb 2022, at 20:26, Jen Kris via Python-list <python-list@python.org> wrote:
>
> I am using multiprocesssing.shared_memory to pass data between NASM and Python. The shared memory is created in NASM before Python is called. Python connects to the shm: shm_00 = shared_memory.SharedMemory(name='shm_object_00',create=False).
>
> I have used shared memory at other points in this project to pass text data from Python back to NASM with no problems. But now this time I need to pass a 32-bit integer (specifically 32,894) from NASM to Python.
>
> First I convert the integer to bytes in a C program linked into NASM:
>
> unsigned char bytes[4]
> unsigned long int_to_convert = 32894;
>
> bytes[0] = (int_to_convert >> 24) & 0xFF;
> bytes[1] = (int_to_convert >> 16) & 0xFF;
> bytes[2] = (int_to_convert >> 8) & 0xFF;
> bytes[3] = int_to_convert & 0xFF;
> memcpy(outbuf, bytes, 4);
>
> where outbuf is a pointer to the shared memory. On return from C to NASM, I verify that the first four bytes of the shared memory contain what I want, and they are 0, 0, -128, 126 which is binary 00000000 00000000 10000000 01111110, and that's correct (32,894).
>
> Next I send a message to Python through a FIFO to read the data from shared memory. Python uses the following code to read the first four bytes of the shared memory:
>
> byte_val = shm_00.buf[:4]
> print(shm_00.buf[0])
> print(shm_00.buf[1])
> print(shm_00.buf[2])
> print(shm_00.buf[3])
>
> But the bytes show as 40 39 96 96, which is exactly what the first four bytes of this shared memory contained before I called C to overwrite them with the bytes 0, 0, -128, 126. So Python does not see the updated bytes, and naturally int.from_bytes(byte_val, "little") does not return the result I want.
>
> I know that Python refers to shm00.buf, using the buffer protocol. Is that the reason that Python can't see the data that has been updated by another language?
>
> So my question is, how can I alter the data in shared memory in a non-Python language to pass back to Python?
>
> Maybe you need to use a memory barrier to force the data to be seen by another cpu?
> Maybe use shm lock operation to sync both sides?
> Googling I see people talking about using stdatomic.h for this.
>
> But I am far from clear what you would need to do.
>
> Barry
>
> Thanks,
>
> Jen
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Subject	Author
Re: Data unchanged when passing data to Python in multiprocessing	Barry Scott