Hi there,

I'm a bit confused. In my scenario I a mixing threading with
multiprocessing. Threading by itself would be nice, but for GIL reasons
I need both, unfortunately. I've encountered a weird situation in which
multiprocessing Process()es which are started in a new thread don't
actually start and so they deadlock on join.

I've created a minimal example that demonstrates the issue. I'm running
on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
([GCC 10.3.0] on linux).

Here's the code:

import time
import multiprocessing
import threading

def myfnc():
print("myfnc")

def run(result_queue, callback):
result = callback()
result_queue.put(result)

def start(fnc):
def background_thread():
queue = multiprocessing.Queue()
proc = multiprocessing.Process(target = run, args = (queue, fnc))
proc.start()
print("join?")
proc.join()
print("joined.")
result = queue.get()
threading.Thread(target = background_thread).start()

start(myfnc)
start(myfnc)
start(myfnc)
start(myfnc)
while True:
time.sleep(1)

What you'll see is that "join?" and "joined." nondeterministically does
*not* appear in pairs. For example:

join?
join?
myfnc
myfnc
join?
join?
joined.
joined.

What's worse is that when this happens and I Ctrl-C out of Python, the
started Thread is still running in the background:

$ ps ax | grep minimal
370167 pts/0 S 0:00 python3 minimal.py
370175 pts/2 S+ 0:00 grep minimal

Can someone figure out what is going on there?

Best,
Johannes

Hi!, in short your code should work.

I think that the join-joined problem is just an interpretation problem.

In pseudo code the background_thread function does:

def background_thread()
# bla
print("join?")
# bla
print("joined")

When running this function in parallel using threads, you will probably
get a few "join?" first before receiving any "joined?". That is because
the functions are running in parallel.

The order "join?" then "joined" is preserved within a thread but not
preserved globally.

Now, I see another issue in the output (and perhaps you was asking about
this one):

join?
join?
myfnc
myfnc
join?
join?
joined.
joined.

So you have 4 "join?" that correspond to the 4 background_thread
function calls in threads but only 2 "myfnc" and 2 "joined".

Could be possible that the output is truncated by accident?

I ran the same program and I got a reasonable output (4 "join?", "myfnc"
and "joined"):

join?
join?
myfnc
join?
myfnc
join?
joined.
myfnc
joined.
joined.
myfnc
joined.

Another issue that I see is that you are not joining the threads that
you spawned (background_thread functions).

I hope that this can guide you to fix or at least narrow the issue.

Thanks,
Martin.

On Mon, Dec 06, 2021 at 12:50:11AM +0100, Johannes Bauer wrote:
>Hi there,
>
>I'm a bit confused. In my scenario I a mixing threading with
>multiprocessing. Threading by itself would be nice, but for GIL reasons
>I need both, unfortunately. I've encountered a weird situation in which
>multiprocessing Process()es which are started in a new thread don't
>actually start and so they deadlock on join.
>
>I've created a minimal example that demonstrates the issue. I'm running
>on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
>([GCC 10.3.0] on linux).
>
>Here's the code:
>
>
>import time
>import multiprocessing
>import threading
>
>def myfnc():
> print("myfnc")
>
>def run(result_queue, callback):
> result = callback()
> result_queue.put(result)
>
>def start(fnc):
> def background_thread():
> queue = multiprocessing.Queue()
> proc = multiprocessing.Process(target = run, args = (queue, fnc))
> proc.start()
> print("join?")
> proc.join()
> print("joined.")
> result = queue.get()
> threading.Thread(target = background_thread).start()
>
>start(myfnc)
>start(myfnc)
>start(myfnc)
>start(myfnc)
>while True:
> time.sleep(1)
>
>
>What you'll see is that "join?" and "joined." nondeterministically does
>*not* appear in pairs. For example:
>
>join?
>join?
>myfnc
>myfnc
>join?
>join?
>joined.
>joined.
>
>What's worse is that when this happens and I Ctrl-C out of Python, the
>started Thread is still running in the background:
>
>$ ps ax | grep minimal
> 370167 pts/0 S 0:00 python3 minimal.py
> 370175 pts/2 S+ 0:00 grep minimal
>
>Can someone figure out what is going on there?
>
>Best,
>Johannes
>--
>https://mail.python.org/mailman/listinfo/python-list

Am 06.12.21 um 13:56 schrieb Martin Di Paola:
> Hi!, in short your code should work.
>
> I think that the join-joined problem is just an interpretation problem.
>
> In pseudo code the background_thread function does:
>
> def background_thread()
>   # bla
>   print("join?")
>   # bla
>   print("joined")
>
> When running this function in parallel using threads, you will probably
> get a few "join?" first before receiving any "joined?". That is because
> the functions are running in parallel.
>
> The order "join?" then "joined" is preserved within a thread but not
> preserved globally.

Yes, completely understood and really not the issue. That these pairs
are not in sequence is fine.

> Now, I see another issue in the output (and perhaps you was asking about
> this one):
>
> join?
> join?
> myfnc
> myfnc
> join?
> join?
> joined.
> joined.
>
> So you have 4 "join?" that correspond to the 4 background_thread
> function calls in threads but only 2 "myfnc" and 2 "joined".

Exactly that is the issue. Then it hangs. Deadlocked.

> Could be possible that the output is truncated by accident?

No. This is it. The exact output varies, but when it hangs, it always
also does not execute the function (note the lack of "myfnc"). For example:

join?
join?
myfnc
join?
myfnc
join?
myfnc
joined.
joined.
joined.

(only three threads get started there)

join?
myfnc
join?
join?
join?
joined.

(this time only a single one made it)

join?
join?
join?
myfnc
join?
myfnc
joined.
myfnc
joined.
joined.

(three get started)

> I ran the same program and I got a reasonable output (4 "join?", "myfnc"
> and "joined"):
>
> join?
> join?
> myfnc
> join?
> myfnc
> join?
> joined.
> myfnc
> joined.
> joined.
> myfnc
> joined.

This happens to me occasionally, but most of the time one of the
processes deadlocks. Did you consistently get four of each? What
OS/Python version were you using?

> Another issue that I see is that you are not joining the threads that
> you spawned (background_thread functions).

True, I kindof assumed those would be detached threads.

> I hope that this can guide you to fix or at least narrow the issue.

Depending on what OS/Python version you're using, that points in that
direction and kindof reinforces my belief that the code is correct.

Very curious.

Thanks & all the best,
Joe

> On 5 Dec 2021, at 23:50, Johannes Bauer <dfnsonfsduifb@gmx.de> wrote:
>
> Hi there,
>
> I'm a bit confused. In my scenario I a mixing threading with
> multiprocessing. Threading by itself would be nice, but for GIL reasons
> I need both, unfortunately. I've encountered a weird situation in which
> multiprocessing Process()es which are started in a new thread don't
> actually start and so they deadlock on join.
>
> I've created a minimal example that demonstrates the issue. I'm running
> on x86_64 Linux using Python 3.9.5 (default, May 11 2021, 08:20:37)
> ([GCC 10.3.0] on linux).
>
> Here's the code:

I suggest that you include the threading.current_thread() in your messages.
Then you can see which thread is saying what.

Barry

>
>
> import time
> import multiprocessing
> import threading
>
> def myfnc():
> print("myfnc")
>
> def run(result_queue, callback):
> result = callback()
> result_queue.put(result)
>
> def start(fnc):
> def background_thread():
> queue = multiprocessing.Queue()
> proc = multiprocessing.Process(target = run, args = (queue, fnc))
> proc.start()
> print("join?")
> proc.join()
> print("joined.")
> result = queue.get()
> threading.Thread(target = background_thread).start()
>
> start(myfnc)
> start(myfnc)
> start(myfnc)
> start(myfnc)
> while True:
> time.sleep(1)
>
>
> What you'll see is that "join?" and "joined." nondeterministically does
> *not* appear in pairs. For example:
>
> join?
> join?
> myfnc
> myfnc
> join?
> join?
> joined.
> joined.
>
> What's worse is that when this happens and I Ctrl-C out of Python, the
> started Thread is still running in the background:
>
> $ ps ax | grep minimal
> 370167 pts/0 S 0:00 python3 minimal.py
> 370175 pts/2 S+ 0:00 grep minimal
>
> Can someone figure out what is going on there?
>
> Best,
> Johannes
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Johannes Bauer wrote at 2021-12-6 00:50 +0100:
>I'm a bit confused. In my scenario I a mixing threading with
>multiprocessing. Threading by itself would be nice, but for GIL reasons
>I need both, unfortunately. I've encountered a weird situation in which
>multiprocessing Process()es which are started in a new thread don't
>actually start and so they deadlock on join.

The `multiprocessing` doc
(--> "https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing")
has the following warning:
"Note that safely forking a multithreaded process is problematic."

Thus, if you use the `fork` method to start processes, some
surprises are to be expected.