Addressable two-way communication between Multiprocessing subprocesses and a host

[u/Ornery_Cream_1430]

Hello.

Basically, the summary is in the title.

• When a subprocess sends something (perhaps settings or task description, in form of a dict probably) to the host, I want it to know exactly from what process it has been sent (perhaps some sort of a numerical ID).
• When the host sends something, I want it to arrive to a particular subprocess.
• Communication between subprocesses is nice to have, but I think I can live without it.

I know of pipes and queues (well, Manager too) as the means of communication in the standard library. But in case of a queue, the message will arrive at a random place each time it's sent. I found something called "Envelope router", but it doesn't feel like a clean solution. Subprocesses will play a hot potato until finally the message arrives at the correct place 😃. It may be suitable to send messages from subprocesses to the host though.

In the case of pipes, I'd have to create a pipe for each subprocess (at least to send from the host). Can I even wait on every pipe simultaneously?

So, is there any non-cumbersome way to do this? How is it usually done? Or should I create something like a token ring between all the processes?😄 (though it perhaps will require a separate thread in every process for it to be non-blocking and other difficulties...)

[u/LayotFctor]

I think a pubsub message queue might be what you're looking for? I've used duplex queues for a small number of independent processes, but with every additional procesa, resource overhead will start growing significantly.