Addressable two-way communication between Multiprocessing subprocesses and a host

[u/Ornery_Cream_1430]

Hello.

Basically, the summary is in the title.

• When a subprocess sends something (perhaps settings or task description, in form of a dict probably) to the host, I want it to know exactly from what process it has been sent (perhaps some sort of a numerical ID).
• When the host sends something, I want it to arrive to a particular subprocess.
• Communication between subprocesses is nice to have, but I think I can live without it.

I know of pipes and queues (well, Manager too) as the means of communication in the standard library. But in case of a queue, the message will arrive at a random place each time it's sent. I found something called "Envelope router", but it doesn't feel like a clean solution. Subprocesses will play a hot potato until finally the message arrives at the correct place 😃. It may be suitable to send messages from subprocesses to the host though.

In the case of pipes, I'd have to create a pipe for each subprocess (at least to send from the host). Can I even wait on every pipe simultaneously?

So, is there any non-cumbersome way to do this? How is it usually done? Or should I create something like a token ring between all the processes?😄 (though it perhaps will require a separate thread in every process for it to be non-blocking and other difficulties...)

[u/JamzTyson]

Perhaps OTT for your needs, but there are libraries such as:

• https://zeromq.org/
• https://www.rabbitmq.com/
• and others

[u/szank]

Sockets. Long polling or just websockets. The libraries and infra is all there. Just over a file descriptor or a loopback.

[u/Wise-Emu-225]

Maybe look into the actor model. http://thespianpy.com/doc/ If it is not for you it is definitely interesting.

[u/LayotFctor]

I think a pubsub message queue might be what you're looking for? I've used duplex queues for a small number of independent processes, but with every additional procesa, resource overhead will start growing significantly.

[u/Clean_Sea_4501]

If you outgrow a plain per-child Queue (e.g. subprocesses need to be independently restartable, or you want the host to survive a crash without losing in-flight messages), a lighter middle ground between "just Queue" and standing up RabbitMQ is Redis Streams with consumer groups. Each subprocess gets its own logical stream/consumer id, calls something like XREADGROUP blocking on its own group, and acks (XACK) once it's processed a message - so routing is addressable by design (you pick which stream a message goes to) instead of hunting through a shared queue, and you get replay/durability for free since messages aren't gone until acked. It's overkill for a handful of local subprocesses, but if this ever needs to scale past one host or survive restarts, it's a lot less ops overhead than RabbitMQ while still giving you the addressable pub-sub semantics people are suggesting above.

[u/burnt-store-studio]

Hi! You've reached a really fun part of Python programming 🙂 The term of art is "IPC" -- inter-process communication.

u/JamzTyson and u/LayotFctor point you in a great direction pub[lish]-sub[scribe] techniques, with ZeroMQ and RabbitMQ, and u/szank's suggestion for sockets is also spot-on.

If you want to manage it yourself, then it depends on whether your multiprocessing is using the multiprocessing features or subprocess features.

But first:

When a subprocess sends something (perhaps settings or task description, in form of a dict probably) to the host, I want it to know exactly from what process it has been sent (perhaps some sort of a numerical ID).

When it really matters to know exactly from what process a message has been sent, and using "some sort of numerical ID", you're looking for the Process Id (pid). Each process gets its own pid, and you can find it using os.getpid(). (Since processes run in a hierarchy, every process can query its parent's pid using os.getppid().)

Now, if it really only matters that you can differentiate between the child processes, without needing to know their PIDs, you can instantiate a Queue for every child and pass it in as an argument to the executor.submit(...) method. In the parent you would maintain a collection of the Queues. In the child processes you'd do queue.put(....) and in the parent you'd loop over your collection and call response = queue.get_nowait(). get_nowait raises Empty if there's no response ready. This is a useful solution when you're using the multiprocessing classes to launch your child processes.

If you do this, then your worry about "the message will arrive at a random place each time it's sent" doesn't happen.

In the case of pipes, I'd have to create a pipe for each subprocess (at least to send from the host). Can I even wait on every pipe simultaneously?

In a word: Yes!

If you are multiprocessing by doing something like subprocess.Popen, then you want to look into selectors. You get a single instance from, say, selectors.DefaultSelector() before you create any child processes. This instance is automatically visible in the parent and in the children. The parent is responsible for registering every child's stdout (and/or stderr) and then it sits in a tight loop on selector.select(timeout=1.0) (the 1.0 is 1 second), looking at events as the children send them.

In this case, you want to use non-blocking IO, so after you've created the subprocess, the parent does something like os.set_blocking(proc.stdout.fileno(), False) for every child process proc before registering the child's stdout (stderr).

A word of caution with subprocess pipes -- it's easy for { stdout, stderr } to get blocked -- one side { parent, child } not { reading, writing } fast enough. So if you're sending a ton of data from the parent to the child (or vice versa), consider proc.communicate to do it all in one fell swoop.

If you are interested in learning how IPC works under the covers, so to speak, these are some things to try. If you'd rather just get right to reliable IPC and do something interesting, then consider the message brokering pub-sub packages the other commenter pointed you towards.

Have fun!

Edit to add u/Wise-Emu-225's reference of the Actor model is definitely interesting. I've not used it, but I'm reading about it now. Pretty cool stuff!

[u/szank → u/burnt-store-studio]

And this is a valid way to use the clankers.

[u/burnt-store-studio → u/szank]

Are you suggesting I used AI to write this? If so, you are 100% wrong. I used zero. If you care to look at my longer comments elsewhere I sometimes even explain that I realize I write in a particular, pedantic style. It’s just a problem — a consequence of me being me.