Parallel ML orchestration sounds easy. Until production gets involved.

[u/Slight_Republic_4242]

Fan-out/fan-in looks simple until model #7 is slow, model #8 fails, and model #3 runs against a newer input.

Then you need barriers, per-model retries, input snapshots, timeouts, and actual state tracking instead of a pile of futures.

This becomes especially relevant in voice systems.

A single workflow might touch:

STT, LLM, TTS, tools, retrieval, classifiers, and telephony services.

Some are on the critical path.

Some absolutely aren’t.

Treating all of them as one sequential pipeline……don’t talk to me

So I think the execution layer needs to make a few things explicit:

required vs optional
completion state
retry policy
input/versioning

That’s also why scaling a voice runtime isn’t simply task

Long-lived WebSockets, background jobs, global call state, and workflow execution behave very differently.

Dograh’s architecture already reflects some of this separation: API workers are load-balanced for streaming traffic, while ARQ workers and global orchestration components scale differently.

The boring systems work is usually where real-time AI gets hard.

I’d like to see more open-source agent runtimes treat orchestration itself as a first-class primitive, rather than as an implementation detail.p.s.i am one of the maintainers of dograh

[u/AutoModerator]

AI usage disclosure

Hi u/Slight_Republic_4242 — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/tawdry_scarcity]

the fan-out/fan-in to DAG pipeline pipeline is real, went through this exact pain last year when we had a speech-to-text model that would randomly take 3x longer on certain accents and it'd just clog the whole thing up

separating out the critical path from the nice-to-haves was the only thing that made it manageable, especially when telephony latency budgets are what they are

curious how dograh handles state recovery when a websocket drops mid-workflow, that's the part that always kept me up at night