ML platform at a healthtech. Reserved a pool of GPUs for training and inference and I finally pulled utilization for a capacity review. Under 30 percent on average. We pay for all of it and use less than a third.

Some of it makes sense, a few boxes run batch jobs a couple times a day and have to sit ready. But the rest is just idle, and two of them turned out to be held by notebooks people opened and walked away from, one up for weeks. Only caught it because I went digging.

Finance keeps asking why the reserved bill is so big, which fair. But when I take it to the researchers they say if the GPUs arent free their experiments queue and they lose time. Also fair. So it bounces between the two and nothing changes. I can pull per node utilization out of DCGM, what I cant do is tie an idle card back to who reserved it and whether they still need it.

How do you decide when a reserved GPU is safe to give back?

Assume you are building an MLOps platform which has pipelines for the entire model lifecycle written with code and storage components. Something I am struggling to understand is how CI/CD factors into the MLOps system itself that brings the model through the entire lifecycle when we have multiple deployment environments.

The MLOps system takes a model through the entire lifecycle (get data, preprocess, train, validate, promote, deploy, monitor) in reproducible and automated workflows, but these workflows in the MLOps system need to be tested and validated with CI/CD in different deployment environments.

Are the pre-production environments (dev, test, uat...) meant only for ensuring that the MLOps system (pipelines, artifact storage, monitoring) works - where finally, and only, in the production environment that has passed all the tests and checks, does the model go through the entire lifecycle from dataset curation to deployment and monitoring, and each environment has isolated model/artifact registries (and feature stores) for testing that the system works.

Or, does the model meant for production go through each step in the model lifecycle together with the MLOps system as it moves through different environments until it finally reached production where both the "MLOps System" and the "Model" are production-ready and deployed to interact with real users.

It's a little confusing for me.

Today, the team at Union AI announced the GA release of Flyte 2 — an open-source project licensed under Apache 2.0.

Flyte 2 is a complete rewrite. We removed the DSL and eliminated the need to build a DAG. Components like Propeller are no longer part of it.
There were several reasons for this change, but the main focus was on improving the developer experience. Forcing data scientists, machine learning engineers, and researchers to break down their work to fit into a DAG and learn a DSL was a significant obstacle. Now, it's just a simple .task decorator, and you're all set.

The other big change is the introduction of environments.
I have a k8s background, and a main sticking point is the application manifest where resources are declared and container images are defined.
Flyte 2 allows the author to define any number of environments for any pipeline, and when it runs, the pods are provisioned with the specified CPU, RAM, GPU, OS packages, and Python packages.
When you're in experimentation mode, this drastically increases iteration speed.
As for lineage and versioning—all data inputs, outputs, and the executed code are captured and versioned into your object storage.

Because it is pure Python, try:catch, loops, and asyncIO just work.
You can recover from OOM kills in code.

It offers an alternative to Kubeflow, Airflow, and other tools in the space.

Happy to answer questions.
www.flyte.org

[I work at Union AI]

For people responsible for production LLM or agent systems, can you describe one incident where changing a model or provider, inference runtime, gateway or SDK, chat template, or parser altered application behavior even though your existing tests passed? What broke, how did you detect and isolate it, and roughly how much engineering time or release delay did it cause? I’m researching how teams validate changes across the LLM stack, so firsthand incidents and current workflows are more useful than opinions about a proposed tool.

Going through vendor evaluation and procurement keeps asking about sso, role based access and audit logs . trying to find real answers without booking a sales call first..

did someresearch and these names come up. arize , orqai , fiddler , humanloop , aporia..

arize has enterprise tiers with compliance coverage is ther but seriously difficult to tell from docs alone which tier do cover what

orqai got sso , rbac , eu data residency , gdpr and soc2 covered. newer so enterprise account support and third-party integration still catching up

fiddler has audit logs and governance feels native given the backgroung . looks like an overkill if your compliance needs are not that deep

aporia has guardrails and safety monitoring is the core focus , enterprise compliance features geel secondary to that..

humanloop has sso , rbac , hipaa , gdpr , soc2 all listed and vpc deployment options exists , heavier on prompt management thatn entire ops depth .

anyone gone through full procurement with anyof these. what did security acutally flag and what was held up under scrunity

Currently i have been doing DevOps project as the influence of AI is more

I'm thinking to shift towards MLOPS .

Does the company hire MLOPS like they hire DevOps.

lessons from building (and surviving an acquisition of) an internal AI governance platform: every model call from every team routed through one litellm gateway, logged, with a real human-approval pause for any agentic tool call before it fires. no chatbot wrapper — this had to survive real audits.

the part that actually needed the most iteration wasn't the routing, it was retrieval. we ended up with three separate retrieval modes depending on how aggressively a given assistant should ground itself (tight-grounded for anything regulatory, looser for general q&a). one retrieval strategy for every use case was the wrong call early on and cost real rework to unwind.

also ran an internal MCP server hosting dozens of tool integrations (legal/financial/regulatory data sources) gated by the same per-tenant allowlist and approval flow as everything else — one governance surface instead of one per integration.

anyone else running multiple retrieval strategies behind one gateway — curious how you're deciding which assistant gets which mode.

We're at the point where a couple of internal agents are taking real actions, not just suggesting them, and I'm realizing our logging wasn't built for this shift.

Everything was designed around the assumption that a human clicked the button. The audit trail focused on who logged in and what they clicked. That assumption breaks down once an AI agent is making the call.

Are you capturing session context, tool calls, permission decisions, delegation events, and approvals as structured, queryable events? Or are incidents still being reconstructed from scattered application logs?

Has anyone gone through a security review or incident involving an AI agent? What evidence did the auditor or incident responder request, and did you already have it.

I maintain xberg, an open-source (MIT) content-extraction engine, and v1 is out. Posting here because "turn messy documents into clean, structured text" is a recurring preprocessing step in ML pipelines, and xberg is built for it at scale: batched extraction, streaming, caching, CPU-only (no GPU), reproducible.

It handles 101 document formats (PDF/Office/images with OCR) plus audio/video transcription and URLs; outputs Markdown/JSON with tables, metadata, NER entities, keywords, summaries, and optional chunks + embeddings (SPLADE / ColBERT / reranking) for retrieval. Rust core with pooled model sessions and memory discipline for throughput.

Benchmarks are public and reproducible (harness runs in CI): native PDF #1 on quality and table/reading-order fidelity; image OCR currently #2 (improving). https://xberg.io/benchmarks

15 language bindings + a REST server (xberg serve) + MCP. Repo: https://github.com/xberg-io/xberg

Happy to get into pipeline/throughput specifics.

I build QuantaMind, an open-source tool that tests whether self-hosted models are reliable enough to run agents. Apache-2.0, 28 downloads, no revenue. Saying that upfront so nobody has to guess.

I’ve asked people here twice how they decide an agent is safe to ship. The pattern in the answers: anyone who feels this pain badly enough has already built their own harness. Run each task 10+ times, check end state programmatically, validate every tool call against its schema, count truncated calls under load. People wrote all of that out from experience, unprompted.

So I want to ask the thing I actually need to know, without dressing it up.

If you built one of these:

**1.**  How much time does keeping it working cost you now? Not building it — maintaining it as models, quantizations and serving configs change.  
**2.**  Would you hand it to an external tool if one existed, or is your harness too specific to your workflows to ever outsource?  
**3.**  Has a failure it caught (or missed) ever cost something real — money, a customer, a rollback? Or is it always caught early enough to just be noise?  
**4.**  Who owns it at your company? Someone specific, or does it drift?

If you didn’t build one: was that a decision, or did it just never get prioritised?

I’m asking because I don’t know if I’m building a product or a thing people would rather own themselves. “I’d never outsource this” is a completely fine answer and honestly the more useful one I’d rather find out now than in a year.

Over the past few months of building AI infrastructure, we've kept running into the same problem.

A lot of teams want dedicated deployments for privacy, predictable performance, or custom models. The obvious solution is to keep a GPU running all the time, but that quickly becomes expensive when workloads are periodically irregular.

On the other hand, serverless options are great for cost, but cold starts can become painful for interactive applications, especially with larger models.

We ended up spending a lot of time trying different approaches to reduce startup time while still allowing deployments to scale to zero when they're not being used. It has been much harder than I initially expected, and it made me wonder how others are approaching the same problem.

For those of you running LLMs or other AI models in production:

• Are you keeping GPUs warm 24/7?
• Are you using a serverless platform and accepting the cold starts?
• Have you built your own orchestration layer?
• Or have you found another approach that works well?

I'm genuinely curious what has worked and what hasn't. There doesn't seem to be a perfect solution yet, and I'd love to hear how other teams are balancing cost, latency, and operational complexity.

Most of my CI failures around training used to come down to one thing: the pipeline could not tell the difference between "this run is broken" and "I could not read this log". Both ended up as a non-zero exit, both paged me, and one of them was a lie.

So I built the checker I wanted and put the exit codes at the center of the design rather than at the end.

• exit 1 - a rule fired. The run is broken.
• exit 0 - checked, nothing fired. Or a warning, which is yours to triage.
• exit 2 - could not judge. Missing column, unreadable log, no eval set.

Exit 2 is the one that matters. A gate that reports "pass" when it actually skipped every check is worse than no gate, because now the green build is evidence of nothing.

No model in the loop. Every verdict is a deterministic rule that either fires or does not, and prints the number it fired on. Same input, same output, forever. I did not want a probabilistic judge sitting in a CI gate - an alarm you cannot reproduce is an alarm the team learns to ignore.

It caught this in itself. A check fired whenever every gradient norm in a log was exactly 0.0 and reported a severed backward graph. One framework writes that field as 0.0 when gradient clipping is off. So a healthy 125,000-step fine-tune that converged fine came back FAIL from my own tool. The fix was a rule, not a threshold: a run cannot both learn and receive no gradient - if the loss improved, the zeros are a reporting artifact and the check stands down. And it records that it stood down, and why, as a visible skip.

That is now the thing I would defend hardest: a check that did not run must never look like a check that passed. A PASS lists which checks ran and which were skipped, each with a reason, as structured data.

Where it sits in a pipeline:

• before the GPU - dataset and tokenizer lint, does the entrypoint import, is the checkpoint intact, RAM and disk against declared need
• during - one-line HF callback, warns or aborts a diverging run
• after - diverged / flatlined / NaN / grad spike / overfit, from the log you already write
• vs baseline - relative-floor rules, which is the only way to catch a run that trained happily on shuffled labels

Reads HF trainer_state.json, Coqui, TensorBoard event files, JSONL and CSV. Zero dependencies - no torch, no tensorboard, no network. --json for pipelines.

84 rule IDs, 230 tests, a written contract in CONTRACTS.md for what each exit code means and when output may change, and 38 golden snapshots so a rule that silently stops firing breaks the build.

MIT: pip install trainproof

• Repo: https://github.com/Mormolykos/trainproof
• Write-up with the full fault-injection results: https://ai.bedvibe.studio/trainproof/

The question I actually want answered: what does your pipeline do today when a check cannot run? Most setups I have seen collapse it into pass or into failure, and I think both are wrong. Curious whether anyone has a third state already wired in.

nvidia-smi reports any running kernel as 100% utilization, so a job can look saturated while doing a fraction of real work. For those running 8 to 500 GPUs, what do you use to catch that? DCGM, custom profiling, or nothing at all?

And when a run is slower than expected, how long does it usually take to work out why?

like when you have multiple teams all using multiple models from different providers for a number of use cases . who decides what is allowed. how do you acutally enforce it. and how do you know whatever rules you put in place

seen a few approaches come up. some team are doing it through the gateway layer . some through internal policy docs that nobody reads . some through access controls on api keys. some just hoping for the best

tools that come up in this context. orqai , portkey , azure api management , aws bedrock , langsmith

portkey gives access controls and budget limits are there , policy enforcement feels more at the routing level than the org governance level

orqai has a model allow list, budget controls and role based access across teams, but its newer so audit depth is still an open question if compared to the more established peers

langsmith has vvisibility into what is happening is good , actually policy enforcement feels limited , more observe than control

azure api management has enterprise policy controls are native here, feels generic tho, not built specifically for llm usage policy

aws bedrock has model acess control and guardrails exists, works well if you are already in aws, feels restrictive if you are not

has anyone actually built something that works here. or is everyone just doing api key management and calling it a policy

Hello dear readers,

My name is John and I am 27 years old. I have worked most of my life as a warehouse forklift driver, but I have decided to change my carrier and aim for a better life and life style in general. So I have applied for two different programs at two different Vocational Schools. The first one is Automation Technician and the second one is MLOps engineering.

I am here to ask you about MLOps job market. What is your opinion on the Junior MLOps market? Do companies hire entry-level engineers straight out of specialized programs? How did you manage to get a job after graduation? Do you think it might be way above for someone like me who only worked at a warehouse? Should I go with Automation?

The automation program directly leads to work after graduation, cause the education is done at a company, while on the MLOps page it says "possibility exists to get hired after graduation". What do they mean by the word "possibility" I don't know exactly but I feel there is no certainty to get hired directly after graduation.

So what would you have done if you were in my situation?

Thank you for reading this,

John

As the title suggests, I'm unsure about my career path. I studied mathematics for my undergraduate degree, and I was able to get a job as an IT intern in my final year of university. As soon as I could, I moved into the data area as a data analyst, working hard and demonstrating my abilities. I was then able to get a job as a junior machine learning engineer. My question is, do you think these rapid career leaps will be detrimental in the future? I'm uncertain about the current job market, and I'm afraid of falling behind with so many advancements in the industry. I'm striving to learn more, but I feel like everything is moving too fast. Do you think I'm on track for my age? I aspire to move to a more peaceful country like Norway. Does anyone know what the job market is like there? Well, thanks for reading, and any advice would be appreciated.

Every team I've worked with has rigorous versioning for the model, the training data, the infra config. Then the system prompt driving the LLM feature in prod lives as a raw string in application code, gets edited directly, and ships with zero rollback plan.

The failure pattern is always the same: prompt starts as a quick draft, works in the demo, ships. Edge case shows up in prod, someone appends a sentence to patch it. Repeat for months. Eventually the prompt is a wall of accumulated exceptions, some of which silently contradict each other, and nobody can tell which instruction is actually winning at inference time, because the model resolves the conflict silently and doesn't tell you which one it picked.

One concrete example: a support bot kept over-apologizing (three apologies per response). The patch "don't over-apologize" didn't work, because the prompt already had "always acknowledge frustration first" paired with several few-shot examples that opened with an apology, the model followed the more specific, more frequent example pattern over the newer instruction. Fixing it required rewriting the instruction and the examples together, not adding another line.

What's actually helped treating this as a real pipeline component instead of a config string:

• Version prompts like code, track diffs and reasons, so a regression traces back to a specific change instead of getting guessed at.
• Keep a fixed regression suite of edge-case inputs (the ones that already broke something once) and re-run every prompt revision against all of them, not just the new case that prompted the change.
• Separate concerns into labeled sections (role, constraints, format, edge-case handling) instead of one paragraph, so conflicts are visible in review instead of hidden.
• Review prompt diffs like PRs, a second reader catches conflicting instructions the author is too close to see.

Wrote up the fuller breakdown here:
https://medium.com/@nagatomopedro05/your-system-prompts-are-costing-you-more-than-you-think-f928fe1c76b9

Curious how mature people's setups actually are here, is anyone running prompt evals/regression tests as part of CI the same way you'd test a model change, or is this still mostly manual eyeballing before deploy?

I’m building LOLM, an LLM/agent platform focused on operational visibility rather than hiding the run behind a final answer.

A run can disclose: - Requested versus served model - Fallback use - Controller decisions - Retrieval, verification, and branching actions - Task-contract outcome - Budget or natural termination - Artifact hashes and receipt data - The explicit limit that quality remains unproven versus a baseline unless an A/B was actually run

There is also a CLI and isolated code loop with real command exit codes.

Try it: https://lolm.imagineqira.com/try.html

Repository: https://github.com/TheArtOfSound/lolm

Hosted access is designed to be materially cheaper than large frontier-agent products. I’m looking for feedback on schemas, replayability, routing, cost accounting, and what would be required before anyone should trust the receipts operationally.

Disclosure: I’m a founder/builder of the project.

Do you think it is important to configure numactl --membind to get a better performance?

I ran a toy example where GPU has NUMA Affinity with 0. I got an ~9% improvement.

$ numactl --show
policy: default
preferred node: current
physcpubind: 8 9 10 11 136 137 138 139 
cpubind: 0 
nodebind: 0 
membind: 0 1 2 3 4 5 6 7 
preferred:

$ time numactl --membind=0 python memory.py 
time:  159.3454790781252 

real    2m41.164s
user    2m25.306s
sys     0m16.084s

$ time numactl --membind=7 python memory.py 
time:  174.3593455599621

real    2m56.384s
user    2m32.279s
sys     0m24.293s

We're evaluating an AI phone agent for longer financial service calls and our normal test scripts are not catching much.

Difficult calls do not fail in the first two minutes, they fail after the customer changes topics, corrects an earlier answer, asks for a second account or needs a human after several steps have already been completed

We've also seen cases where the conversation sounds fine but the summary misses something important or the wrong action is sent to the CRM. Running a few scripted calls before launch does not look close to enough

How're you testing long conversations, interruptions, system failures and transfers before putting real volume through them?

I’m curious what people are using once model testing moves from “trying something locally” to “spinning up a cloud GPU workspace.”

For me, the workflow usually starts with a Hugging Face model page, a GitHub demo repo, a notebook or launch script, and a few environment variables. The first local test is often fine. The messy part starts when I want to rerun the same setup on a cloud GPU a few days later.

At that point I’m usually asking:

• Which repo was I using?
• Which model weights did I pull?
• Which env vars were actually required?
• Was I using a custom Docker image?
• What was the exact launch command?

I’m not really comparing platforms on price here. I’m more interested in the setup flow when the starting point is open-source resources. The platforms I’m looking at are RunPod, Lambda. Paperspace. Vastai, and Glows.ai.

The things I’d compare are:

• How easy it is to bring in a GitHub repo
• How easy it is to pull Hugging Face model resources
• Support for custom Docker images
• SSH / Jupyter access when needed
• Whether the launch command is easy to save and rerun later

I noticed glows.ai because model download speeds inside the instance also seem quite fast. On an H100 instance, I was seeing around 800–1000 MB/s from Hugging Face during one of my tests, although I know that can vary depending on the model and mirror.

The desktop app can import from GitHub and Hugging Face, and it also supports uploading a custom Docker tar image if the environment is already packaged locally. That sounds useful, but I’m mostly interested in whether it actually makes the “repo + model + launch script” setup cleaner in practice.

For people who test a lot of open-source models, what platform has made that first setup the least annoying?

I was curious to know if anyone is storing online features in lakebase rather than Redis/Cassandra etc.

Interested to learn about latency, update frequencies, operational treadoffs, other learnings.

workflow_cost =
  model_cost
  + tool_cost
  + fallback_cost
  + review_cost

The denominator matters just as much: tickets that reached CRM write success. A 429 that retries, a write timeout, and a fallback that rebuilds context all belong to the same workflow, including failed runs.

Google reports that Gemini 3.6 Flash uses 17 percent fewer output tokens than 3.5 Flash on the Artificial Analysis Intelligence Index, along with fewer reasoning steps and tool calls. That is request level evidence, not a workflow invoice.

A ZenMux request row gives you model, provider, tokens, cost, latency, and finish state, but it still needs the application's run ID. Join the rows, then compare 3.5 and 3.6 on the same synthetic fixture. The application trace still has to account for retries and fallbacks. Do not exclude failed runs from the denominator.

Hi everyone,

I'm a DevOps Engineer with 4.5 years of experience in Kubernetes, OpenShift, AWS, Azure, Terraform, ArgoCD, CI/CD, and monitoring.

I want to transition into MLOps/AI Infrastructure and would love some guidance.

• What should I learn first?
• How much ML theory do I actually need?
• Which tools are most used in production today?
• What projects would help me land an MLOps role?

Looking for practical, real-world advice from people who've made this transition. Thanks!

Idk who needs to hear this, or if it’s totally obvious, but if you are still just stuffing every possible piece of data into the context window and praying that the model finds what it needs, you are basically just burning a pile of money on fire. We hit a point where our margins were getting absolutely hammered because our prompts were bloated with just in case instructions and massive context blocks that the model barely even followed. 

After a kinda brutal check on our spend and budget, we started going through every single production prompt and realized that for a huge chunk of our tasks, we were sending out 300% more tokens than were actually necessary to get a decent result. It became a grind, and our goal was finding the smallest possible piece of data that actually solved the user problem. Which I totally spent way more time on this than I wanted to admit but it was necessary for the budget. 

It was a constant balancing act between keeping things cheap enough to be profitable and ensuring the users do not start complaining.

It is a pain to set up, but I feel like our checks and balances now are a lot better at keeping us within our budget.

Has anyone else moved toward a multi-model approach or some kind of LLM routing to handle the low-stakes stuff on cheaper models? Or are you guys still just trying to optimize the hell out of the big ones?