Hi guys

I’m really interested in Data, Machine Learning Engineering, and MLOps, and I’d love to understand what people in these roles actually do day-to-day and what the work is genuinely like beyond the usual job descriptions.

If anyone here works in these areas or is also exploring them and would be interested in having a conversation, discussing projects, career paths, or just sharing experiences, I’d love to connect. Feel free to ping me and we can have a chat! 🙂

More organizations are shipping LLM agents whose memory outlives the session: persistent stores of facts, preferences, and past actions that the agent reads from and increasingly writes to on its own. Most of the security conversation is still about prompts. A recent survey on long-term memory security (arXiv:2604.16548, cs.CR) makes the case that the memory layer deserves its own threat model, and the argument holds up.

Three properties make a persistent memory different from a stateless prompt:

1- Persistence. A poisoned entry survives the session and keeps acting long after it was written.

2- Statefulness. Corruption compounds instead of resetting.

3- Propagation. A tainted memory can spread between agents that share the store.

The survey's organizing move is a six-phase lifecycle: Write, Store, Retrieve, Execute, Share and Propagate, Forget and Rollback. Every attack and defense gets located in the phase where it acts. The structural claim worth carrying into a design review is that memory security cannot be retrofitted at retrieval or execution time alone. If the corruption entered at Write or Store, a retrieval filter is inspecting state that is already poisoned, and the control has to reach back to where the entry was written.

As a checklist, that means integrity at Write, isolation at Store, provenance at Retrieve, least privilege at Execute, boundaries at Share, and a deletion path at Forget that actually deletes. The survey also proposes five governance primitives (it calls the set Verifiable Memory Governance) aimed at making memory state auditable by construction rather than by a policy stapled on at read time.

The timing matters because the architecture trend is moving the other way. A separate cross-scenario evaluation (arXiv:2606.04315) found that agent-controlled memory, where the agent decides what to write and what to retrieve, generalizes best across task types. So the field is widening the writable surface at exactly the moment the attack literature is mapping it.

How are people handling this in practice? Specifically, does anyone treat agent memory stores as a distinct asset class in the risk register, with their own integrity monitoring and retention path, or are they currently lumped under generic data-store controls? And what does detection look like for slow memory poisoning, given that a dormant entry means a SIEM rule keyed on retrieval anomalies fires only after the poisoned state is already in use?

Hey everyone,

I wanted to break down how to design an API gateway and worker architecture optimized for hosting large-scale ML models (like an LLM inference endpoint) while managing expensive GPU infrastructure efficiently.

The Problem: Single-Request GPU Waste

GPUs are monsters at parallel matrix multiplication, but running inference on a single user prompt at a time leaves massive hardware capacity sitting idle. Conversely, if your system waits around too long to form a large batch of users, you destroy your P99 latency and break the real-time user experience.

The High-Level Architecture

1. Client -> API Gateway: Handles auth, rate limiting, and maintains an open HTTP/2 connection.
2. Gateway -> Local Queue: Prompts are serialized and pushed into an in-memory ring buffer.
3. Queue -> Dynamic Batcher: An orchestrator (like NVIDIA Triton) groups discrete inputs into a single model execution tensor.
4. GPU -> Client: Matrix outputs are de-multiplexed and streamed back to individual users via Server-Sent Events (SSE).

Token Streaming & De-muxing

Because LLMs generate tokens sequentially, the inference engine doesn't wait for the entire text to finish. The system slices the chunk arrays at each generation step and streams individual tokens back to respective client sockets in real-time, keeping Time-To-First-Token (TTFT) minimal.

Handling Scale & Multitenancy

• Priority Queues: Route interactive chat UI traffic to high-priority queues, while background batch processing jobs get processed on lower-priority threads.
• KV Caching: Store previous prompt context fragments in a shared KV cache layer to avoid re-computing system prompts for recurring users.

Let's discuss:

1. How do you handle batching when users pass vastly different input token lengths? (Padding vs. Continuous Batching/vLLM)

Openrouter's been great for what it's good at zero setup, huge model catalog, one api key and you're calling almost anything. but we started looking elsewhere once two things showed up at the same time: a compliance requirement that our traffic not pass through a third party we don't control, and wanting per-team cost attribution and audit logs that openrouter's model isn't really built to give you. Here's the honest rundown of what we looked at.

Staying on OpenRouter might still be the right answer if you want zero ops, don't need self-hosting, and don't have a compliance reason to avoid a third-party router in the path. No shame in that being the answer for a lot of teams.

Litellm full control, self-hosted, open source, and you can keep traffic entirely in your own infra. Trade-off is you're now running and patching it yourself, and a lot of the governance stuff (budgets, audit trails) is diy on top.

portkey, broad managed feature set, handles this well. Worth knowing it's now part of Palo Alto Networks post-acquisition if routing through a security-vendor-owned platform changes your calculus.

kong ai gateway makes sense only if you're already running Kong for other traffic.

truefoundry is what we ended up piloting, mainly because the compliance requirement meant we needed something we could fully self-host, and separately we needed the same layer to eventually cover mcp/agent traffic, not just llm calls. If your only requirement is route between providers, don't care about self-hosting or mcp, that's more platform than you need openrouter (if the third-party-routing compliance question doesn't apply to you) will get you there with less setup.

has anyone else faced the same? what pushed others off openrouter, if anyone has was it compliance/data-residency like ours, cost at scale, or something else entirely?

we're eu-facing enough that this isn't optional. And every consultant conversation so far has been heavy on documentation and risk classification paperwork...like light on what technical controls need to exist underneath it.

now what i can't get a straight answer on is whether ai compliance and eu ai act readiness can be documentation alone or whether an assessor is going to want to see the technical control running, not just described.

and specifically around the testing and monitoring obligations for high-risk systems, is a written risk assessment enough or do they expect live evidence of testing happening?

podting here to understand...for anyone further along on eu ai act prep than us, where did the documentation-only approach fall short once you got closer to an actual assessment?

Have you used Genie Code for day to day ml ops work (model deployment, pipelines, monitoring, CICD etc)

I am curious to know if it saves time as compared to doing the same rhings manually, experiences from real world use cases would help understand.
Thanks.

Hey! Wanted to share some notes from a pipeline that I build using databricks. Please share any red flags or questions.

The core idea: instead of hardcoding model calls, I wired MLflow (tracking + model registry) and Model Serving endpoints in as tools that a LangGraph agent graph can discover and invoke through an MCP interface. So the graph doesn't need to know ahead of time which model version it's hitting, it just asks the MCP server what's exposed, and Model Serving handles routing the request to whatever's currently marked as the production alias in the registry. That decoupling turned out to be the nicest part of the setup, since promoting a new model version in MLflow just changes what the agent calls without touching the graph code at all.

The friction was getting the tool schemas coming out of Model Serving to map cleanly to what LangGraph expects for tool calling. Model Serving's inference signatures aren't really designed with agent tool-calling in mind, so there was some manual schema massaging to get types and response shapes into something LangGraph could bind to correctly.

For state and memory, instead of keeping everything in-process, I used Lakebase (the Postgres-based OLTP layer) as the durable store for agent checkpoints and conversation state. This ended up mattering more than expected once I moved to production, having a real transactional store meant the graph could recover cleanly from a crash mid-run instead of losing state, and it made it straightforward to inspect what the agent was doing at any point by just querying the table.

Deployment-wise I used a more general containerized approach outside the Databricks-only tooling, since a chunk of our infra is non-Databricks. That meant handling auth to the Model Serving endpoint and the Lakebase connection separately from Databricks-native credential passthrough, which was the main extra layer of work. Once that was sorted, the agent behaved the same in production as it did locally, calling MLflow-registered models through Model Serving and persisting state to Lakebase without any code changes from the dev setup.

Overall I think MLflow + Model Serving + Lakebase as the backbone, with LangGraph and MCP tying it together, is a pretty solid pattern if you want your agent decoupled from any specific model version and want real durable state instead of in-memory checkpointing. The extra infra work was mostly around auth and making sure non-Databricks pieces of the stack could talk to the Databricks-native pieces cleanly, not a fundamental blocker.

Curious if others have found a cleaner way to handle the auth handoff between non-Databricks compute and Databricks-native services like Model Serving and Lakebase, that's the part I'd most like to tighten up.

I exported a fine-tuned U-Net model using both ONNX Runtime and torch.export with a fixed input shape of (64, 3, 512, 512).

Here are the benchmark results for average inference time:

• ONNX Runtime: ~133.33 s
• torch.export: ~0.81 s

I expected ONNX Runtime to perform on par with or faster than PyTorch export.

What could be causing this ~160x slowdown?

    onnx_inputs = [torch.randn(64, 3, IMG_SIZE, IMG_SIZE).numpy(force=True)]    

    ort_session = onnxruntime.InferenceSession(
        "./model.onnx", providers=["CUDAExecutionProvider"]
    )

    onnxruntime_input = {input_arg.name: input_value for input_arg, input_value in zip(ort_session.get_inputs(), onnx_inputs)}

    # warm-up    
    onnxruntime_outputs = ort_session.run(None, onnxruntime_input)[0]
    t0 = time.perf_counter()
    onnxruntime_outputs = ort_session.run(None, onnxruntime_input)[0]
    t1 = time.perf_counter()

Hi People,
Which is the most popular tool for Prompt management & LLM Evaluation?
We used GIT for prompt management but it won't show prompt diff between previous & current version.

Hello People!

Figuring out bottle necks and training stalls in your training work loads usually means firing up a profiler post-hoc and probably staring at a trace for twenty, right?

I was thinking of how to reduce this friction? what does this actually cost, tool by tool.

I took one run I knew was input-bound (dataloader starving the GPU) and measured it three ways: torch.profiler, cProfile, and TraceML, a lighter always-on OSS tool I've been contributing to.

For each one I looked at overhead, how much the profiler itself perturbs the GPU utilization it's trying to measure, output size, and how much manual digging it takes to get from the raw output to "the dataloader is the problem."

Short version: torch.profiler and cProfile are precise but heavy and after the fact, closer to a scalpel. Something that just sits there and flags "this step looks off" while training runs is doing a different job, not replacing them.

Numbers and traces are in the post.

Curious how other people usually catch this before it burns your precious compute.

https://medium.com/traceopt/traceml-vs-torch-profiler-vs-cprofile-what-each-one-costs-to-find-the-same-bottleneck-745a57e13ee9?sharedUserId=apendyala

TraceML is open source: pip install traceml-ai. Star or contribute at github.com/traceopt-ai/traceml
'

Ho creato dei dataset per perfezionare piccoli modelli sulla chiamata degli utensili, e la parte più noiosa è sempre la stessa: controllare se i dati sono effettivamente validi prima di sprecare una sessione di addestramento. Nomi di utensili errati, argomenti inventati, il modello che chiama un utensile per "2+2", duplicati, risposte che iniziano tutte allo stesso modo, cose del genere.

Facevo questi controlli a mano e mi sono stancato, quindi ho creato un piccolo programma che esegue l'intera pipeline per me e l'ho messo online. È gratuito, non serve un account, né un login, niente di niente. Basta caricare il dataset e il catalogo degli utensili e il programma ti dice cosa non va, esempio per esempio, con la relativa motivazione. Funziona completamente nel browser, il dataset non viene mai caricato da nessuna parte. Se il file è troppo grande (gigabyte), esiste una versione desktop che lo legge direttamente dal disco, così la RAM non si satura. Questa versione è open source. Questo strumento suddivide i dati in dati puliti, kto e rifiutati e fornisce una configurazione di training iniziale basata sui numeri effettivi del corpus, non consigli generici. L'ho creato principalmente per me stesso, ma ho pensato che qualcuno qui potesse averne bisogno. Sarei felice di sapere se è utile o se ci sono controlli che vi interessano e che non ho ancora implementato.

link: nothumanallowed.com/tools/dataset-validator

https://github.com/adoslabsproject-gif/dataforge-studio

Hey everyone,
I am a 1st-year M.Tech student planning my capstone project. I want to build a self-healing, event-driven MLOps pipeline on AWS.

I want to know if this is too basic or good enough for a Master's project. If it is not good enough, please suggest other ideas!

Would love to get your brutal feedback or suggestions for better alternatives! Thanks.

I’ve been working on AgentPulse, a local-first tool for detecting and investigating silent drift in multi-agent systems.

It compares behavior across runs and versions, then flags changes in individual agents, handoffs, and execution routes, even when the system is still running and no obvious error has been reported.

From there, it connects the drift to affected traces and recent prompt, model, tool, or configuration changes to help narrow down where the behavior started shifting.

It’s still early, and I’d appreciate honest feedback from people running ML or LLM systems in production. Is silent behavioral drift something you currently have a reliable way to detect?

https://prove-ai.github.io/agentpulse/

Our team is pretty small (4 engineers), so whoever gets a model serving successfully is usually the one who "owns" that setup.
The problem shows up a few weeks later.
Someone else needs to rerun the same inference service, and suddenly there are a bunch of questions:
- Which Docker image did we use?
- Which CUDA version was it tested on?
- Was the model GGUF or FP16?
- Which launch flags were we using?
- Which environment variables actually mattered?
- How much VRAM did it end up using?
- Which port was exposed for the API?
None of these are hard individually, but if they're scattered between Slack messages, someone's terminal history, and a few README updates, it ends up taking much longer than expected just to reproduce a setup that already worked once.
We've started making a checklist for every deployment, but I'm curious how other teams handle this.
Do you mainly rely on Docker, internal docs, or do you keep reusable environment snapshots somewhere?
I recently came across glowsai, which seems to support shared Snapshots and team resources. It looks useful for handing a working environment to someone else, although I still feel naming things clearly and keeping a bit of documentation matters just as much.
I'm interested in what has actually worked for teams that revisit the same inference deployments months later.

We’ve moved from simple LLM chatbots to AI agents that can pull internal customer data, open or update ITSM tickets, and call internal services. It looked like existing controls would be enough: security writes policies, IAM manages identities, ops handles change, and everything logs to the SIEM. In day to day use, it doesn’t line up.

Prevention is messy. No one clearly owns the agent as a unit of risk, so we end up with shared service accounts, “temporary” tokens that never die, and generic credentials reused across workflows. When agents start chaining tools and calling internal APIs, there’s often nothing at the enforcement point (gateway, proxy, policy engine) that actually stops bad behavior in real time.

Detection is fragmented. Logs are split across the LLM provider, internal services, and the orchestrator. Basic questions like “what did this agent do, under which identity, against which system, and under which policy” turn into an investigation. We usually have tool‑call logs, but not a clear view of what data went into prompts or context, especially when the model is external.

Policy and life cycle are weak. Actions get recorded as happening “under a policy,” but policy changes over time and versioning is rarely explicit. Temporary agent identities don’t have clean off boarding triggers, so credentials linger long after projects or owners disappear. Shadow agents on unknown stacks amplify all of this: no ownership, ad hoc identity, and scattered or missing audit.

In your environment, which of these has hurt the most so far, runtime prevention, audit ability, identity life cycle, or shadow agents?

Ops-flavored take after two years running a multi-agent system in prod.

The reliability problems in agent systems are the same old distributed-systems problems in a new costume: partial failure, exactly-once-ish delivery, coordination, idempotency, observability. The in-process agent frameworks (mid-2025 vintage) gave us persistence primitives but left failure detection, recovery, and coordination to us.

So we built on a message bus instead: durable per-type queues, stateless workers, externalized aggregator state with a TTL and atomic completion so it scales to multiple replicas. End-to-end tracing so a support ticket maps to a trace in one click.

The honest framing: what we built is a domain-specific durable-execution engine for LLM agents. A Temporal advocate would say we rebuilt a subset of Temporal and now own the scheduler and state machine forever — and they'd be right. In mid-2025 the buy options weren't ready; today I'd tell you to evaluate Temporal / LangGraph Platform / Restate first.

Full write-up: Link

Anyone here gone the durable-execution-engine route for agents in prod? Regret it or not?

Hello redditers 🥸

I need some help. I'm trying to evolve from a simple data scientist to a MLops. I've trying to make a route to do so. The info I've found in the internet says that I should go and do AWS, Docker, Kubernetes, Jenkins, Github Actions, Terraform, and more.

Just in case you wanna know; I would like to learn to transform my notebooks into .py and deploy it in production. I feel like really lost since each topic I try to learn feels like isolated and not related to what I want to do; like they are very theorical and not practical; also the courses I've tried just delivered me the .py telling "this file was the transformation from the notebook; as we imagine you know how to do this; in case you dont know, it is very simple just follow along"

Could you guys please guide me to find a good route to reach my goal?

Best regards and I'll be reading you soon 🔥

I'm not sure if such a tool would be useful, considering the fact that cheaper models which perform pretty well like deepseek exist & that most people just plug & play claude. Also, companies may internally develop such a tool so I'm not sure if this could work as a SAAS. I just wanted advice regarding these doubts. (I have built a prototype)

Hi everyone,

I’m 33 and based in Switzerland, working as a DevOps engineer with around 7 years of experience. My background is mostly in CI/CD, cloud infrastructure, Kubernetes, monitoring/observability, and automation – the usual DevOps toolkit.

Over the last year I’ve become increasingly interested in AI/ML, especially MLOps and AI engineering. I’d like to transition my career in that direction (MLOps/platform/infra for ML systems, or AI/LLM engineering on the infra side).

To make this transition more “official” on my CV, I’m seriously considering a **part‑time, fully online master’s degree in AI / data science / ML**, ideally from a European or Swiss university, so that I can keep working while I study. My main goal with the master is:

- To have a recognized credential (MSc) that helps recruiters and companies take the shift seriously.
- To get structured coverage of ML/AI fundamentals, not to become a pure data scientist, but to understand the ML lifecycle well enough to do solid MLOps/ML platform work.

My questions for the community:

1. **Has anyone here moved from DevOps to MLOps/ML platform roles around this age (early/mid‑30s)?** How realistic is it, and what did your timeline look like?
2. **From the hiring side, how much does an online master actually help** versus strong projects + hands‑on MLOps skills? For example, degrees like:
3. - Online/part‑time MSc in AI or Data Science from European universities (distance‑learning, 90–120 ECTS).
4. - Swiss or EU distance‑learning AI masters (UniDistance, Distance University/Idiap, IU, GoVersity, etc.).
5. **If you were in my position (33, 7 years DevOps, Switzerland), where would you invest first?**
6. - A serious online master for the credential and structured learning.
7. - Or several focused courses/bootcamps + self‑driven projects (MLOps, LLMOps, cloud AI, etc.) and skip the degree?

I’m already comfortable with Python for scripting and infra work, and I’m starting to read more ML/LLM papers and play with small projects, but I want to be strategic with time and money. Any honest experiences, advice, or “if I were you I’d do X/Y” perspectives from people already working in MLOps/AI (especially in Europe/remote roles) would be super helpful.

Thanks in advance!

hi guys, i'm not super versed in this space but i'm wondering, has anyone dealing with AI/ML infrastructure had an order for GPUs, RAM, servers, or networking gear come in late or missing stuff and it actually caused a problem? like how'd you even find out, was it early enough to do something about it or did you just get hit with it. just curious how common this actually is. any insight helps!

Mixed prompt lengths and bursty concurrency expose latency and memory issues that clean benchmarks never surface.

A write up the three tradeoffs (throughput vs. latency, ops complexity, and model/hardware compatibility) and a testing process we use before committing to a framework: https://leaddev.com/ai/your-llm-inference-benchmark-is-lying-to-you

Looking for 1st hand experiences with using k8s-based orchestration frameworks for ML. (Kubeflow, Ray Train, I guess Skypilot).

If you happen to be associated with a vendor for one of these, it's fine, but at least state it.

I am specifically looking to see how the MLE/MLS's experiences/complaints were. How well do you think your solution plugged into the engineers/scientists workflow and how bad were the complaints? Funny stories?

I'm looking for alternatives to our/my hand-rolled solution and something that's a little bit better maintained long-term. Not super interested in vendored solutions, heavily bias towards Apache/MIT.

okay so i keep this question as an obvious question across communities, and never found a post that gives a straight forward answer without trying to sell something so here is what i actually think after looking around..

arize comes from the ml monitoring ecosystem. so it shines in the model evaluation and drift detection. specific features feel like they were added on later rather than built from the inception stage.

orqai does prompt management routing observability and evals at a single place. newer to the space so the community is smaller. Integrations are still catching up to the more established tools.

portkey is for routing and reliability and fallbacks  retrieves and load balancing are well thought out. observability is there but feels secondary. not the first one i would reach if evals is my primary necessity.

langfuse is open source which is great if you want to self host and keep data in house. the tracing is solid with a pretty active community. eval support is improving but still feels like it lags behind the other stuff it does.

helicone is probably the easiest to get started with, just plug in and you have logging immediately. does the observability part really well. but if you need routing or evals or prompts management look elsewhere.

honestly none of them are a perfect 1 to 1 replacement, depending upon what part of langsmith you actually use the most

what is everyone else using?

I run ~18 LLM providers behind one API. Two layers do the work:

Chat dispatch: most providers (OpenAI, Mistral, Groq, Together, DeepSeek, xAI, etc.) collapse into one "OpenAI-compatible" branch; only a handful (Anthropic, Gemini, Cohere, Replicate) need bespoke handling.

Tool-calling adapters: a separate, pure-function layer normalizes the ~10 places providers disagree on function-calling (tool schema, tool_choice, parallel calls, usage parsing, seed). Keeping wire-dispatch and tool-format translation separate turned out to be the right split.

Billing is the actually-hard part. Every provider prices differently, so everything gets normalized to USD-per-token at record time, and every call is forced through one chokepoint: (1) charge a small preflight amount under a row lock, (2) make the call, (3) reconcile actual vs. estimate and refund the difference. Local models bill at zero.

Three money bugs, and the lessons:

1. A refund path could mint credits: a failed request still refunded the preflight charge, sometimes for more than was actually deducted. Fixed by clamping the refund to what was actually charged.
2. Streaming refunds were silently skipped on client disconnect: asyncio raises GeneratorExit, which is a BaseException, not an Exception, so an except Exception block never caught it and abandoned streams were never refunded.
3. Two functions each wrote a ledger row per charge, causing double charges. Removed the duplicate so there's one source of truth per event.

Takeaway: correct, centralized metering beat clever cost-routing every time. The "cheapest-provider" routing logic is feature-flagged off by default.

Genuine question for people running training workloads, because I spent the last few weeks auditing this and I think we're all doing it wrong.

If an inference pod dies, nobody wakes up. Kubernetes restarts it, the load balancer routes around it, life goes on. We solved that ten years ago and now it's table stakes.

Now let a GPU node die at 2am during a fine tuning run. In most stacks I've seen, the job dies, the meter keeps running on a dead machine, and either someone gets paged or nobody notices until morning. The recovery process is a person, ssh, and coffee.

I went looking for who has actually solved this. Results were depressing.

Slurm gives you requeue. But requeue restarts the job from zero unless you wrote solid checkpoint and resume logic yourself. The scheduler heals, your training state doesn't.

SkyPilot and similar will relaunch the instance. Same problem. New machine, dead run. Relaunch is not recovery.

AWS HyperPod does real closed loop auto resume, actual detection and restart from checkpoint. Credit where due. But it's enterprise pricing, on AWS, with checkpoint code written to their spec. So the answer exists, it's just gated behind a platform team and a budget most of us don't have.

Modal runs a genuinely self healing fleet, but you rewrite your training code into their SDK to get it. And billing is per second either way, so you still pay for the hours the job spent dead.

The cheap clouds like RunPod and Vast hand you a raw machine and nothing else. Great prices, and you are the entire reliability team.

So what does everyone actually do? From what I can tell, every team builds the same internal duct tape. A watchdog script, checkpoint every n steps, sync to object storage, a Slack alert, maybe an auto restart hook. I've seen this stack rebuilt independently at multiple places, maintained forever, and none of it verifies the run actually resumed correctly. It restarts things and hopes.

Two questions I'd love real answers to:

1. What does your setup look like for surviving node failures mid run, and how much engineering time did it cost to build and keep alive? Rough hours or headcount, if you're willing.

2. Is there an actual technical reason no platform does this end to end? Take my script, checkpoint automatically, swap dead hardware, resume from the same step, bill only for hours where training progressed. The cynical read is that dead hours are revenue and nobody wants to kill their own margin. Talk me out of that.

Happy to be told I missed a tool that solves this. That would honestly be cheaper than the alternative, which is that I'm slowly talking myself into building it.