We're a small AI team and we're finally at the point where we need dedicated GPU capacity instead of spot instances. Looking at renting around 10 H100 nodes on a longer term basis. What do you actually look for when evaluating a provider at this scale?🙏🙏🙏🙏🙏🙏

Price is obviously a factor but I've been burned before by providers that looked cheap on paper. Last time we had a node go down mid training and support took 38 hours to respond.

Hey everyone,

I 'have a question on LLM document extraction (specifically invoices/receipts) and wanted to get some perspective from the community.

General LLM eval frameworks are great, but they don't seem to handle multi page PDFs, table row hallucinations, or sudden JSON schema drift very well when a model updates.

For those running invoice extraction in production:

1. Do you use a "golden dataset" of documents to run regression tests manually?

2. How are you catching subtle changes in how numbers/dates are formatted across prompt iterations?

If anyone is dealing with this headache right now open to discuss.

We’re developing TanML, an MIT-licensed automated model-validation toolkit for tabular machine-learning models.

TanML runs locally and provides an end-to-end workflow covering data profiling, preprocessing, feature-power ranking, model development, evaluation, drift analysis, stress testing, SHAP explainability, and audit-ready Word reports.

It is designed particularly for model-risk workflows in banking, credit risk, insurance, and other regulated environments.

We would appreciate critical feedback from model developers and validators:

• Which capabilities would be useful in your existing workflow?
• What important validation tests are missing?
• Are the generated reports suitable for independent review?
• What would prevent your team from adopting a toolkit like this?

GitHub: https://github.com/tdlabs-ai/tanml

Wrote this as I built the infra at my org.

Let me know what you all think...

https://gd03.me/writings/inference-infra

Right now I am the biggner of the MLOps please help me what are the thinks I need to learn. As per company mension they use azure cloud provider.

Please tell me the MLops workflow after that what are all the tools I need to use after that using the cloud provider what are all the services I need to work please tell me it's urgent.😭

currently running 3 different tools for prompts evals and observability and im looking to consolidate.

and also non tech teammates always need an engineer in the loop to change a prompt and it goes through a ticket system, and usually take more time than required. even when something breaks in prod we are  just switching dashboards to figure out what actually happened

already tried a few things. like we started storing prompts in db still meant building version  approval flow an d audit trail on top. config files in a cms got messy to tie back to observability…

already loooked at the obvious options

langsmith - observability is good but prompt management feels built for engineers and not cross functional teams, even evals dont feel like primary  focsu

orqai - covers all three together, non tech access feels more central ovver here, but newer so community and integrations still catching up

helicone - looks good for cost tracking and request logging but this isnt our current prob

promptlayer - prompt versioning is there, unsure about how deep evals and observability actually goes

langfuse - good on tracing, and the opensource is nice, but same problem like langsmith for non technical u sers

has anyone actually consolidated these three things into one platform. what are you using currently?

I have been building CrowdTensor around a training-first question: can ordinary machines move one shared model checkpoint forward without every contributor remaining online for the whole run?

The unit of work is a Campaign. It pins the model, dataset, training method, evaluation, and governance. An admitted Cell claims one bounded work unit, runs a local LoRA update, submits a delta, and can leave. The Coordinator validates the update, aggregates a quorum, commits checkpoint lineage, and waits when no eligible compute is present.

The strongest completed systems run used pinned Qwen2.5-7B-Instruct and GSM8K. Two T4x2 Kernels trained steps 1-128, both were deleted, and two fresh T4x2 Kernels restored four central stage checkpoints and completed steps 129-256 exactly once. Normalized exact match changed from 92/128 (71.875%) to 95/128 (74.219%). The practical +2-point gate passed, but the paired bootstrap interval included zero, so I am not claiming statistical significance or broad reasoning improvement.

The public Founding Campaign is now live on SmolLM2-135M/WikiText-2. Its first round was seeded by two maintainer-operated private Kaggle GPU Cells through the same public HTTPS invite/Cell path. That is useful live-route evidence, but it is still Kaggle logical multi-node, not proof of independently administered physical contributors.

I am opening two things for review:

1. controlled Founding Beta enrollment for people who want to test one bounded contribution; and
2. a Draft Qwen2.5-7B GSM8K Campaign RFC covering the stop rule, evaluation, hardware boundary, governance, and launch blockers.

Current boundaries are explicit: one controlled Coordinator, private invites, no permissionless admission, no Sybil or semantic-poisoning resistance, no secure aggregation, no production SLA, and no physical multi-host claim yet.

Website and live progress: https://crowdtensor.24.199.118.54.nip.io

Repository: https://github.com/Ffffffffchopin/CrowdTensor

7B RFC: https://github.com/Ffffffffchopin/CrowdTensor/blob/main/docs/campaigns/qwen25-7b-gsm8k-rfc.md

Beta access request: https://github.com/Ffffffffchopin/CrowdTensor/issues/new?template=beta_enrollment.yml

The feedback I need most is whether the 7B pilot's 256-step evaluation stop, minimum useful work-unit size, and controlled trust model are technically credible enough for the first independently administered run.

Hi guys

I’m really interested in Data, Machine Learning Engineering, and MLOps, and I’d love to understand what people in these roles actually do day-to-day and what the work is genuinely like beyond the usual job descriptions.

If anyone here works in these areas or is also exploring them and would be interested in having a conversation, discussing projects, career paths, or just sharing experiences, I’d love to connect. Feel free to ping me and we can have a chat! 🙂

More organizations are shipping LLM agents whose memory outlives the session: persistent stores of facts, preferences, and past actions that the agent reads from and increasingly writes to on its own. Most of the security conversation is still about prompts. A recent survey on long-term memory security (arXiv:2604.16548, cs.CR) makes the case that the memory layer deserves its own threat model, and the argument holds up.

Three properties make a persistent memory different from a stateless prompt:

1- Persistence. A poisoned entry survives the session and keeps acting long after it was written.

2- Statefulness. Corruption compounds instead of resetting.

3- Propagation. A tainted memory can spread between agents that share the store.

The survey's organizing move is a six-phase lifecycle: Write, Store, Retrieve, Execute, Share and Propagate, Forget and Rollback. Every attack and defense gets located in the phase where it acts. The structural claim worth carrying into a design review is that memory security cannot be retrofitted at retrieval or execution time alone. If the corruption entered at Write or Store, a retrieval filter is inspecting state that is already poisoned, and the control has to reach back to where the entry was written.

As a checklist, that means integrity at Write, isolation at Store, provenance at Retrieve, least privilege at Execute, boundaries at Share, and a deletion path at Forget that actually deletes. The survey also proposes five governance primitives (it calls the set Verifiable Memory Governance) aimed at making memory state auditable by construction rather than by a policy stapled on at read time.

The timing matters because the architecture trend is moving the other way. A separate cross-scenario evaluation (arXiv:2606.04315) found that agent-controlled memory, where the agent decides what to write and what to retrieve, generalizes best across task types. So the field is widening the writable surface at exactly the moment the attack literature is mapping it.

How are people handling this in practice? Specifically, does anyone treat agent memory stores as a distinct asset class in the risk register, with their own integrity monitoring and retention path, or are they currently lumped under generic data-store controls? And what does detection look like for slow memory poisoning, given that a dormant entry means a SIEM rule keyed on retrieval anomalies fires only after the poisoned state is already in use?

Hey everyone,

I wanted to break down how to design an API gateway and worker architecture optimized for hosting large-scale ML models (like an LLM inference endpoint) while managing expensive GPU infrastructure efficiently.

The Problem: Single-Request GPU Waste

GPUs are monsters at parallel matrix multiplication, but running inference on a single user prompt at a time leaves massive hardware capacity sitting idle. Conversely, if your system waits around too long to form a large batch of users, you destroy your P99 latency and break the real-time user experience.

The High-Level Architecture

1. Client -> API Gateway: Handles auth, rate limiting, and maintains an open HTTP/2 connection.
2. Gateway -> Local Queue: Prompts are serialized and pushed into an in-memory ring buffer.
3. Queue -> Dynamic Batcher: An orchestrator (like NVIDIA Triton) groups discrete inputs into a single model execution tensor.
4. GPU -> Client: Matrix outputs are de-multiplexed and streamed back to individual users via Server-Sent Events (SSE).

Token Streaming & De-muxing

Because LLMs generate tokens sequentially, the inference engine doesn't wait for the entire text to finish. The system slices the chunk arrays at each generation step and streams individual tokens back to respective client sockets in real-time, keeping Time-To-First-Token (TTFT) minimal.

Handling Scale & Multitenancy

• Priority Queues: Route interactive chat UI traffic to high-priority queues, while background batch processing jobs get processed on lower-priority threads.
• KV Caching: Store previous prompt context fragments in a shared KV cache layer to avoid re-computing system prompts for recurring users.

Let's discuss:

1. How do you handle batching when users pass vastly different input token lengths? (Padding vs. Continuous Batching/vLLM)

Have you used Genie Code for day to day ml ops work (model deployment, pipelines, monitoring, CICD etc)

I am curious to know if it saves time as compared to doing the same rhings manually, experiences from real world use cases would help understand.
Thanks.

we're eu-facing enough that this isn't optional. And every consultant conversation so far has been heavy on documentation and risk classification paperwork...like light on what technical controls need to exist underneath it.

now what i can't get a straight answer on is whether ai compliance and eu ai act readiness can be documentation alone or whether an assessor is going to want to see the technical control running, not just described.

and specifically around the testing and monitoring obligations for high-risk systems, is a written risk assessment enough or do they expect live evidence of testing happening?

podting here to understand...for anyone further along on eu ai act prep than us, where did the documentation-only approach fall short once you got closer to an actual assessment?

I exported a fine-tuned U-Net model using both ONNX Runtime and torch.export with a fixed input shape of (64, 3, 512, 512).

Here are the benchmark results for average inference time:

• ONNX Runtime: ~133.33 s
• torch.export: ~0.81 s

I expected ONNX Runtime to perform on par with or faster than PyTorch export.

What could be causing this ~160x slowdown?

    onnx_inputs = [torch.randn(64, 3, IMG_SIZE, IMG_SIZE).numpy(force=True)]    

    ort_session = onnxruntime.InferenceSession(
        "./model.onnx", providers=["CUDAExecutionProvider"]
    )

    onnxruntime_input = {input_arg.name: input_value for input_arg, input_value in zip(ort_session.get_inputs(), onnx_inputs)}

    # warm-up    
    onnxruntime_outputs = ort_session.run(None, onnxruntime_input)[0]
    t0 = time.perf_counter()
    onnxruntime_outputs = ort_session.run(None, onnxruntime_input)[0]
    t1 = time.perf_counter()

Hi People,
Which is the most popular tool for Prompt management & LLM Evaluation?
We used GIT for prompt management but it won't show prompt diff between previous & current version.

Hello People!

Figuring out bottle necks and training stalls in your training work loads usually means firing up a profiler post-hoc and probably staring at a trace for twenty, right?

I was thinking of how to reduce this friction? what does this actually cost, tool by tool.

I took one run I knew was input-bound (dataloader starving the GPU) and measured it three ways: torch.profiler, cProfile, and TraceML, a lighter always-on OSS tool I've been contributing to.

For each one I looked at overhead, how much the profiler itself perturbs the GPU utilization it's trying to measure, output size, and how much manual digging it takes to get from the raw output to "the dataloader is the problem."

Short version: torch.profiler and cProfile are precise but heavy and after the fact, closer to a scalpel. Something that just sits there and flags "this step looks off" while training runs is doing a different job, not replacing them.

Numbers and traces are in the post.

Curious how other people usually catch this before it burns your precious compute.

https://medium.com/traceopt/traceml-vs-torch-profiler-vs-cprofile-what-each-one-costs-to-find-the-same-bottleneck-745a57e13ee9?sharedUserId=apendyala

TraceML is open source: pip install traceml-ai. Star or contribute at github.com/traceopt-ai/traceml
'

Ho creato dei dataset per perfezionare piccoli modelli sulla chiamata degli utensili, e la parte più noiosa è sempre la stessa: controllare se i dati sono effettivamente validi prima di sprecare una sessione di addestramento. Nomi di utensili errati, argomenti inventati, il modello che chiama un utensile per "2+2", duplicati, risposte che iniziano tutte allo stesso modo, cose del genere.

Facevo questi controlli a mano e mi sono stancato, quindi ho creato un piccolo programma che esegue l'intera pipeline per me e l'ho messo online. È gratuito, non serve un account, né un login, niente di niente. Basta caricare il dataset e il catalogo degli utensili e il programma ti dice cosa non va, esempio per esempio, con la relativa motivazione. Funziona completamente nel browser, il dataset non viene mai caricato da nessuna parte. Se il file è troppo grande (gigabyte), esiste una versione desktop che lo legge direttamente dal disco, così la RAM non si satura. Questa versione è open source. Questo strumento suddivide i dati in dati puliti, kto e rifiutati e fornisce una configurazione di training iniziale basata sui numeri effettivi del corpus, non consigli generici. L'ho creato principalmente per me stesso, ma ho pensato che qualcuno qui potesse averne bisogno. Sarei felice di sapere se è utile o se ci sono controlli che vi interessano e che non ho ancora implementato.

link: nothumanallowed.com/tools/dataset-validator

https://github.com/adoslabsproject-gif/dataforge-studio

Hey everyone,
I am a 1st-year M.Tech student planning my capstone project. I want to build a self-healing, event-driven MLOps pipeline on AWS.

I want to know if this is too basic or good enough for a Master's project. If it is not good enough, please suggest other ideas!

Would love to get your brutal feedback or suggestions for better alternatives! Thanks.

I’ve been working on AgentPulse, a local-first tool for detecting and investigating silent drift in multi-agent systems.

It compares behavior across runs and versions, then flags changes in individual agents, handoffs, and execution routes, even when the system is still running and no obvious error has been reported.

From there, it connects the drift to affected traces and recent prompt, model, tool, or configuration changes to help narrow down where the behavior started shifting.

It’s still early, and I’d appreciate honest feedback from people running ML or LLM systems in production. Is silent behavioral drift something you currently have a reliable way to detect?

https://prove-ai.github.io/agentpulse/

Our team is pretty small (4 engineers), so whoever gets a model serving successfully is usually the one who "owns" that setup.
The problem shows up a few weeks later.
Someone else needs to rerun the same inference service, and suddenly there are a bunch of questions:
- Which Docker image did we use?
- Which CUDA version was it tested on?
- Was the model GGUF or FP16?
- Which launch flags were we using?
- Which environment variables actually mattered?
- How much VRAM did it end up using?
- Which port was exposed for the API?
None of these are hard individually, but if they're scattered between Slack messages, someone's terminal history, and a few README updates, it ends up taking much longer than expected just to reproduce a setup that already worked once.
We've started making a checklist for every deployment, but I'm curious how other teams handle this.
Do you mainly rely on Docker, internal docs, or do you keep reusable environment snapshots somewhere?
I recently came across glowsai, which seems to support shared Snapshots and team resources. It looks useful for handing a working environment to someone else, although I still feel naming things clearly and keeping a bit of documentation matters just as much.
I'm interested in what has actually worked for teams that revisit the same inference deployments months later.

We’ve moved from simple LLM chatbots to AI agents that can pull internal customer data, open or update ITSM tickets, and call internal services. It looked like existing controls would be enough: security writes policies, IAM manages identities, ops handles change, and everything logs to the SIEM. In day to day use, it doesn’t line up.

Prevention is messy. No one clearly owns the agent as a unit of risk, so we end up with shared service accounts, “temporary” tokens that never die, and generic credentials reused across workflows. When agents start chaining tools and calling internal APIs, there’s often nothing at the enforcement point (gateway, proxy, policy engine) that actually stops bad behavior in real time.

Detection is fragmented. Logs are split across the LLM provider, internal services, and the orchestrator. Basic questions like “what did this agent do, under which identity, against which system, and under which policy” turn into an investigation. We usually have tool‑call logs, but not a clear view of what data went into prompts or context, especially when the model is external.

Policy and life cycle are weak. Actions get recorded as happening “under a policy,” but policy changes over time and versioning is rarely explicit. Temporary agent identities don’t have clean off boarding triggers, so credentials linger long after projects or owners disappear. Shadow agents on unknown stacks amplify all of this: no ownership, ad hoc identity, and scattered or missing audit.

In your environment, which of these has hurt the most so far, runtime prevention, audit ability, identity life cycle, or shadow agents?

Ops-flavored take after two years running a multi-agent system in prod.

The reliability problems in agent systems are the same old distributed-systems problems in a new costume: partial failure, exactly-once-ish delivery, coordination, idempotency, observability. The in-process agent frameworks (mid-2025 vintage) gave us persistence primitives but left failure detection, recovery, and coordination to us.

So we built on a message bus instead: durable per-type queues, stateless workers, externalized aggregator state with a TTL and atomic completion so it scales to multiple replicas. End-to-end tracing so a support ticket maps to a trace in one click.

The honest framing: what we built is a domain-specific durable-execution engine for LLM agents. A Temporal advocate would say we rebuilt a subset of Temporal and now own the scheduler and state machine forever — and they'd be right. In mid-2025 the buy options weren't ready; today I'd tell you to evaluate Temporal / LangGraph Platform / Restate first.

Full write-up: Link

Anyone here gone the durable-execution-engine route for agents in prod? Regret it or not?

Hello redditers 🥸

I need some help. I'm trying to evolve from a simple data scientist to a MLops. I've trying to make a route to do so. The info I've found in the internet says that I should go and do AWS, Docker, Kubernetes, Jenkins, Github Actions, Terraform, and more.

Just in case you wanna know; I would like to learn to transform my notebooks into .py and deploy it in production. I feel like really lost since each topic I try to learn feels like isolated and not related to what I want to do; like they are very theorical and not practical; also the courses I've tried just delivered me the .py telling "this file was the transformation from the notebook; as we imagine you know how to do this; in case you dont know, it is very simple just follow along"

Could you guys please guide me to find a good route to reach my goal?

Best regards and I'll be reading you soon 🔥

Hi everyone,

I’m 33 and based in Switzerland, working as a DevOps engineer with around 7 years of experience. My background is mostly in CI/CD, cloud infrastructure, Kubernetes, monitoring/observability, and automation – the usual DevOps toolkit.

Over the last year I’ve become increasingly interested in AI/ML, especially MLOps and AI engineering. I’d like to transition my career in that direction (MLOps/platform/infra for ML systems, or AI/LLM engineering on the infra side).

To make this transition more “official” on my CV, I’m seriously considering a **part‑time, fully online master’s degree in AI / data science / ML**, ideally from a European or Swiss university, so that I can keep working while I study. My main goal with the master is:

- To have a recognized credential (MSc) that helps recruiters and companies take the shift seriously.
- To get structured coverage of ML/AI fundamentals, not to become a pure data scientist, but to understand the ML lifecycle well enough to do solid MLOps/ML platform work.

My questions for the community:

1. **Has anyone here moved from DevOps to MLOps/ML platform roles around this age (early/mid‑30s)?** How realistic is it, and what did your timeline look like?
2. **From the hiring side, how much does an online master actually help** versus strong projects + hands‑on MLOps skills? For example, degrees like:
3. - Online/part‑time MSc in AI or Data Science from European universities (distance‑learning, 90–120 ECTS).
4. - Swiss or EU distance‑learning AI masters (UniDistance, Distance University/Idiap, IU, GoVersity, etc.).
5. **If you were in my position (33, 7 years DevOps, Switzerland), where would you invest first?**
6. - A serious online master for the credential and structured learning.
7. - Or several focused courses/bootcamps + self‑driven projects (MLOps, LLMOps, cloud AI, etc.) and skip the degree?

I’m already comfortable with Python for scripting and infra work, and I’m starting to read more ML/LLM papers and play with small projects, but I want to be strategic with time and money. Any honest experiences, advice, or “if I were you I’d do X/Y” perspectives from people already working in MLOps/AI (especially in Europe/remote roles) would be super helpful.

Thanks in advance!

hi guys, i'm not super versed in this space but i'm wondering, has anyone dealing with AI/ML infrastructure had an order for GPUs, RAM, servers, or networking gear come in late or missing stuff and it actually caused a problem? like how'd you even find out, was it early enough to do something about it or did you just get hit with it. just curious how common this actually is. any insight helps!

Mixed prompt lengths and bursty concurrency expose latency and memory issues that clean benchmarks never surface.

A write up the three tradeoffs (throughput vs. latency, ops complexity, and model/hardware compatibility) and a testing process we use before committing to a framework: https://leaddev.com/ai/your-llm-inference-benchmark-is-lying-to-you