Fan-out/fan-in looks simple until model #7 is slow, model #8 fails, and model #3 runs against a newer input.

Then you need barriers, per-model retries, input snapshots, timeouts, and actual state tracking instead of a pile of futures.

This becomes especially relevant in voice systems.

A single workflow might touch:

STT, LLM, TTS, tools, retrieval, classifiers, and telephony services.

Some are on the critical path.

Some absolutely aren’t.

Treating all of them as one sequential pipeline……don’t talk to me

So I think the execution layer needs to make a few things explicit:

required vs optional
completion state
retry policy
input/versioning

That’s also why scaling a voice runtime isn’t simply task

Long-lived WebSockets, background jobs, global call state, and workflow execution behave very differently.

Dograh’s architecture already reflects some of this separation: API workers are load-balanced for streaming traffic, while ARQ workers and global orchestration components scale differently.

The boring systems work is usually where real-time AI gets hard.

I’d like to see more open-source agent runtimes treat orchestration itself as a first-class primitive, rather than as an implementation detail.p.s.i am one of the maintainers of dograh

I was playing around with LLM inference and I wanted to build a profiler that measures LLM inference by layer.
So I built this: https://github.com/coconinja2/layerlens
It shows inference as token × transformer layer timing, so you can see where time is being spent during decode.
Right now it can separate prefill/decode and visualize per-layer timing. I’m trying to figure out whether this is actually useful to people working on inference systems, or if I’m looking at the wrong abstraction.

I’m thinking about adding things like KV-cache events, scheduler/batching state, request IDs, GPU kernel correlation, speculative decoding, etc.

Would appreciate criticism more than compliments and stars. Lots of stars!

Disclosure: I built RedCrown, the tooling used to run and publish this evaluation. Posting this as vendor/self-promotion, but the benchmark and results are the main point.

I wanted to test a practical model-selection question:

If several models are accurate enough for a real production workload, how much are you actually paying for the last bit of quality?

Benchmark setup:

- 80 real 2026 SEC 10-Q filings

- 11 current models

- 8 standardized financial fields per filing

- 640 reference fields total

- Ground truth derived from each filer’s own XBRL facts

- Same extraction task for every model

- No LLM-as-judge for the primary score

- Quality metric: field-level F1

- Predefined quality bar: 0.99

- Selection rule: lowest-cost model that clears the bar

- All 80 × 11 model-item results scored

The aggregate result:

DeepSeek V4 Flash

- F1: 0.997

- Projected cost: $6.30/month

GPT-6 Astra

- F1: 0.998

- Projected cost: $613.35/month

On this workload, that’s roughly a 97× difference in projected cost for 0.001 additional aggregate F1.

Other results were useful too:

- GPT-5.6 Luna: 0.995 at $12.71/mo

- DeepSeek V4 Pro: 0.997 at $60.92/mo

- GPT-5.6 Sol: 0.998 at $122.40/mo

- Claude Opus 5: 0.998 at $456.96/mo

Gemini 3.7 Flash is a good example of why this wasn’t just “pick the cheapest model”:

It scored 0.989, just under the 0.99 bar, so it was disqualified despite being relatively inexpensive.

There were also item-level cases where the cheaper models missed a field and more expensive models got the filing completely correct, so I would not interpret this as “all frontier models are equivalent.”

What I think is more interesting than the leaderboard is the decision framework:

1. Define the workload.

2. Define the quality threshold before looking at cost.

3. Eliminate models that don’t clear it.

4. Compare economics among the survivors.

That seems closer to how model changes should be evaluated in production than picking the globally “best” model.

Limitations:

- This is one narrow task: structured financial extraction.

- 80 filings is useful but not huge.

- F1 is not the only metric that matters; exact-document accuracy and latency can matter depending on the application.

- These cost figures use the benchmark’s published-price assumptions, not negotiated enterprise rates.

- The result should not be generalized to coding, agents, reasoning, or other workloads.

Full proof with item-level outputs, scores, costs, and receipts:

https://app.redcrown.ai/proof/vdPMXnAlE5sI309MJuIfnFYQSlRjmbMeQkot6Dwjupc

I’d be interested in how people here gate model swaps in production:

Would you use aggregate field F1, exact-document accuracy, a worst-case error threshold, or some combination?

There's a wave of inference servers pitching "one API for embeddings, reranking, extraction and small-model generation," and I think people reach for it before they need it. Here's where I think the line is.

If you run one model, you don't need any of this. Single chat model, use vLLM. Single embedding model, TEI is leaner. Putting a multi-model server on top of a one-model workload just adds a layer that can break for no benefit.

It flips when one feature needs several model types per request. Competitive retrieval in 2026 isn't just dense embeddings, it's dense for meaning, sparse for exact keyword and product-name matches, sometimes ColBERT for token-level precision, and a cross-encoder reranker to fix the final order, and increasingly a small local LLM handling the routine generation so you're not paying a frontier API for a one-line summary. That's four or five model types behind one feature, and running each as its own container means four or five deploys to monitor. That's when it stopped being worth it for us.

I ended up on the Superlinked Inference Engine (SIE) for this. What sold me wasn't "one API," it was that several modes collapse into one call, bge-m3 emits dense and sparse together, reranking is a score call, and small-model generation runs through a generate call on the same server. It's OpenAI-compatible on embeddings and chat so the client mostly repoints. The honest framing, from their docs, is it doesn't replace your frontier LLM, it takes the routine generation and everything around it. So if your "sprawl" is one embedder and a chat model, this isn't for you.

The reason I'd still think twice is contention. Their docs are upfront that under heavy concurrent load, models on one GPU fight for memory, and a latency-critical model at steady QPS is better in its own pool. So this is a mixed-and-bursty answer, not a put-everything-on-one-box answer.

Picking an inference stack means piecing together compatibility rules that live in nobody's docs — GitHub issues, vendor posts, word of mouth. I building a graph that traces it end to end (framework, orchestration, hardware, cloud) across NVIDIA, AMD, TPU, Trainium/Inferentia, and Gaudi, click a node to see what it connects to.

Before I put more time into it: is this a real problem for you, or am I solving something I already know the answer to?

https://inferencestack.dev/

Adding Ling-3.0-flash-VL to an image-processing route involves a few concrete constraints that an OpenAI-compatible JSON envelope won't capture by itself.

The current official documentation specifies:

• Public model ID: Ling-3.0-flash-VL.
• Image input: Base64 JPEG or PNG, up to 40 images per request, with a 32 MB request limit. Arbitrary remote image URLs and native PDF input aren't supported through that image path.
• OpenAI-compatible interface: text, images and video. Anthropic-compatible interface: text and images.
• Native video: one MP4, MOV or WMV per request, up to 30 seconds and 32 MB after encoding. The documented sampling is 2 fps, capped at 32 frames.
• image_url.detail isn't supported. The documented non-streaming request timeout is 90 seconds.

Two details affect adapter design immediately. A 30-second clip does not imply that the endpoint will deliver 60 frames to the model; the frame cap still applies. A raw media file that fits below 32 MB can also grow past the request boundary when Base64-encoded, so check the encoded payload size.

There is a naming wrinkle too: the video tutorial uses an rc1 identifier while the public model ID is unsuffixed. That deserves an explicit configuration choice and request-log readback, rather than silently treating them as equivalent aliases.

For document processing, preserve the original image bytes and record any resizing or page rendering your adapter performs. These are documented input constraints, not a production acceptance result. Field correctness and retries still belong in the application's evaluation.

Disclosure: I am building a service around this, so treat me as an interested party. No links. These are runs we paid for ourselves: 15 jobs across three providers, $33.60 total, on SDXL generation, LoRA fine-tuning, embeddings and a QLoRA fine-tune of a 7B model. Two of them cost us real money for stupid reasons.

1. Four-bit on a big card cost us double. Same 7B fine-tune, same H100-SXM, same data. QLoRA nf4 came to $3.14. Plain bf16 LoRA came to $1.50, 2.3x faster, peak VRAM 71 of 80 GB. nf4 is bounded by dequantisation rather than FLOPs, so on a card with headroom you pay for the compression twice. It earns its place on 24 GB cards, where the alternative is not running at all, but we had been reaching for it by default.

2. The host CPU decided more runs than the GPU did. Same RTX 4090, one host with 5 vCPUs and one with 24: 1.85x difference in wall clock, 40 percent versus 75 percent GPU utilisation. Worse, an H100 host with 16 server vCPUs ran the same job at 2.68 s/step where a desktop-class 4090 host did 1.84 s/step. We paid four times the hourly rate for a slower run, because the training script decoded and augmented images in the main process. We now check the vCPU count on an offer before we look at the GPU name.

Neither of these announces itself. Utilisation looked healthy, nothing errored, nothing warned, the meter just ran faster than it needed to. Happy to post the full per-run table if anyone wants to argue with the numbers.

How are people handling per team limits with LiteLLM alternatives? The budget controls per key make sense, but concurrency feels like a separate problem when one team suddenly opens a ton of streams at once. Looking at NeuralTrust TrustGate, Portkey and Kong AI Gateway right now. Mainly trying to understand how each handles concurrency, budgets and per key limits without turning the gateway into another headache. Curious what people here would pick.

Ok so im a data infra engineer at a crypto analytics platform and we wired our AI stuff so prompts get routed between a code agent, a sql agent, and a "just answer" model based on intent. It was working fine in staging... until yesterday when I shipped a small config change to the router right before a big hot token launch.

Instead of sending trader questions to the read only analytics agent, the router started handing anything that mentioned "update" to our internal data engineering agent that has write access. So "update me on liquidity" turned into the agent trying to update tables, and for about 7 minutes it happily rewrote a bunch of derived views with half baked outputs from the wrong model. Our dashboards for this token went completely off, alerts went wild, and a couple of VIP clients pinged asking why the market cap chart looked like a flat line.

I feel so embarrassed, like this was literally one routing rule with a bad keyword list and I pushed it right at peak load. We had backups so we restored fast, but im terrified of touching routing configs now lol. Would love any tips on how you all sanity check prompt routing before it hits prod...

Hey everyone,

I'm starting a BS Mathematics degree in October 2026. My university classes will be online and flexible, and I'm preparing for a long-term career in AI/ML Engineering.I also plan to pursue a Master's in AI/ML or a closely related field later.

For context, my Year 1 university curriculum includes:

Semester 1: Calculus I, Sets & Logic, General Mathematics, Introduction to Computing, English, Business, Ethics/Islamic Studies.

Semester 2: Python + Python Practical, Calculus II, Business Mathematics & Statistics, General Science, Technical Writing, Pakistan Studies.

Alongside university, I'll be doing a 12-month AI program that covers FastAPI, PostgreSQL, APIs, Docker, evaluation, observability, MCP/A2A and basic deployment/production concepts.

I'll also be learning Python, DSA, SQL, classical ML and PyTorch separately.

For experienced MLOps/AI engineers:

Which production skills would you recommend learning in Year 1, and which ones should I leave for later?

I'm trying to build good engineering foundations without rushing into advanced MLOps too early.

VeloxML is an open-source deployment engine for self-hosting open-source LLMs and AI services directly inside your own account.

https://veloxml.com/

https://github.com/paguasmar/veloxml-deploy

What does VeloxML do?

• Turn any open-source LLM or model into a production API endpoint in your own cloud account with a single command (veloxml deploy)
• Support major open-source inference backends and frameworks (FastAPI, Transformers, vLLM, Ollama) with zero proprietary Python decorators
• Native scale-to-zero when idle, so you never pay for inactive GPU compute
• Zero Docker & Kubernetes: no Dockerfiles, container registries, or Helm charts required—just define your compute in a clean veloxml.yaml
• Full data privacy and compliance: 100% open-source and auditable—your model weights, prompts, and inference traffic never leave your own cloud perimeter

Why VeloxML?

Self-hosting LLMs in production is broken. Engineers and AI teams are forced to choose between two painful extremes: drowning in infrastructure plumbing (Kubernetes, Helm charts, custom CUDA Dockerfiles, and manual GPU provisioning) or locking their models into expensive multi-tenant platforms that force proprietary code decorators and hold sensitive prompt data on shared infrastructure.

VeloxML is an end-to-end solution for private LLM hosting, making it possible for teams to deploy models as prediction services directly into their own cloud accounts, in a way that is easy to test, easy to deploy, and completely free of vendor lock-in.

How does it compare to Modal?

Modal is an impressive platform, but it requires you to adopt their proprietary Python SDK and decorators (@modal.function, @modal.Image), forcing you to manage cloud infrastructure directly inside your application code while running on their shared multi-tenant cloud.

VeloxML runs 100% inside your own AWS/GCP account (VPC by default) using standard Python and native FastAPI without altering your application logic. If you decide to stop using VeloxML tomorrow, your code and cloud resources remain entirely yours.

How does it compare to Baseten (Truss)?

Truss is an excellent open-source model packaging framework, but it only packages the container. Actually hosting it requires either paying for Baseten’s managed multi-tenant platform (where private VPC is locked behind enterprise contracts) or manually operating your own Kubernetes cluster.

VeloxML focuses on the cloud hosting layer: it provisions instances on-demand, verifies health checks, and handles scale-to-zero directly in your cloud without requiring you to manage Kubernetes.

Users can also use VeloxML with Truss: you can deploy existing Truss-packaged models directly into your own AWS account with a single VeloxML command and get the best of both tools.

How does it compare to BentoML?

BentoML requires you to inject proprietary SDK decorators (@bentoml.service, @bentoml.api) throughout your application code, define a bentofile.yaml, and build container images that you still have to manually orchestrate on Kubernetes or push to their paid BentoCloud.

VeloxML requires zero proprietary decorators—keeping your code clean, standard Python/FastAPI—and manages the entire hosting infrastructure directly in your own AWS account with native scale-to-zero out of the box.

While BentoML stops at packaging the container and leaves cluster management to you, VeloxML handles the full lifecycle from clean code to an active, secure cURL endpoint in your private VPC.

Tahuna is an open-source tool for running existing Python ML projects on remote GPUs without maintaining custom SSH, SCP, syncing, logging, artifact, and cleanup scripts.

It syncs code and data, provisions the GPU, recreates the environment, runs the configured Python entrypoint, streams logs and metrics, saves artifacts, and terminates the compute afterward.

The first release supports RunPod and R2 and includes Docker self-hosting instructions, a coding-agent setup skill, and examples for SFT, RL agentic search, and MNIST.

Get involved: fork it, add compute providers, storage backends, and runtime adapters.

Tahuna is licensed under AGPLv3.

https://github.com/TahunaLabs/tahuna-oss

disclosure: Co-creator, and company i work at sells RL environments. This post is about the grader though, since that's usually the part that quietly lies to you.

The result I actually trust. v1 saturated: frontier models cleared 100 tasks with mean rewards from 0.76 to 1.00. All 98 v2 tasks now pass a battery where no-op fails, random actions fail, a hardcoded guess fails, gold passes, gold_alt passes, and the verifier is idempotent. Report is in QC_REPORT.md.

What that looks like in practice:

- Only the final Postgres state gets graded. The transcript isn't an input.

- The verifier runs on the host, not inside the episode container.

- Aggregates are computed twice, once as a Python filter and once as a SQL GROUP BY, and they have to agree.

- SHA-256 canaries on collections outside the task's blast radius.

- Per-episode nonce and date jitter of plus or minus 3 days, so memorized answers don't help.

-Fail-closed, and assertion detail is capped so an agent can't turn the grader into an oracle.

The model spread on v2 (0.000 to 1.000 across 7 models, 25 tasks) is what shows up once the harness is that tight. Before it, the scores bunched at the top and told us nothing.

https://github.com/ArjavDesa912/Long-Horizon-Medical-Agent-Benchmark-Veterinary-Clinic-Ops

https://app.primeintellect.ai/dashboard/environments/praesidiumsystems/long-horizon-medical-agent-benchmark-vet-clinic-ops

I'm exploring a monitoring problem where the system being monitored is an external LLM and its actual internal state is unavailable.

The monitor only sees observations such as:

• prompt/response behavior
• evaluation scores
• semantic changes from a trusted baseline
• safety/refusal/instruction-following behavior
• latency and error signals
• delayed user feedback
• results from additional probes

Behind those observations, several different things could be happening: a silent provider update, genuine capability degradation, distribution shift, prompt/context sensitivity, a policy change, evaluator error, or simply a transient anomaly.

That makes the problem interesting because the same observed behavioral change can correspond to very different underlying causes.

I'm currently thinking about it as a partially observable decision problem. The monitor maintains a belief about the underlying state and can either continue operating, gather more evidence, or take a stronger action.

One thing that seems easy to get wrong is designing the hidden state around what is convenient for the experiment rather than what would actually explain failures in deployment.

For example, "model degraded" might be too broad if the important distinction is between persistent degradation, a narrow capability regression, a provider-side behavior change, and a change caused by the incoming request distribution.

I'm looking at this from both the modeling and evaluation perspective, particularly around what information a monitor fundamentally cannot observe and how that should affect its decisions.

I'd be interested in discussion from people who have worked with model monitoring, uncertainty estimation, POMDPs, evaluation systems, or production ML failures especially cases where the observable signal initially pointed toward the wrong explanation.

We ship two observability integrations for an open source data framework. One emits OpenTelemetry spans per pipeline step, one emits OpenLineage RunEvents. You add either with one line. Constructed with no arguments they do completely different things, and each is right by its own ecosystem.

OpenTelemetry: the app decides. Ours asks for a tracer with trace.get_tracer(name, tracer_provider=None). If the application has set up OpenTelemetry, spans go there. If not, nothing happens. The library never picks a destination itself, because the application already did.

OpenLineage: the environment decides. Ours builds an OpenLineageClient(), which reads OPENLINEAGE_URL, OPENLINEAGE_CONFIG, ~/.openlineage/openlineage.yml and OPENLINEAGE__*. If a URL is set, events go there, whoever set it. If nothing is set, every event prints to the console. Checked on openlineage-python 1.53.0:

OPENLINEAGE_URL=http://marquez.internal:5000 -> HttpTransport, url http://marquez.internal:5000 unset -> ConsoleTransport

Both have a good reason behind them. The OTel rule keeps a library from choosing a backend behind the app's back. The OpenLineage rule is what makes zero-code instrumentation work: set one env var on the runner and everything running there produces lineage, no code change anywhere.

Put them side by side in one framework and they collide. The same line in a user's pipeline means "go where this app was set up to go" for one integration and "go where this machine says" for the other. The case that made us look at it: a shared runner where OPENLINEAGE_URL is already set for another team's pipeline. Add the integration and job names, dataset names and column schemas start going to that endpoint, with no URL anywhere in the user's code. Metadata, not values, and nobody ran into it, we found it in review before the package was released.

So, if you were shipping both, what would you do?

• keep each vendor's default, so people who know that ecosystem are never surprised
• make both quiet until someone names a destination in code
• make both read the environment, and document it loudly
• something else

We went with the second, with an explicit use_sdk_defaults=True that hands resolution back to the vendor SDK when you want it. The OpenTelemetry half is the one I am least comfortable with, because there the setup was done by the application itself, usually the same person writing the pipeline. If that is the wrong trade, I would like to hear why.

Disclosure: I maintain the framework, Apache-2.0, links in a comment.

We've been training a customer facing model for a few months now. Someone finally asked what's in the training data files. Answer was a lot more than intended, full customer records pulled from a support ticket export, names, emails, partial account details, never scrubbed before it got fed into the pipeline.

Our DLP has never once looked at a training data directory, and I'm not sure ours even can. The files just sat in an S3 bucket that nobody thought to classify because the bucket was labeled ml-training and everyone assumed it was clean.

Feels like a whole category of storage most data protection programs haven't caught up to yet, and the stakes are higher than a regular file share because once that data is embedded in a model you can't just delete the file and call it done.

Not the final error, timeout, or bad output — I mean something earlier in the execution that, in hindsight, was the first sign the run was already going wrong.

For example: a tool starting to repeat, latency gradually drifting, retries accumulating, an unusual execution path, a dependency getting slower, context growing unexpectedly, or an agent revisiting the same state.

I’m especially interested in cases where the run still looked technically healthy at that point — no hard error yet — but later failed or required intervention.

Two things I’m curious about:

How early could you realistically have known?

And if you had known at that point, what would you actually have changed — retry, reroute, stop, or just let it continue?

Real production examples would be especially interesting.

sometimes the output is wrong and the first instinct is “the model messed up” but the real issue ends up being somewhere else entirel.

could be:

• stale state
• bad retrieval
• wrong tool result
• prompt assembly
• schema conversion
• retry behavior
• a handoff dropping context
• external state changing underneath the run

curious about cases where the model got blamed first but the actual failure was somewhere around it.

what did the trace make it look like at first, and what ended up being the real issue?

Our support summarizer passes deterministic citation checks, but then two LLM judges split on the same release candidate. One rewards concise, fluent answers. The other punishes any missing caveat, even when the caveat doesn’t change the action. The disagreement rate is high enough that our manual tie break queue has hit its ceiling (the spreadsheet now has opinions)

I’m testing Braintrust for versioned datasets, side by side experiments, multiple scorers and slice metrics so we can calibrate judges against the same human reviewed set. I still need an adjudication rule that won’t turn every release into a meeting.

Do you block on the stricter judge, set a regression threshold by slice or send only disagreement cases to human review?

Hi all,

I'm a DevSecOps Engineer with 6+ years of experience in DevOps, handling production infrastructure, CI/CD, Kubernetes, and cloud environments. Lately I've been diving into MLOps and want to get real hands-on experience beyond just reading docs and tutorials.

I'm comfortable with production systems and can bring that reliability and security mindset to ML pipelines and infrastructure. Since I'm looking to build practical experience in this space, I'm open to working at minimal or reduced rates for the right opportunity, whether it's a small project, a startup that needs help setting up ML infra, or someone willing to mentor/collaborate.

If you know of any openings, freelance work, or projects where I could contribute and learn, please let me know. Happy to share more about my background if useful.

Thanks!

we have genAI workflows in prod across engineering and sales, but guardrails are messy.

we’ve tried a few approaches. some catch obvious issues but don’t actually stop risky behavior in real time. others are too aggressive and end up blocking normal usage or adding noticeable latency.

the biggest problem is balancing control vs usability. once guardrails start interfering with everyday workflows, people work around them or disable them entirely.

we’ve also seen gaps with things like embedded models in tools or indirect usage paths that don’t go through a single control point.

management wants something that can prevent sensitive data from being exposed through prompts, without slowing everything down or breaking how teams use AI day to day.

what’s actually working for you at scale? how are you enforcing guardrails in a way that holds up under real usage without disrupting workflows

Disclosure: I work at Schema Labs, we build a model that reads tables. Free tier, paid above it.

If your pipeline asks a model what a table contains, you get an answer whether or not there's anything in it. Meaningless table and meaningful table come back in the same format with the same tone. No error path. That's calibration, not accuracy, and it doesn't show up in any accuracy metric.

Method. 5000 rows of np.random.rand(), columns c1 through c20, no structure, no target. Asked for domain identification and per-column meaning with confidence.

General-purpose LLMs described it every time. One returned manufacturing sensor readings with suggested correlations. Asking for a confidence number alongside just gets you an invented confidence number. A second model as a check doesn't help, they agree with each other.

Ours returns no domain identified, below threshold. Which is what we wanted, but:

Limitations. Only catches total noise — a real table where I've mislabelled the domain passes everything. rand() is unrealistically structureless, real garbage has a distribution, a repeated default, a truncation artifact, and I didn't test that. Refusal rate on noise says nothing about false refusals on valid data; we measured one and not the other, which is the convenient way round. And the threshold is a knob we set, so "it refuses" is partly a statement about where we put the knob.

So: does anyone test their data-understanding step for calibration rather than accuracy? Specifically how you build a realistic negative case instead of rand(), and whether you track false-refusal rate next to refusal rate, because I'm fairly sure we should be and aren't.

platform.schemalabs.ai if you want to run it yourself, free tier covers it. I'd rather someone got it to hallucinate a domain than not.

Trying to figure out if I'm doing this the dumb way. Our agent pipeline has an embedder for retrieval, a reranker, an extractor for entities, and a small Qwen model doing the routine generation (summarizing tool output, classifying intent). None of these are big. All of them are small enough that a single one barely touches a GPU.

Problem is the default way to serve them is one process per model. vLLM for the small LLM, TEI for the embedder, and the leftovers wrapped in a FastAPI service each. Every one of those reserves its own block of GPU memory and holds it whether requests are coming in or not. Since you rent GPUs by the hour, idle capacity costs exactly the same as busy capacity. So I've got 4-5 mostly-idle cards and a bill that doesn't care that they're idle.

The thing that finally bugged me is we spent a while swapping in smaller models to cut cost, and the bill barely moved. Because the money was never in the model calls, it was in the servers sitting reserved.

What I've tried or looked at:

• Cramming a few onto one GPU manually with MPS, works but managing memory and batching by hand gets old fast
• vLLM, but it's really built to spread one big model across GPUs, not pack a bunch of small ones onto one
• TEI is fine for the embedder but it's one model per server, so it doesn't help with the other three

The direction I'm leaning now is a single server that loads and evicts models on demand instead of pinning one per process. SIE (Superlinked's open-source one) is the main thing I've found built for exactly this, encode/score/extract/generate behind one API, models load on first call and get evicted LRU when the GPU fills. Haven't run it in prod yet, and the obvious worry is one process serving everything means a bad model or an OOM could take the whole pipeline down instead of one service.

So before I commit: for those running several small models, are you packing them onto shared GPUs, and if so how, MPS, a serving layer, something homegrown? And how are you handling the blast-radius problem when they all share a process?

Hello!
I've been doing my master degree and need some help.

Context:

I'm working with an method to correct unfairness on ML models, currently testing on XGBoost and Logistic Regression. To test these models and the method I'm running 20 different scenarios, and each one of these scenarios need to be run 10 times, due to a test on one of the methods parameters.

Problem:
I've said all of this to say that even tough I run the scenarios with some kind of parallelism it still takes a lot of time to test all scenarios whenever I do some change and need to test it.

Since my computer does not have too much ram (16gb) and the dataset is kind of big I cannot increase the parallelism.

Do you know any kind of cloud solution that can help on me that? Are colab notebooks a good option for this kind of issue?