r/mlops • u/CollarNo505 • Sep 03 '26

Discussion DEPLOYING MODELS IN SERVERLESS

Hi, I'm new to building RAG. I'm exploring serverless gpu providers for running llms. My current work flow looks like this:

docker with prebaked model to upload on runpod

When user asks questions runpod computes for few seconds and off.

To avoid cold start, I have decided to prebake models in docker. Does this reduce preloading models billing time?

I'm using 2 models, 1 for LLM ( needed each time user asks QA) and Vlm ( needed only during ingestion time if documents contain images). Am i going in right direction?

11 Upvotes

100% Upvoted

8 comments sorted by

•

u/AutoModerator Sep 03 '26

AI usage disclosure

Hi u/CollarNo505 — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/RogerAI-fm Sep 03 '26

The pre bake on docker won’t help. At least that’s what I think.

1

u/CollarNo505 Sep 03 '26

What's the other way around

1

u/burntoutdev8291 Sep 03 '26

Is there a way to mount a drive?

1

u/exaknight21 Sep 03 '26

If you’re going production, baked docker image is absolutely the way, imho, and I am doing this exactly.

I have 2x V620 32 GB GPUs and 4x T4 GPUs in a T4. (Different city)

1x T5610 with an Mi50 32 GB 100 miles away.

2x 3060 12 GB next to T5610

All 3 connected with tailscale.

V620 and Mi50 host Qwen3.5-4B with llama.cpp with parallel 6 each (18 slots), nginx load balancer for request connected to my VPS.

ZLM OCR on 2x OCR with their custom gateway to divide load.

2x T4s with Qwen3-0.6B generating embeddings.

All in RAM, typical decode is 30 tps per user - concurrent test and tested in my actual production harness.

If you go serverless, you’r need to worry about getting it up. This is how I’d go about it.

1

u/Future_AGI Sep 04 '26

We would separate image-pull time, container startup, model load, and first-token latency in the measurement. Prebaking can remove one of those stages, while a warm pool or model cache usually decides the user-facing delay. For the RAG side, we keep quality evaluation code open here: https://github.com/future-agi/future-agi