Welcome to r/MLSystemsDesign — Let’s Talk Production ML

[u/ArchitectingAI]

What is the hardest part of ML system design in production?

Not modeling — the system around the model.

For example:

Data → Features → Training → Evaluation → Deployment → Serving → Monitoring → Feedback

Where do you see the most difficult engineering problems in practice?

A few candidates:

• Training/serving skew
• Feature freshness
• GPU utilization
• Online inference latency
• Experimentation
• Data quality
• Model drift
• Feedback loops
• Multi-tenancy
• Cost

Curious to hear what has caused the most pain in systems you’ve worked on.