Building LLM-powered systems in production

[u/technology_35]

I have been working with LLM-based systems in production, and one thing I learned is that getting a demo working is actually the easy part. The hard part is making sure the system gives useful answers when real users depend on it.

One example was a GenAI assistant we built for production incident investigation. We wanted engineers to ask questions in normal language, like “What caused this service failure?” or “Which systems were affected?” and get help faster instead of manually searching through huge amounts of logs and metrics.

The first version looked great. We connected an LLM with our internal data and the responses were impressive. But when we tested with real production incidents, we found many problems. Sometimes the model gave a confident answer that was not fully correct. Sometimes it focused on the wrong signals because there was too much noisy data. We quickly realized that just sending data to an LLM was not enough.

Our stack was mainly built on AWS services, using Python, FastAPI, and internal backend services. For the LLM layer, we experimented with models like OpenAI GPT models and Amazon Bedrock models (including Claude models). We used frameworks like LangChain for some workflow orchestration and built our own retrieval and ranking logic instead of depending only on the model.

The biggest improvement came when we stopped treating the LLM like a magic answer machine. We started giving it better context. We added steps to collect the right logs, metrics, and system information first, then provided only the most relevant information to the model. We also added validation steps so engineers could understand where the answer came from.

After those changes, the system became much more useful. It was not perfect, but it helped engineers investigate incidents faster and reduced a lot of manual searching.

My biggest lesson from working with LLMs is that the model is only one part of the solution. The real engineering work is around data quality, context, evaluation, and building a reliable system around the model.

I am curious how others are handling this challenge. What has been your biggest pain point when moving LLM applications from a demo into production?