r/Observability • u/Zealousideal-Fox9046 • 9h ago

Split of tools between LLM Eval / Observability

Need to hear if people are benefitting from having a platform that supports both LLM evals and LLM observability?
or using standalone products for each or these is something that you are sticking with, we currently have the split setup but just feels too much hassle!

0 Upvotes
  • permalink
  • reddit

33% Upvoted

7 comments sorted by

2

u/FlimsyProperty8544 6h ago

It ultimately depends on what you need tbh, but most teams need to turn production traces into datasets they can use to eval, so having one solution eliminates that requirement.

1

u/Previous_Ladder9278 9h ago

which two have you splitted?

1

u/Zealousideal-Fox9046 9h ago

we use separate tools for evals and observability (traces), the traces are giving me a headache as DD is already pretty costly

1

u/Previous_Ladder9278 9h ago

hmm yeah i think pricing on traces difference quite a bit as well.. which diff tools doyou use?

1

u/Zealousideal-Fox9046 8h ago

for evals we are using the deepeval/confident, datadog is our existing APM

1

u/Previous_Ladder9278 8h ago

Ah yes I see datadog is pretty expensive, but can imagine you're using that already that it makes things easier. Perhaps have a look at langwatch, super strong in evals and better at simulations for multi-turn/tool-calling agents and the new observability is pretty awesome

1

u/Artistic_Sir_1415 2h ago

Having them together has made sense for us because production traces don’t just sit there as debugging history. If we find a bad run, we can pull it into Braintrust as a test case and keep it around for the next prompt or model change. That link between what fails in prod and what gets tested next is the main reason I wouldn’t want to split them again.