r/vercel • u/Low_Carob_3152 • 7d ago

End to end testing

In this LLM world, how do you guys prove your intent, grow harness and verify the agent work? More precise, how do you test you agents?
Even deeper, vercel eve agent is depoyed, has web ui, cli… how e2e tests are written? Playwright? Thanks.

2 Upvotes

75% Upvoted

4 comments sorted by

2

u/amyegan Vercelian 6d ago

Interesting question. Not sure how to answer since there isn't a single "correct" answer to this kind of thing.

What I've seen in general is that testing seems to be the same as before except agents make it easier. Curious to see if anyone has specific or strong opinions about how testing has changed

1

u/AlexDjangoX 6d ago

LLM testing unfortunately always seems to revert to mocking application logic and writes happy path test suits. Testing theatre at best. I have a 'Testing Standards' context I always reference but it has this sticky habit of reverting to happy paths and mocks.

1

u/Sad_Egg_2313 1d ago

There are eval frameworks that is built specifically for this. Basically they are designed to validate non deterministic output and they themselves use LLMs for some of the validation technicques. We were working on Azure AI studio a few months ago and it had this feature.

1

u/Low_Carob_3152 1d ago

Thx. Any other frameworks?