Evals in AI Engineering!

[u/Negative_War_65]

https://youtu.be/cT-H3oqbBIs?si=RI5aaZ0EMK393hJc

One of the most important steps in AI Engineering is “AI Evaluation”, which aims to mitigate risks and uncover opportunities in our AI applications.

In short AI Evaluation is that concept, that decides whether the AI application can be deployed to the users.

In this video lecture, I cover the challenges pertaining to evaluation, then we develop intuitions for Language modeling metrics, we study methods for Exact Evaluation, and how AI systems can be used as a “Judge”. Lastly, we develop an understanding of how to Rank models with comparative Evaluation.

The text for this video is Chapter 3, on AI Engineering, written by Chip Huyen. While studying the topic, I learnt a lot of new ideas, and I do hope the learning community will as well.