Building agents for Alexa+

I spent the last two years figuring out how to evaluate and measure agents for Alexa+ at Amazon. These are notes on the problems I found surprising or challenging along the way.

2024 The Measurement Problem On building evaluation and measurement infrastructure for tool-calling agents — traces, synthetic data, LLM judges, and the loop that holds it all together.