AWS launches UAEF to compare AI agent performance
The free, open-source tool supports multiple agent frameworks, but AI-based scoring and results storage rely on AWS services.
Amazon Web Services has introduced an open-source testing tool that helps developers check how AI agents perform after changes to prompts or models. Called the Universal Agent Evaluation Framework (UAEF), it evaluates performance using 36 measures, AWS said.
UAEF standardizes activity records from different software kits so developers can assess agents across frameworks. Supported kits include LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore, which otherwise record agents’ actions in different formats.
The assessments span seven areas, covering answer quality, safety, speed, tool selection and use, long conversations and cooperation among agents. Some results come from direct calculations; others rely on another AI model running on Amazon Bedrock.
Saved tests allow comparisons with earlier runs. A drop of more than 5% on any measure generates a regression flag, with severity labeled minor, moderate or severe. According to AWS, 12 organizations in telecommunications, healthcare and financial services have used the tool to compare agent versions before deployment.
Although UAEF itself is free, AWS services are required for AI judging and storing results. AWS cautions that the framework is not intended for making security decisions.
