Are evals just a ritual?

On taste in AI engineering

· 4 min read

Trustworthy evals take far more careful human work than most teams put into them - evals must be a significant investment, and my experience is that they require constant upkeep and validation that they are providing real signal. Evals are a double-edged sword - we need some way to track outcomes over time, but AI features are inherently unbounded, and over-optimising for specific scenarios can lead to worse behaviour on net. Building a flexible harness with an expansive range of tools to handle the vast array of scenarios, and having taste in prompting and construction of the harness, is more important. Then, step-change improvements in quality still come from the underlying model.

Conceptually, evals are very simple. You’re running end-to-end tests against an agent, and rating the results based on judges or deterministic metrics. The agent can then be tuned to improve these outputs, and also these can be run on a schedule to track trends over time. The judge/metrics can be applied to an “offline” curated dataset, and also to track success for “online” customer scenarios. Due to data sharing policies, often online data is much more limited, and the metrics may be higher level such as user feedback and thumbs up/down rates.

The three most difficult parts of creating accurate evals are the following:

  1. Defining an appropriate dataset that is comprehensive and reflective of the key customer scenarios
  2. Constructing a realistic environment for the agent to run within
  3. Ensuring that the judges are accurately tuned to evaluate what humans also consider problematic

This is a difficult task, and at this point in time is a rare skill.

Consider a personal agent application, and let’s dig into each.

  1. Users can ask their personal agent literally anything, so of course it is not possible to have an infinitely large dataset. The problem with settling on a fixed dataset is that naturally over time to improve the results, the engineering team may overly bias the agent’s prompts for that dataset. What matters most is that the agent scores well because it generalises, not because it’s been tuned to the test. The reality is that this requires thoughtful skill/tool/subagent design, and deep analysis of the agent traces by both humans and other agents/judges.
  2. A personal agent must search for information across a range of connected services, and may provide proactive briefs and suggestions. For these tasks, the agent must successfully find the appropriate data, discard irrelevant junk, and then extract the essential parts for surfacing to a user. This implies constructing a sandbox environment for the agent with generated data, that has data relevant to all tasks in the dataset, and additional noise so that the environment is realistic. Building out and populating this test environment is tedious, and curating the data is challenging. As above, this comes down to highly detail-oriented engineers using agents to produce the data, and then analysing in detail the “realness” of the data.
  3. A personal agent should have a friendly, concise output style. It may also create reports for the user, or dynamically generated UIs. The quality of these is inherently subjective, and therefore LLM judges are used. LLMs still don’t see the bigger picture, and the judge prompt will overly bias what the judge sees and critiques. For example, a judge might score a brief highly for its friendly tone while missing that it surfaced the wrong meeting. Therefore, it’s essential to frequently go through these outputs as a human and baseline the judge - find what the judge is missing, or where it is penalising too hard - and use that feedback to update the judge prompt to be closer to the human assessment.

So what is the takeaway here? There is no shortcut to building out useful evals - it requires dedicated, thoughtful, and thorough human work. A lot of mediocre, lazy evals will be created to check a box saying that a feature has evals. Evals are a great way to keep core capabilities from regressing over time. We must be able to critically look at our evals and clearly say that they are bringing value, or else there is no point to them.

Although evals are very important, the step-change improvements in AI features over the past year come from underlying model improvements. Evals and training environments matter most to the labs. But notably, we have seen some large AI applications that didn’t use evals. They were built mainly through taste. Take OpenClaw - it revolutionised personal agents, but it had no evals, just manual testing and taste. The underlying models had to be good enough for OpenClaw to work, and the real power was due to its vast tool set and extensibility.

Personal agents are just one example. For many other AI features, we need to define something much narrower and really need to tune the outputs.

Personal agents sit at one end of a spectrum. At the other end are narrow features with hard rules, like a visual output that has to use a proprietary format or follow a specific style. Here evals are essential, but even so, a model upgrade can do more for format adherence than our own tuning - the existing evals can prove this.

We have no control over how the models are trained, so we need evals as guardrails. But the more closely we tune a feature to one model’s quirks, the harder it is to move to the next one. Features need to stay unopinionated enough to evolve with the models. Some tuning can’t be avoided, but we should keep it to a minimum.