I attended a 1-day conference this past weekend on Embedding Evaluators in Practice, hosted by the EvalEval Coalition. Here are some (unstructured) reflections on the panels and discussions (the conference observed the Chatham House Rule, so I am not attributing these insights to specific individuals).

For broader context, this conference comes on the heels of high-profile proposals to embed third-party evaluators in AI companies (from the companies themselves and from policy orgs), as well as the recent passage of California’s SB 813, which creates regulatory infrastructure for certifying independent verification organizations (IVOs) for third-party audits of frontier AI companies. My primary goal in attending this conference was really just to learn stuff - in particular, about the operational details of existing embedded evaluation arrangements and about the opportunities/challenges in the evaluation ecosystem more broadly.

Some of the key questions that surfaced (and interesting proposals) were as follows:

• Where can evaluator organizations get the funding to conduct costly evaluations/hire strong technical talent, in a way that doesn’t compromise independence? One partial idea floated here was to have the AISIs of the world facilitate this (assuming that governments have a source of/allocate funding) - they can then dispense money to individual organizations.
• If these evaluator organizations are themselves private companies, what kind of business model can support this work without creating misaligned incentives? It seems like the most straightforward revenue play for eval companies is to sell training data. But that can be dicey, because a data vendor might be incentivized to construct benchmarks that show results which promote more spend on data
• How does one avoid “evaluator shopping”, e.g. model providers picking evaluators that are more likely to produce findings that align with their commercial interests?
• What are the kinds of findings that should be shared with broader audiences, and who are those audiences (e.g. governmental bodies, the public at large)? Evaluations related to misuse like bioterrorism, for example, represent cases where it does seem in the public interest to withhold details from broad audiences, to avoid revealing information about potential vulnerabilities. But at the same time, public disclosure of procedures feels important to ensure that the evaluations themselves are rigorous.
• What information should evaluators have access to? Model checkpoints were raised as a valuable artifact (in addition to release candidates) - I hadn’t considered that previously but that seems really useful, to study a model’s “ontogeny” for certain capabilities/behaviors
• What are the actual artifacts that should be evaluated? Most of the work that I was familiar with involved testing of individual models. Here I learned that in the embedded evaluation pilots with OpenAI, Redwood Research and METR evaluated something a bit different - safety systems (in this case internal monitoring systems). It also seems plausible to evaluate operational practices, kind of analogous to how food safety instructors check for Current Good Manufacturing Practice, like washing hands, pest control, etc

The discussions also spurred me to look more into the provisions of SB 813. The law creates a state-level agency that will develop a set of criteria for third-party organizations to be accredited as IVOs. Model developers are not required to engage an IVO to build or deploy models; the idea is that the stamp of approval from a certified IVO can be a competitive advantage, which creates an incentive for companies to pursue auditing. The policy creates a “regulatory market”, where multiple IVOs compete to provide high-quality services, and the government serves as a sort of meta-regulator defining high-level constraints. One of the key architects of this regulatory markets model is Prof. Gillian Hadfield - one of the first books I read in grad school, back in 2023, was her book Rules for a Flat World, which explores regulatory markets as a legal tool to allow governance to keep up with fast moving economic and technological developments (I read it for reasons completely unrelated to AI). Funny how things resurface!

Overall, I found the conference really informative - for me, the most valuable parts were hearing directly from individuals with boots-on-the-ground experience conducting this work. There were other components that were more high-level and accordingly felt somewhat untethered from the scientific practices/implementational considerations of evaluation. I don’t think that existing practices should necessarily constrain imagination - in no small part because the continued improvement in model capabilities also create the potential for more sophisticated forms of measurement. But when thinking about organizational design, it feels important to take into account the limitations of evaluations - namely the fact that they are highly imperfect proxies for fuzzy concepts, and currently cannot provide (or really even get close to) guarantees about real-world behavior. This is even before the practical/logistical considerations, perhaps the most important of which is the difficulty/long timelines of government regulation.

Maybe the most salient takeaway is that policy people and AI people working on the frontier need to get themselves into the same room more often - the technical and scientific details are needed to make sure policy actually achieves its desired goals, and awareness of the policy landscape is likewise crucial for steering technical/scientific work in fruitful directions.

RSS feed