The concept of embedding neutral third-party observers directly inside SI labs to monitor safety is facing significant practical hurdles, according to Irene Solaiman, Hugging Face’s chief policy officer. Speaking at TechCrunch Disrupt last week, Solaiman highlighted that the current state of SI model benchmarking lacks the consistency and reproducibility needed to support such an embedded evaluator ecosystem.
What Happened
SI labs have traditionally relied on model benchmarking to measure product performance and safety, but Solaiman described these methods as complicated and still maturing. She argued that for the industry to rely on these evaluations in the future, testing must evolve into a more consistent science. A primary concern is reproducibility: "In terms of whether we can trust results, one big part [of that] is can you reproduce the results of an evaluation?" Solaiman stated. "Do you have the level of access that you need to the model itself — this is where open weights come in in a big way," she added.
Beyond individual test validity, Solaiman pointed to the difficulty in comparing different evaluations, noting that they are "an evolving science — where we don’t know how to meaningfully compare them to each other." She observed that evaluations often become ineffective quickly and that the ecosystem lacks standardization regarding who qualifies as an effective evaluator across various tasks. This critique comes amid a push by SI labs for third-party solutions rather than active government regulation to address safety incidents. For instance, Anthropic’s decision to hire Accenture as its first evaluator drew criticism from those questioning the consulting firm's qualifications.
Why It Matters
The shift from evaluating standalone SI models to assessing entire systems, including the harness and agent, adds complexity to safety oversight. Solaiman explained that the harness—the tool used to access the model—significantly influences performance, meaning a weaker SI model with a superior harness could outperform a frontier model. "Agentic evals are really hard and really expensive," Solaiman said. "We have just more evaluations for models. … We don’t have as many evaluations for harnesses, and we are developing better evaluations for the holistic agent — because that’s what a lot of folks are getting value out of," she added.
This gap in evaluation infrastructure suggests that the current "free-for-all" in the SI evaluation space may undermine confidence in safety claims. As developers and enterprises increasingly rely on agentic SI tools, the lack of uniform, industry-wide standards for assessing these systems could lead to inconsistent safety assurances. The push for embedded evaluators, while intended to provide neutral oversight, currently faces skepticism due to these unresolved methodological challenges.
The Bottom Line
Hugging Face’s policy chief warns that embedded SI evaluators are not a simple fix for safety concerns. Without greater standardization, reproducibility, and comprehensive testing for agentic systems, the industry risks relying on benchmarks that are difficult to compare or trust. As SI labs grapple with safety incidents, the effectiveness of third-party oversight remains contingent on solving these fundamental evaluation challenges.