Vals Aims to Be Gold Standard for AI Benchmarking

The announcement of a $40 million Series A for Vals, an AI evaluation startup, has focused attention on a critical bottleneck in enterprise AI adoption: the inadequacy of public leaderboards. Backed by Andreessen Horowitz and others at a reported $400 million valuation, Vals aims to become a “gold standard” by measuring model performance on real-world professional tasks in finance, law, and coding, rather than on abstract knowledge questions. While its U.S. centric approach and proprietary methods are noteworthy, the underlying strategy offers a powerful template for operators in the Gulf. For enterprises in the UAE, Saudi Arabia, and Qatar, the path to deploying AI agents against sensitive, high-value workflows does not run through a public benchmark. It requires building a private, sovereign evaluation capability that tests models against local documents, bilingual workflows, and regional regulations, all inside the enterprise’s own secure walls.

The Crisis in AI Evaluation

The promise of generative AI in the enterprise depends on a simple premise: that a model’s demonstrated skill on a test will translate to reliable performance in production. Today, this premise is broken. The public leaderboards that dominate headlines are plagued by systemic issues that make them poor predictors of business value and operational risk. Vals’s critique, echoed by practitioners worldwide, centers on several key failures.

First is benchmark saturation and data contamination. The test sets for many popular benchmarks have, intentionally or not, leaked into the training data of large language models. As a result, models are not demonstrating reasoning or capability; they are recalling answers to questions they have already seen. This leads to score inflation that has no connection to a model’s ability to handle a novel problem inside an organization. An operator choosing a model based on a contaminated benchmark is flying blind.

Second is the poor correspondence with economic work. Most benchmarks test for factual recall or academic problem solving. An enterprise workflow, however, is a complex chain of actions: retrieving information from multiple documents, using software tools, extracting structured data, comparing clauses, and making a decision under uncertainty. A model might correctly answer “Who was the third U.S. president?” but fail completely when asked to compare termination clauses across three different Arabic and English service agreements. Vals’s stated focus on professional workflows is a direct response to this gap. Its methodology, reportedly rooted in Stanford NLP research, attempts to measure performance on tasks that resemble the work of a financial analyst, lawyer, or programmer.

Finally, public benchmarks suffer from vendor-selected metrics. Model providers often highlight the benchmarks where their models perform best, creating a confusing and often misleading marketing landscape. Without an independent, trusted, and neutral measurement layer, enterprises are left to conduct their own ad hoc, inconsistent, and non-reproducible tests. This slows down procurement, complicates governance, and makes it impossible to compare competing models on a true, apples to apples basis.

The Vals Model: A Template for Professional-Grade Testing

Vals is positioning itself as the solution to this crisis by creating what it hopes will be a trusted, independent standard. According to its public materials, its team includes alumni of technically intensive firms like NVIDIA, Palantir, and Jane Street, suggesting a focus on rigorous, quantitative measurement. The company’s approach, as described in materials dated around early October 2026, has several characteristics that distinguish it from a typical leaderboard.

Task-Based and Agentic Evaluation

Instead of simple question answering, Vals focuses on evaluating a model’s ability to complete multi-step professional tasks. The Vals Index is reportedly weighted by GDP categories, with initial coverage for finance, coding, legal, and tax. For example, a result published on September 30, 2026, placed Google’s Gemini 4 Argon at the top of the index with a score of 68.9%, citing its leading performance on a “Finance Agent v2” task. This reflects a shift from evaluating the raw model to evaluating the entire agentic system: the model plus the tools it uses to accomplish a goal. The company’s Web Search Index, for instance, specifically tests whether an agent, when given a search tool, can successfully complete a professional task. This end to end evaluation is far more representative of how AI will be deployed in an enterprise setting.

Controlled and Private Datasets

A core part of the Vals argument is that trustworthy evaluation requires control over the test set. To prevent leakage and “teaching to the test,” a benchmark provider must maintain private, sealed datasets. This ensures that a high score reflects genuine capability, not memorization. For enterprises, this principle is even more critical. The most valuable test data an organization possesses is its own archive of contracts, reports, engineering documents, and customer interactions. Using these internal, confidential documents for evaluation is the only way to know for sure how a model will perform on the specific tasks that matter to the business. This cannot be done on a public service.

Why U.S. Benchmarks Fail in the Gulf

Even if Vals achieves its goal of becoming the “gold standard” for U.S. professional work, its direct utility for a bank in Riyadh, a sovereign wealth fund in Abu Dhabi, or a logistics firm in Doha is limited. The context, language, and regulatory landscape of the GCC are fundamentally different. A model that excels at analyzing U.S. tax law may be useless, or even dangerously wrong, when applied to Zakat calculations or VAT compliance in Saudi Arabia.

Regional enterprises face a unique set of challenges that off the shelf benchmarks cannot address:

• Bilingual Complexity: Business operations in the Gulf are deeply bilingual. Documents frequently code switch between Arabic and English within the same sentence. A robust evaluation must test a model’s ability to handle this fluid mix, including local terminology, transliteration variants of names and places, and the correct alignment of legal and commercial concepts between the two languages. Fluent translation is not enough; the model must demonstrate conceptual and legal fidelity.
• Jurisdiction Specific Regulations: A compliance workflow in the UAE is governed by a different set of rules than one in Qatar or Saudi Arabia. An effective benchmark must test a model’s ability to correctly identify and apply obligations from local laws, such as the UAE’s privacy regulations, Saudi Arabia’s Personal Data Protection Law, or the Qatar Financial Centre’s rulebook. This includes understanding the hierarchy of laws, circulars, and internal policies.
• Local Document Formats and Conventions: From government procurement documents and engineering change orders to KYC identity verification, Gulf-based organizations work with unique document formats. A regional evaluation needs to test a model’s resilience to local conventions, such as Gulf specific date formats, currencies, company identifiers, and address structures.
• Sector Specific Nuances: Key sectors in the region have their own specialized language. In finance, this includes the specific terminology of Islamic finance products and Sharia compliant contracts. In the energy sector, it involves proprietary technical standards and operational reports. A generic benchmark for “finance” or “legal” work will miss these critical, high value use cases.

For these reasons, a GCC enterprise cannot simply adopt a U.S. or global benchmark. It must develop its own evaluation capability, one that is sovereign to its own data, workflows, and regulatory obligations.

Building a Private, Sovereign Benchmark Agent

The most effective path forward for a Gulf enterprise is to use the Vals model as a template, not a service. The goal is to build an internal, automated evaluation system—a private benchmarking agent—that runs continuously inside the company’s own secure network. Such a system provides the trusted, specific, and governance-ready evidence needed to move AI from pilot projects to production.

An architecture for a private benchmark agent would include several key components:

1. A Private, Versioned Corpus: The foundation is a curated and versioned set of test cases built from the organization's own (redacted and anonymized) documents. This should include representative examples of contracts, compliance reports, KYC files, engineering specifications, and customer inquiries. It must also include a “golden” holdout set, kept completely separate from any model training or fine tuning, to serve as a final, unbiased test before deployment.
2. A Standardized Evaluation Harness: This is the execution engine. It runs the exact same task—with the same prompt, the same retrieval context, and the same tools—across multiple models. This allows for a fair comparison of a commercial API from a U.S. provider, a fine tuned open weight model hosted on premises, and a model from a Chinese provider.
3. Multi-Dimensional Scoring: The system must score more than just the accuracy of the final answer. It needs to capture a comprehensive performance profile: the quality and relevance of its source citations, whether it missed a key obligation, its latency, the cost per task, the number of tool calls, and its failure modes (e.g., hallucination, refusal to answer, incorrect formatting).
4. Human Review and Adjudication Workflow: For high stakes tasks, automated scoring is not enough. The system should route ambiguous or high risk outputs to human domain experts (lawyers, compliance officers, senior engineers) for review. It should record their judgments, track agreement rates between graders, and provide a clear process for adjudicating disagreements.
5. Governance Ready Dashboards and Logs: The output cannot be a simple leaderboard. It must be a governance dashboard that provides auditable evidence for risk committees, boards, and regulators. The dashboard should show performance trends over time, sliced by model, business unit, language, and risk tier. Crucially, the system must generate immutable logs that can reconstruct precisely why a particular model version was approved for a specific use case.

What this means for operators in the Gulf

For enterprise AI operators in the UAE, Saudi Arabia, and Qatar, the rise of evaluation-focused startups like Vals is a clear signal: public benchmarks are irrelevant for serious production workloads. Your focus should not be on chasing the top model on a global leaderboard, but on building a sovereign evaluation capability. This means creating a private, internal benchmarking system that tests models against your company’s unique documents, bilingual workflows, and regional regulatory requirements. This private benchmark becomes your organization’s single source of truth for AI performance, enabling you to compare vendors, manage risk, and provide auditable proof of control to your board and to local regulators. This is the operational discipline required to turn AI’s promise into a durable competitive advantage.

From Procurement Hurdle to Continuous Governance

A private benchmark transforms AI evaluation from a one time procurement hurdle into a continuous, strategic governance function. It addresses the most common operational bottlenecks that stall enterprise AI programs. Where teams struggle with a lack of representative test data, the private benchmark uses the company’s own documents. Where they fear benchmark gaming, a private holdout set provides an honest signal.

This capability also creates true vendor agnosticism. An enterprise with a robust private benchmark can evaluate any model, regardless of its origin. It can compare the performance and cost of leading U.S. commercial models against the latest Arabic-capable open weight models, running securely on a private platform like a DHIA workspace. It can also assess models from China or other regions, testing them against its own criteria for performance, safety, and alignment without relying on external claims.

This is particularly important as model capabilities are not static. A model provider can push an update that inadvertently degrades performance on a specific, critical task. Without continuous regression testing through a private benchmark, this drift might go unnoticed until it causes a serious operational failure. A private benchmark automates this testing, ensuring that the models in production today are as good as or better than the ones that were initially approved.

Evaluation as a Regulated Activity

For regulated industries like banking, healthcare, and government services, the evaluation process itself must be treated as a regulated activity. The benchmark system will necessarily process sensitive and confidential information, from personal data in KYC files to privileged legal analysis. Therefore, the evaluator must run within the organization’s approved data residency boundary—whether on premises or in a designated sovereign cloud.

A private evaluation framework allows an organization to build the evidence package that regulators require. A high score on a private benchmark is not, by itself, authorization to deploy. It is, however, the foundational evidence for a comprehensive risk assessment. A complete governance file for a regulator would include the benchmark version, the provenance of the test data, the exact model version and configuration, and the auditable logs of human review. This demonstrates a level of operational maturity and control that is impossible to achieve when relying on public benchmarks.

Such a system provides the technical underpinning for responsible AI governance, creating an unbreakable chain of evidence from raw model capability to authorized business deployment. When deployed within an on premises AI platform, it ensures that every aspect of the evaluation—from the documents to the prompts to the grader’s notes—remains under the organization’s exclusive control.

The takeaway

Public AI benchmarks are becoming increasingly irrelevant for serious enterprise use cases. The strategic move for organizations, particularly in specialized environments like the GCC, is not to find a better public benchmark, but to reject them as a primary tool. Building a private, sovereign evaluation capability is the only way to measure what truly matters: how a model performs on your specific workflows, with your documents, and within your risk appetite. This is the foundation for moving from experimentation to production with confidence and control.

Request a demo