ARC-AGI-3 Drops Frontier AI Models Below 1%: The First Benchmark That Tests Whether AI Can Actually Learn

Update: The sub-1% and 12.58% figures below are launch-day, March 25, 2026 snapshots. Scores moved materially in the following weeks as teams adapted agent architectures to the benchmark. Read the follow-up: why current models now score in the low double digits.

The ARC Prize Foundation launched ARC-AGI-3 on March 25, 2026, a new interactive AI benchmark designed to test whether systems can explore unfamiliar environments, infer goals, and solve problems without any instructions. At launch, every frontier model tested scored below 1%: Gemini 3.1 Pro hit 0.37%, GPT-5.4 reached 0.26%, Claude Opus 4.6 managed 0.25%, and Grok-4.20 scored 0.00%. Humans solved 100% of environments with no prior training. The competition offers $2 million in prizes, with a $700,000 grand prize for the first agent to achieve human-level performance. All winning solutions must be open-sourced.

Two days before the launch, NVIDIA CEO Jensen Huang told Lex Fridman “I think we’ve achieved AGI.” ARC-AGI-3’s launch-day results arrived as a stark counterpoint on this specific axis. The benchmark does not test knowledge, coding ability, or language comprehension. It tests whether AI systems can adapt to completely novel situations the way humans naturally do. At launch, the gap on that specific metric was enormous; how much of it closed with dedicated engineering effort is exactly what the follow-up piece linked above covers.

What Changed From ARC-AGI-1 and ARC-AGI-2

ARC-AGI-1 (2019) and ARC-AGI-2 (2025) presented static grid puzzles: show a model input-output pairs, ask it to infer the transformation rule and produce the correct output for a new instance. Frontier models reached 90%+ on version 1 by 2025, largely through scaffolding techniques (wrapping models in test-time compute loops with verification). ARC-AGI-2 raised difficulty with compositional puzzles, but the format remained the same: observe patterns, produce outputs.

ARC-AGI-3 abandons static puzzles entirely. Each of the 135 environments is a turn-based interactive game built by an in-house game studio. The agent sees a visual state, takes an action, observes the result, and must figure out both what it is trying to do and how to do it. There are no instructions. No stated goals. No hints. No descriptions. The agent must explore, form hypotheses about the game’s rules, and execute a plan. This is the first major format change since Chollet introduced the original benchmark in 2019.

How the Scoring Works

ARC-AGI-3 uses Relative Human Action Efficiency (RHAE). The baseline is the second-best first-run human performance on each environment. If a human completes a level in 10 actions and an AI takes 100 actions, the AI does not score 10%. The formula squares the ratio: (human actions / AI actions) squared. So 10x more actions produces a 1% score, not 10%. The penalty for inefficiency is deliberately harsh. Wandering, backtracking, and guessing are punished quadratically.

A hard cutoff stops AI agents at 5x the human action count. If a human takes 10 actions to complete a level, the AI is terminated after 50 actions. This prevents models from brute-forcing solutions through exhaustive exploration. The scoring system measures learning efficiency, not just task completion: can the agent figure out the rules and act on them with human-like economy of action?

Why Frontier LLMs Failed This Badly at Launch

The sub-1% launch scores were not a function of perception. A Duke University team built a custom harness for Claude Opus 4.6 that scored 97.1% on a single known environment variant (TR87). The same model scored 0% on unfamiliar environments. This demonstrates that the bottleneck is not visual processing or API format comprehension. Claude can see the game state clearly. At launch, it could not generalize strategies to environments it had not been specifically engineered to handle.

The interactive format exposes a limitation that static benchmarks never tested: sustained sequential reasoning across hundreds of steps, state tracking over long horizons, and learning from environmental feedback in real time. Language models are trained to produce the most likely next token given a context. ARC-AGI-3 requires forming a model of an unknown dynamic system, testing hypotheses through action, and revising understanding based on results. That capability does not emerge from scale alone, though the follow-up piece covers how much of it emerged from better agent scaffolding.

The 12.58% That Mattered More Than 0.37% at Launch

During the 30-day developer preview, the best-performing system scored 12.58%. It was not a frontier LLM. It was a simpler RL and graph-search approach built by Tufa Labs. That score outperformed every frontier model at the time by more than 30x. The implication at launch was direct: the path to solving ARC-AGI-3 runs through algorithmic innovation in sequential decision-making under uncertainty, not through scaling language models alone. Classical AI techniques (reinforcement learning, search, planning) outperformed the most expensive models in the world on tasks that require genuine adaptation, at least in this early snapshot.

This finding aligns with what researchers have observed in agentic AI more broadly: strong early results often come from hybrid approaches that combine LLM reasoning with structured search and planning, rather than from end-to-end LLM generation. ARC-AGI-3 provides a quantitative benchmark for measuring this gap, though as the follow-up piece shows, the gap is not static.

What ARC-AGI-3 Does and Does Not Measure

What This Means for the AGI Timeline

OpenAI, Google DeepMind, Anthropic, and xAI all report ARC scores on their model cards. At launch, none of them were close on ARC-AGI-3. The benchmark’s competition runs through December 2026 with milestone checkpoints in June and September. The competition requires open-source solutions with no external API calls during evaluation, meaning entrants cannot rely on frontier model inference alone.

Huang’s “AGI is here” and ARC-AGI-3’s 0.37% launch-day score coexist because they measure fundamentally different things. Huang means AI can perform most economically valuable tasks better than most humans most of the time, which is defensible on its own terms. ARC-AGI-3 measures adaptive reasoning in environments where training data provides zero advantage, where models must learn from scratch through interaction. At launch, the gap on that specific metric was wide. Whether AGI has arrived depends entirely on which definition you use. ARC-AGI-3 makes the definitional choice explicit and measurable, and its scores as of this update should be read as a snapshot in a moving series, not a final verdict.

Sources: ARC-AGI-3 Technical Report, ARC Prize Foundation, March 25, 2026; ARC Prize 2025 Results Analysis; Decrypt analysis; The Decoder coverage; DEV Community technical breakdown; ARC Prize 2026 Kaggle competition page. Updated 2026-08-18: reframed launch-day figures explicitly as launch-day figures, softened “first benchmark that tests whether AI can actually learn” to “new interactive AI benchmark designed to test” exploration and adaptation, and added a dated update linking to this publication’s later coverage of how scores moved.

Francois Chollet created the original ARC in 2019 alongside his paper “On the Measure of Intelligence,” which argued that intelligence should be measured as skill-acquisition efficiency rather than task-specific performance. ARC-AGI-3 is among the more complete implementations of that philosophy to date: a benchmark where the only way to score well is to learn quickly from scratch. The $2 million prize pool, the open-source requirement, and the Kaggle infrastructure mean that the solutions will be public and reproducible.