Tag: LLMs

A zero-parameter algorithm that copies directly from its own input context outperforms every major time-series foundation model on predicting chaotic systems, turbulence, coupled oscillators, and electrocardiograms. It costs six orders of magnitude less to run. The paper, accepted at ICLR 2026, is not proposing a replacement for foundation models. It is exposing what those models actually do when they appear to work, and it is not what anyone assumed.

Yuanzhao Zhang of the Santa Fe Institute and William Gilpin of the University of Texas at Austin built the simplest possible forecasting algorithm. Given a time series context, scan it for nearly repeating motifs. Find the best match to the current state. Copy whatever came after that match as your prediction. No learned weights. No training data. No gradient descent. The entire algorithm is a nearest-neighbor lookup in delay-coordinate space, executable on a CPU in milliseconds.

They tested it against Chronos, Chronos Bolt, TimesFM, TimeMoE, and Moirai across chaotic attractors (Lorenz, Rössler, double pendulum), turbulent fluid dynamics, coupled Kuramoto oscillators, and real-world EKG recordings. Context parroting won on both forecast error (sMAPE) and attractor reconstruction fidelity (KL divergence) across every system tested. The computational gap ranged from five to six orders of magnitude.

How Context Parroting Works

The algorithm operates in delay-coordinate embedding space, a technique from nonlinear dynamics dating to the 1981 Takens embedding theorem. Given a scalar time series x(t), construct delay vectors by taking D consecutive values: [x(t), x(t+1), …, x(t+D-1)]. Each delay vector represents the state of the system at time t in a D-dimensional space. Takens proved that for a deterministic system with an attractor of dimension d, choosing D greater than 2d reconstructs the topology of the attractor from the scalar measurements alone.

Context parroting uses this embedding to find the best match to the current state within the context window. The algorithm constructs delay vectors from the entire context, computes the Euclidean distance between the most recent delay vector and every earlier delay vector, finds the nearest neighbor, and copies the trajectory following that neighbor as the forecast. If the nearest neighbor occurred at time t* in the context, the forecast is simply x(t*+1), x(t*+2), and so on for as many steps as needed.

This is mathematically identical to a first-order local model in the sense of Farmer and Sidorowich (1987), one of the foundational methods in nonlinear time series prediction. The difference is that context parroting runs entirely inside the context window with no separate training phase. It is, functionally, an in-context nearest-neighbor algorithm.

The connection to Takens embedding is not incidental. It is the reason the method works at all. Takens’ theorem guarantees that the delay-coordinate reconstruction preserves the diffeomorphic structure of the original attractor. Nearby points in the reconstruction correspond to nearby points on the true attractor, which means nearby states evolve similarly in time. This is why nearest-neighbor forecasting in delay space produces accurate predictions: it exploits the geometric continuity of the dynamics. Without the embedding theorem, copying from the nearest neighbor would be random guessing. With it, copying is a geometrically principled operation grounded in 45 years of dynamical systems theory.

Why It Beats Foundation Models

The paper identifies a specific failure mode shared by TimesFM, TimeMoE, and Chronos Bolt: they systematically underestimate oscillations and converge toward the mean. Given a chaotic system that swings between extremes, the foundation models predict a trajectory that dampens too quickly and settles near the average value. This is consistent with training objectives that minimize average prediction error across diverse datasets. Predicting the mean is the safest strategy for minimizing loss across many different distributions. It is also the wrong strategy for any specific dynamical system.

Chronos is the exception. It performs well precisely because it implements something close to parroting internally. The paper shows that Chronos frequently copies motifs from the context window when forecasting chaotic systems. When Chronos works, it works because it parrots. When foundation models fail, they fail because they don’t parrot enough and instead fall back on mean-convergent predictions learned from pretraining.

This explains a finding that puzzled the time-series community: large language models trained on text, with no time series in their training data, can sometimes forecast dynamical systems competitively. The mechanism is induction heads, the attention pattern that identifies repeated sequences and copies what follows. Induction heads are a form of context parroting. LLMs can forecast time series not because they understand physics but because they learned to copy repeating patterns from text, and that same copy mechanism transfers to time series.

The Fractal Dimension Scaling Law

The paper’s most original contribution is connecting forecast accuracy to the fractal dimension of the underlying attractor. Context parroting works by finding near-recurrences in the context. The Poincaré recurrence theorem guarantees that an ergodic system will eventually return arbitrarily close to any previous state, but the waiting time depends on the dimensionality of the attractor. For a system with correlation dimension d, the expected recurrence time scales as L ~ epsilon^(-d), where epsilon is the matching tolerance and L is the required context length.

This produces a scaling law: forecast accuracy improves as a power law in context length, with the exponent determined by the fractal dimension of the attractor. Low-dimensional chaotic systems (Lorenz, d approximately 2.05) need shorter contexts for accurate parroting. High-dimensional systems (turbulence, d much larger) need exponentially longer contexts. The paper validates this scaling law empirically across multiple systems and shows it explains previously observed in-context neural scaling laws for time series forecasting.

The practical implication is quantitative. For a system with known fractal dimension, you can calculate exactly how much context data you need for parroting to reach a target accuracy. This is something no foundation model can tell you because their performance depends on training data composition, not on the mathematical structure of the target system.

What This Does Not Mean

The authors state explicitly that they are not proposing to replace foundation models with context parroting. The value of parroting is as a baseline that reveals gaps. When a foundation model underperforms relative to parroting, it means the model has not learned to use the context data effectively. The failure is not that the model is bad. The failure is that a copy-paste algorithm does better, which means the model is leaving information on the table.

Context parroting has clear limitations. It assumes stationarity: the underlying dynamics must not change over the forecast horizon. It cannot handle distribution shifts, trend changes, or regime transitions. It struggles with non-stationary real-world time series (weather, financial markets, traffic) where the generating process itself evolves. Foundation models handle simple nonstationarity like baseline drift because their pretraining covers such patterns. The authors suggest generalizing parroting to handle nonstationarity as a future direction.

The algorithm also requires that the context contains a near-recurrence of the current state. For high-dimensional systems, the context may not be long enough to contain a good match. In these cases, parroting produces poor forecasts and foundation models that generalize from pretraining would outperform it. The fractal dimension scaling law tells you exactly when this happens: when the required context length exceeds the available context window.

Why This Matters for the Foundation Model Race

Every major AI lab is building or acquiring time-series foundation models. Google has TimesFM. Salesforce has Moirai. Amazon backed Chronos. The premise is that pretraining on massive time series datasets produces models that generalize to unseen systems. Context parroting challenges that premise by showing that, for an important class of systems, generalization from pretraining adds nothing. The context alone is sufficient.

This does not kill the foundation model thesis. It narrows it. Time-series foundation models add value when they handle nonstationarity, distribution shifts, and systems where the context window is too short for recurrence. They fail to add value, and actively harm performance, when the system is stationary and the context contains enough recurrences. Knowing which regime you are in determines whether a billion-parameter model is worth its inference cost.

For practitioners running time series forecasting in production, the actionable takeaway is to benchmark against context parroting before deploying a foundation model. If parroting beats your model, you are paying for compute that is worse than free. If your model beats parroting, you have evidence that pretraining is contributing something beyond pattern copying. Either answer is useful. Not knowing which regime you are in is not.

The deeper implication connects to a recurring pattern in machine learning: the simplest baseline, properly constructed, often outperforms complex systems that were never tested against it. When the baseline is missing, the community overestimates how much the complex system has learned. Context parroting fills that gap for time series. The question it forces every foundation model team to answer: what, exactly, did you learn from pretraining that a copy-paste algorithm cannot recover from the context alone?

Sources: arXiv:2505.11349 (Zhang and Gilpin, ICLR 2026). OpenReview (ICLR 2026 acceptance). Takens, “Detecting Strange Attractors in Turbulence” (1981). Farmer and Sidorowich, “Predicting Chaotic Time Series” (1987). Chronos (Ansari et al., 2024). TimesFM (Das et al., 2024). Moirai (Salesforce, 2024).

Sakana AI, the University of British Columbia, and the Vector Institute presented a paper at ICLR 2026 describing the Darwin Gödel Machine (DGM), an AI system that rewrites its own source code to become better at programming tasks. On SWE-bench, a benchmark requiring agents to resolve real-world GitHub issues, DGM improved its own score from 20.0% to 50.0%. On Polyglot, a multi-language coding benchmark, it jumped from 14.2% to 30.7%. These are real performance gains produced by automated self-modification. They are not what the headline “self-improving AI” implies.

What the System Actually Modifies

The DGM does not modify the underlying foundation model. It does not rewrite neural network weights. It does not retrain itself. The system modifies its own Python codebase: the tools, workflows, prompts, and control logic that surround a frozen pretrained language model. The foundation model (Claude 3.5 Sonnet in the primary experiments) stays exactly the same throughout the entire process. The “self” in “self-improving” is the agent environment, not the neural network.

This distinction matters. A system that can rewrite its own scaffolding code to become better at coding tasks is interesting and useful. A system that can rewrite its own neural architecture to become smarter at everything is something else entirely. The DGM is the former, not the latter. The paper’s authors are clear about this. Their framework “envisions agents that can rewrite their own training scripts (including training a new foundation model),” but they explicitly state that retraining models is computationally intensive and left as future work.

How the Self-Modification Loop Works

The DGM alternates between two phases: self-modification and evaluation. During self-modification, the system reads its own Python codebase and proposes changes. These might be adding a new tool (like a patch validator), improving file viewing capabilities, building better editing commands, implementing a system that generates multiple solutions and ranks them, or adding a memory of what has been tried before and why it failed.

During evaluation, the modified agent is tested on coding benchmarks. If the modified version scores better, it gets added to an archive of agents. If it scores worse, it may still be kept if it represents an interesting variation that could lead to future improvements. This is the “Darwin” part: inspired by biological evolution, the system maintains a growing population of diverse agents rather than keeping only the single best performer.

The evolutionary archive is the key innovation. Traditional optimization would keep only the highest-scoring agent and modify from there, risking getting stuck in local optima. The DGM maintains an archive of diverse agents and can branch new modifications from any of them. The paper shows that some low-scoring “ancestor” agents produced descendants that eventually outperformed the best agents found through greedy optimization. The branching exploration, not just the self-modification, drives the results.

The Results Transfer Across Models and Languages

The improvements discovered by the DGM generalize beyond the specific setup used during self-modification. An agent optimized using Claude 3.5 Sonnet also showed improved performance when run with o3-mini or Claude 3.7 Sonnet as the underlying model. A DGM whose self-improvement was guided exclusively by Python tasks showed significant gains on Rust, C++, Go, and other languages in the Polyglot benchmark.

This transferability suggests the DGM is discovering general agent design improvements (better tools, smarter workflows, more effective prompting strategies) rather than model-specific tricks or task-specific overfitting. The improvements work because they change how the agent approaches problems, not because they exploit quirks of a particular model or language.

What the DGM Discovered

The paper documents specific innovations the DGM invented for itself. Early in the SWE-bench run, it developed improved file viewing and editing tools. Later, it discovered a patch generation strategy that creates multiple candidate patches and ranks them by quality before applying the best one. It built a memory system tracking which approaches failed on similar problems. These are the same types of improvements that human developers make when building coding agents by hand, but the DGM found them through automated search rather than human engineering.

What the DGM Cannot Do

The system requires substantial computational resources. Each self-modification cycle involves running the modified agent on benchmark problems, which means hundreds of API calls to the underlying foundation model per evaluation. The process scales with the number of agents explored and benchmark problems evaluated.

The DGM’s exploration process and archive management are fixed algorithms that the system cannot modify. The agent can rewrite its coding tools, workflows, and prompts, but not the meta-algorithm that governs how self-modification happens. This is a deliberate safety constraint but also a fundamental limitation: the system cannot improve the way it improves. True recursive self-improvement would require the meta-algorithm itself to be subject to modification, which the authors leave as future work.

All experiments ran in sandboxed environments with human oversight. The safety considerations around self-modifying AI are not hypothetical. The DGM’s modifications are constrained to Python code changes evaluated on benchmarks, not arbitrary system-level access. But as these systems become more capable, the gap between “can modify its own coding tools” and “can modify anything” narrows, and the sandboxing requirements become more demanding.

Where This Fits in the Research Trajectory

The Gödel Machine concept dates to Jürgen Schmidhuber’s theoretical proposal decades ago: an AI that proves its own modifications are beneficial before applying them. The DGM drops the requirement for formal proof and replaces it with empirical testing, trading theoretical guarantees for practical applicability. Concurrent work by Robeyns et al. (2025) explores a similar concept (single agent recursively modifying itself) but without the DGM’s open-ended archive, which the paper shows is necessary to avoid stagnation.

The practical implication is that automated agent design may soon match hand-designed agents. If the pattern holds, teams building AI coding agents will shift from manually engineering tools and workflows to running DGM-style search over agent designs. The DGM’s 50% on SWE-bench is not state-of-the-art (hand-designed agents score higher), but the rate of improvement suggests automated search could close that gap as compute budgets and foundation model capabilities increase.

The DGM is not self-improving AI in the science fiction sense. It is automated engineering of AI agent scaffolding, validated by benchmarks, constrained by sandboxes, and limited to the capabilities of its frozen foundation model. That is a more boring description. It is also a more accurate one, and the results it produces are real.

Sources: Zhang et al., arXiv: 2505.22954 (v3, March 2026). Sakana AI official page. GitHub: jennyzzt/dgm. ICLR 2026 poster. Schmidhuber, Gödel Machine (2007). SWE-bench original benchmark.

A paper published in Nature on March 25, 2026 presents the first AI system that autonomously completed the entire scientific research lifecycle: generating ideas, writing code, running experiments, analyzing results, producing a complete manuscript, and performing its own peer review. The manuscript it generated passed the first round of human peer review at a workshop affiliated with a top-tier machine learning conference. The workshop had a 70% acceptance rate.

The system is called The AI Scientist. It was built by researchers at Sakana AI, the University of Oxford, and the University of British Columbia, led by Chris Lu, Cong Lu, Robert Tjarko Lange, and Yutaro Yamada, with senior authors David Ha and Jeff Clune. The paper has already accumulated over 101,000 accesses and an Altmetric score of 481 in its first five days online. It is the most concrete demonstration to date that foundation models can produce research-grade scientific output without continuous human intervention.

Before the celebration or panic starts, two things need to be said plainly. First, the generated manuscript passed peer review at a workshop with a 70% acceptance rate, not a flagship conference or high-impact journal. Second, the system could not have built itself. It depends on human-designed templates, human-created evaluation criteria, and foundation models trained on human-written scientific literature. This is automation of a process, not replacement of the intelligence behind it.

How the System Works: Seven Stages, No Human in the Loop

The AI Scientist operates as a complex agentic system built on top of foundation models from OpenAI, Anthropic, and Meta. The pipeline has seven discrete stages, each handled autonomously.

Stage 1: Idea generation. The system generates research ideas by combining prompts with information about the current state of a research area. In “focused mode,” it receives a human-provided code template as a starting scaffold. In “open-ended mode,” it uses agentic search to explore research questions without templates.

Stage 2: Code implementation. The system writes the experimental code to test its idea. It generates Python scripts, sets up training loops, configures hyperparameters, and creates the infrastructure needed to run experiments.

Stage 3: Experiment execution. The system runs its own experiments on compute infrastructure. It manages training, handles errors, and collects results across multiple trials.

Stage 4: Data analysis. Results are processed, visualized, and statistically analyzed. The system generates plots, computes metrics, and identifies the key findings from its experimental runs.

Stage 5: Manuscript writing. The system produces a complete scientific paper. Introduction, related work, methodology, experiments, results, discussion, conclusion. The output follows standard machine learning paper conventions, including proper citation formatting.

Stage 6: Self-review. The system performs its own peer review, evaluating the manuscript for clarity, rigor, and contribution. This internal review can trigger revisions before the manuscript is submitted.

Stage 7: Automated review. A separate instance of the system evaluates the final manuscript using review criteria consistent with major ML conferences.

The system was evaluated in two settings. The focused mode used human-provided code templates as starting points for research on specific topics. The open-ended mode used AIDE (AI-driven exploration in the space of code) for wider scientific exploration without templates. Both settings produced diverse research ideas and complete, reviewable manuscripts.

What “Passed Peer Review” Actually Means

The most cited claim from the paper is that an AI-generated manuscript “passed peer review.” The specifics matter. The manuscript was submitted to a workshop co-located with a top-tier ML conference (ICLR). Workshops at major conferences operate with higher acceptance rates and less rigorous review standards than the main conference. This workshop accepted 70% of submissions.

Passing the first round of review means the manuscript was not desk-rejected and received reviewer scores consistent with acceptance. It does not mean the paper was published in a peer-reviewed journal. It does not mean the research was independently validated. It means the AI-generated paper looked enough like a competent machine learning workshop submission to pass initial screening by human reviewers who did not know the paper was machine-generated.

That achievement is still significant. A 70% acceptance rate means 30% of submissions were rejected. The AI system’s manuscript cleared a bar that nearly one-third of human-written papers failed to meet. But the framing matters: this is closer to “AI can write a passable conference workshop paper” than “AI can do science.”

The Architecture: Why It Works Now

Previous attempts at automated scientific research failed at the integration points between stages. A system might generate ideas but fail to implement them in working code. A system might run experiments but fail to interpret results. A system might write a manuscript but produce incoherent analysis. The AI Scientist succeeds because foundation models like GPT-4, Claude, and Llama 3 have become capable enough at each individual stage that the full pipeline holds together.

The key architectural decision is treating each stage as an independent agent task with well-defined inputs and outputs. Idea generation produces a research plan. Code implementation takes that plan and produces executable scripts. Experiment execution takes scripts and produces data. Each transition is a structured handoff, not a free-form conversation. This modular design means failures in one stage can be caught and addressed without cascading through the entire pipeline.

The system also uses what the authors call “agentic search,” particularly in the open-ended mode. Instead of exploring research questions randomly, the system uses a search process inspired by evolutionary algorithms to generate, evaluate, and refine ideas before committing compute to experiments. This produces more diverse and higher-quality research directions than pure random exploration.

What It Cannot Do

The honest limitations section is where this paper distinguishes itself from the hype cycle around AI research automation.

The AI Scientist cannot design novel experimental methodologies. It works within existing paradigms: standard ML training loops, established evaluation metrics, known architectures. The “ideas” it generates are variations and combinations of existing approaches, not conceptual breakthroughs. This is optimization within a defined search space, not the kind of creative leap that produces genuinely new scientific directions.

The system’s self-review is not independent verification. A system that generates a manuscript and then reviews its own work using the same underlying model cannot catch systematic errors in its own reasoning. The self-review functions as a quality filter (rejecting obviously bad output) rather than a genuine peer review (identifying subtle flaws in methodology or interpretation).

The manuscripts the system produces, while structurally correct, lack the contextual judgment that human researchers bring. A human scientist chooses a research question partly based on years of intuition about what the field needs, which problems are tractable, and which results would be surprising. The AI Scientist generates ideas that are technically executable, not ideas that advance scientific understanding in ways the research community recognizes as important.

The authors are explicit about risks. Taxing overwhelmed peer review systems with machine-generated submissions is a concrete near-term harm. Adding noise to the scientific literature, making it harder for researchers to identify genuinely useful work, is another. The same dynamics reshaping the software industry through AI automation apply here: more output at lower cost is only valuable if quality holds.

What This Means for Working Scientists

The immediate practical impact is on the grunt work of ML research. Running ablation studies, exploring hyperparameter spaces, writing up results in standard formats: these are time-consuming tasks where the AI Scientist could function as a research assistant. A human researcher who uses the system to quickly test ten variations of an idea, discards nine, and publishes the one that works has genuinely saved weeks of work.

The danger is the inverse: using the system to mass-produce papers that technically pass review but add nothing to scientific knowledge. ML conferences already face a submission volume crisis, with reviewers overwhelmed by thousands of papers per venue. A tool that makes it trivially easy to generate additional submissions could break the peer review system entirely.

A related paper published in Nature in January 2026, titled “Artificial Intelligence Tools Expand Scientists’ Impact but Contract Science’s Focus,” found that AI tools tend to narrow the range of topics researchers explore even as they increase output. If automated research systems follow the same pattern, the result could be more papers covering fewer ideas, the opposite of scientific progress.

The Competitive Context

Google DeepMind‘s AlphaEvolve, a Gemini-powered coding agent that pairs language models with evolutionary algorithms, has been used to discover new mathematical structures. Sakana AI, one of the institutions behind The AI Scientist, is a Tokyo-based startup founded by former Google Brain researchers David Ha and Llion Jones (one of the original “Attention Is All You Need” co-authors). The company raised $200 million in its Series A in 2024.

The paper’s publication in Nature rather than a preprint server signals that the journal’s reviewers found the work meets the bar for a flagship science publication. Nature’s acceptance rate is approximately 8%. The irony is thick: a paper about AI passing peer review had to pass a much more selective peer review process to be published.

What Happens Next

The open-ended mode of The AI Scientist, where the system explores research questions without human-provided templates, is the more consequential contribution. If that mode can produce papers that pass review at higher-quality venues (main conferences rather than workshops, journals rather than proceedings), the implications change from “useful research tool” to “credible research agent.”

The authors plan to extend the system to other scientific domains beyond machine learning. Chemistry, materials science, and biology all involve experimental workflows that could, in principle, be automated in the same way. Each domain introduces new challenges: physical experiments require robotic lab infrastructure, biological experiments require safety protocols that software experiments do not, and the gap between “technically correct” and “scientifically meaningful” widens in fields where human judgment plays a larger role in defining research questions.

For now, The AI Scientist is best understood as a proof of concept that works within narrow constraints. It can do machine learning research in domains where the experimental infrastructure is fully digital. It cannot yet do science in the way most scientists understand the word. The gap between those two statements is where the next decade of research automation will be built.

Sources: Lu et al., “Towards End-to-End Automation of AI Research,” Nature 651, 914-919 (March 25, 2026), AIDE: AI-Driven Exploration in the Space of Code (arXiv, 2025), “AI Tools Expand Impact but Contract Focus,” Nature (January 14, 2026), “Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy,” Nature Communications (September 2025).