Part IV: From LLM Wiki to the AI Scientist

Chapter 12: AI Scientist in Practice — AI/Robotics, Bio/Chem/Drug, Autoresearch

Written: 2026-04-28 Last updated: 2026-06-11

12.1 Two Existence Proofs, One Week Apart

In April 2026, two papers appeared within a week of each other.

Anthropic's AAR (Automated Alignment Researchers) [13]: 9 Claude Opus 4.6 instances ran weak-to-strong supervision research autonomously over 5 days. Result: PGR (Performance Gap Recovered) 0.97 vs human baseline 0.23. Total cost was about $18,000, or roughly $22 per AAR-hour. Each agent worked in its own sandbox and shared results through a forum and code storage [13].

Karpathy's autoresearch [4]: about 700 automated experiments over 2 days on a nanochat/GPT-2-style training loop. Public summaries report around 20 genuine improvements and an 11% improvement on the "Time to GPT-2" benchmark [15]. What humans had been tuning manually for months became a loop of hypothesis, code change, short training run, evaluation, and keep/revert.

In one week, autonomous agents produced production-grade results in both alignment research (Anthropic) and ML research (Karpathy). This is the evidence that the AI Scientist is not "future talk" but "happening now."

12.2 Three Stages of Research Democratization

Um[24] classifies AI-driven research into three stages:

S1: Document-bound Research

Already AI > human. Literature search, paper summarization, comparative analysis, review writing. Doable right now with Claude Code + LLM Wiki.

S2: In-silico Experiments

Experiments that complete inside a computer. ML training, simulation, data analysis. Karpathy's autoresearch and Anthropic's AAR are the clearest examples of this stage.

S3: Physical Experiments

Requires real labs, real chemicals, real robots. Still early, but the direction is clear.

S8 adds a more gradual ladder between these three stages [24]. LLM Wiki is the memory layer. Paper-to-agent workflows turn documents into runnable research tasks. Research-associate agents run bounded literature and analysis loops. Dry-lab AI Scientist systems close in-silico loops. Wet-lab systems add physical actuation and safety constraints. The stages are therefore not a slogan; they are a path from markdown memory to bounded autonomy.

Figure 12.1: Three stages of research democratization — S1 document-bound (AI greater than human now), S2 in-silico (autoresearch and AAR are production examples), S3 physical experiments (self-driving labs emerging). illustration by author Gemini assisted

12.3 ML / Alignment Research — Autonomous Agents

Karpathy's Autoresearch

Karpathy[4]'s design:

  1. Define the search space: GPT-2 training hyperparameter combinations (learning rate, batch size, architecture variants)
  2. Agent loop: propose → run → evaluate → propose next
  3. Record results: store all experiment configs and outcomes in the filesystem
  4. Meta-harness pattern: Chapter 9's meta-harness is exactly this loop
Figure 12.2: Karpathy autoresearch loop — define search space, propose experiment, run and evaluate, record to filesystem. 700 experiments in 2 days; roughly 11% Time to GPT-2 improvement. illustration by author Gemini assisted

Karpathy[4] trained a small GPT-4-sized model (nanochat) the same way in one day. Demonstrates the pattern's scalability.

Anthropic's AAR

Key finding from Anthropic[13]: 9 agents working independently was more effective than a single long-running agent. Distinct starting directions delayed idea convergence and improved hill-climbing efficiency. PGR 0.97 is more than 4x the 0.23 achieved by human experts on the same task.

But the result should not be oversold. Anthropic reported that the best method did not produce a statistically significant improvement on production-scale Claude Sonnet 4, and the AARs also found reward-hacking strategies [13]. The point is not "alignment research is automated"; it is that well-scored research slices with a sandbox can already be accelerated by agent loops.

Common implication of both cases: agent loop + filesystem-based experience accumulation + meta-harness pattern + metric design is the core structure of research automation.

The S8 lesson is to publish negative results honestly inside the loop. A research agent that only preserves successful hypotheses will overfit its own history. Failed experiments, dead ends, and contradiction notes are not clutter; they are the experience accumulation that prevents the next agent from repeating the same path.

12.4 Medical AI — Clinical Task Benchmarks

Wu et al.[13]'s Med-AI Bench (arXiv 2603.28589, summarized in Um[24]):

  • 19 clinical tasks
  • 171 cases
  • Scope: diagnosis, treatment planning, medical report generation, clinical reasoning

Results: AI exceeded human baselines on multiple clinical tasks. The gap was largest in documentation-heavy tasks (report generation, coding).

Note: This benchmark demonstrates AI's potential in clinical settings, but actual clinical deployment requires separate regulatory, ethical, and validation processes. AI Scientist can support clinical research; that is a different question from replacing clinical judgment.

12.5 AI Co-Scientist — Google's Multi-Agent Research System

Google's AI Co-scientist is a Gemini 2.0-based multi-agent research system published as arXiv 2502.18864 [11] [12]:

  • Architecture: Multi-agent system on Gemini 2.0
  • Loop: Generate → Debate → Evolve + Elo tournament
  • Validated domains: AML (Acute Myeloid Leukemia) drug repurposing, liver fibrosis

Key innovation: agents debate and critique each other's hypotheses. An Elo tournament automatically selects the strongest hypotheses. A March 2026 Nature Medicine feature frames this as a shift from chat toward hypothesis generation, with some AI-generated ideas moving into organoids, animals, and early-stage clinical work [17].

This is Chapter 9's dependency graph + multi-agent verification + meta-harness applied to the research domain.

Figure 12.3: AI Co-Scientist loop — Generate, Debate, Evolve. Nine agents propose hypotheses, critique each other, and an Elo tournament automatically selects the strongest. illustration by author Gemini assisted

12.6 Self-Driving Labs — Chemistry / Biology

In the last three months, the strongest S3 examples come from chemistry and biology self-driving labs. Rachel Brazil's March 2026 Nature feature describes the pattern as a physical research loop: AI forms a strategy, robots execute experiments, measurements feed the next round [18]. The robot is not just automation equipment; it is the actuator layer of hypothesis → experiment → measurement → next hypothesis.

RoboChem-Flex makes this concrete [19]. The April 2026 Nature Synthesis paper introduces a low-cost, modular self-driving laboratory with a Python control framework and Bayesian optimization. It validates autonomous closed-loop optimization across six case studies, including photocatalysis, biocatalysis, thermal cross-couplings, and enantioselective catalysis. The direction is important: from "only a very large lab can build this" toward platforms resource-limited labs can realistically adopt.

The strong parts are clear:

  • Experiment execution: liquid handling, robotic arms, and analytical instruments repeat protocols reliably
  • Search optimization: Bayesian, multi-objective, and transfer-learning workflows choose next conditions
  • Reproducibility: protocol, condition, and result logs become machine-readable

The weak parts are just as clear:

  • Problem framing: humans still define much of what should be discovered
  • Safety and regulation: chemicals, biological samples, instrument failure, and waste handling require separate systems
  • Data bias: AI-driven labs can reinforce existing reaction, reagent, and molecule-space biases [20]

The conclusion is not "labs are fully autonomous." It is more precise: the S3 bottleneck is shifting from human handwork to closed-loop physical infrastructure. If LLM Wiki is the memory layer for S1, self-driving labs are the actuator layer for S3.

S8 also connects this to the longer self-driving-lab lineage: Adam showed automated biological discovery as early as 2009; ChemCrow and related 2023 systems connected LLM planning to chemistry tools; RoboChem-Flex showed that closed-loop reaction optimization can move toward cheaper modular platforms. The arc is not one sudden jump to autonomous science. It is the gradual coupling of memory, planning, instruments, and safety gates.

12.7 Robotics — Not a Full AI Scientist Yet, But the Loops Exist

The previous version of this section said there were not enough robotics-specific AI Scientist examples. As of May 2026, that is too weak. A complete robotics AI Scientist is still early, but the last three months show at least three partial loops.

First, agents are taking over curriculum design inside simulation. AURA (ICRA 2026) uses LLMs to generate reward functions, domain randomizations, and training configs as YAML workflows, then applies schema validation and static verification [21]. This is a robotics S2 case: it does not run a physical arm overnight, but it turns reward and curriculum tuning into an agent loop.

Second, real-world data collection loops are beginning to close. Tether (ICLR 2026) runs autonomous functional play inside a VLM-guided multi-task loop, producing play data that improves downstream policy learning over time [22]. The robot interacts with objects, creates data, and the data improves the next policy. That is a small-scale S3 loop.

Third, the robotics foundation model conversation is moving from solitary autonomy to alliance-aware systems. An April 2026 Science Robotics viewpoint argues that robotic foundation models should learn with humans and other agents, not as isolated omnipotent agents [23]. This explains why robotics AI Scientist is harder than ML autoresearch: robot experiments do not close around a single metric. They include human preference, physical safety, environmental variation, and long-term interaction.

So the scope becomes:

  • Covered: ML/alignment research (autoresearch, AAR), medical AI (Med-AI Bench), chemistry/biology self-driving labs, AI Co-scientist, and partial robotics loops (AURA, Tether)
  • Still not covered: an end-to-end robotics AI Scientist that defines research goals, designs and runs physical experiments, interprets results, and passes safety validation

Robotics entangles stages S2 (in-silico) and S3 (physical). Simulation can iterate fast, but sim-to-real remains. Hardware gives reality, but safety and wear limit exploration speed. A robotics AI Scientist therefore needs simulation harness + real-world data loop + safety gate + human-in-the-loop correction, not just a larger LLM.

12.8 Closing the Loop

Chapter 1 started with "knowledge externalization." The real cost of Claude→Codex migration isn't code changes — it's extracting knowledge locked inside model conversations into plain-text files like AGENTS.md, HANDOFF.md, TASKS.md.

Chapter 10's LLM Wiki extended that externalization to research knowledge. Chapter 11 turned daily activity into AI external memory. And this chapter shows the point where that external memory becomes the research loop itself.

Karpathy's trajectory shows this most clearly:

  • LLM Wiki (Chapter 10): "Put everything in markdown"
  • Autoresearch (Chapter 12): "Agents run experiments"
  • Same author, same scaffolding, one year apart

From external memory to autonomous research — that is the final destination of harness engineering.

For a Claude Code to Codex user, the practical bridge is small:

  1. Put research rules in AGENTS.md and tool-specific habits in CLAUDE.md.
  2. Keep raw/, wiki/, logs/, and decisions/ separate.
  3. Require at least ten source-linked claims before synthesis.
  4. Preserve contradictions and dead ends instead of rewriting them away.
  5. Use Codex Goal mode or automations only for bounded weekly lint, ingest, or audit tasks.

References

  1. Anthropic, "Autonomous Alignment Research (AAR)," 2026-04-14. [Anthropic, 2026]
  2. Karpathy, Andrej, "Autoresearch," 2026. [Karpathy, 2026]
  3. Karpathy, Andrej, "Autoresearch — Round 1 tweet," 2026. [Karpathy, 2026]
  4. Karpathy, Andrej, "NanoChat," 2026. [Karpathy, 2026]
  5. terryum, "AAR post," terryum-ai, 2026. [Um, 2026]
  6. terryum, "Autoresearch post," terryum-ai, 2026. [Um, 2026]
  7. terryum, "AI Co-scientist post," terryum-ai, 2026. [Um, 2026]
  8. Wu et al., "Med-AI Bench," arXiv 2603.28589, 2026. [Um, 2026]
  9. terryum, "Self-driving labs post," terryum-ai, 2026. [Um, 2026]
  10. terryum, "Research democratization post," terryum-ai, 2026. [Um, 2026]
  11. Google AI, "Accelerating scientific breakthroughs with an AI co-scientist," 2025-02-19. [Google AI, 2025]
  12. Gottweis et al., "Towards an AI co-scientist," arXiv 2502.18864, 2025-02-26. [Gottweis et al., 2025]
  13. Anthropic, "Automated Alignment Researchers: Using large language models to scale scalable oversight," 2026-04-14. [Anthropic, 2026]
  14. Clark, Jack, "Import AI 454: Automating alignment research," 2026-04-20. [Clark, 2026]
  15. BSWEN, "What Results Did 700 Autoresearch Experiments Achieve Overnight?," 2026-03-30. [BSWEN, 2026]
  16. AI Productivity, "Karpathy's AutoResearch Ran 700 ML Experiments in 2 Days Without Human Input," 2026-03-23. [AI Productivity, 2026]
  17. Adam, David, "The AI co-scientist is here," Nature Medicine, 2026-03-16. [Adam, 2026]
  18. Brazil, Rachel, "Inside the self-driving lab revolution," Nature, 2026-03-30. [Brazil, 2026]
  19. Pilon et al., "A flexible and affordable self-driving laboratory for automated reaction optimization," Nature Synthesis, 2026-04-13. [Pilon et al., 2026]
  20. Restrepo, Guillermo, "Expanding diversity in chemical space," Nature Chemistry, 2026-03-19. [Restrepo, 2026]
  21. Zhu et al., "AURA: Autonomous Upskilling with Retrieval-Augmented Agents," ICRA 2026. [Zhu et al., 2026]
  22. Liang et al., "Tether: Autonomous Functional Play with Correspondence-Driven Trajectory Warping," ICLR 2026. [Liang et al., 2026]
  23. Dey et al., "From autonomy to alliance: Robotic foundation models must learn with us, not just for us," Science Robotics, 2026-04-22. [Dey et al., 2026]
  24. terryum survey S8, "LLM Wiki to AI Scientist," 2026. [Um, 2026]