WAIT — Missing license — unclear to depend on

• Last commit 6w ago
• 9 active contributors
• ⚠ Concentrated ownership — top contributor handles 78% of recent commits
• ⚠ No license — legally unclear to depend on
• ⚠ No CI workflows detected
• ⚠ No test directory detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

autoresearch is an autonomous AI research agent framework that runs on a single GPU to continuously experiment with and improve LLM training code. The agent autonomously modifies train.py (which contains a GPT model, optimizer stack, and training loop), trains for exactly 5 minutes, evaluates against val_bpb (validation bits-per-byte), and decides whether to keep or discard changes—all without human intervention. It's a meta-research tool where the AI itself conducts the hyperparameter search and architectural experimentation. Flat structure: three core files: prepare.py (static data prep, dataloader, eval utilities—never modified), train.py (the mutable target file the agent edits—architecture, hyperparams, optimizer logic), and program.md (human-written agent instructions—the 'research org constitution'). Supporting: analysis.ipynb for post-hoc analysis, pyproject.toml for project config, uv.lock for pinned dependencies.

ML researchers and practitioners who want to run overnight autonomous experiments on single-GPU setups, and who are interested in delegating hyperparameter tuning and architectural decisions to AI agents. Specifically: people building on Karpathy's nanochat training framework who want to parallelize research via autonomous iteration rather than manual trial-and-error.

Highly experimental and research-stage. The repo is deliberately minimal (3 core files) and treated as a proof-of-concept playground rather than production infrastructure. No CI/tests are visible, no multi-decade commit history expected. This is a recent proof-of-concept by Karpathy himself (March 2026 fictional date in README) demonstrating the concept, not a hardened framework.

Single-maintainer risk (Karpathy's personal research project), no visible test suite or validation layer, and the agent-driven code modification loop has no built-in safety mechanisms—a poorly-designed agent instruction in program.md could corrupt train.py in unhelpful ways. The 5-minute fixed training window is a constraint, not validated. GPU-specific (NVIDIA H100 mentioned) with no fallback for CPU or other accelerators.

This is a released proof-of-concept snapshot. The repo is not an active development project with a backlog—it's a demonstration/template. Future iteration would focus on evolving program.md to be a more sophisticated agent prompt, adding more agents, and learning what 'research org code' works best, but that is future work outside this snapshot.

Clone: git clone https://github.com/karpathy/autoresearch.git && cd autoresearch. Install deps: uv sync (uses uv package manager as indicated by uv.lock). Prep data once: python prepare.py. Then: point an AI agent at program.md and let it run train.py in a loop, or manually iterate by editing train.py and running python train.py.

Daily commands: One-time setup: python prepare.py (downloads data, builds BPE tokenizer). Then run a single 5-minute training trial: python train.py (writes logs, checkpoints, and final val_bpb metric). To iterate: manually edit train.py, re-run python train.py. Or: have an AI agent do this in a loop via instructions in program.md.

• train.py — Core LLM training loop that agents will modify; implements the nanochat training setup on single GPU.
• program.md — System prompt and instructions for the AI agent describing what experiments to run and how to modify the codebase.
• prepare.py — Data preparation script that sets up training datasets; must run before training and agents may modify.
• analysis.ipynb — Jupyter notebook for analyzing experimental results and progress; documents what worked and what didn't.
• pyproject.toml — Project dependencies and Python version specification; defines the reproducible environment for agent runs.

• AI Agent (external orchestrator) (LLM + code generation (external to repo)) — Autonomously reads program.md, modifies train.py and prepare.py, launches training, parses results, decides whether to keep changes.
  • Failure mode: Generated invalid Python code, infinite loops, out-of-memory, or catastrophic loss spikes; corrupts experiment log.
• train.py (PyTorch, Python) — Implements nanochat model training loop; forwards metrics to stdout for agent parsing.
  • Failure mode: Syntax/runtime errors from agent modifications; diverging loss; GPU OOM.
• prepare.py (Python, file I/O) — Data loading and preprocessing pipeline; creates train/val datasets.
  • Failure mode: Data corruption, missing files, encoding errors if agent modifies incorrectly.
• program.md (Markdown, natural language) — High-level prompt and constraints; guides agent's modification decisions.
  • Failure mode: Ambiguous or contradictory instructions cause agent to make suboptimal or harmful changes.
• analysis.ipynb (Jupyter, Pandas, Matplotlib) — Post-run aggregation and visualization; human reviews progress in the morning.
  • Failure mode: Corrupted notebook state if agent writes invalid JSON or cells; hard to parse results.

• prepare.py → train.py — Preprocessed dataset (likely pickle or memmap) loaded at train start.
• train.py → stdout / log file — Training loss, validation metrics, and hyperparameters printed for agent and human review.
• Agent (external) → train.py — Modifications to model architecture, learning rate, regularization, or loss function.
• Agent (external) → program.md — Agent reads current objectives and constraints before each iteration.
• train.py output → analysis.ipynb — Metrics and logs ingested for plotting and trend analysis.
• analysis.ipynb → Human — Morning review: progress.png and notebook show which experiments succeeded.

Add a New Hyperparameter for Agent to Tune

1. Define the hyperparameter in train.py with a default value and add it to the argument parser. (train.py)
2. Add the parameter to program.md instructions so the agent knows it can modify this knob. (program.md)
3. Update analysis.ipynb to log and plot this parameter's effect on model performance. (analysis.ipynb)

Add a New Evaluation Metric for Agent to Optimize

1. Implement the metric calculation in train.py during the evaluation loop. (train.py)
2. Log the metric to stdout so the agent's feedback loop can parse success/failure. (train.py)
3. Document the metric in program.md and specify whether higher or lower is better. (program.md)
4. Add visualization and tracking code in analysis.ipynb. (analysis.ipynb)

Modify Agent Experiment Constraints or Objectives

1. Edit program.md to change training duration, search space, or optimization objective. (program.md)
2. Ensure pyproject.toml has all required dependencies for the new constraints. (pyproject.toml)
3. Update analysis.ipynb if new metrics or thresholds need to be monitored. (analysis.ipynb)

• Python + PyTorch (implied by nanochat reference) — Standard framework for LLM training; allows agent to easily modify model code and hyperparameters.
• Jupyter Notebook (analysis.ipynb) — Interactive analysis and visualization of training runs; human-readable results for morning review.
• Simple file-based experiment logging — No database overhead; agent can modify and append results easily; minimal system dependencies.
• Single GPU training — Keeps compute footprint manageable overnight; allows fast iteration loops (5 min per run).

• Single GPU only, no distributed training

  • Why: Simplicity and ability for agent to modify code safely; faster iteration cycles for overnight experiments.
  • Consequence: Limited model size and batch scale; not suitable for state-of-the-art production models, only experimental nanochat.

• 5-minute training runs per experiment

  • Why: Allows ~100–150 iterations overnight; agent gets fast feedback loop for hypothesis testing.
  • Consequence: Models cannot converge fully; optimization must focus on relative improvements within short windows.

• Agent modifies Python files directly

  • Why: Maximum flexibility for autonomous code generation and experimentation.
  • Consequence: High risk of syntax errors, logical bugs, or training instability; requires robust error handling and rollback.

• Markdown-based system prompt (program.md) for agent

  • Why: Human-readable instructions; easy to update agent objectives without code changes.
  • Consequence: Prompt injection risks; agent behavior is less formally specified than code.

• Does not implement distributed multi-GPU or multi-machine training.
• Does not handle long-running, production-grade convergence (fixed to ~5 min per iteration).
• Not a real-time experiment dashboard or web UI for live monitoring.
• Does not provide formal guardrails against agent-generated code breaking the training loop.
• Not designed for large-scale model deployment or inference serving.

• Unbounded agent code modification without versioning (High) — train.py (modified by external agent): Agent may introduce breaking changes (e.g., incompatible function signatures, missing imports) without storing prior versions or recovery points.
• Implicit success criteria parsing from stdout (Medium) — train.py (output format) + agent (parsing): Agent must parse training metrics from unstructured stdout; fragile to format changes. Better to emit JSON or write to a dedicated results file.
• No sandboxing or resource limits on agent modifications (High) — prepare.py, train.py: Agent can modify code to allocate unlimited GPU memory, spawn infinite loops, or corrupt the filesystem.
• Single agent objective without fallback (Medium) — program.md: If optimization objective in program.md is poorly specified, agent may pursue degenerate solutions (e.g., gaming metrics without actual improvement).

• train.py training loop (Compute / Hardware) — Single GPU becomes bottleneck after model reaches ~100M+ parameters; fixed 5-minute wall-clock limit prevents exploring larger architectures.
• prepare.py data loading (I/O) — If dataset is large or preprocessing is slow, data prep phase delays agent iterations; no caching across runs if agent modifies preprocessing logic.

Fixed 5-minute wall-clock training budget regardless of GPU—if your GPU is slower, fewer iterations run in that window. val_bpb metric is vocabulary-size-independent (good for fair comparison of architecture changes), but only meaningful relative to a fixed dataset and tokenizer (both prepared once by prepare.py, not re-run per trial). Agent-modified train.py must remain valid Python and return a numeric val_bpb to stdout or the agent's decision logic breaks. No rollback mechanism if agent corrupts train.py syntax. Requires NVIDIA GPU (tested on H100; no CPU fallback documented).

• Bits Per Byte (BPB) — The evaluation metric used to judge all agent experiments; it's vocab-size-independent and thus fair across architectural changes, so understanding what val_bpb means and how to interpret it is critical
• Byte-Pair Encoding (BPE) — prepare.py trains a BPE tokenizer once; the agent never modifies tokenization, so understanding BPE vocabulary size trade-offs helps predict how model changes will behave
• Muon Optimizer — An optimizer alternative to AdamW used in train.py; the agent may modify optimizer choice or hyperparams, so understanding Muon (or why it was chosen) informs architectural decisions
• Transformer Architecture (GPT) — train.py implements a GPT-style decoder model; the agent modifies layer counts, hidden dims, attention patterns—familiarity with transformer building blocks is essential
• Agent-in-the-Loop Optimization — The core loop: agent modifies code → train 5 min → measure val_bpb → keep/discard → repeat. This is a form of black-box optimization where the agent must learn what code changes correlate with improvement
• Hyperparameter Tuning / Neural Architecture Search (NAS) — autoresearch automates what researchers manually do: searching the space of learning rates, layer counts, batch sizes, etc.; the agent is running a form of NAS via code mutation
• Fixed Compute Budget (5-minute wall clock) — Training always runs for exactly 5 minutes regardless of GPU speed or batch size; this constraint forces the agent to trade off batch size, learning rate, and iteration count fairly across trials

• karpathy/nanochat — The base LLM training framework that autoresearch simplifies and automates; train.py is a single-GPU extraction of nanochat's design
• karpathy/nanoGPT — Karpathy's foundational minimal GPT implementation; provides the core model and training loop patterns that train.py builds on
• openai/gpt-2 — Reference GPT-2 model architecture and training; helps understand the model class structure and hyperparameter baselines that autoresearch tunes
• EleutherAI/gpt-neox — Large-scale GPT training codebase; relevant for understanding optimizer choices (Muon, AdamW) and distributed training patterns that autoresearch simplifies to single-GPU
• anthropic/constitutional-ai — Parallel research concept: using AI to improve AI behavior via constrained iteration; shares the philosophy of autonomous improvement that autoresearch demonstrates

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add experiment tracking and result logging to train.py

The core premise of autoresearch is autonomous experimentation with comparison of results, but there's no visible mechanism in train.py to log experiments, track metrics across runs, or persist results for analysis. The analysis.ipynb suggests experiments are being run, but there's no clear instrumentation in train.py to enable the agent to 'check if the result improved'. This PR should add structured logging that records hyperparameters, training metrics, and model checkpoints so the AI agent can make informed decisions.

• [ ] Create an Experiment class or logging module to track runs (hyperparams, loss curves, final metrics)
• [ ] Instrument train.py to log metrics at each checkpoint and end-of-training
• [ ] Add a results.json or experiment_log.csv file that stores structured experiment history
• [ ] Update analysis.ipynb to read and visualize this experiment log
• [ ] Document the experiment schema in README.md so contributors understand what the agent sees

Implement agent decision logic in program.md as executable Python module

program.md appears to be documentation or a specification, but the actual autonomous agent logic that modifies code and decides whether to keep changes is missing from the repo. This should be a concrete Python module that the agent can execute to: propose code modifications, run train.py, compare metrics, and commit/discard changes. This is the core loop described in the README.

• [ ] Create agent.py with classes for CodeModifier, ExperimentRunner, and ResultComparator
• [ ] Implement logic to read train.py, propose hyperparameter/architecture changes, and write modified versions
• [ ] Add a main loop that: (1) modifies code, (2) calls train.py, (3) compares results to baseline, (4) keeps or discards
• [ ] Parse and execute the logic from program.md into agent.py
• [ ] Add CLI entrypoint (e.g., python -m autoresearch.agent --num_experiments 10 --max_runtime_minutes 480)

Add integration tests for the full autoresearch loop in prepare.py and train.py

There's a prepare.py that likely sets up the training data/environment, and train.py that runs training, but no tests verify that the pipeline works end-to-end. For an autonomous system to safely modify code overnight, we need tests that ensure prepare.py and train.py don't break when the agent makes changes. This PR should add a test suite that validates the training pipeline.

• [ ] Create tests/ directory with test_prepare.py validating prepare.py output (data shapes, file existence)
• [ ] Create tests/test_train.py that runs a minimal training loop and validates output (checkpoint creation, loss logging)
• [ ] Add a tests/test_agent_safety.py that verifies the agent can't introduce syntax errors or break imports
• [ ] Add GitHub Actions workflow (.github/workflows/test.yml) to run pytest on each commit
• [ ] Update pyproject.toml with test dependencies (pytest, pytest-cov) and test command

• idea: Add integration tests for train.py: verify that a single training run completes in ~5 minutes and returns a valid val_bpb number; catch syntax errors before agent iteration
• idea: Extend analysis.ipynb to plot agent decision history (which modifications were accepted vs rejected and why); helps understand agent reasoning
• idea: Document or add validation in program.md about 'safe' code mutations (e.g., learning rate bounds, layer count bounds) to reduce risk of agent proposals that break training

The codebase presents significant security risks due to its core design of autonomous, self-modifying code execution. The primary concerns are unvalidated dynamic code execution, lack of sandboxing, and absence of resource controls. While the use of uv.lock suggests dependency management awareness, the autonomous agent architecture requires substantial hardening before production use. Key recommendations: implement strict sandboxing, avoid eval/exec patterns, add comprehensive input validation, enforce resource limits, and develop detailed security documentation.

• High · Autonomous Code Modification Without Sandboxing — train.py, program.md (core agent logic). The core functionality allows AI agents to autonomously modify and execute training code. This presents a significant security risk as self-modifying code could introduce malicious patterns, unsafe operations, or unintended behaviors. Without proper sandboxing, an agent could modify critical files, access sensitive data, or execute arbitrary system commands. Fix: Implement strict sandboxing for code modifications (e.g., using containers, restricted Python interpreters). Use allowlisting for permitted code changes. Add cryptographic integrity checks. Implement comprehensive logging and rollback mechanisms. Run agents with minimal privilege (principle of least privilege).
• High · Unvalidated Dynamic Code Execution — train.py (suspected agent execution loop). AI agents that modify and execute code likely use eval(), exec(), or dynamic imports. These functions are inherently dangerous as they can execute arbitrary Python code if the input is not properly validated and sanitized. Fix: Avoid eval() and exec() entirely. Use ast.parse() with ast.literal_eval() for safe evaluation of literals only. Implement a Domain Specific Language (DSL) for agent modifications with restricted syntax. Use abstract syntax trees (AST) analysis to validate code before execution.
• Medium · Missing Dependency Integrity Verification — pyproject.toml, uv.lock. While uv.lock is present (good), the actual dependency list in pyproject.toml was not provided. Without visibility into pinned versions, there's risk of outdated packages with known vulnerabilities. Dependencies should be explicitly pinned and regularly audited. Fix: Ensure all dependencies are pinned to specific versions in pyproject.toml. Regularly run 'pip-audit' or 'safety check' to identify vulnerable packages. Set up dependency update monitoring and automated security scanning in CI/CD.
• Medium · Insufficient Input Validation on Agent Parameters — train.py, prepare.py (parameter handling). AI agents modify code parameters autonomously. Without strict validation, agents could introduce malformed inputs, path traversal attacks, or resource exhaustion attacks (e.g., setting batch_size to extreme values causing DoS). Fix: Implement strict schema validation for all modifiable parameters using pydantic or similar. Define min/max ranges for numeric parameters. Use allowlisting for categorical parameters. Validate file paths with pathlib.Path.resolve() to prevent path traversal.
• Medium · Lack of Resource Limits and DoS Protection — train.py (training loop). Autonomous agents could inadvertently or maliciously cause denial of service by consuming excessive GPU memory, CPU, disk space, or network resources. No apparent resource quotas or limits are visible. Fix: Implement resource limits using cgroups or container limits (CPU, memory, disk I/O). Set timeouts for training iterations. Monitor resource usage in real-time. Implement circuit breakers that halt experiments if resources exceed thresholds.
• Low · No Apparent Access Control on Experimental Results — progress.png, experiment logs (implicit). The codebase logs experiments and generates model artifacts (progress.png, logs). Without proper access control, unauthorized parties could read sensitive experiment data or tamper with results. Fix: Implement proper file permissions (chmod 600 for sensitive files). Store artifacts in secure storage with encryption. Use role-based access control (RBAC) if running in shared environments. Audit all file access.
• Low · Missing Security Documentation — README.md. The README provides creative context but lacks security considerations, threat models, or safety guidelines for autonomous agent execution. This could lead to unsafe deployments. Fix: Add a SECURITY.md file documenting: threat models, safety constraints, resource limits, audit logging capabilities, and incident response procedures. Document all security assumptions and limitations.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/karpathy/autoresearch shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live karpathy/autoresearch repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/karpathy/autoresearch.

What it runs against: a local clone of karpathy/autoresearch — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in karpathy/autoresearch | Confirms the artifact applies here, not a fork | | 2 | Default branch master exists | Catches branch renames | | 3 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 4 | Last commit ≤ 74 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of karpathy/autoresearch. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/karpathy/autoresearch.git
#   cd autoresearch
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of karpathy/autoresearch and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "karpathy/autoresearch(\\.git)?\\b" \\
  && ok "origin remote is karpathy/autoresearch" \\
  || miss "origin remote is not karpathy/autoresearch (artifact may be from a fork)"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "train.py" \\
  && ok "train.py" \\
  || miss "missing critical file: train.py"
test -f "program.md" \\
  && ok "program.md" \\
  || miss "missing critical file: program.md"
test -f "prepare.py" \\
  && ok "prepare.py" \\
  || miss "missing critical file: prepare.py"
test -f "analysis.ipynb" \\
  && ok "analysis.ipynb" \\
  || miss "missing critical file: analysis.ipynb"
test -f "pyproject.toml" \\
  && ok "pyproject.toml" \\
  || miss "missing critical file: pyproject.toml"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 74 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~44d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/karpathy/autoresearch"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.