If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/openai/spinningup shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across all four use cases

• 26+ active contributors
• Distributed ownership (top contributor 43% of recent commits)
• MIT licensed
• CI configured
• Tests present
• ⚠ Stale — last commit 2y ago

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live openai/spinningup repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/openai/spinningup.

What it runs against: a local clone of openai/spinningup — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in openai/spinningup | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | Last commit ≤ 670 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of openai/spinningup. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/openai/spinningup.git
#   cd spinningup
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of openai/spinningup and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "openai/spinningup(\\.git)?\\b" \\
  && ok "origin remote is openai/spinningup" \\
  || miss "origin remote is not openai/spinningup (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 670 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~640d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/openai/spinningup"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Spinning Up is an OpenAI educational resource that provides clean, well-documented standalone implementations of core deep reinforcement learning algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC) alongside theoretical explanations. It teaches RL fundamentals through runnable Python code examples designed for learning, not production performance. Single-package structure: spinningup/ directory contains algorithm implementations (vpg.py, trpo.py, ppo.py, ddpg.py, td3.py, sac.py), docs/ holds Sphinx documentation with algorithm walkthroughs (docs/algorithms/*.rst) and educational essays, and examples/ likely contains runnable training scripts. Modular design with each algorithm self-contained.

Students and researchers new to deep RL who want to understand how canonical algorithms work by reading and running well-commented reference implementations. Also useful for practitioners validating algorithmic understanding or prototyping RL solutions before moving to optimized frameworks.

Maintenance mode (per README status badge). The codebase is stable and established (2018 publication with structured docs in docs/algorithms/*.rst for all 6 algorithms), has CI via .travis.yml, but is not under active feature development—expect bug fixes and documentation updates only.

Low risk for learning purposes; minimal external dependencies by design (simplicity is intentional). However, TensorFlow/PyTorch ecosystem dependencies may drift over time. Single-maintainer educational project means slower response to breaking dependency updates. No visible open issue tracking in provided data, suggesting either well-maintained or low community engagement.

Repo is in maintenance mode. No active development sprints visible, but the comprehensive documentation (docs/algorithms with detailed .rst files for each algorithm) and benchmark results (docs/images/plots/ showing performance on Ant, HalfCheetah, Hopper, Swimmer, Walker2d) suggest ongoing stability maintenance. Last status check indicates bug fixes expected.

git clone https://github.com/openai/spinningup.git
cd spinningup
pip install -e .
# Install optional dependencies: pip install tensorflow torch mujoco-py

Note: Review docs/docs_requirements.txt for documentation building dependencies.

Daily commands: After installation, run individual algorithm training scripts (e.g., python spinningup/examples/pytorch/vpg_cartpole.py or equivalent). Build docs via cd docs && make html. Most learning happens by reading source code directly—see spinningup/algos/ for implementations and docs/algorithms/*.rst for theory walkthroughs.

• spinningup/algos/vpg.py: Vanilla Policy Gradient baseline—simplest on-policy algorithm, good learning entry point
• spinningup/algos/ppo.py: Proximal Policy Optimization—workhorse algorithm used across industry, critical implementation reference
• spinningup/algos/ddpg.py: Deep Deterministic Policy Gradient—canonical off-policy actor-critic for continuous control
• docs/algorithms/ppo.rst: Detailed PPO theory walkthrough and implementation guide
• docs/spinningup/rl_intro.rst: Core RL terminology and conceptual foundations for all algorithms
• .travis.yml: CI configuration showing test/validation pipeline

For algorithm improvements: edit spinningup/algos/{vpg,trpo,ppo,ddpg,td3,sac}.py. For documentation fixes: edit docs/algorithms/.rst or docs/spinningup/.rst. For new educational examples: add to examples/ directory following existing structure (separate TensorFlow/PyTorch variants). For tests: contribute to tests/ directory (if it exists) or create it with pytest fixtures.

MuJoCo dependency required for continuous control benchmarks—requires separate physics simulator license/installation. Algorithm implementations assume PyTorch or TensorFlow available (not auto-installed). Benchmark results in docs/images/plots/ are generated offline; no automated benchmarking pipeline visible in repo. Documentation is Sphinx-based requiring make/build step, not auto-served.

• Policy Gradient Methods — VPG and PPO use policy gradients; understanding gradient ascent on log-probability distributions is foundational for on-policy learning
• Actor-Critic Architecture — DDPG, TD3, SAC all use separate actor (policy) and critic (value) networks; this pattern reduces variance in gradient estimates
• Trust Region Optimization — TRPO and PPO use trust regions to constrain policy updates; prevents catastrophic forgetting in on-policy learning
• Experience Replay Buffer — DDPG, TD3, SAC all use offline replay buffers to break temporal correlation in off-policy learning and improve sample efficiency
• Deterministic Policy Gradient — DDPG and TD3 use deterministic policies with stochastic exploration; enables off-policy learning in continuous action spaces
• Entropy Regularization — SAC uses maximum entropy RL to balance exploration and exploitation through learned temperature parameter
• Generalized Advantage Estimation (GAE) — VPG, TRPO, PPO all use GAE to compute advantage estimates; critical for reducing variance while maintaining bias control

• openai/gym — Standard RL benchmark environment suite used by Spinning Up examples for CartPole, MuJoCo, Atari testing
• openai/baselines — OpenAI's optimized RL algorithm implementations; Spinning Up is pedagogical alternative with cleaner code for learning
• berkeleydeeprlcourse/homework — Companion educational resource from UC Berkeley covering similar RL fundamentals with different implementation approach
• raisimGym/raisimGym — Modern alternative physics simulator and environment suite beyond MuJoCo for benchmarking deep RL algorithms
• railroad2/railrl — Predecessor-adjacent RL codebase exploring meta-RL extensions of core algorithms taught in Spinning Up

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add missing algorithm documentation pages and implementation guides

The docs/algorithms/ directory contains RST files for 6 algorithms (DDPG, PPO, SAC, TD3, TRPO, VPG), but there are no corresponding implementation walkthroughs or code examples embedded in these pages. New contributors could add detailed sections explaining the pseudocode, key hyperparameters, and links to the actual implementation files in spinup/algos/. This would make the repo significantly more useful for learners trying to understand the connection between theory and code.

• [ ] Review existing docs/algorithms/*.rst files to identify missing sections (e.g., implementation details, hyperparameter explanations)
• [ ] Check spinup/algos/ directory structure to identify which algorithm implementations exist and need documentation links
• [ ] Add 'Implementation Details' and 'Code Walkthrough' sections to at least 2-3 algorithm RST files with code snippets and file references
• [ ] Ensure all algorithm docs follow consistent formatting and include links to corresponding source files

Create GitHub Actions CI workflow for multi-version Python testing

The repo has a .travis.yml file (legacy Travis CI), but modern open source projects use GitHub Actions. Spinningup likely needs to test against multiple Python versions (3.7, 3.8, 3.9, 3.10+) and different backend versions (TensorFlow 1.x vs 2.x, PyTorch versions). A contributor could create a .github/workflows/test.yml file to run unit tests, linting, and documentation builds on push/PR, replacing the outdated Travis config.

• [ ] Create .github/workflows/ directory structure
• [ ] Review existing .travis.yml to understand current test commands and matrix configurations
• [ ] Create test.yml workflow file that tests against Python 3.8+ and relevant backend versions (TensorFlow/PyTorch)
• [ ] Add linting job (flake8/pylint) and documentation build verification
• [ ] Test locally with act tool and document the workflow in CONTRIBUTING.md if it exists

Add benchmark reproduction scripts and documentation in docs/

The repo contains extensive benchmark plots (docs/images/plots/ddpg/, ppo/, sac/, td3/, etc.) showing algorithm performance on MuJoCo tasks (Ant, HalfCheetah, Hopper, Swimmer, Walker2d), but there is no documentation on how to reproduce these benchmarks. A contributor should create docs/benchmarking.rst with clear instructions, hyperparameter settings, random seeds, and shell scripts to regenerate these plots, making the repo more reproducible and trustworthy for learners.

• [ ] Create docs/benchmarking.rst file with benchmark reproduction methodology
• [ ] Document exact hyperparameters, seeds, and environment settings used for each algorithm
• [ ] Create scripts/ directory with Python scripts (e.g., reproduce_ddpg_benchmarks.py) that run experiments and generate plots matching docs/images/plots/
• [ ] Include instructions for installing MuJoCo dependencies and running benchmarks locally
• [ ] Link to this documentation from the main docs/index.rst and algorithm pages

• Add comprehensive unit tests to spinningup/algos/ covering gradient computation, loss functions, and policy updates for each of the 6 algorithms—currently no tests/ directory visible
• Expand docs/algorithms/*.rst with worked examples showing step-by-step walkthroughs of hyperparameter sensitivity (learning rate, network size, batch size) on simple tasks like CartPole
• Create comparative benchmarking notebook showing performance trade-offs between VPG, TRPO, PPO on same tasks with wall-clock time and sample efficiency metrics

The Spinning Up in Deep RL repository appears to be a low-risk educational project with primarily static content (documentation and code examples). No obvious hardcoded secrets, injection vulnerabilities, or infrastructure misconfigurations were detected in the file structure provided. However, the main security concern is the absence of visible dependency management files, which prevents verification of whether the project uses vulnerable versions of its dependencies. The project should maintain explicit dependency manifests with pinned versions and regularly audit them for vulnerabilities. The use of CI/CD (Travis CI) is present and should be verified to not expose credentials. Overall, this is an educational resource with a reasonable security posture for its purpose, but dependency management practices should be formalized and documented.

• Medium · Missing Dependency Pinning Information — Repository root - dependency files not provided. No dependency file (requirements.txt, setup.py, setup.cfg, pyproject.toml, or Pipfile) was provided for analysis. This makes it impossible to verify if the project uses vulnerable versions of dependencies. Educational repositories often have outdated dependencies that may contain known vulnerabilities. Fix: Provide and maintain a requirements.txt or equivalent dependency manifest. Regularly audit dependencies using tools like 'pip audit' or 'safety' and keep all packages updated to their latest secure versions.
• Low · Travis CI Configuration Present — .travis.yml. The presence of .travis.yml indicates the project uses Travis CI for continuous integration. While not inherently a vulnerability, the configuration file should be reviewed to ensure it doesn't expose secrets or execute untrusted code without proper isolation. Fix: Review the Travis CI configuration to ensure: (1) No secrets or API keys are hardcoded, (2) Only trusted dependencies are installed, (3) Build environment variables are properly managed through secure CI/CD settings, not in the config file.
• Low · Documentation Build Configuration — docs/conf.py, docs/docs_requirements.txt, docs/Makefile. The docs directory contains configuration files (docs/conf.py, docs/Makefile) and a requirements file (docs/docs_requirements.txt). Documentation build processes can potentially introduce vulnerabilities if dependencies are outdated or if the configuration is insecure. Fix: Review docs/docs_requirements.txt for pinned versions and known vulnerabilities. Ensure docs/conf.py doesn't execute arbitrary code during the build process. Consider using a separate, minimal dependency set for documentation.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.