If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/github/scientist shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across all four use cases

• Last commit 6mo ago
• 30+ active contributors
• Distributed ownership (top contributor 36% of recent commits)
• MIT licensed
• CI configured
• Tests present
• ⚠ Slowing — last commit 6mo ago

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live github/scientist repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/github/scientist.

What it runs against: a local clone of github/scientist — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in github/scientist | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | Last commit ≤ 197 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of github/scientist. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/github/scientist.git
#   cd scientist
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of github/scientist and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "github/scientist(\\.git)?\\b" \\
  && ok "origin remote is github/scientist" \\
  || miss "origin remote is not github/scientist (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 197 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~167d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/github/scientist"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Scientist is a Ruby library for safely refactoring critical code paths by running experiments that execute both old (control) and new (candidate) code paths in parallel, measure their performance and results, and publish findings for comparison. It lets teams validate refactored logic under real production load before fully committing to changes, solving the problem of risky refactors in high-stakes code. Simple modular design: lib/scientist.rb is the entry point exporting the Scientist module; lib/scientist/experiment.rb contains the Experiment base class that users subclass to customize behavior; lib/scientist/observation.rb and lib/scientist/result.rb model the data flow; lib/scientist/default.rb provides a minimal no-op implementation. Tests in test/ mirror the lib/ structure with one test file per module.

Ruby backend developers and DevOps engineers at organizations with critical business logic who need to refactor permissions, billing, or other sensitive code paths with confidence. GitHub uses it internally for their own refactoring work.

Production-ready and stable. The repo shows deliberate, conservative development: it has comprehensive test coverage (test/ directory with dedicated test files for each module), GitHub Actions CI configured (.github/workflows/ci.yml), clear changelog documentation (doc/changelog.md), and a formal contribution guide (CONTRIBUTING.md). Last activity appears recent based on the presence of dependabot automation.

Low risk. The codebase is minimal (~51KB Ruby, minimal dependencies inferred from sparse Gemfile), making it easy to audit. Single-file core (lib/scientist.rb) and modular design reduce breaking-change surface. No obvious red flags: well-tested, published via gemspec, and backed by GitHub's maintenance. Main risk is typical of small libraries: dependency on Ruby ecosystem stability.

The repo appears to be in stable maintenance mode. Dependabot is configured (.github/dependabot.yml) for automated dependency updates. No specific ongoing feature work is visible from the file list, suggesting the library is feature-complete and focused on reliability and compatibility.

Clone the repo, install dependencies, and run tests:

git clone https://github.com/github/scientist.git
cd scientist
script/bootstrap
script/test

The script/ directory contains shell helpers for setup and testing (see script/bootstrap and script/test in the file list).

Daily commands: This is a library, not an application. To use it in your own Ruby project:

require 'scientist'

class MyClass
  include Scientist

  def critical_method(arg)
    science "experiment-name" do |experiment|
      experiment.use { old_implementation(arg) }
      experiment.try { new_implementation(arg) }
    end
  end
end

To develop on Scientist itself, run script/test to execute the test suite.

• lib/scientist.rb: Main entry point defining the Scientist module with the science helper that users call; sets up the experiment DSL
• lib/scientist/experiment.rb: Abstract base class defining the Experiment interface (enabled?, raised, publish, run); users subclass this to customize behavior
• lib/scientist/observation.rb: Models a single observation (control or candidate run) capturing value, error, duration, and CPU time; used internally to compare results
• lib/scientist/result.rb: Aggregates observations (control + candidate) and provides methods to check if results match; passed to publish callbacks
• lib/scientist/default.rb: Minimal no-op Experiment implementation used if no custom experiment class is set; useful for initial testing
• test/scientist_test.rb: Main integration tests for the science helper and Scientist module behavior; shows how the library is intended to be used
• scientist.gemspec: Gem metadata and dependency declaration; defines how the library is packaged and distributed to RubyGems

1. Add a new Experiment feature: Subclass Scientist::Experiment in your code (see lib/scientist/experiment.rb for interface) and override enabled?, raised(operation, error), or publish(result) methods. 2. Fix a bug: Start in lib/scientist/ to understand the module structure, then add a test case to test/scientist/ mirroring the file you're fixing. 3. Extend behavior: The science helper (defined in lib/scientist.rb via the Scientist module) is the main entry point; modifying it or Scientist::Experiment base class will ripple to all users.

1. Order randomization is deliberate: the library randomizes whether control or candidate runs first (mentioned in README); tests must account for side effects. 2. Exceptions in candidate are swallowed by default: the raised callback in your Experiment subclass must call super to re-raise; if you override it and don't call super, errors silently disappear into observations. 3. No try block = no science: if you don't define any experiment.try blocks, the entire Scientist machinery is skipped and only the control value runs (see README: 'If you don't declare any try blocks, none of the Scientist machinery is invoked'). 4. Ruby version constraint: look at .github/workflows/ci.yml and Gemfile for minimum Ruby version supported; this is not explicitly stated in the README.

• A/B testing via parallel execution — Scientist runs both control and candidate code on every request (not just some percentage), measuring both to compare safety—this is not the same as traditional A/B testing; understanding the difference prevents misuse of the library for user-facing experimentation
• Observation and Result aggregation — The library wraps each code path execution in an Observation (lib/scientist/observation.rb) to capture value, error, and timing, then combines control + candidate Observations into a Result for comparison; understanding this flow is key to extending Scientist's behavior
• Exception swallowing in candidate code — The candidate (try) code's exceptions are caught and stored in observations by default, not raised—only the raised callback (which you can override) decides whether to re-raise; this is critical to prevent candidate bugs from breaking production, but requires explicit understanding
• Pluggable experiment strategy via subclassing — Users don't use Scientist::Experiment directly; they subclass it and override enabled?, raised, and publish to control when experiments run, how errors are handled, and where results are logged; this strategy pattern is the core extension mechanism
• Randomized execution order — Scientist intentionally randomizes whether control or candidate runs first to avoid order-dependent bugs; code that relies on execution timing or side effects will fail unpredictably—this is a feature, not a bug, designed to surface non-determinism
• Nil-result matching (truthiness-based comparison) — Scientist's default comparison (in lib/scientist/result.rb) uses == to check if control and candidate results match; understanding how Ruby equality works and how to customize comparison is needed for non-trivial refactors (e.g., comparing object identity vs. deep equality)
• DSL via module inclusion — The science helper is injected into classes via include Scientist; this metaprogramming pattern (module methods becoming instance methods) is how Scientist provides a clean syntax without boilerplate—understanding module inclusion is essential to extending the DSL

• github/flipper — Companion library from GitHub for feature flags; often used with Scientist to control whether experiments run (checking if feature flag is on before running candidate code)
• test-double/suture — Similar Ruby A/B testing library for safely replacing legacy code with new implementations; direct alternative with different API and philosophy (Suture focuses on gradual replacement, Scientist on parallel observation)
• LaunchDarkly/ruby-server-sdk — Industry-standard feature flag service with Ruby SDK; often integrated with Scientist experiments to gate which users see the candidate behavior
• github/scientist-js — JavaScript port of Scientist for Node.js; same API and philosophy adapted for JavaScript—reference for cross-language consistency
• petergrace/laboratory — Earlier Ruby experiment library that inspired or paralleled Scientist; useful for understanding the design space of A/B testing in Ruby

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add integration test suite for end-to-end experiment workflows

The test directory has unit tests for individual components (experiment_test.rb, observation_test.rb, result_test.rb) but lacks comprehensive integration tests that validate complete experiment workflows with multiple scenarios (control/candidate mismatches, error handling, timing comparisons). This would catch regressions in the core experiment execution flow.

• [ ] Create test/scientist/integration_test.rb with multi-step experiment workflows
• [ ] Test scenarios: successful runs, exceptions in try block, mismatched results, timing validation
• [ ] Verify integration with Scientist::Default, Scientist::Experiment, and Scientist::Result
• [ ] Run full test suite with script/test to ensure no regressions

Add GitHub Actions workflow for testing against multiple Ruby versions

The .github/workflows/ci.yml exists but the current setup likely tests against a single Ruby version. The scientist library needs to maintain compatibility across Ruby 2.7+ versions, but there's no evidence of a matrix strategy testing multiple versions. This is critical for a widely-used refactoring library.

• [ ] Review current .github/workflows/ci.yml configuration
• [ ] Add ruby-version matrix strategy for [2.7, 3.0, 3.1, 3.2, 3.3]
• [ ] Ensure Gemfile/gemspec dependencies are compatible with all versions
• [ ] Verify script/test runs successfully across all matrix combinations

Document experiment configuration and lifecycle in CONTRIBUTING.md with example patterns

CONTRIBUTING.md exists but the README snippet cuts off mid-sentence. New contributors need specific guidance on Scientist::Experiment configuration options (enabled/disabled experiments, custom comparators, publishing logic) which aren't fully documented. Adding architecture documentation will lower contribution friction.

• [ ] Expand CONTRIBUTING.md with 'Development Guide' section referencing lib/scientist/experiment.rb
• [ ] Document key configuration patterns: publishing results, custom comparison logic, experiment filtering
• [ ] Add code examples showing how to extend Scientist::Default vs Scientist::Experiment
• [ ] Cross-reference with lib/scientist/errors.rb to document error handling patterns

• Add timing precision documentation: lib/scientist/observation.rb stores duration in seconds (as floats), but the README doesn't specify precision or whether wall time vs CPU time is recommended for different use cases. Add a doc/timing.md file explaining when to use each metric.
• Improve error message specificity: lib/scientist/errors.rb likely defines custom exceptions, but test coverage for error scenarios is light. Add test cases in test/scientist/errors_test.rb (new file) to verify each error type is raised with clear messages in edge cases (e.g., control and candidate both raise, only candidate raises).
• Add an integration example for Rails: the README shows generic Ruby, but no Rails-specific example is visible. Add examples/rails_permissions_refactor.rb showing how to wrap a Pundit policy refactor or CanCanCan upgrade using Scientist, then link it from the README.

The Scientist library codebase demonstrates a secure posture overall. As a Ruby library for experimental code execution, it does not contain obvious injection vulnerabilities, hardcoded secrets, or exposed credentials. The primary security concerns are process-related rather than code-based: the absence of a responsible disclosure policy (SECURITY.md) and lack of code ownership definitions (CODEOWNERS). The library's purpose (safely comparing old vs. new code behavior) is inherently security-conscious. Dependency management appears sound with dependabot configured. No Docker infrastructure issues detected. Recommendations focus on improving security governance processes.

• Low · Missing CODEOWNERS file — .github/CODEOWNERS (missing). The repository lacks a CODEOWNERS file, which helps enforce code review requirements and ensures security-sensitive changes are reviewed by appropriate maintainers. Fix: Create a .github/CODEOWNERS file to define ownership and enforce review requirements for security-critical changes.
• Low · Incomplete security documentation — SECURITY.md (missing). No SECURITY.md file found for disclosing security vulnerabilities responsibly. This makes it difficult for security researchers to report issues privately. Fix: Create a SECURITY.md file with clear instructions for reporting security vulnerabilities responsibly, following GitHub's security advisory guidelines.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.