If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/zombocom/derailed_benchmarks shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

AVOID — Stale and unlicensed — last commit 1y ago

• 15 active contributors
• CI configured
• Tests present
• ⚠ Stale — last commit 1y ago
• ⚠ Concentrated ownership — top contributor handles 68% of recent commits
• ⚠ No license — legally unclear to depend on

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live zombocom/derailed_benchmarks repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/zombocom/derailed_benchmarks.

What it runs against: a local clone of zombocom/derailed_benchmarks — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in zombocom/derailed_benchmarks | Confirms the artifact applies here, not a fork | | 2 | Default branch main exists | Catches branch renames | | 3 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 4 | Last commit ≤ 523 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of zombocom/derailed_benchmarks. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/zombocom/derailed_benchmarks.git
#   cd derailed_benchmarks
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of zombocom/derailed_benchmarks and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "zombocom/derailed_benchmarks(\\.git)?\\b" \\
  && ok "origin remote is zombocom/derailed_benchmarks" \\
  || miss "origin remote is not zombocom/derailed_benchmarks (artifact may be from a fork)"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "lib/derailed_benchmarks.rb" \\
  && ok "lib/derailed_benchmarks.rb" \\
  || miss "missing critical file: lib/derailed_benchmarks.rb"
test -f "lib/derailed_benchmarks/tasks.rb" \\
  && ok "lib/derailed_benchmarks/tasks.rb" \\
  || miss "missing critical file: lib/derailed_benchmarks/tasks.rb"
test -f "lib/derailed_benchmarks/core_ext/kernel_require.rb" \\
  && ok "lib/derailed_benchmarks/core_ext/kernel_require.rb" \\
  || miss "missing critical file: lib/derailed_benchmarks/core_ext/kernel_require.rb"
test -f "lib/derailed_benchmarks/require_tree.rb" \\
  && ok "lib/derailed_benchmarks/require_tree.rb" \\
  || miss "missing critical file: lib/derailed_benchmarks/require_tree.rb"
test -f "lib/derailed_benchmarks/stats_from_dir.rb" \\
  && ok "lib/derailed_benchmarks/stats_from_dir.rb" \\
  || miss "missing critical file: lib/derailed_benchmarks/stats_from_dir.rb"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 523 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~493d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/zombocom/derailed_benchmarks"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Derailed Benchmarks is a Ruby gem that profiles and benchmarks entire Rails applications to identify memory usage, load time bottlenecks, and performance hotspots. It can either statically analyze gem dependencies in your Gemfile (via derailed bundle:mem) or dynamically boot your Rails app and run instrumented HTTP requests to measure real-world performance across different Rails versions (5.1–8.0). Single-gem structure: /lib/derailed_benchmarks contains the main logic organized by concern (auth helpers, git utilities, stats calculators). Entry point is bin/derailed CLI. Rake tasks defined in lib/derailed_benchmarks/tasks.rb. Multi-version testing via Appraisals gemfiles in /gemfiles (one per Rails version). Tests in /test mirror the lib structure.

Rails developers and DevOps engineers who need to optimize application startup time, memory footprint, and request performance in production environments. Specifically targets teams debugging why their Rails app is slow or memory-hungry and need visibility into which gems/code paths are responsible.

Actively maintained and production-ready. The repo has CI/CD via GitHub Actions, comprehensive test coverage in /test, support for 9 Rails versions (5.1 through 8.0+HEAD), and a detailed CHANGELOG. Last activity appears recent based on the Rails 8.0 gemfile inclusion.

Low risk for a mature tool. Single maintainer (schneems/Schneems visible in repo), but the tool's narrow scope (benchmarking, not core app logic) limits blast radius. No apparent breaking changes in recent commits. Dependency surface is small since it wraps Ruby's built-in benchmark and optional stackprof gem.

Actively supporting latest Rails versions—Rails 8.0 and rails_head gemfiles present. CI pipeline runs against all supported versions. Recent work likely focused on maintaining compatibility as Rails evolves (8.0 is recent).

git clone https://github.com/zombocom/derailed_benchmarks.git
cd derailed_benchmarks
bundle install
bundle exec rake test

Daily commands:

# Static gem memory analysis
bundle exec derailed bundle:mem

# Dynamic app benchmarking (requires Rails app context)
bundle exec derailed exec perf:ips
bundle exec derailed exec perf:mem

# Tests
bundle exec rake test

• lib/derailed_benchmarks.rb — Main entry point and module definition—defines the public API and loads core components.
• lib/derailed_benchmarks/tasks.rb — Rake task definitions that drive all benchmarking workflows—essential to understand how benchmarks are invoked.
• lib/derailed_benchmarks/core_ext/kernel_require.rb — Patches Ruby's require/load to track file loading metrics—core instrumentation for the entire framework.
• lib/derailed_benchmarks/require_tree.rb — Analyzes require/load call trees and generates statistics—fundamental to dependency and boot-time analysis.
• lib/derailed_benchmarks/stats_from_dir.rb — Aggregates benchmark statistics from test fixture directories—used by comparison and reporting tasks.
• bin/derailed — CLI entry point—how end-users invoke derailed benchmarks as a command-line tool.
• derailed_benchmarks.gemspec — Gem metadata and dependency declarations—defines what versions of Rails and Ruby are supported.

• Kernel#require Patch (Ruby Kernel module, metaprogramming) — Intercepts and instruments Ruby's require/load calls to track which files are loaded, when, and their impact on memory and timing.
  • Failure mode: If patching fails or is incomplete, benchmark results are inaccurate or missing; circular requires may cause infinite loops.
• Tasks Orchestrator (Rake, Ruby) — Orchestrates the full benchmark lifecycle: booting app, running requests, collecting stats, comparing branches.
  • Failure mode: If a task fails, the entire benchmark run is aborted; partial results are lost.
• Statistics Collector & Aggregator (Ruby Hash/Array, file I/O) — Accumulates metrics from require patches and compares results across runs or git branches.
  • Failure mode: If stats are lost or corrupted, comparisons become unreliable; no rollback mechanism.
• Git Manager (Git CLI) — Switches git branches/commits and re-runs benchmarks to enable automated A/B comparisons.
  • Failure mode: If git operations fail (e.g., dirty working tree), benchmark runs are aborted; may leave repo in inconsistent state.
• Auth Helpers — Supports authenticating HTTP requests (e.g., via Devise) so benchmarks

Add a New Benchmark Task

1. Define a new Rake task method in the DerailedBenchmarks::Tasks class (lib/derailed_benchmarks/tasks.rb)
2. Use Kernel#require patching via kernel_require.rb to track metrics during task execution (lib/derailed_benchmarks/core_ext/kernel_require.rb)
3. Call stats_for_file.rb or stats_from_dir.rb to aggregate and return results (lib/derailed_benchmarks/stats_for_file.rb)
4. Register the task in load_tasks.rb so it's available in Rails apps (lib/derailed_benchmarks/load_tasks.rb)

Support a New Authentication Method

1. Create a new file in lib/derailed_benchmarks/auth_helpers/ (e.g., auth_helpers/oauth.rb) (lib/derailed_benchmarks/auth_helpers)
2. Implement the authentication interface defined in auth_helper.rb (lib/derailed_benchmarks/auth_helper.rb)
3. Register the new auth helper in the auth_helper.rb dispatcher or in tasks.rb where auth is used (lib/derailed_benchmarks/tasks.rb)
4. Test with the test Rails app by adding test users or config in test/rails_app/config/initializers/ (test/rails_app/config/initializers)

Add Support for a New Rails Version

1. Create a new gemfile in gemfiles/ (e.g., gemfiles/rails_8_1.gemfile) (gemfiles)
2. Update Appraisals to define the test matrix for the new version (Appraisals)
3. Update CI workflow if needed to test against the new version (.github/workflows/ci.yml)
4. Update gemspec compatibility constraints if the new version requires different dependencies (derailed_benchmarks.gemspec)

• Ruby metaprogramming (Kernel patching) — Enables non-intrusive instrumentation of the entire require/load chain without modifying user app code.
• Rake tasks — Standard Rails/Ruby convention for background jobs; integrates naturally into Rails workflows.
• Git integration — Allows automated A/B benchmarking across commits without manual branch switching.
• Fixture Rails app (test/rails_app) — Provides a consistent, repeatable test environment for validating benchmark behavior across Rails versions.

• Kernel#require patching vs explicit instrumentation

  • Why: Patching is less intrusive and captures all loading, but introduces metaprogramming complexity and potential edge cases.
  • Consequence: Benchmarks are transparent to user code but harder to debug; requires careful handling of circular requires.

• Single fixture Rails app vs multi-app fixtures

  • Why: Simpler maintenance and faster CI, but limits testing diversity.
  • Consequence: May not catch issues specific to complex multi-engine Rails apps; trade-off favors development velocity.

• Git-based branch switching in tasks vs separate run invocations

  • Why: Automatic switching allows automated comparison pipelines; manual approach gives user control.
  • Consequence: Users get comparison out-of-the-box but less flexibility for custom workflows.

• Does not provide real-time performance monitoring or continuous profiling in production.
• Does not replace full APM or production profiling tools (e.g., New Relic, Datadog).
• Does not benchmark database queries in isolation (focuses on Rails boot and app startup).
• Does not support Windows natively (relies on curl and shell scripts; Linux/macOS focus).

1. stackprof gem is optional but required for certain profiling features—install separately if desired. 2. Benchmarking dynamic app performance requires a bootable Rails app; won't work on static gem analysis alone. 3. curl binary must be on PATH for HTTP request benchmarking (noted in README). 4. Git switching features in /lib/derailed_benchmarks/git assume you're in a Git repo; will fail silently otherwise. 5. Some tasks may be slow on first run due to full gem tree traversal (bundle:mem can take minutes on large Gemfiles).

• Memory introspection via require hooking — Derailed patches Ruby's require in kernel_require.rb to measure heap delta per gem—understanding this metacircular approach is key to how bundle:mem works.
• Sampling profilers vs. instrumentation — Derailed uses stackprof (sampling) for low-overhead wall-clock profiling vs. ruby-prof (instrumentation); knowing when each applies helps interpret benchmark results.
• Rails boot lifecycle and lazy loading — Derailed's ability to intercept require calls and measure gem loading depends on understanding Rails' two-phase boot (eager load vs. autoload) and Zeitwerk.
• Appraisals for multi-version testing — The /gemfiles directory uses Appraisals to generate per-Rails-version Gemfiles; this pattern lets Derailed test against 9 major Rails versions in CI without conditional code.
• Process substitution & curl for request benchmarking — Derailed shells out to curl to generate real HTTP traffic against a booted Rails process; understanding this subprocess model is needed to debug request-based benchmarks.
• Git ref switching for comparative benchmarks — Derailed can switch Git commits and re-benchmark to measure performance deltas across versions; /lib/derailed_benchmarks/git abstracts this, useful for regression detection.

• fasterer/fasterer — Static code analyzer for Rails apps that finds performance anti-patterns; complements Derailed's runtime/load-time focus with linting.
• ruby-prof/ruby-prof — Lower-level Ruby profiler that Derailed wraps; understanding ruby-prof's output helps interpret Derailed's stackprof integration.
• evanphx/benchmark-ips — Gem that Derailed bundles for perf:ips task; Derailed is a thin harness around this for Rails-specific scenarios.
• tmm1/stackprof — Sampling profiler that Derailed optionally uses for flame graphs and stack sampling; optional dependency for deeper profiling.
• rails/rails — Derailed's primary target—tests compatibility across Rails versions via Appraisals and provides the framework context being benchmarked.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add integration tests for Rails 8.0 and Rails head gemfiles

The repo has gemfiles for Rails 5.1 through 8.0 and rails_head.gemfile, but test/integration/tasks_test.rb likely doesn't validate all versions. Given that derailed_benchmarks is a Rails-specific tool that must work across versions, adding CI matrix tests and integration test coverage for the newest Rails versions (8.0 and head) ensures compatibility and catches breakage early.

• [ ] Review test/integration/tasks_test.rb to identify current Rails version coverage
• [ ] Add parameterized integration tests for Rails 8.0 and rails_head.gemfile variants
• [ ] Verify all major benchmark tasks (perf, memory, stackprof) execute without errors on these versions
• [ ] Update .github/workflows/ci.yml to test against multiple gemfile variants if not already present

Add unit tests for lib/derailed_benchmarks/stats_for_file.rb

The file structure shows lib/derailed_benchmarks/stats_for_file.rb and test/derailed_benchmarks/stats_from_dir_test.rb exists, but there's no corresponding test/derailed_benchmarks/stats_for_file_test.rb. This utility is critical for parsing and analyzing benchmark output files, so dedicated unit tests would improve reliability and make refactoring safer.

• [ ] Create test/derailed_benchmarks/stats_for_file_test.rb
• [ ] Write tests covering single file parsing, edge cases (empty files, malformed data, missing metrics)
• [ ] Add fixture files in test/fixtures/stats/ for various benchmark output formats
• [ ] Test integration with existing stats comparison logic used by tasks

Add comprehensive tests for lib/derailed_benchmarks/auth_helpers/devise.rb

The devise.rb auth helper exists in lib/derailed_benchmarks/auth_helpers/ but there are no corresponding unit tests. Authentication helpers are security-sensitive code used for benchmarking authenticated requests, and lack of test coverage makes it difficult to maintain and verify correctness across Rails versions.

• [ ] Create test/derailed_benchmarks/auth_helpers/devise_test.rb
• [ ] Write tests for devise session setup, user creation, and authentication mocking
• [ ] Add fixtures or factories for test user generation
• [ ] Test integration with the main auth_helper.rb to ensure proper fallback and delegation

• Add test coverage for /lib/derailed_benchmarks/auth_helpers/devise.rb—visible in file list but /test/derailed_benchmarks/auth_helpers/ is absent, meaning Devise auth integration is untested.
• Document which benchmarking tasks work in which contexts (static vs. booted app) in a guide under /docs—README mentions the distinction but lacks per-task clarity.
• Add a --json-output flag to derailed bundle:mem and other tasks to emit JSON instead of plain text, enabling easier machine parsing for CI/CD pipelines (visible gap in task.rb).

The derailed_benchmarks gem has moderate security posture. Primary concerns are around dynamic code loading, external command execution (git, curl), and potential code injection vectors. The codebase lacks visible hardcoded credentials or critical misconfigurations. Main recommendations: (1) Implement strict input validation for all user-supplied data, particularly in git and require operations; (2) Use parameterized command execution instead of string interpolation; (3) Enforce environment checks to prevent deployment in production; (4) Replace external curl calls with native Ruby HTTP libraries where possible. The gem appears designed for development use, but additional safeguards should be implemented to prevent accidental production exposure.

• Medium · External Command Execution via Git Operations — lib/derailed_benchmarks/git/commit.rb, lib/derailed_benchmarks/git/switch_project.rb, lib/derailed_benchmarks/git_switch_project.rb. The codebase contains git operation utilities (lib/derailed_benchmarks/git/) that execute external commands. If user input is passed unsanitized to git commands, this could lead to command injection attacks. Fix: Ensure all git commands use parameterized execution (e.g., using Ruby's Kernel#system with array arguments rather than string interpolation). Validate and sanitize all user inputs before passing to shell commands.
• Medium · Potential Code Injection via Dynamic Require — lib/derailed_benchmarks/core_ext/kernel_require.rb, lib/derailed_benchmarks/require_tree.rb. The lib/derailed_benchmarks/core_ext/kernel_require.rb and lib/derailed_benchmarks/require_tree.rb files handle dynamic file loading and requiring. If file paths are derived from user input without proper validation, arbitrary code execution is possible. Fix: Implement strict whitelist validation for any dynamically required files. Use absolute paths and verify that file paths resolve within expected directories. Never allow user-controlled input to directly influence require paths.
• Medium · Curl Dependency External Process Execution — README.md (curl requirement), likely in benchmark tasks. The README indicates curl is used as an external dependency for benchmarking. If URLs or curl parameters are user-supplied without validation, command injection or SSRF attacks are possible. Fix: Validate and sanitize all URLs before passing to curl. Use Ruby's built-in HTTP libraries (Net::HTTP, HTTPClient) instead of shell curl where possible, or use parameterized curl execution to prevent injection.
• Low · Missing HTTPS Configuration in DevContainer — .devcontainer/devcontainer.json. The .devcontainer/devcontainer.json configuration file may expose services without enforcing HTTPS, which could be a concern if the container runs on a network. Fix: Review the devcontainer configuration for any exposed ports. If services are exposed, ensure they use HTTPS or are restricted to localhost only.
• Low · Test Fixtures May Contain Exploitable Code Paths — test/fixtures/. Test fixtures in test/fixtures/require/ contain files designed to test require behavior (including raise_child.rb and relative imports). These should not be included in production builds but could be accidentally deployed. Fix: Ensure test directories and fixtures are excluded from gem package (verify gemspec excludes test/ directory). Add test/ to .gitignore if not already present for built artifacts.
• Low · Insufficient Access Control in Auth Helpers — lib/derailed_benchmarks/auth_helper.rb, lib/derailed_benchmarks/auth_helpers/devise.rb. The auth_helpers directory (lib/derailed_benchmarks/auth_helpers/devise.rb) handles authentication. If misconfigured, it could allow unauthorized access to benchmark endpoints. Fix: Ensure auth helpers enforce proper authentication/authorization checks. Verify that benchmark endpoints are only accessible in development/test environments. Add environment checks to prevent exposure in production.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.