If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/TechEmpower/FrameworkBenchmarks shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across the board

• Last commit 7w ago
• 18 active contributors
• Distributed ownership (top contributor 36% of recent commits)
• Other licensed
• CI configured
• Tests present
• ⚠ Non-standard license (Other) — review terms

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live TechEmpower/FrameworkBenchmarks repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/TechEmpower/FrameworkBenchmarks.

What it runs against: a local clone of TechEmpower/FrameworkBenchmarks — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in TechEmpower/FrameworkBenchmarks | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 77 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of TechEmpower/FrameworkBenchmarks. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/TechEmpower/FrameworkBenchmarks.git
#   cd FrameworkBenchmarks
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of TechEmpower/FrameworkBenchmarks and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "TechEmpower/FrameworkBenchmarks(\\.git)?\\b" \\
  && ok "origin remote is TechEmpower/FrameworkBenchmarks" \\
  || miss "origin remote is not TechEmpower/FrameworkBenchmarks (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "frameworks" \\
  && ok "frameworks" \\
  || miss "missing critical file: frameworks"
test -f ".github/workflows/build.yml" \\
  && ok ".github/workflows/build.yml" \\
  || miss "missing critical file: .github/workflows/build.yml"
test -f "Dockerfile" \\
  && ok "Dockerfile" \\
  || miss "missing critical file: Dockerfile"
test -f "entrypoint.sh" \\
  && ok "entrypoint.sh" \\
  || miss "missing critical file: entrypoint.sh"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 77 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~47d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/TechEmpower/FrameworkBenchmarks"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

TechEmpower Framework Benchmarks is a comprehensive performance testing suite that benchmarks web application frameworks across 50+ programming languages (C++, Java, Python, Go, Rust, C#, etc.) under controlled conditions. It measures throughput, latency, and resource usage for standardized workloads: plaintext responses, JSON serialization, database ORM operations, and server-side template rendering. The project runs tests both on cloud infrastructure and physical hardware to produce authoritative performance comparisons published at techempower.com/benchmarks/. Monorepo structure: frameworks/[LANGUAGE]/[FRAMEWORK]/ directories each contain a benchmark_config.json (test metadata), Dockerfile or .dockerfile (containerized build), README.md (framework-specific instructions), and language-specific source (main.cpp, Main.java, etc.). Root-level entrypoint.sh and ./tfb script orchestrate test execution. .github/workflows/ automates build validation. deployment/vagrant/ provides alternative non-Docker test environments.

Framework maintainers and core contributors who need to validate their framework's performance characteristics; infrastructure engineers and architects evaluating technology stacks for production workloads; language ecosystem communities (Go, Rust, Java, Python teams) competing for performance credibility; TechEmpower itself, which publishes results as an industry-standard benchmark suite.

Highly mature and actively maintained. The repo shows 50+ languages with multiple frameworks per language, CI/CD via GitHub Actions (.github/workflows/build.yml), Docker containerization for reproducibility (Dockerfile, framework-specific .dockerfile files), and a published website with historical results. Regular contributions and framework additions indicate ongoing community activity. This is a de facto standard in web framework benchmarking.

Standard open source risks apply.

Based on file structure, active areas include: (1) framework version updates across C++, Java, Rust, Go stacks; (2) CI pipeline enhancements (build.yml, label-failing-pr.yml suggest handling of flaky tests); (3) new language/framework additions (Zig, Nim, V language folders present); (4) Docker/containerization refinement (multiple .dockerfile variants per framework). The label-failing-pr.yml and ping-maintainers.yml workflows suggest active triaging of broken tests.

git clone https://github.com/TechEmpower/FrameworkBenchmarks.git
cd FrameworkBenchmarks
# Requires Docker
./tfb --mode verify --test gemini

Alternatively, use Vagrant: cd deployment/vagrant && vagrant up then follow vagrant/README.md.

Daily commands:

./tfb --mode verify --test [framework_name]
# e.g., ./tfb --mode verify --test aspnetcore

The tfb script wraps: docker run -it --rm --network tfb -v /var/run/docker.sock:/var/run/docker.sock -v [FWROOT]:/FrameworkBenchmarks techempower/tfb [ARGS]. See README.md Quick Start section.

• README.md — Defines the project's mission to benchmark web framework performance across languages and test types; essential for understanding benchmark scope and participation.
• frameworks — Root directory containing all framework implementations; contributors must understand how frameworks are organized by language and configuration.
• .github/workflows/build.yml — CI/CD pipeline that validates all framework implementations; critical for understanding how benchmarks are tested and deployed.
• Dockerfile — Primary containerization configuration for the benchmark environment; essential for reproducible, isolated benchmark execution.
• entrypoint.sh — Main orchestration script that bootstraps the benchmark suite and manages framework deployment; the entry point for test execution.
• deployment/vagrant — Infrastructure-as-code for local development and testing; helps new contributors replicate the benchmark environment.

• Framework Implementation (C++, Python, Java, Go, Ruby, PHP, Scala, Clojure, Erlang, Haskell, Perl, Lua, JavaScript, Dart, F#, C#) — Language-specific web application implementing standardized test endpoints (plaintext, JSON, ORM queries, template rendering)
  • Failure mode: Failed health check, connection timeout, incorrect response format, or performance degradation triggers test failure
• Docker Orchestration (Docker, Docker Compose) — Builds and launches isolated containers for each framework variant, manages networking and port exposure
  • Failure mode: Image build failure, container startup timeout, or port binding conflict aborts benchmark run
• Configuration Management (JSON, TOML) — benchmark_config.json and config.toml define test endpoints, database variants, threading models, and connection pooling behavior
  • Failure mode: Invalid configuration syntax or missing required endpoint definitions prevent benchmark execution
• Database Connectivity (PostgreSQL, MySQL, Database drivers (libpq, ODBC, JDBC, etc) — ORM layer and connection pooling handle database operations for /db and /update test endpoints

Add a New Web Framework Benchmark

1. Create framework directory under frameworks/{Language}/{FrameworkName}/ (frameworks/C++/aeronet)
2. Create benchmark_config.json defining test endpoints (json, plaintext, db, fortune, update) (frameworks/C++/aeronet/benchmark_config.json)
3. Implement framework application with required endpoints (plaintext, JSON, database ORM operations) (frameworks/C++/aeronet/main.cpp)
4. Create Dockerfile for isolated, reproducible containerized benchmark execution (frameworks/C++/aeronet/aeronet.dockerfile)
5. Create config.toml with runtime parameters (threads, connection pooling, port) (frameworks/C++/aeronet/config.toml)
6. Add framework README with build/run instructions and known characteristics (frameworks/C++/aeronet/README.md)

Add a Database Variant Configuration

1. Create alternate Dockerfile with different database driver (MySQL, PostgreSQL) (frameworks/C++/cppcms/cppcms-postgres.dockerfile)
2. Create database-specific config file with connection string and pooling parameters (frameworks/C++/cppcms/config-nginx-postgresql.json)
3. Update benchmark_config.json with new test variant entry pointing to alternate Dockerfile (frameworks/C++/cppcms/benchmark_config.json)

Add a Framework Language Family

1. Create top-level language directory under frameworks/{NewLanguage}/ (frameworks/C++)
2. Create first framework subdirectory with standard structure (main app, Dockerfile, benchmark_config.json, config.toml, README.md) (frameworks/C++/aeronet)
3. Implement all required test endpoints: plaintext, json, db read, db update, fortune template rendering (frameworks/C++/aeronet/main.cpp)

• Docker — Ensures reproducible, isolated benchmark environments across diverse infrastructure and eliminates dependency conflicts between frameworks
• Multiple Languages (C++, Python, Java, Go, Ruby, PHP, etc.) — Enables fair cross-language performance comparison and demonstrates framework suitability across different technology ecosystems
• Vagrant — Provides local development and testing environment that mirrors production benchmark infrastructure for contributors
• GitHub Actions CI/CD — Automates validation of framework implementations and prevents regressions in benchmark configurations
• JSON & TOML Configuration — Standardizes framework metadata (test endpoints, database backends) and runtime behavior in human-readable, version-controllable format

• Framework-specific Dockerfiles vs. monolithic build system

  • Why: Each framework has unique build dependencies, languages, and compilation requirements
  • Consequence: Higher maintenance burden but ensures accurate representation of each framework's real-world deployment

• Benchmark suite includes both synchronous and asynchronous frameworks

  • Why: Different frameworks excel under different concurrency models (threads vs. async I/O)
  • Consequence: Requires multiple test configurations but provides holistic performance comparison

• Multiple database backend variants (MySQL, PostgreSQL) per framework

  • Why: Different databases exhibit different performance characteristics for the same ORM code
  • Consequence: Combinatorial explosion of test scenarios but reveals database-specific optimization opportunities

• Real-world application benchmarking with complex business logic—scope limited to standard test patterns (plaintext, JSON, ORM queries, templates)
• Security or vulnerability assessment—benchmarks focus on performance, not security hardening
• Framework functionality testing—does not validate correctness, only measures performance on standardized endpoints
• Continuous monitoring or production telemetry—one-time benchmark runs, not ongoing observability
• Support for JavaScript/Node.js frameworks beyond standard HTTP benchmarking (no WebSocket or real-time protocol variants)

Docker network 'tfb' must exist: The ./tfb script uses --network tfb; if not present, container will fail. Requires docker network create tfb first. Framework-specific DB dependencies: Each framework's benchmark_config.json likely points to a Postgres instance (framework-specific .dockerfile variants mention postgres-raw.dockerfile); test infrastructure must spin up database containers or the test hangs. Port conflicts: Multiple frameworks running simultaneously on localhost can fail if ports collide; Docker isolation mitigates but manual testing requires explicit port mapping. Vagrant requires VirtualBox: deployment/vagrant/Vagrantfile assumes VirtualBox; on Windows/Apple Silicon, requires workarounds. entrypoint.sh logic opaque: The actual test harness (request generation, latency measurement, result aggregation) is likely inside entrypoint.sh, not visible in file tree—familiarize yourself with it before modifying test behavior.Test durations: Full benchmark runs (all frameworks, all tests) take hours; use --mode verify for quick spot-checks.

• ORM (Object-Relational Mapping) Benchmarking — TFB includes 'db' and 'queries' test types that measure ORM performance; understanding how frameworks abstract database I/O (query batching, lazy loading, N+1 problem) is critical to interpreting TFB results and optimizing real applications.
• Containerized Benchmarking & Reproducibility — TFB uses Docker containers (framework-specific .dockerfile files) to isolate each test and eliminate system-level variance; understanding Docker layer caching, image sizes, and startup overhead is essential when adding new frameworks or diagnosing flaky tests.
• Plaintext vs JSON Encoding Overhead — TFB's 'plaintext' and 'json' test types measure the framework's serialization speed; JSON encoding becomes a bottleneck in high-throughput scenarios, and frameworks that optimize serialization (e.g., code generation, memory pooling) rank higher on TFB.
• Connection Pooling & Database Concurrency — Frameworks' 'db' tests depend on Postgres connection pooling; understanding connection limits, pool exhaustion, and query queueing explains why some frameworks handle concurrent requests better than others—critical for TFB results validity.
• Server-Side Template Rendering Performance — TFB includes 'fortunes' (HTML rendering) test workload; frameworks differ in template compilation (JIT vs interpreted) and caching strategies—this measures real-world web app performance beyond JSON APIs.
• Throughput vs Latency Trade-offs — TFB measures both requests-per-second (throughput) and response time distributions (latency percentiles); some frameworks optimize for median latency while sacrificing tail latency; understanding this distinction is key to choosing the right framework for your SLA.
• Async I/O & Event-Driven Architectures — High-performing TFB frameworks (Rust Actix, Go stdlib, Node.js) use non-blocking I/O and event loops; understanding async runtime design (tokio, libuv, epoll) explains why some languages dominate TFB leaderboards despite similar algorithmic complexity.

• mahmoud/boltons — Utility library for Python benchmarking and profiling; used by some Python frameworks in TFB for measurement.
• wg/wrk — Modern HTTP load-testing tool; likely underlying driver for the request generation workload in TFB's test harness (plaintext, JSON load testing).
• brendangregg/FlameGraph — Flame graph visualization; used for performance profiling of top-performing frameworks to identify bottlenecks in CPU/memory.
• grpc/grpc — Modern RPC framework used in some TFB implementations (e.g., Rust, Go); represents high-performance alternative to HTTP REST benchmarks.
• RedisLabs/memtier_benchmark — Cache benchmarking tool; related to database workloads in TFB; some frameworks may use Redis as backing store.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Create a centralized benchmark_config.json schema validator and documentation

Each framework has a benchmark_config.json file (e.g., frameworks/C++/aeronet/benchmark_config.json, frameworks/C++/cinatra/benchmark_config.json), but there's no visible schema documentation or validation tool. This causes configuration inconsistencies across 100+ frameworks. A JSON schema file with a validator script would prevent invalid configs and help new contributors understand required fields.

• [ ] Create benchmarks/benchmark_config.schema.json with complete JSON Schema specification for benchmark_config.json structure
• [ ] Add a Python/shell validation script in the root that runs ajv validate or similar against all framework configs
• [ ] Add validation step to .github/workflows/build.yml to catch invalid configs in PRs early
• [ ] Document the schema in a new BENCHMARK_CONFIG.md file with examples from existing frameworks

Add GitHub Actions workflow for framework-specific Docker image validation

The repo contains 100+ Dockerfiles across frameworks (e.g., frameworks/C++/aeronet/aeronet.dockerfile, frameworks/C++/cpoll_cppsp/cpoll_cppsp.dockerfile) but there's no automated validation that these build successfully. The current .github/workflows/build.yml doesn't show framework-by-framework Docker build checks. This causes Dockerfile regressions to slip through.

• [ ] Create .github/workflows/validate-dockerfiles.yml that builds a sample of framework Dockerfiles (start with C++, Go, Python)
• [ ] Add a script in tools/validate_dockerfiles.sh that tests Docker build for frameworks modified in the PR
• [ ] Configure the workflow to run on PR events with path filtering for frameworks/*/
• [ ] Add failure annotations to comment on PRs with specific Docker build errors

Consolidate and document the Vagrant deployment setup used across deployment/vagrant/

The deployment/vagrant/ directory has multiple files (Vagrantfile, bootstrap.sh, core.rb, custom_motd.sh) but there's no clear README explaining the Vagrant setup process or how to contribute benchmark infrastructure. New contributors wanting to test locally or add a new framework don't have clear guidance on using this environment.

• [ ] Expand deployment/vagrant/README.md with step-by-step setup instructions, system requirements, and troubleshooting
• [ ] Add documented examples showing how to run benchmarks locally via Vagrant for specific frameworks
• [ ] Create deployment/vagrant/CONTRIBUTING.md explaining how to modify Vagrant config when adding new frameworks/dependencies
• [ ] Add comments to Vagrantfile, bootstrap.sh, and core.rb explaining each section's purpose and dependencies

• Add missing framework version pinning in Dockerfiles: Review frameworks/[LANGUAGE]/*/[framework].dockerfile and identify any that use unversioned base images (FROM node:latest instead of node:18.x) or unversioned dependency installs (apt-get install, pip install without version); submit PR adding explicit versions to ensure reproducible builds across CI runs.: Unversioned dependencies cause flaky tests (label-failing-pr.yml suggests current breakages); pinning improves CI reliability.
• Document missing benchmark_config.json schemas: Several frameworks/ subdirectories may have outdated or minimal README.md files. Create a canonical BENCHMARK_CONFIG_SCHEMA.md in the root or wiki, then audit frameworks///benchmark_config.json files and update README.md for any framework with <100 lines of documentation, adding examples of test types (plaintext, json, db) and database connection fields.: New contributors can't understand how to add frameworks without clear examples; this lowers barrier to contribution.
• Add CI check for unmaintained frameworks: Parse .github/workflows/build.yml and check for frameworks whose Dockerfile builds have failed in the last N commits. Create a simple script that identifies deprecated or broken framework entries, then submit issue or PR to remove them or flag them as 'community-maintained' in a FRAMEWORKS_STATUS.md file.: 300+ frameworks → maintenance burden; automated detection of broken builds prevents stale entries from inflating the test suite and slowing CI.

• High · Insecure pip Installation with --break-system-packages — Dockerfile, lines 25-29. The Dockerfile uses 'pip3 install --break-system-packages' which bypasses Python's safety mechanisms designed to prevent conflicts with system-managed packages. This could allow malicious packages to overwrite critical system files. Fix: Use virtual environments or containers to isolate Python dependencies instead of --break-system-packages. Consider using 'python3 -m venv' or Docker's multi-stage builds to manage dependencies safely.
• High · Hardcoded Package Versions Without Hash Verification — Dockerfile, lines 27-29. Docker installation pins specific package versions (docker==7.0.0, mysqlclient==2.2.4, pymongo==4.7.2) but does not use hash verification (--require-hashes). This allows package tampering or man-in-the-middle attacks during installation. Fix: Add hash verification to pip install commands: 'pip3 install --require-hashes docker==7.0.0 --hash=sha256:...' or use a lock file (requirements.lock with hashes). Implement package signature verification.
• Medium · Outdated Base Image with Potential Vulnerabilities — Dockerfile, line 1. The Dockerfile uses 'ubuntu:24.04' without a specific date tag, which is subject to change. While Ubuntu 24.04 is recent, relying on floating tags can introduce unexpected security updates or breaking changes. Fix: Use a specific dated base image tag (e.g., 'ubuntu:24.04-20240101') to ensure reproducible builds and control when security patches are applied. Implement regular image scanning with tools like Trivy.
• Medium · Excessive Package Installation Without Minimization — Dockerfile, lines 6-24. The Dockerfile installs multiple development packages (gcc, git-core, pkg-config, python3-dev) that may not be needed at runtime. This increases the attack surface and container size. The Dockerfile appears incomplete (cut off at 'Collect resour'). Fix: Implement multi-stage Docker builds: separate build stage from runtime stage. Remove unnecessary development tools from the final image. Clean up apt cache and temporary files with 'apt-get clean && rm -rf /var/lib/apt/lists/*'.
• Medium · Missing Security Headers in Framework Configurations — frameworks/C++/cppcms/nginx.conf and similar configuration files. The file structure shows multiple nginx configuration files (e.g., frameworks/C++/cppcms/nginx.conf, frameworks/C++/cutelyst/config) that are not visible in the content. Web frameworks commonly miss critical security headers like Content-Security-Policy, X-Frame-Options, and X-Content-Type-Options. Fix: Add security headers to all nginx and framework configurations: 'X-Frame-Options: DENY', 'X-Content-Type-Options: nosniff', 'Content-Security-Policy', 'Strict-Transport-Security', 'X-XSS-Protection'.
• Medium · Potential SQL Injection in Multiple Frameworks — frameworks/*/db, frameworks/*/src/*db*.cpp, and similar database-related files. The codebase contains multiple database query implementations across C++, Python, Java, and other languages (e.g., frameworks/C++/cpoll_cppsp/www/db, frameworks/C++/cppcms/src/test_db.cpp). Without visibility into implementation details, raw SQL queries are common in benchmarking code. Fix: Use parameterized queries and prepared statements exclusively. Never concatenate user input into SQL strings. Implement input validation and sanitization. Use ORM frameworks with built-in SQL injection protection where applicable.
• Low · Incomplete Dockerfile (Truncated Content) — Dockerfile, end of file. The provided Dockerfile is truncated at 'Collect resour' (line 31), indicating incomplete configuration. This prevents full security analysis of the Docker setup. Fix: Review the complete Dockerfile to identify additional security issues. Ensure all subsequent sections follow security best practices including explicit USER declarations, read-only filesystems where possible, and security scanning.
• Low · Exposed Database Credentials in Configuration Files — undefined. undefined Fix: undefined

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.