If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/pytorch/serve shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across all four use cases

• Last commit 9mo ago
• 31+ active contributors
• Distributed ownership (top contributor 22% of recent commits)
• Apache-2.0 licensed
• CI configured
• Tests present
• ⚠ Slowing — last commit 9mo ago

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live pytorch/serve repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/pytorch/serve.

What it runs against: a local clone of pytorch/serve — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in pytorch/serve | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 305 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of pytorch/serve. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/pytorch/serve.git
#   cd serve
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of pytorch/serve and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "pytorch/serve(\\.git)?\\b" \\
  && ok "origin remote is pytorch/serve" \\
  || miss "origin remote is not pytorch/serve (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f ".github/workflows/ci_cpu.yml" \\
  && ok ".github/workflows/ci_cpu.yml" \\
  || miss "missing critical file: .github/workflows/ci_cpu.yml"
test -f "CONTRIBUTING.md" \\
  && ok "CONTRIBUTING.md" \\
  || miss "missing critical file: CONTRIBUTING.md"
test -f "benchmarks/utils/benchmarks.py" \\
  && ok "benchmarks/utils/benchmarks.py" \\
  || miss "missing critical file: benchmarks/utils/benchmarks.py"
test -f ".github/workflows/official_release.yml" \\
  && ok ".github/workflows/official_release.yml" \\
  || miss "missing critical file: .github/workflows/official_release.yml"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 305 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~275d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/pytorch/serve"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

TorchServe is a production-grade model serving system for PyTorch that packages, deploys, and scales trained models via REST/gRPC APIs with minimal boilerplate. It handles model versioning, batching, metrics collection, and worker process management to serve inference workloads at scale without requiring users to write deployment code. Monorepo structure: ts_scripts/ contains setup & dependency installation; benchmarks/ holds performance testing suites; .github/workflows/ defines 30+ CI pipelines (CPU, GPU, Graviton, Docker, KServe, Kubernetes). Core backend is split between Java (model server, worker lifecycle) and Python (inference handlers, model archiving via torch-model-archiver and torch-workflow-archiver).

ML engineers and PyTorch researchers deploying models to production who need HTTP/gRPC endpoints, multi-model serving, canary deployments, and autoscaling without building custom serving infrastructure. Also used by platform teams building internal model serving infrastructure on Kubernetes.

Production-ready but no longer actively maintained — the README explicitly states limited maintenance with no planned updates or security patches. The codebase is mature (extensive CI/CD with 30+ GitHub Actions workflows, comprehensive Java/Python backend) but deprecated, meaning existing deployments work but contributions and fixes are not being accepted.

Critical risk: project is in maintenance-only mode per the README disclaimer; vulnerabilities may not be patched. Dependency on aging Java framework (1.1M LOC Java) and Python 3.8+ requirement. Multiple heavy dependencies (boto3, fabric2, retrying) without visibility into update cadence. Breaking changes to security defaults (token auth enforced, model API control disabled by default in recent versions) could impact existing deployments.

Project is in maintenance mode with no active development. The most recent changes visible are security-focused: enforcing token authorization by default and disabling model API control as opt-in (documented in token_authorization_api.md and model_api_control.md). CI/CD is comprehensive but static — regression tests run nightly across CPU, GPU, Graviton, Docker, and Kubernetes.

# Clone the repo
git clone https://github.com/pytorch/serve.git
cd serve

# Install dependencies (choose your accelerator or CPU)
python ./ts_scripts/install_dependencies.py
# OR with CUDA 12.1: python ./ts_scripts/install_dependencies.py --cuda=cu121
# OR with ROCm 6.1: python ./ts_scripts/install_dependencies.py --rocm=rocm61

# Install TorchServe and archiving tools
pip install torchserve torch-model-archiver torch-workflow-archiver

# Start the server (default: http://127.0.0.1:8080)
torchserve --start --model-store /path/to/models

# Test with a curl request
curl http://127.0.0.1:8080/predictions/your-model -T input.txt

Daily commands: After install, start with: torchserve --start --model-store ./models. For development: check .github/workflows/ci_cpu.yml for test execution patterns; benchmarks run via python ./benchmarks/benchmark.py with config files like benchmark_config_cpu.yaml. Docker: docker pull pytorch/torchserve && docker run --rm -p 8080:8080 pytorch/torchserve.

• README.md — Primary entry point documenting TorchServe's purpose, security announcements, and limited maintenance status that every contributor must understand
• .github/workflows/ci_cpu.yml — Primary CI/CD pipeline for CPU testing; defines validation gates for all contributions
• CONTRIBUTING.md — Contribution guidelines and development workflow; mandatory reading for all contributors
• benchmarks/utils/benchmarks.py — Core benchmarking framework used across nightly workflows and performance validation
• .github/workflows/official_release.yml — Release automation pipeline; critical for understanding versioning and deployment strategy
• MANIFEST.in — Python package distribution configuration; controls what gets packaged and deployed
• SECURITY.md — Security policy and vulnerability disclosure process; essential context given recent token authorization enforcement

Add a new benchmark configuration for a model

1. Create a new YAML file in benchmarks/models_config/ following the naming convention {model}_{platform}.yaml (benchmarks/models_config/bert_cpu.yaml)
2. Define model metadata, batch size, and hardware requirements matching the template structure (benchmarks/benchmark_config_template.yaml)
3. Reference your config in the appropriate benchmark_config_{platform}.yaml (CPU/GPU/Neuron) (benchmarks/benchmark_config_cpu.yaml)
4. Add corresponding model loader logic to benchmarks/utils/benchmarks.py if custom inference handling needed (benchmarks/utils/benchmarks.py)
5. Run validation via benchmarks/auto_benchmark.py to ensure config is properly picked up (benchmarks/auto_benchmark.py)

Add a new CI/CD workflow for a platform or test type

1. Create new YAML file in .github/workflows/ following existing naming patterns (e.g., ci_{platform}_{language}.yml) (.github/workflows/ci_cpu.yml)
2. Define triggers (push, pull_request, schedule) and job matrix for multi-version testing (.github/workflows/regression_tests_cpu.yml)
3. Reference appropriate Docker image or setup steps (see docker-ci.yaml for containerization patterns) (.github/workflows/docker-ci.yaml)
4. Integrate with existing dependabot and PR automation (pytorch-probot.yml) (.github/pytorch-probot.yml)
5. Document the new workflow in CONTRIBUTING.md with required approvals and expected runtime (CONTRIBUTING.md)

Add new benchmarking metrics or reporting

1. Extend benchmarks/utils/benchmarks.py to collect new metric data during model inference (benchmarks/utils/benchmarks.py)
2. Add metric serialization logic in benchmarks/utils/gen_metrics_json.py (benchmarks/utils/gen_metrics_json.py)
3. Create markdown report generator in benchmarks/utils/gen_md_report.py if human-readable output needed (benchmarks/utils/gen_md_report.py)
4. Update benchmarks/auto_benchmark.py orchestration to include new metric collection steps (benchmarks/auto_benchmark.py)
5. Validate output format with benchmarks/validate_report.py test suite (benchmarks/validate_report.py)

Add support for a new hardware backend (e.g., new Neuron instance type)

1. Create new benchmark config file benchmarks/benchmark_config_{backend}.yaml following existing patterns (benchmarks/benchmark_config_neuronx.yaml)
2. Add hardware-specific model configs in benchmarks/models_config/{model}_{backend}.yaml (benchmarks/models_config/opt_6.7b_neuronx.yaml)
3. Create CI workflow in .github/workflows/ci_{backend}.yml for automated testing on that hardware (.github/workflows/ci_graviton_cpu.yml)
4. Update benchmarks/utils/system_under_test.py to collect backend-specific metrics (CPU, memory, accelerator utilization) (benchmarks/utils/system_under_test.py)
5. Document backend setup in benchmarks/README.md with prerequisites and installation steps (benchmarks/README.md)

• GitHub Actions — Native GitHub integration for CI/CD with native secret management, matrix testing across platforms, and scheduled nightly runs
• Python + YAML — PyTorch ecosystem standard; YAML configs are readable for model/benchmark parameter specification across diverse hardware targets
• Apache JMeter — Industry-standard load testing tool; supports concurrent user simulation, batch requests, and visual test plan design via GUI
• Docker — Containerization ensures reproducible testing environments across CPU, GPU, and specialized hardware (Trainium, Neuron)
• boto3 — AWS SDK for automated artifact uploads and S3 storage of benchmark results across nightly runs

• Limited maintenance mode (no planned updates/security patches)

  • Why: Project transitioned to maintenance-only status after reaching maturity
  • Consequence: Users cannot expect new features or proactive vulnerability fixes; must fork or use alternative solutions for production security

• Token authorization and model API control enforced by default

  • Why: Security hardening to prevent unauthorized API calls and code injection
  • Consequence: Backward compatibility breaking change; existing deployments require explicit configuration to disable

• Multiple backend support (CPU, GPU, Neuron, NeuronX, Graviton, torch.compile)

  • Why: Flexibility for diverse deployment targets and optimization strategies
  • Consequence: Increased test matrix complexity and maintenance burden across platforms; some tests may only run on specialized hardware

• Benchmark suite separate from core Tor

  • Why: undefined
  • Consequence: undefined

1. Token authorization is enforced by default — clients must provide Authorization headers or requests fail silently; disable only if you understand the security implications (see docs/token_authorization_api.md). 2. Model API control disabled by default — custom handler endpoints must be explicitly re-enabled per model via configuration. 3. Python 3.8+ required — older runtimes will fail; check your environment before contributing. 4. Nightly builds differ from releases — pip packages torch*-nightly may have incompatibilities; regression tests run nightly but may not represent stable releases. 5. Docker setup varies by accelerator — pytorch/torchserve image is CPU; GPU variants require CUDA installation outside the base image. 6. Benchmark config files are environment-specific — benchmark_config_gpu.yaml, benchmark_config_neuron.yaml require matching hardware; CI validates against actual runners.

• Model Archiving (.mar format) — TorchServe requires models packaged as .mar archives containing weights, handler code, and metadata; understanding this format is essential for any deployment
• Worker Pool & Process Isolation — TorchServe isolates inference in worker processes per model; understanding worker lifecycle, batching, and scaling is critical for production performance tuning
• Token-Based Authorization (JWT) — TorchServe enforces token authorization by default (breaking change in recent versions); contributors must understand OAuth2/JWT flows and how to issue/validate tokens
• Canary Deployments & Model Versioning — TorchServe supports multi-version models with traffic splitting; this pattern is critical for A/B testing and gradual rollouts in production
• Batching & Dynamic Batching — TorchServe can batch inference requests to improve throughput; understanding batch_size, max_wait_ms, and handler batching logic is essential for performance optimization
• gRPC vs. REST API Serving — TorchServe supports both REST (default port 8080) and gRPC (port 7070) endpoints; different clients require different protocol implementations
• Custom Handler Lifecycle (initialize, handle, shutdown) — Handlers are Python classes that implement model-specific inference logic; bugs in handler code are the most common production issues and require understanding the full lifecycle

• pytorch/pytorch — Core runtime dependency; TorchServe models require PyTorch and ts_scripts/install_dependencies.py pins specific PyTorch versions per CUDA/ROCm variant
• pytorch/serve-examples — Companion repo with reference model handlers and archiving scripts; essential for understanding handler.py patterns and model packaging workflows
• ray-project/ray — Alternative inference serving framework; Ray Serve offers similar multi-model scaling but with different orchestration model (Actors vs. worker processes)
• kubeflow/kfserving — KServe (formerly KFServing) integrates TorchServe as a Predictor runtime; kubernetes_tests.yml validates this integration nightly
• onnx/onnx-runtime — Competing inference backend supporting PyTorch via ONNX conversion; covers different optimization goals (quantization, compression) not in TorchServe's scope

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive benchmarking CI workflow for torch.compile optimization

The repo has benchmark_torch_compile_nightly.yml workflow but lacks integration tests validating torch.compile improvements across different model architectures. Given the existence of benchmark_config_torch_compile_gpu.yaml and multiple benchmark scripts (benchmark.py, auto_benchmark.py), there's an opportunity to create a structured regression test that compares torch.compile performance gains against baseline models and alerts on performance regressions.

• [ ] Create new GitHub Actions workflow .github/workflows/torch_compile_regression_tests.yml
• [ ] Develop benchmark validation script in benchmarks/ that parses torch.compile metrics and compares against baseline thresholds defined in benchmark_config_torch_compile_gpu.yaml
• [ ] Add integration tests that verify torch.compile models achieve expected speedup percentages, failing if improvements fall below defined thresholds
• [ ] Document expected performance gains in benchmarks/README.md with specific model examples

Add security-focused unit tests for token authorization and model API control enforcement

The README announces critical security changes (token authorization enforced, model API control disabled by default). However, there's no visible test coverage in the file structure for these security features. Given the security-critical nature and the existence of docs/token_authorization_api.md and docs/model_api_control.md, comprehensive unit tests are essential to prevent regression.

• [ ] Create test/unit/test_token_authorization.py covering token validation, expiration, and unauthorized request rejection
• [ ] Create test/unit/test_model_api_control.py validating API method filtering and enforcement of disabled-by-default behavior
• [ ] Add security edge case tests: expired tokens, malformed tokens, missing auth headers, and API bypass attempts
• [ ] Integrate tests into ci_cpu.yml and ci_gpu.yml workflows to run on every commit

Implement distributed load testing suite with KServe integration validation

The repo has kserve_cpu_tests.yml and kserve_gpu_tests.yml workflows plus JMeter benchmark configs (in benchmarks/jmx/), but lacks automated distributed load testing that validates TorchServe behavior under sustained multi-client stress. The existing benchmark framework (jmeter.md, auto_benchmark.py) and JMeter test plans provide infrastructure, but no CI automation ensures load test stability across releases.

• [ ] Create new workflow .github/workflows/load_tests_distributed.yml that orchestrates multi-node JMeter tests using concurrentLoadPlan.jmx and concurrentScaleCalls.jmx
• [ ] Develop test harness (benchmarks/distributed_load_test.py) that validates KServe integration stability under sustained 1000+ concurrent requests
• [ ] Add failure detection and reporting for timeout/error rate threshold breaches (e.g. >5% error rate fails the test)
• [ ] Document expected load test SLAs in benchmarks/README.md with pass/fail criteria for different hardware configurations

• Add missing test coverage for torch-workflow-archiver integration — benchmark_config_*.yaml files have workflow configs but .github/workflows/ lacks a dedicated workflow_archiving_tests.yml job that validates .wf archive creation and deployment: Workflow archiving is a core feature but tests appear absent from CI
• Expand kserve_cpu_tests.yml with explicit handler validation tests — currently tests KServe deployment but doesn't validate custom handler execution paths (BaseHandler subclasses with different input/output formats): Handler bugs in production are common; explicit test scenarios would catch regressions early
• Document environment variable requirements for benchmarks/ — auto_benchmark.py references config.properties and config_template.properties but setup guide (docs/getting_started.md) doesn't list required vars (e.g., CUDA_VISIBLE_DEVICES, TORCHSERVE_JAR_PATH for custom builds): New contributors attempting to run benchmarks locally hit undocumented setup failures

• Critical · Project No Longer Actively Maintained — SECURITY.md, README.md. According to SECURITY.md and README, this project is no longer actively maintained. There are no planned updates, bug fixes, new features, or security patches. Vulnerabilities may not be addressed, and the project relies on automated tools (Dependabot, Snyk, CodeQL) without active human review. Fix: Users should consider using an actively maintained alternative for production deployments. If using this project, implement additional security monitoring and maintain a fork with security patches.
• High · Outdated and Unpatched Dependencies — Dependencies: fabric2, boto3, retrying. The codebase depends on fabric2, boto3, and retrying packages. Without active maintenance, these dependencies will not receive security updates. Older versions of these packages may contain known vulnerabilities. Fix: Conduct a full dependency audit using tools like 'pip-audit' or 'safety'. Lock dependency versions and implement a process for regular security updates even though the main project is unmaintained.
• High · Multiple Open Network Ports — SECURITY.md - Port Configuration. TorchServe listens on multiple ports (HTTP: 8080, 8081, 8082 and gRPC: 7070, 7071). While documented as accessible to localhost by default, misconfiguration could expose these ports publicly, allowing unauthorized access. Fix: Ensure strict firewall rules restrict access to these ports. Document and enforce localhost-only binding. Implement network segmentation and regularly audit port exposure.
• High · Token Authorization and API Control Default Configuration — README.md - Security Announcement. While token authorization is now enforced by default and model API control is disabled by default as a security feature, the announcement indicates these were previously not secure by default. Historical versions may have insecure configurations. Fix: Ensure all deployed instances run version 0.11.1 or later with token authorization enabled. Review authentication mechanisms and implement additional API access controls.
• Medium · Limited Security Support Window — SECURITY.md - Supported Versions Table. Only version 0.11.1 is listed as supported in the security policy. All other versions are unsupported and will not receive security patches. Fix: Maintain an upgrade path to version 0.11.1. Implement monitoring to detect usage of unsupported versions in production.
• Medium · Automated Security Scanning Without Active Review — SECURITY.md - How we do security section. The project relies on automated tools (Dependabot, Snyk, CodeQL) for security scanning, but without active maintenance, findings may not be addressed or reviewed by human security experts. Fix: Configure strict alert thresholds for automated tools. Implement automated CI/CD gates that block deployments with critical vulnerabilities. Consider periodic manual security audits.
• Medium · Benchmark and Configuration Files in Repository — benchmarks/ directory - configuration files. Multiple configuration files (benchmark_config_.yaml, config.properties) are present in the repository. These could potentially contain sensitive information or default credentials if not properly managed. Fix: Audit all configuration files for hardcoded credentials, API keys, or sensitive data. Use environment variables or secrets management for sensitive values. Implement pre-commit hooks to prevent credential commits.
• Medium · AWS Boto3 Dependency Without Clear Usage Documentation — Dependencies: boto3. The boto3 package (AWS SDK) is a dependency, but its usage context is unclear from the visible file structure. This could indicate cloud integration points that may be misconfigured or expose AWS credentials. Fix: Document all AWS integrations. Ensure AWS credentials are managed through IAM roles and not hardcoded. Implement least-privilege IAM policies for any AWS service access.
• Low · Docker Images Built Without Explicit Security Baseline — .github/workflows/docker-*.yml, .github/workflows/official_release_docker.yml. Multiple Docker-related workflows exist (docker-ci.yaml, docker-nightly-build.yml, official_release_docker.yml) but without visible Dockerfile content or explicit security hardening guidelines in the provided context. Fix: Implement Docker security best practices

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.