If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/triton-inference-server/server shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across the board

• Last commit today
• 8 active contributors
• Distributed ownership (top contributor 36% of recent commits)
• BSD-3-Clause licensed
• CI configured
• Tests present

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live triton-inference-server/server repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/triton-inference-server/server.

What it runs against: a local clone of triton-inference-server/server — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in triton-inference-server/server | Confirms the artifact applies here, not a fork | | 2 | License is still BSD-3-Clause | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 30 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of triton-inference-server/server. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/triton-inference-server/server.git
#   cd server
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of triton-inference-server/server and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "triton-inference-server/server(\\.git)?\\b" \\
  && ok "origin remote is triton-inference-server/server" \\
  || miss "origin remote is not triton-inference-server/server (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(BSD-3-Clause)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"BSD-3-Clause\"" package.json 2>/dev/null) \\
  && ok "license is BSD-3-Clause" \\
  || miss "license drift — was BSD-3-Clause at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "CMakeLists.txt" \\
  && ok "CMakeLists.txt" \\
  || miss "missing critical file: CMakeLists.txt"
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "build.py" \\
  && ok "build.py" \\
  || miss "missing critical file: build.py"
test -f "compose.py" \\
  && ok "compose.py" \\
  || miss "missing critical file: compose.py"
test -f "CONTRIBUTING.md" \\
  && ok "CONTRIBUTING.md" \\
  || miss "missing critical file: CONTRIBUTING.md"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 30 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~0d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/triton-inference-server/server"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Triton Inference Server is a production-grade inference serving platform that deploys trained AI models from multiple frameworks (TensorRT, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL) across cloud, data center, edge, and embedded devices. It optimizes inference performance for real-time, batched, ensemble, and streaming workloads on NVIDIA GPUs, x86/ARM CPUs, and AWS Inferentia hardware. Monolithic repository with server core logic (likely in src/core based on CMakeLists.txt patterns), multiple language bindings and client libraries, deployment templates in /deploy (AWS Helm charts, GCP, Alibaba Cloud, Fleet Command), Python SDK and examples, and QA/testing infrastructure (Dockerfile.QA). CMake drives the C++ build; Python components use standard setuptools patterns. CI/CD via GitHub Actions (.github/workflows/codeql.yml, pre-commit.yml).

MLOps engineers and data scientists who need to deploy multiple machine learning models in production at scale, serving real-time inference requests with low latency and high throughput across heterogeneous hardware (GPU clusters, edge devices, CPUs). Also used by inference framework maintainers building Triton backends.

Highly mature and production-ready. Current stable release is v2.68.0 corresponding to NGC 26.04 container. The codebase contains 3.7M lines of Python, 1.2M C++, comprehensive CMake build system (91k lines), and is actively developed (main branch tracking next release). Multiple deployment templates (AWS, GCP, Alibaba Cloud) and production-grade Docker containers indicate extensive real-world usage.

Low technical risk for stable features, but moderate complexity risk due to the breadth of supported backends and frameworks. Dependencies on external backends (TensorRT, PyTorch, ONNX runtimes) create version matrix complexity. The main branch is explicitly under-development, so using main directly is higher risk than stable releases. Monolithic C++/Python codebase means changes can have wide blast radius.

Active development toward next release (post-2.68.0) on main branch. Pre-commit hooks and CodeQL security scanning enabled. Recent dependencies pinned in requirements files (FastAPI 0.121.2, scipy 1.16.3 with specific fix for segfault TRI-620, openai 1.107.3). Integration with LLM backends (OpenAI client, partial JSON parser) suggests expansion into generative AI inference.

Clone the repository: git clone https://github.com/triton-inference-server/server.git. Build via python build.py (main build script). Docker option: docker build -f Dockerfile.QA -t triton:qa . for QA environment or docker build -f Dockerfile.sdk -t triton:sdk . for SDK. See CMakeLists.txt for direct CMake build if modifying C++ core.

Daily commands: Execute python build.py for full build with all backends enabled. For local development server: python -m triton_server (once built and installed). Docker: docker run --gpus all triton:latest tritonserver --model-repository=/models (mount your model directory). Configuration via model_repository/ with model.pbtxt files defining model properties, backend type, and input/output specifications.

• CMakeLists.txt — Root build configuration for the entire Triton server; defines compilation, dependencies, and deployment targets across all platforms.
• README.md — Primary documentation covering architecture, build instructions, deployment options, and high-level design—essential onboarding reference.
• build.py — Main build orchestration script that automates compilation, containerization, and artifact generation; critical for understanding development workflow.
• compose.py — Docker Compose configuration generator for local development and testing; key entry point for running Triton locally.
• CONTRIBUTING.md — Contribution guidelines and code standards; mandatory reading for any code submission or pull request.
• .github/workflows — CI/CD pipeline definitions (codeql.yml, pre-commit.yml) that enforce quality gates and automated testing on every commit.
• TRITON_VERSION — Version control file; single source of truth for release versioning across the entire codebase and containers.

Add Support for a New Cloud Provider Deployment

1. Create a new cloud provider directory under deploy/ (e.g., deploy/azure/) (deploy/aws)
2. Copy and customize Chart.yaml from deploy/aws/Chart.yaml as template (deploy/aws/Chart.yaml)
3. Create provider-specific Helm templates in templates/ subdirectory (deployment.yaml, service.yaml, secrets.yaml) (deploy/aws/templates/deployment.yaml)
4. Add values.yaml with provider-specific defaults (instance types, networking, storage class) (deploy/aws/values.yaml)
5. Update README.md with new provider deployment instructions (README.md)

Add a New Benchmark Model Configuration

1. Create a model directory under deploy/gke-marketplace-app/benchmark/model-store/ (e.g., model-store/my_model/) (deploy/gke-marketplace-app/benchmark/model-store/bert_base_tf_cpu)
2. Add config.pbtxt with model configuration (backend, inputs/outputs, instance groups) (deploy/gke-marketplace-app/benchmark/model-store/bert_base_tf_cpu/config.pbtxt)
3. Create or update perf_query.sh with model-specific performance test parameters (deploy/gke-marketplace-app/benchmark/perf-analyzer-script/perf_query.sh)
4. Add example client request JSON in deploy/gke-marketplace-app/client-sample/ (deploy/gke-marketplace-app/client-sample/bert_request.json)

Modify Build Process or Add Build Step

1. Define new compilation targets or dependency handling in CMakeLists.txt (CMakeLists.txt)
2. Add corresponding Python build logic in build.py (e.g., new artifact generation, test suite) (build.py)
3. Update container definitions if new tooling or dependencies are required (Dockerfile.QA)
4. Update TRITON_VERSION if version bump is required (TRITON_VERSION)
5. Document new build flags and options in README.md (README.md)

Enforce New Code Quality Standard

1. Update .clang-format to reflect new C++ style rules (.clang-format)
2. Add new pre-commit hook or tool configuration in .pre-commit-config.yaml (.pre-commit-config.yaml)
3. Create or update GitHub Actions workflow in .github/workflows/ to validate new standard (.github/workflows/pre-commit.yml)
4. Update CONTRIBUTING.md with new code standard documentation (CONTRIBUTING.md)

• C++ Core Engine — High-performance inference server requires low-latency request handling and efficient resource utilization; C++ provides direct hardware access and minimal overhead.
• gRPC & REST APIs — Multi-protocol support enables diverse client ecosystems (Python, Go, Java, etc.) and interoperability with cloud-native platforms; REST for broad compatibility, gRPC for performance-critical paths.
• Kubernetes & Helm — Cloud-native deployment target; Helm charts provide declarative, reproducible multi-cloud deployments across AWS/GCP/on-premises.
• Docker Containers — Ensures reproducible builds, dependency isolation, and seamless portability across development/staging/production environments.
• Python Build Orchestration — Cross-platform build complexity requires abstraction over CMake; Python enables consistent build experience on Linux, macOS, and Windows.
• Pre-commit & CodeQL — Automated code quality enforcement prevents regressions early; CodeQL detects security vulnerabilities in C++ code at commit time.

• C++ core + Python build scripts + optional language bindings

  • Why: Maximum performance in inference path while maintaining developer productivity in build/test infrastructure.
  • Consequence: Developers must be proficient in multiple languages; more complex contribution onboarding but faster production inference.

• Support multiple model backends (TensorFlow, PyTorch, TensorRT, ONNX, etc.)

  • Why: Maximizes market applicability and prevents vendor lock-in; allows users to optimize per workload.
  • Consequence: Significant engineering complexity, testing burden, and maintenance surface area; higher risk of backend-specific bugs.

• Single-machine and distributed inference support via Kubernetes

  • Why: Covers both edge deployment (single container) and enterprise scaling (multi-node Kubernetes).
  • Consequence: Architecture must abstract scheduling/batching logic; not optimized for any single deployment mode.

• Cloud-agnostic deployment (AWS/GCP/on-prem via Helm)

  • Why: Reduces customer lock-in and increases address
  • Consequence: undefined

Model repository structure required: models/ directory with model_name/version_number/model.pbtxt convention. Backend mismatch risk when upgrading Triton if models reference TensorRT/PyTorch/ONNX backends built for different CUDA versions (pin NGC container versions carefully). scipy 1.16.3 specifically pinned due to segfault in TRI-620 — upgrading breaks tests. FastAPI httpx version pinned to 0.27.2 for OpenAI client compatibility (TRI-641) — do not bump without testing. When using main branch, breaking API changes possible (not stable). GPU device visibility requires --gpus flag or proper CUDA_VISIBLE_DEVICES when containerized.

• Model Repository Convention — Triton's model serving is entirely directory-structure driven (models/model_name/version/model.pbtxt); understanding this hierarchy is critical to deployment and model lifecycle management
• Dynamic Batching & Request Scheduling — Triton's core value proposition is intelligent batching of requests to maximize GPU utilization; understanding batching policies, sequence handling, and scheduler behavior is essential for performance tuning
• Ensemble Models — Triton supports multi-model inference pipelines where outputs of one model feed into another; essential for complex inference workflows without application-level orchestration
• gRPC vs HTTP Backend Protocol — Triton supports both gRPC (high performance, protobuf) and HTTP (FastAPI, JSON) frontends; choosing between them impacts latency, throughput, and client compatibility
• Custom Backend Development — Triton's extensibility comes from custom backends (C++ or Python implementations); understanding the backend API allows deployment of proprietary models or custom preprocessing logic
• Metrics & Prometheus Integration — Triton exposes detailed inference metrics (latency, throughput, GPU memory) via Prometheus; critical for observability and production monitoring in distributed deployments
• Multi-GPU and Heterogeneous Device Support — Triton orchestrates inference across multiple GPUs, CPUs, and specialized accelerators (Inferentia); understanding device affinity and model placement is crucial for large-scale deployments

• triton-inference-server/backend — Official repository for Triton backend implementations (TensorRT, PyTorch, ONNX runtimes); understand this to develop or debug specific framework backends
• triton-inference-server/client — Official Python/C++/Java client libraries for connecting to Triton servers; essential for testing and integrating Triton into applications
• NVIDIA/TensorRT — Deep learning inference optimizer and runtime frequently deployed as a Triton backend; understanding TensorRT patterns improves optimization in Triton deployments
• onnx/onnx-runtime — ONNX Runtime backend support in Triton; key for model format compatibility and cross-framework inference
• pytorch/serve — Competing inference serving solution for PyTorch models; useful comparison for feature parity and architectural patterns (TorchServe vs Triton)

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add pre-commit hook validation for Dockerfile and deployment manifests

The repo has .pre-commit-config.yaml and .clang-format but lacks validation for Docker and Kubernetes manifests. With multiple Dockerfiles (Dockerfile.QA, Dockerfile.sdk) and extensive Helm charts in deploy/ (aws, gcp, fleetcommand, alibaba-cloud), adding pre-commit hooks for hadolint (Dockerfile linting) and kubeval/helm lint would catch security issues and configuration errors early. This aligns with the existing pre-commit infrastructure and would prevent deployment issues.

• [ ] Update .pre-commit-config.yaml to add hadolint for Dockerfile validation
• [ ] Add helm lint and kubeval for all deploy//Chart.yaml and deploy//templates/ files
• [ ] Test the pre-commit hooks against existing Dockerfiles and Helm charts
• [ ] Document the new pre-commit hooks in CONTRIBUTING.md with examples

Add GitHub Actions workflow for dependency vulnerability scanning and license compliance

The repo has .github/workflows/codeql.yml and .github/workflows/pre-commit.yml but lacks automated dependency security scanning. With pinned dependencies like fastapi==0.121.2, openai==1.107.3, and scipy==1.16.3 (with known FIXME comment in TRI-641), plus multiple Helm charts with complex dependencies, adding Dependabot and/or Snyk workflow would catch vulnerabilities proactively. This is critical for a cloud inference server that users deploy widely.

• [ ] Create .github/workflows/dependency-scan.yml with Dependabot configuration or Snyk scanning
• [ ] Configure the workflow to scan Python dependencies (requirements files), Docker base images, and Helm chart dependencies
• [ ] Add SBOM (Software Bill of Materials) generation for release artifacts
• [ ] Document security scanning process in SECURITY.md with remediation steps

Add integration tests for Kubernetes deployment configurations across cloud providers

The deploy/ directory contains production Helm charts for aws, gcp, fleetcommand, and alibaba-cloud, but there's no evidence of automated testing for these configurations. New contributors cannot verify that changes don't break deployments. Creating a test suite that validates Helm chart syntax, template rendering, and basic Kubernetes compatibility would prevent deployment regressions and make the repo more contributor-friendly.

• [ ] Create tests/deploy/ directory with helm lint scripts for each deploy/*/Chart.yaml
• [ ] Add template rendering tests (helm template) to verify all Helm charts produce valid Kubernetes manifests
• [ ] Create a GitHub Actions workflow (.github/workflows/deploy-validation.yml) to run these tests on PRs affecting deploy/
• [ ] Document deployment testing requirements in CONTRIBUTING.md with examples for adding new cloud providers

• Add integration tests for the new OpenAI/LLM inference paths referenced in dependencies (openai==1.107.3, partial-json-parser) — likely missing test coverage for generative model responses in src/tests/
• Document the scipy 1.16.3 segfault workaround (TRI-620/TRI-641) in CONTRIBUTING.md with a section on dependency pinning rationale and version matrix compatibility, since these are critical for onboarding
• Create example model.pbtxt templates in examples/ for common backends (ONNX, TensorRT, PyTorch) showing best practices for batching, dynamic shapes, and ensemble configuration — currently no concrete examples visible in file list

Failed to generate security analysis.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.