If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/huggingface/text-generation-inference shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across the board

• Last commit 7w ago
• 22+ active contributors
• Distributed ownership (top contributor 27% of recent commits)
• Apache-2.0 licensed
• CI configured
• Tests present

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live huggingface/text-generation-inference repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/huggingface/text-generation-inference.

What it runs against: a local clone of huggingface/text-generation-inference — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in huggingface/text-generation-inference | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | Last commit ≤ 77 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of huggingface/text-generation-inference. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/huggingface/text-generation-inference.git
#   cd text-generation-inference
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of huggingface/text-generation-inference and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "huggingface/text-generation-inference(\\.git)?\\b" \\
  && ok "origin remote is huggingface/text-generation-inference" \\
  || miss "origin remote is not huggingface/text-generation-inference (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 77 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~47d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/huggingface/text-generation-inference"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Text Generation Inference (TGI) is a production-ready Rust/Python/gRPC server for deploying and serving Large Language Models with optimized inference on GPUs. It powers Hugging Face's Hugging Chat and Inference API with features like continuous batching, tensor parallelism, Flash Attention, token streaming via SSE, and OpenAI-compatible Messages API—enabling high-throughput, low-latency text generation for models like Llama, Falcon, and BLOOM. Monorepo (Cargo.toml workspace) with five core components: backends/v2 and backends/v3 (inference engines), launcher/ (model bootstrapping), router/ (request routing and batching), and backends/grpc-metadata/ (gRPC protocol definitions). Backends are gRPC-based microservices, router is the HTTP ingress point, and launcher orchestrates the full stack. Python wrapper in the repo root binds to Rust via PyO3.

ML engineers and platform teams deploying LLMs to production who need a battle-tested, high-performance inference server with distributed tracing, Prometheus metrics, multi-GPU support, and zero-boilerplate model serving. Also relevant for researchers benchmarking inference performance across model architectures.

Production-mature but now in maintenance mode (as of the README caution notice). The project has significant GitHub stars, comprehensive CI/CD workflows (.github/workflows/ contains 15+ automation pipelines), extensive test coverage (integration_tests.yaml, client-tests.yaml, nix_tests.yaml), and was actively maintained until transition to maintenance. Verdict: production-ready and battle-tested, but no longer receiving major feature development.

Maintenance-mode status means no new features or architecture changes will be accepted—only bug fixes and lightweight maintenance. The README explicitly recommends users migrate to vLLM, SGLang, or llama.cpp for new projects. Dependency risk is moderate: heavy Rust ecosystem usage (hf_hub, tokenizers, pyo3, tokio) and Python bindings add complexity. Breaking changes unlikely given maintenance stance, but stack is tied to specific transformer model architectures.

The repository is in maintenance mode per the README caution—active work has shifted to downstream projects (vLLM, SGLang). Commits are likely bug fixes and dependency updates. The 15 active GitHub workflows suggest CI/CD remains healthy, but feature work is paused. New model additions will be rejected per ISSUE_TEMPLATE/new-model-addition.yml.

git clone https://github.com/huggingface/text-generation-inference.git
cd text-generation-inference
cd launcher
cargo build --release
# Or use Docker: docker build -t tgi:latest .

For Python development: install Rust nightly, then pip install -e . in the repo root (PyO3 bindings will compile Rust).

Daily commands:

# Option 1: Docker (recommended for production)
docker run -it --gpus all ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-2-7b --port 8080

# Option 2: Local Rust build (development)
cargo build --release -p launcher
./target/release/text-generation-launcher --model-id meta-llama/Llama-2-7b --port 8080

# Option 3: Python wrapper
python -m text_generation.launcher --model-id meta-llama/Llama-2-7b

Server starts on port 8080; test with curl http://localhost:8080/docs (Swagger UI).

• launcher/src/main.rs: Entry point that orchestrates the full stack—spawns backends, router, and handles SIGTERM gracefully.
• router/src/main.rs: HTTP/SSE server implementation; defines /generate, /generate_stream, /v1/chat/completions endpoints.
• backends/v3/src/lib.rs: Latest inference engine (V3); implements continuous batching, prefill/decode phases, and gRPC server.
• backends/v2/src/lib.rs: Previous-generation V2 engine; maintained for compatibility but V3 is preferred.
• backends/client/src/v3/client.rs: gRPC client stub; router uses this to communicate with backends.
• Cargo.toml: Workspace root; defines all workspace members and shared dependency versions.
• .github/workflows/integration_tests.yaml: End-to-end test pipeline; shows how to test model inference, batching, and streaming.
• Dockerfile: Production-grade multi-stage build; includes CUDA toolchain, quantization, and model caching setup.

Adding backend optimizations: modify backends/v2/src/ or backends/v3/src/ (Rust CUDA kernels). Changing HTTP API: edit router/src/ (Axum handlers). New model support: fork and run locally; new-model PRs are rejected per maintenance policy. Fixing inference bugs: backends/{v2,v3}/src/models/*/mod.rs contain model-specific code. Modifying tokenizer integration: backends/client/src/ handles tokenization. CI/test changes: .github/workflows/tests.yaml and .github/workflows/integration_tests.yaml. Dockerfile updates: Dockerfile (main), Dockerfile_amd, Dockerfile_neuron for hardware variants.

1. gRPC port collision: backends spawn on hardcoded ports (usually 50051+); if running multiple instances locally, they will fail silently. 2. Shared memory (shm) requirement: batching and prefill/decode caching require /dev/shm; Docker deployments need --shm-size=1g or higher. 3. Model quantization flags (GPTQ, AWQ) must match the model's published format—mismatches cause silent inference failure. 4. PyO3 rebuild requirement: modifying Rust code invalidates the Python wheel; pip install -e . with --no-cache-dir is needed. 5. Maintenance mode blocks PRs: new model additions or feature PRs will be rejected per .github/ISSUE_TEMPLATE/new-model-addition.yml. 6. Flash Attention GPU requirement: some kernels only work on A100/H100; older GPUs fall back to slower paths without error messages.

• Continuous Batching — TGI's core throughput optimization—allows new requests to join a batch mid-generation rather than waiting; critical for understanding router/src/ batching logic.
• Tensor Parallelism — Enables serving models larger than single-GPU memory; backends/v3/src/ implements TP for distributed prefill/decode across GPUs.
• Flash Attention — GPU kernel optimization used in TGI's attention implementation; dramatically reduces memory bandwidth and latency for inference.
• Paged Attention (KV Cache Management) — Virtual memory-like KV cache management in backends/v3; reduces memory fragmentation and enables dynamic batching.
• Server-Sent Events (SSE) — Protocol used by /generate_stream endpoint in router/src/; allows clients to receive tokens as they're generated without polling.
• gRPC with Protocol Buffers — Inter-service communication protocol between router and backends (backends/grpc-metadata/); enables typed, efficient serialization of requests/responses.
• Prefill vs Decode Phases — Two-phase inference in backends/v3: prefill processes all input tokens in parallel, decode generates one token at a time; understanding this separation explains batching strategy.

• vllm-project/vllm — Direct successor/alternative in the same problem space; TGI README explicitly recommends it for new deployments with PagedAttention-based optimization.
• sgl-project/sglang — Sibling inference engine recommended in TGI README; focuses on structured generation and serving multiple backends (vLLM, llama.cpp).
• ggerganov/llama.cpp — Lightweight CPU/GPU inference alternative recommended by TGI README; used as a backend option via Dockerfile_llamacpp.
• huggingface/transformers — Core model architecture library; TGI inference kernels are optimized transformers code for models defined here.
• huggingface/tokenizers — Tokenizer library (workspace dependency in Cargo.toml); used by launcher and backends for prompt tokenization.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add integration tests for backends/client sharded client communication

The repo has v2 and v3 sharded client implementations (backends/client/src/v2/sharded_client.rs and backends/client/src/v3/sharded_client.rs) but the .github/workflows/integration_tests.yaml workflow doesn't appear to have specific test coverage for multi-shard scenarios. Given that sharding is critical for LLM inference at scale, adding integration tests would catch regressions early. This is especially important since the project is in maintenance mode and needs robust testing for existing features.

• [ ] Review existing integration_tests.yaml workflow to understand test structure
• [ ] Create integration test scenarios in backends/client/tests/ for sharded_client.rs v2 and v3
• [ ] Add tests covering: shard initialization, request routing across shards, error handling when shards fail
• [ ] Extend integration_tests.yaml to run these new backend client tests
• [ ] Document test setup in CONTRIBUTING.md

Add missing Dockerfile CI workflow for variant validation

The repo contains 7 different Dockerfiles (Dockerfile, Dockerfile.neuron, Dockerfile.nix, Dockerfile_amd, Dockerfile_gaudi, Dockerfile_intel, Dockerfile_llamacpp, Dockerfile_trtllm) but there's no dedicated CI workflow to validate they build successfully on PRs. The build.yaml and ci_build.yaml workflows don't explicitly mention these variants. A PR adding a matrix-based workflow would prevent merge of changes that break specific hardware backend builds.

• [ ] Review existing build.yaml and ci_build.yaml to understand current build matrix
• [ ] Create new .github/workflows/dockerfile_matrix_build.yaml with matrix strategy for each Dockerfile variant
• [ ] Configure appropriate runners/tags for hardware-specific builds (amd, gaudi, intel, neuron if available)
• [ ] Add fallback validation for variants without dedicated runners
• [ ] Update README.md to document which variants are validated in CI

Add comprehensive benchmarking regression tests for router performance

The repo has a benchmark/ member in the workspace and load_test.yaml workflow, but there's no automated performance regression detection. Given that TGI is a performance-critical inference engine, a PR adding baseline performance thresholds and regression detection would help maintainers catch performance regressions before merge. This aligns with the caution note about maintenance mode - preventing performance degradation is critical.

• [ ] Review benchmark/ crate Cargo.toml and existing benchmark implementations
• [ ] Extend benchmark/ to generate baseline metrics for router latency, throughput, and memory
• [ ] Create .github/workflows/performance_regression_check.yaml that runs benchmarks and compares against baseline
• [ ] Store baseline metrics in repository (JSON or similar) with versioning strategy
• [ ] Add step to comment on PRs with performance impact analysis (% change vs baseline)
• [ ] Document performance testing guidelines in CONTRIBUTING.md

• Add missing integration tests for the /v1/chat/completions (OpenAI-compatible) endpoint in .github/workflows/integration_tests.yaml—currently tested via client-tests.yaml but lacks end-to-end router tests.
• Document the gRPC protocol schema and V2 vs V3 differences in a DEVELOPMENT.md file, since backends/grpc-metadata/ exists but no developer guide explains when to use each backend or how to extend them.
• Add Prometheus metric emission for router request latency percentiles (p50, p99) and batch size distribution in router/src/main.rs—currently only basic counters exist per metrics-exporter-prometheus usage.

The codebase presents moderate security risks primarily stemming from its maintenance mode status which limits patch velocity, combined with supply chain risks in the Docker build process (unverified protobuf download) and dependency management. While the project uses reasonable practices like Cargo.lock and pinned dependency versions, the explicit statement that security is not a priority going forward, combined with incomplete infrastructure hardening in Dockerfiles, results in a concerning security posture. Immediate actions should focus on establishing security patch procedures for maintenance mode and hardening the build pipeline with hash verification and minimal container images.

• High · Hardcoded Protobuf Version in Dockerfile — Dockerfile (builder stage). The Dockerfile downloads and installs protoc version 21.12 from GitHub without version pinning via hash verification. This could be vulnerable to man-in-the-middle attacks or supply chain compromises if the download is intercepted. Fix: Add SHA256 hash verification for the downloaded protoc binary. Use: echo 'EXPECTED_HASH protoc-21.12-linux-x86_64.zip' | sha256sum -c - after download and before extraction.
• High · Maintenance Mode Security Risk — README.md, Project Status. Project is in maintenance mode per README, accepting only minor bug fixes and documentation improvements. This significantly limits security patch velocity and increases risk of unpatched vulnerabilities accumulating over time. Fix: Establish a clear security patch policy for maintenance mode. Define critical vs non-critical security issues and SLAs for patches. Consider automated dependency scanning and regular audits.
• Medium · Unverified External Dependencies — Cargo.toml (workspace dependencies). Cargo.toml uses multiple external crates (base64, tokenizers, hf-hub, pyo3, etc.) without lock file hash verification visible in the static structure. The 'tokenizers' dependency has 'http' feature enabled which could introduce network-based vulnerabilities. Fix: Ensure Cargo.lock is committed and verified in CI/CD. Use cargo audit regularly to scan dependencies. Review the necessity of 'http' feature in tokenizers; consider disabling if not needed.
• Medium · Missing Security Headers in Build Process — Dockerfile (builder stage - apt-get installation). Dockerfile uses apt-get install with limited security controls. No signature verification for packages and no explicit security-focused update mechanism defined. Fix: Add --no-install-recommends (already present, good), but add apt-get upgrade before install. Implement package signature verification. Use minimal base image and consider distroless alternatives.
• Medium · Python 3.11-dev Installed in Docker — Dockerfile (builder stage - line: python3.11-dev). Installing python3.11-dev in production/build images increases attack surface. Development tools should not be in production images if possible. Fix: Use multi-stage builds to ensure dev dependencies don't leak into final image. Create separate builder and runtime stages with minimal dependencies in runtime image.
• Low · Incomplete Dockerfile — Dockerfile (end of provided snippet). The provided Dockerfile snippet is truncated at 'COPY Carg' which appears incomplete. This could hide security issues in the final stages. Fix: Complete the Dockerfile review. Ensure final image doesn't run as root, has minimal attack surface, and implements proper health checks.
• Low · Missing SBOM and Provenance — Build process / CI/CD workflows. No visible Software Bill of Materials (SBOM) generation in build pipeline. This impacts supply chain transparency and vulnerability tracking. Fix: Implement SBOM generation using tools like syft or cargo-sbom. Add to release artifacts. Consider SLSA provenance attestation for reproducible builds.
• Low · Maintenance Mode Code Review Risk — Project governance. With maintenance mode accepting only minor changes, comprehensive security code reviews may be deprioritized. Pull request review capacity may be limited. Fix: Establish explicit security review requirements even in maintenance mode. Use automated security scanning tools (SAST, DAST) to supplement manual reviews. Consider community security champions.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.