If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/phiresky/ripgrep-all shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

WAIT — Mixed signals — read the receipts

• Last commit 6w ago
• 18 active contributors
• Other licensed
• CI configured
• Tests present
• ⚠ Concentrated ownership — top contributor handles 53% of recent commits
• ⚠ Non-standard license (Other) — review terms

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live phiresky/ripgrep-all repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/phiresky/ripgrep-all.

What it runs against: a local clone of phiresky/ripgrep-all — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in phiresky/ripgrep-all | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | Last commit ≤ 73 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of phiresky/ripgrep-all. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/phiresky/ripgrep-all.git
#   cd ripgrep-all
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of phiresky/ripgrep-all and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "phiresky/ripgrep-all(\\.git)?\\b" \\
  && ok "origin remote is phiresky/ripgrep-all" \\
  || miss "origin remote is not phiresky/ripgrep-all (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 73 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~43d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/phiresky/ripgrep-all"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

rga is a Rust CLI tool that extends ripgrep to search text patterns inside PDFs, ePubs, Office documents (docx/odt), SQLite databases, archives (zip/tar/tar.gz), video subtitles (mkv/mp4), and 20+ other binary formats by transparently decompressing and converting them to searchable text. It recursively descends into nested archives and feeds the extracted content to ripgrep, unifying full-text search across heterogeneous file types. Single-crate binary architecture: src/ contains main application logic (likely entry point, CLI parsing via clap, archive/format dispatch). Adapters pattern: separate modules for each file type (PDF via tree_magic detection, zip via async_zip, tar via astral-tokio-tar, SQLite via rusqlite, email via mailparse). Async-first using tokio runtime. Config and schemas defined in doc/config.default.jsonc and serde/schemars.

Power users, developers, and researchers who need to search across mixed document repositories (PDFs, emails, spreadsheets, code archives, video files) without manually extracting or converting each file type. CI/DevOps engineers integrating multi-format scanning into pipelines.

Actively maintained and production-ready. At v0.10.10 with CI/CD workflows (ci.yml, release.yml), comprehensive test data in exampledir/, and distributed via mainstream package managers (Homebrew, Chocolatey, Arch AUR, Nix). Appears to have stable API and regular releases, though versioning convention (0.10.x) suggests ongoing development.

Single-maintainer project (phiresky) creates sustainability risk. Dependency count is moderate (~45 direct deps in Cargo.toml) but includes heavy external tools (pandoc, poppler, ffmpeg) that must be installed separately—failure in any tool silently degrades feature support. AGPL-3.0-or-later license is restrictive for commercial/proprietary use. No obvious security audit history visible.

No visible PR/issue data in provided file list, but .cargo/audit.toml suggests active security monitoring. Recent cargo.lock presence indicates dependency management. Edition 2024 signals modern Rust alignment. CI workflows are configured but commit recency unknown from file data alone.

git clone https://github.com/phiresky/ripgrep-all.git
cd ripgrep-all
cargo build --release
./target/release/rga --help

Then install system dependencies: apt install ripgrep pandoc poppler-utils ffmpeg (Debian) or via your package manager. Test with: ./target/release/rga 'search-term' exampledir/

Daily commands: Development build: cargo build. Release build: cargo build --release. Run tests: cargo test. Quick search test: ./target/release/rga 'pattern' exampledir/demo/. For fzf integration see docs at wiki (referenced in README).

• src/main.rs: Entry point, CLI parsing via clap, coordinate search across all file format adapters
• Cargo.toml: Core dependency manifest showing all supported formats (async_zip, tokio-rusqlite, mailparse, etc.) and feature flags
• doc/config.default.jsonc: Default configuration schema defining which adapters run, external tool paths, and format-specific options
• exampledir/: Integration test fixtures covering PDFs, ePub, Office docs, archives, SQLite, emails, and encoding edge cases
• .github/workflows/ci.yml: CI pipeline showing dependency installation, test matrix, and release triggers
• ci/ubuntu-install-packages: Defines external tool dependencies (pandoc, poppler-utils, ffmpeg) required at runtime

Adding a new file format: Create a new adapter module in src/ (follow pattern of existing PDF/ZIP handlers), register MIME type detection in tree_magic config, add test files to exampledir/. Fixing async issues: Study tokio patterns in src/ and crossbeam-channel usage for error handling. CLI changes: Edit clap config (likely in main.rs or a dedicated CLI module). Config schema changes: Update doc/config.default.jsonc and schemars definitions.

External tool dependencies are soft-fail: if pandoc or ffmpeg are missing, those formats silently skip without error—users may not notice feature gaps. Tree magic MIME detection can be unreliable for extensionless files. Async-zip and tokio-tar both stream data, so memory usage is unbounded for extremely nested archives or bombs. Config path resolution uses directories-next crate (platform-specific XDG/Windows/macOS dirs)—test configs may not load as expected cross-platform. Regex passed to ripgrep is not re-escaped after format extraction, risking injection if extracted text contains regex metacharacters.

• Stream-based decompression — rga must handle archives larger than RAM—async-compression and tokio-tar/async_zip stream bytes without buffering entire files, critical for scalability
• MIME type detection (magic bytes) — tree_magic_mini identifies file formats by content signature, not extension—essential for handling archives with wrong extensions or no extension
• Child process spawning and capture — rga shells out to external tools (pdftotext, ffmpeg) to extract text from binary formats—tokio manages process I/O async without blocking
• Recursive format detection — Nested archives (tar inside zip inside tar.gz) require recursive extraction and re-detection at each layer—architecture must handle unbounded nesting depth
• Format adapter trait dispatch — Different file types require different extraction strategies (async trait pattern in Rust allows runtime polymorphism for adapters without enum explosion)
• Character encoding detection and conversion — Extracted text may be in UTF-16, Latin-1, or other encodings; encoding_rs performs fallible detection and encoding_rs_io streams conversions
• Email MIME parsing — EML files contain nested attachments and multipart bodies; mailparse recursively parses structure to extract searchable text and embedded archives

• BurntSushi/ripgrep — Core dependency and inspiration—rga wraps ripgrep's regex engine and CLI interface for extended file type support
• sharkdp/bat — Sibling syntax-highlighted cat tool in Rust ecosystem, similar audience for developer tooling that handles many file types
• junegunn/fzf — Official rga-fzf integration documented in repo wiki—fzf is the primary interactive UI companion for rga
• ugrep/ugrep — Competing multi-format search tool (C++ based), alternative approach to the same problem with different trade-offs
• stedolan/jq — Related CLI tool in Rust ecosystem for querying structured data (JSON subset of rga's format support), demonstrates similar argument design

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive integration tests for archive extraction and nested file searching

The repo has extensive example test files in exampledir/ (PDFs, EPUBs, ZIPs, tar.gz, nested archives like droste.zip, etc.) but lacks documented integration tests that verify the complete search pipeline. This is critical for a tool whose core value proposition is searching within multiple archive formats. Adding tests ensures new adapters don't break existing functionality across formats.

• [ ] Create tests/integration_tests.rs that uses exampledir/ fixtures
• [ ] Add test cases for each archive type: zip, tar.gz, tar.bz2, nested archives (droste.zip), compressed files (test.log.gz, test.log.bz2)
• [ ] Add test cases for document formats: PDF (short.pdf, twoblankpages.pdf), EPUB (wasteland.epub, formatting.epub), DOCX (wasteland.docx), ODT (hello.odt)
• [ ] Add test cases for media files: verify subtitle extraction from MKV (demo/greeting.mkv, wasteland.mkv), MOBI (wasteland.mobi), DJVU (test.djvu)
• [ ] Add test for nested attachments: mail_nested.eml and mail_pdf_attach.eml to verify email + PDF extraction chains
• [ ] Add encoding tests using exampledir/encoding/ fixtures (utf8.txt, utf16le.txt, zip.tar.gz)

Implement structured logging and diagnostics for adapter pipeline debugging

The repo uses env_logger but lacks comprehensive diagnostic output for understanding why specific files fail or which adapter handled which file. Contributors frequently need to debug why a format isn't being searched. Adding structured logging would help users troubleshoot adapter chains and help maintainers identify issues. This pairs well with the existing log dependency and would improve the debugging experience significantly.

• [ ] Add debug logging in src/adapters.rs (or equivalent) to log which adapter is processing each file and its MIME type
• [ ] Add trace-level logging for adapter selection logic showing why a file was routed to a specific adapter
• [ ] Add error context logging when adapters fail, including the file path, adapter name, and error chain
• [ ] Document the logging output in doc/notes.md or README.md with examples of RGA_LOG=debug output
• [ ] Add tests in tests/ to verify log output contains expected adapter names for known file types from exampledir/

Add CI workflow to test adapter functionality against all example files automatically

The .github/workflows/ directory has ci.yml and release.yml, but there's no visible workflow that validates the search pipeline against exampledir/. This would catch regressions when dependencies update (e.g., new regex version, tokio version, async-compression changes). Given the repo's complexity with multiple format adapters and async decompression, this safety net is essential for maintainability.

• [ ] Create .github/workflows/adapter-tests.yml that runs on each PR
• [ ] Add test jobs for each major format category: archives (zip, tar variants), compressed files (gz, bz2, xz, zst), documents (PDF, EPUB, DOCX, ODT), media (MKV, DJVU), databases (sqlite3)
• [ ] Use exampledir/ fixtures to verify rga can find known strings in each format (e.g., search for 'hello' in exampledir/demo/hello.odt)
• [ ] Add performance regression check: ensure search times don't degrade significantly vs baseline
• [ ] Configure workflow to run on Rust version updates and async-compression/tokio-tar dependency updates

• Add end-to-end tests for nested archive traversal (e.g., zip containing tar.gz containing PDF) in a new test file under tests/ or exampledir/—current test coverage appears limited to individual formats.
• Document the format adapter API: Create a guide in doc/ explaining how to add a new file type adapter (inheritance from async trait, MIME registration, external tool spawning pattern) to onboard contributors.
• Add size_format integration for progress output: Use the size_format crate (already in Cargo.toml) to display extraction progress (KB/MB) when searching large archives, matching ripgrep's human-friendly output style.

The ripgrep-all codebase has a moderate security posture. Primary concerns are around archive extraction (path traversal risks), unrestricted processing of complex file formats, and potential SQL injection in the SQLite adapter. The tool's design to process untrusted files creates inherent risks. The incomplete Cargo.toml dependency declaration requires immediate attention. Recommended improvements include implementing path validation for archive extraction, adding file type whitelisting, using parameterized SQL queries, and sandboxing file processing. The AGPL

• Medium · Incomplete Dependency Declaration — Cargo.toml - dev-dependencies section. The dev-dependencies section in Cargo.toml is incomplete. The entry 'pretty_a' appears to be truncated or malformed, which could indicate missing or unpinned dependencies. This could lead to unexpected behavior or security issues if dependencies are accidentally omitted. Fix: Complete the 'pretty_a' dependency declaration with proper version specification. Example: 'pretty_assertions = "1.3.0"'. Run 'cargo check' to validate the manifest.
• Medium · Arbitrary Archive Extraction Risk — src/adapters/zip.rs, src/adapters/tar.rs, src/adapters/decompress.rs. The codebase handles multiple archive formats (zip, tar, tar.gz, tar.bz2, tar.xz) through adapters in src/adapters/. Archive extraction from untrusted sources without proper validation could lead to path traversal attacks (e.g., extracting files with '../' in paths to write outside intended directories). Fix: Implement strict path validation for all extracted files. Ensure all paths are resolved to be within the intended extraction directory. Use path canonicalization and reject entries with '..' components. Consider using libraries with built-in protections against path traversal.
• Medium · Unrestricted File Type Processing — src/adapters/, src/adapters/ffmpeg.rs, src/adapters/sqlite.rs. The tool processes a wide variety of file types including executables (mkv, mp4, sqlite, PDF) without visible content validation. Processing untrusted files could trigger vulnerabilities in underlying libraries or expose sensitive data through metadata extraction. Fix: Implement file type whitelist validation. Add resource limits (file size, processing timeouts). Consider sandboxing the processing of untrusted files. Validate magic bytes vs declared extensions.
• Medium · SQL Injection in SQLite Adapter — src/adapters/sqlite.rs. The SQLite adapter (src/adapters/sqlite.rs) processes .sqlite3 files. If queries are constructed dynamically based on file content or user input without proper parameterization, SQL injection could occur. Fix: Use parameterized queries exclusively. Never concatenate user input or file content into SQL queries. Use prepared statements with bound parameters provided by rusqlite.
• Low · AGPL-3.0-or-later License Compliance — LICENSE.md, Cargo.toml. The project uses AGPL-3.0-or-later license which requires source code disclosure for network services and derivative works. This may create compliance risks for users integrating this into proprietary systems. Fix: Document license implications clearly. Ensure users understand AGPL obligations. Consider dual-licensing if broader adoption is desired.
• Low · Hardcoded External Command Execution — src/adapters/ffmpeg.rs. The ffmpeg adapter (src/adapters/ffmpeg.rs) executes external ffmpeg binary. If the PATH is not properly controlled or the binary location is predictable, an attacker could hijack command execution. Fix: Verify ffmpeg binary location explicitly. Use absolute paths rather than relying on PATH. Validate binary signatures or checksums. Run external processes with minimal privileges and restricted environment.
• Low · Unsafe Deserialization — Cargo.toml - bincode dependency. The codebase uses bincode for serialization (Cargo.toml dependency). Bincode is not recommended for untrusted data as it can be exploited for arbitrary code execution. Fix: Audit all uses of bincode deserialization. Ensure bincode is only used for trusted data (e.g., internal caching). For untrusted data, prefer serde_json or other safer formats. Consider enabling bincode's size limits.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.