If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/opendataloader-project/opendataloader-pdf shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across all four use cases

• Last commit 1d ago
• 8 active contributors
• Apache-2.0 licensed
• CI configured
• Tests present
• ⚠ Single-maintainer risk — top contributor 80% of recent commits

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live opendataloader-project/opendataloader-pdf repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/opendataloader-project/opendataloader-pdf.

What it runs against: a local clone of opendataloader-project/opendataloader-pdf — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in opendataloader-project/opendataloader-pdf | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 31 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of opendataloader-project/opendataloader-pdf. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/opendataloader-project/opendataloader-pdf.git
#   cd opendataloader-pdf
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of opendataloader-project/opendataloader-pdf and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "opendataloader-project/opendataloader-pdf(\\.git)?\\b" \\
  && ok "origin remote is opendataloader-project/opendataloader-pdf" \\
  || miss "origin remote is not opendataloader-project/opendataloader-pdf (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api" \\
  && ok "java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api" \\
  || miss "missing critical file: java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api"
test -f "java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIMain.java" \\
  && ok "java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIMain.java" \\
  || miss "missing critical file: java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIMain.java"
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "CONTRIBUTING.md" \\
  && ok "CONTRIBUTING.md" \\
  || miss "missing critical file: CONTRIBUTING.md"
test -f "java/opendataloader-pdf-core/pom.xml" \\
  && ok "java/opendataloader-pdf-core/pom.xml" \\
  || miss "missing critical file: java/opendataloader-pdf-core/pom.xml"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 31 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~1d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/opendataloader-project/opendataloader-pdf"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

opendataloader-pdf is an open-source PDF parser that extracts structured data (Markdown, JSON with bounding boxes, HTML, Tagged PDF) from digital, scanned, and tagged PDFs with benchmark-leading accuracy (0.907 overall, 0.928 table extraction). It combines deterministic local-mode extraction with hybrid AI mode for complex pages, includes built-in OCR for 80+ languages, and auto-tags PDFs to Tagged PDF/PDF/UA accessibility standards. Multi-language monorepo: Java core (pdfcore, extraction engine, accessibility tagging) exports via Maven Central; Python wrapper (opendataloader-pdf package on PyPI) provides Pythonic API; Node.js bindings (@opendataloader/pdf on npm); build-scripts/ contains version/dependency tooling (set_version.py, fetch_shaded_jar.py); docs/hybrid/ contains active experiments on chunking strategies and speed optimization; THIRD_PARTY/ manages license compliance for bundled dependencies.

Data engineers and ML practitioners building RAG/LLM pipelines who need deterministic, high-accuracy PDF-to-structured-data extraction; accessibility teams automating PDF remediation and Tagged PDF generation; enterprises requiring PDF/UA compliance validation via PDF Association collaboration.

Production-ready and actively developed. The codebase shows enterprise maturity with 1.5M+ lines of Java, comprehensive GitHub workflows (release.yml, test-benchmark.yml), third-party license management, and documented accessibility compliance (Well-Tagged PDF spec, veraPDF collaboration). Multi-platform SDKs (Python, Node.js, Java 11+) with PyPI and npm releases indicate stable, distributed usage.

Moderate risk factors: Java-heavy stack (1.5M LOC) with complex PDF processing logic (layout analysis, OCR integration, XY-Cut++ reading order) creates potential maintenance burden; hybrid AI mode dependency on external services (benchmarks show experimental speed optimizations in docs/hybrid/experiments/) requires infrastructure assumptions; single-digit contributor visibility suggests core team concentration. No visible breaking-change policy in CHANGELOG.md snippet. Last release cadence unclear from file list alone.

Active experimentation on hybrid mode performance optimization visible in docs/hybrid/experiments/ (chunking_strategy/, speed/, triage/ with 2026-01-03 benchmark reports); likely working toward production-grade hybrid AI inference pipeline. Release workflow (release.yml) and benchmark testing (test-benchmark.yml) indicate regular deployment cadence. SUPPORT.md and SECURITY.md suggest mature operational practice.

Clone the repo: git clone https://github.com/opendataloader-project/opendataloader-pdf.git. For Python: pip install opendataloader-pdf>=2.2.1 (requires Python 3.10+). For development, examine build-scripts/set_version.py and Java requirements (Java 11+). Check .github/workflows/ for CI/CD patterns and CONTRIBUTING.md for development setup details.

Daily commands: For Python development: python -m pip install -e . after cloning (infer from pyproject.toml structure). For Java: standard Maven build via pom.xml (likely mvn clean install). For hybrid mode experiments: python docs/hybrid/experiments/speed/docling_page_range_benchmark.py to reproduce benchmark comparisons. CI workflows (test-benchmark.yml) define the authoritative test/build pipeline.

• java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api — Core API module—entry point for all PDF parsing and layout analysis; every contributor must understand the API contracts and extension points.
• java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIMain.java — CLI entry point and primary user-facing interface; demonstrates how the core library is invoked and configured.
• README.md — Project overview, capabilities, and philosophy; establishes context for PDF extraction, accessibility, and AI-ready output formats.
• CONTRIBUTING.md — Contribution guidelines and development workflow; essential for understanding testing, PR process, and code standards.
• java/opendataloader-pdf-core/pom.xml — Maven build configuration and dependency tree; defines external libraries (Apache PDFBox, JJ2000, etc.) that shape architecture.
• docs/hybrid/hybrid-mode-design.md — Design rationale for hybrid AI mode (local + cloud); explains key differentiator and performance tradeoffs.
• .github/workflows/test-benchmark.yml — Benchmark and test automation; reveals quality gates, performance expectations, and CI/CD requirements.

Add a new PDF output format (e.g., Custom XML)

1. Define new output format interface in the core API module (e.g., CustomXmlExporter extends OutputFormatter) (java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api)
2. Implement serialization logic to convert extracted elements (with bounding boxes and reading order) to Custom XML (java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api)
3. Register the new formatter in the core library's OutputFactory or equivalent registration mechanism (java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api)
4. Add CLI flag to CLIMain.java to invoke the new output format (java/opendataloader-pdf-cli/src/main/java/org/opendataloader/pdf/cli/CLIMain.java)
5. Add integration test to CLIMainTest.java verifying custom XML generation (java/opendataloader-pdf-cli/src/test/java/org/opendataloader/pdf/cli/CLIMainTest.java)

Add a new hybrid AI provider (e.g., alternative cloud service)

1. Review hybrid mode design and existing cloud API contracts in docs/hybrid/research/docling-openapi.json (docs/hybrid/research/docling-openapi.json)
2. Create new AIProvider adapter module in core library implementing cloud extraction interface (java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api)
3. Implement request/response mapping to internal IObject structure per iobject-structure.md (docs/hybrid/research/iobject-structure.md)
4. Add configuration option and fallback logic to hybrid mode orchestrator (java/opendataloader-pdf-core/pom.xml)
5. Benchmark the new provider against baseline using speed experiment framework (docs/hybrid/experiments/speed/docling_page_range_benchmark.py)

Enhance PDF/UA compliance for a specific language or font family

1. Review CID font detection design and current Tagged PDF generation in superpowers specs (docs/superpowers/specs/2026-03-16-cid-font-detection-design.md)
2. Extend font detection or character mapping logic in core PDF processing engine (java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api)
3. Update Tagged PDF tagging strategy to handle language-specific or font-specific edge cases (java/opendataloader-pdf-core/src/main/java/org/opendataloader/pdf/api)
4. Add veraPDF validation rules and test cases for the enhanced compliance scenario (java/opendataloader-pdf-cli/src/test/java/org/opendataloader/pdf/cli/CLIMainTest.java)

Add a new Python SDK example for RAG/LLM pipeline

1. Create new example file in examples/python/rag/ (e.g., llamaindex_example.py or custom_llm_example.py) (examples/python/rag)
2. Use opendataloader-pdf to extract PDF and generate Markdown or JSON with bounding boxes (examples/python/rag/langchain_example.py)
3. Demonstrate chunking strategy using docs/hybrid/experiments/chunking_strategy patterns (docs/hybrid/experiments/chunking_strategy)
4. Add requirements.txt with LLM SDK dependencies (examples/python/rag/requirements.txt)
5. Document the example with README explaining use case and expected output (examples/python/rag/README.md)

• Java 11+ with Maven — Cross-platform PDF processing with mature library ecosystem (Apache PDFBox); Maven enables reproducible builds and dependency management for complex JAR shading.
• Apache PDFBox — Industry-standard open-source PDF library; provides low-level PDF parsing, CID font handling, and Tagged PDF annotation support.
• Python SDK (3.10+) — AI/ML workflows and RAG pipelines require Python integration; enables LangChain,

Java 11+ is a hard requirement (not 8 or 10 as older projects might have). Hybrid mode experiments suggest AI inference endpoint dependency (verify FastAPI/subprocess service availability in docs/hybrid/experiments/). OCR and AI features may require additional runtime dependencies or environment variables (check build-scripts/fetch_shaded_jar.py for bundled jar setup). Scanned PDF support requires 300 DPI+ quality — lower resolution will silently degrade. Tagged PDF export is enterprise-tier (not in open-source core), so feature-gating may apply. Third-party license compliance is strictly tracked (THIRD_PARTY/) — modifying dependencies requires audit.

• XY-Cut++ Reading Order — Core algorithm for determining text flow in complex multi-column PDFs; opendataloader-pdf claims it as a key differentiator, so understanding how it outperforms naive Y-coordinate sorting is essential for modifying extraction behavior.
• Tagged PDF and PDF/UA Accessibility — opendataloader-pdf's unique selling point is auto-tagging PDFs to accessibility standards; understanding tag trees, role mappings, and PDF/UA compliance is critical for contributing to the accessibility module.
• Optical Character Recognition (OCR) for Scanned PDFs — Hybrid mode integrates OCR for 80+ languages to handle scanned documents; understanding text detection, language auto-detection, and quality thresholding (300 DPI constraint) is needed for improving scanned PDF support.
• Bounding Boxes and Geometric Layout Analysis — JSON output includes bounding box coordinates for every extracted element; understanding PDF's coordinate system (origin at bottom-left), transformation matrices, and geometric intersection for reading order are fundamental to the extraction engine.
• Hybrid AI Mode and Deterministic vs. Stochastic Extraction — Codebase supports both local deterministic mode and AI-augmented hybrid mode; understanding when to invoke LLM inference vs. rule-based extraction is key to the architecture's flexibility and the experiments in docs/hybrid/.
• Content Stream Parsing and PDF Graphics State — PDF text extraction requires parsing content streams, respecting graphics state (font, matrix transforms, clipping paths); familiarity with these internals is needed to debug edge cases in formula/chart extraction.
• RAG (Retrieval-Augmented Generation) Pipeline Integration — Project explicitly targets RAG/LLM pipelines as end-users; understanding chunking strategies (documented in docs/hybrid/experiments/chunking_strategy/), semantic embedding, and context window constraints shapes extraction design decisions.

• Unstructured-IO/unstructured — Competitor in PDF/document extraction for LLM pipelines; similar JSON/Markdown output goal but different architecture (Python-first, not Java-core).
• VikParuchuri/marker — Alternative open-source PDF-to-Markdown converter; lighter-weight than opendataloader-pdf but lower accuracy on complex tables/formulas.
• duallab-com/veraPDF — Direct ecosystem dependency mentioned in README — veraPDF is the PDF/UA validation tool opendataloader-pdf integrates with for accessibility compliance checks.
• PDF-Association/Well-Tagged-PDF — Standards partner for accessibility tagging — opendataloader-pdf's auto-tagging output is validated against this spec; essential context for PDF/UA features.
• apache/pdfbox — Likely foundational library (inferred from PDF processing scope and Java stack) — understanding PDFBox's Graphics State and content stream model is useful for debugging extraction issues.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add benchmark validation tests for extraction accuracy metrics

The README claims #1 extraction accuracy (0.907) and table extraction (0.928), but there's no automated test suite validating these metrics. The docs/hybrid/experiments/ folder contains benchmark JSON files (chunking_strategy/docling_benchmark_report.json, speed/baseline_results.json) but no CI workflow to continuously validate that code changes don't regress these critical claims. A new contributor could create a test-benchmark.yml workflow enhancement that runs extraction accuracy validations against known PDFs with expected outputs.

• [ ] Analyze existing .github/workflows/test-benchmark.yml to understand current setup
• [ ] Create reference PDFs with known extraction outputs in tests/fixtures/benchmark-pdfs/
• [ ] Implement pytest tests in tests/test_extraction_accuracy.py that validate accuracy metrics match or exceed baseline (0.907 overall, 0.928 tables)
• [ ] Add GitHub Action step to test-benchmark.yml that fails if accuracy drops below thresholds
• [ ] Document benchmark test methodology in docs/testing-benchmarks.md

Implement comprehensive test coverage for hybrid mode integration

The repo documents extensive hybrid mode research and design in docs/hybrid/ (hybrid-mode-design.md, experiments with docling integration, sample responses) but the file structure shows no dedicated test suite for hybrid mode functionality. Since hybrid mode is a core differentiator offering #1 accuracy, a contributor could build integration tests that validate the hybrid extraction pipeline, fallback mechanisms, and response format consistency.

• [ ] Review docs/hybrid/research/docling-openapi.json and docs/hybrid/research/opendataloader-sample-response.json to understand expected response structures
• [ ] Create tests/test_hybrid_mode.py with integration tests validating: docling fallback triggering, response schema validation, and chunking strategy selection (reference docs/hybrid/experiments/chunking_strategy/)
• [ ] Add test fixtures in tests/fixtures/hybrid-mode-responses/ based on sample responses documented in docs/hybrid/research/
• [ ] Ensure tests cover all three extraction modes and verify output format consistency mentioned in README (Markdown, JSON with bounding boxes, HTML, Tagged PDF)
• [ ] Add hybrid mode test documentation to CONTRIBUTING.md

Create accessibility validation test suite using veraPDF integration

The README mentions 'PDF Association collaboration, Well-Tagged PDF specification, veraPDF automated validation' as a key differentiator, and docs/superpowers/specs/ contains CID font detection and hancom-ai options designs, but there's no visible test suite validating PDF/UA compliance or Tagged PDF correctness. A contributor could implement automated accessibility validation tests using veraPDF or internal validators to ensure output PDFs meet Well-Tagged PDF specification.

• [ ] Review THIRD_PARTY/THIRD_PARTY_LICENSES.md to confirm veraPDF license compatibility
• [ ] Create tests/test_pdf_ua_compliance.py that validates: PDF tagging correctness, font embedding, logical reading order, and accessibility metadata
• [ ] Implement helper function in tests/accessibility_validators.py to run veraPDF checks on output PDFs and parse results
• [ ] Add test fixtures in tests/fixtures/accessibility-test-pdfs/ with PDFs of varying complexity (simple text, tables, forms, scanned content)
• [ ] Update docs/superpowers/specs/ with test coverage documentation for PDF/UA validation pipeline

• Add unit tests for XY-Cut++ reading order algorithm logic — benchmark reports in docs/hybrid/experiments/ mention this custom algorithm but it likely lacks test coverage; identify the Java source file and write edge-case tests for multi-column, rotated text, and RTL scenarios.
• Create a developer quickstart guide for the Python SDK — README covers high-level use but no step-by-step example of calling extract_markdown() or extract_json() with bounding box output; add a worked example in docs/ with sample PDF and expected output.
• Document the chunking_strategy experiment conclusions — docs/hybrid/experiments/chunking_strategy/conclusion.json exists but no narrative explanation; convert the JSON findings into a markdown doc explaining why chunking strategy X was chosen and trade-offs with alternatives.

The codebase demonstrates a generally secure posture with proper licensing documentation, code of conduct, and security policy framework in place. No critical vulnerabilities were identified. Main concerns are: (1) loose dependency version constraints that could allow installation of incompatible versions, (2) potential exposure of API specifications and sample data in public documentation, and (3) missing explicit Python version enforcement. The project follows good security practices with established CONTRIBUTING guidelines and issue templates. For a PDF processing tool handling potentially sensitive documents, ensure proper input validation and sanitization of PDF data is implemented in the core parsing logic (not visible in provided file structure).

• Medium · Dependency Version Constraint Not Pinned — requirements.txt or pyproject.toml (dependencies specification). The dependency 'opendataloader-pdf>=2.2.1' uses a minimum version constraint without an upper bound. This allows installation of future versions that may contain breaking changes or security vulnerabilities. Best practice is to use pinned or bounded version constraints (e.g., >=2.2.1,<3.0.0). Fix: Update dependency specification to use bounded version constraints: 'opendataloader-pdf>=2.2.1,<3.0.0' or pin to a specific tested version.
• Low · Potential Sensitive Information in Documentation — docs/hybrid/research/docling-openapi.json, docs/hybrid/research/docling-sample-response.json, docs/hybrid/research/opendataloader-sample-response.json. The presence of experimental results, API specs, and sample responses in docs/hybrid/research/ and docs/superpowers/ directories could potentially expose API structures or sensitive testing data if the repository is public. Fix: Review sample responses and API specifications to ensure no sensitive data, API keys, or authentication tokens are included. Consider using anonymized/sanitized examples.
• Low · Missing Security Policy Details — .github/SECURITY.md. While SECURITY.md exists, the file content was not provided for analysis. Ensure it contains clear vulnerability reporting procedures and supported versions for security updates. Fix: Verify the security policy document includes: (1) how to responsibly report vulnerabilities, (2) supported versions receiving security updates, (3) expected response timeline, (4) security update schedule.
• Low · Python Version Requirement Not Strictly Enforced — Root requirements specification. The comment indicates 'Requires Python 3.10+' but without explicit enforcement in setup.py/pyproject.toml, installation could occur on incompatible Python versions, potentially leading to unexpected behavior. Fix: Add explicit python_requires='>=3.10' constraint in pyproject.toml or setup.py to enforce minimum Python version at installation time.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.