If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/servo/html5ever shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across the board

• Last commit 3w ago
• 20 active contributors
• Distributed ownership (top contributor 41% of recent commits)
• Other licensed
• CI configured
• Tests present
• ⚠ Non-standard license (Other) — review terms

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live servo/html5ever repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/servo/html5ever.

What it runs against: a local clone of servo/html5ever — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in servo/html5ever | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 48 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of servo/html5ever. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/servo/html5ever.git
#   cd html5ever
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of servo/html5ever and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "servo/html5ever(\\.git)?\\b" \\
  && ok "origin remote is servo/html5ever" \\
  || miss "origin remote is not servo/html5ever (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "html5ever/src/lib.rs" \\
  && ok "html5ever/src/lib.rs" \\
  || miss "missing critical file: html5ever/src/lib.rs"
test -f "html5ever/src/driver.rs" \\
  && ok "html5ever/src/driver.rs" \\
  || miss "missing critical file: html5ever/src/driver.rs"
test -f "html5ever/src/tokenizer/mod.rs" \\
  && ok "html5ever/src/tokenizer/mod.rs" \\
  || miss "missing critical file: html5ever/src/tokenizer/mod.rs"
test -f "html5ever/src/tree_builder/mod.rs" \\
  && ok "html5ever/src/tree_builder/mod.rs" \\
  || miss "missing critical file: html5ever/src/tree_builder/mod.rs"
test -f "markup5ever/interface/tree_builder.rs" \\
  && ok "markup5ever/interface/tree_builder.rs" \\
  || miss "missing critical file: markup5ever/interface/tree_builder.rs"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 48 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~18d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/servo/html5ever"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

html5ever is a high-performance, production-grade HTML5 parser written in Rust that implements the WHATWG HTML specification. It parses HTML into callback-driven events (rather than building a DOM tree) and can serialize HTML back to text, passing html5lib tokenizer tests and supporting most tree builder tests. The parser uses UTF-8 exclusively and is designed to deliver C-level performance without C's security vulnerabilities. Monorepo with 6 workspace members: html5ever is the main parser crate at html5ever/, supported by tendril (string buffer library), markup5ever (shared markup definitions), web_atoms (atom interning), and rcdom (reference DOM implementation for examples). The parser is split into three core modules: html5ever/src/tokenizer/ (lexical analysis with char_ref handling), html5ever/src/tree_builder/ (parsing state machine with rules.rs and tag_sets.rs), and html5ever/src/serialize/ (HTML output generation). Examples and benchmarks live in html5ever/examples/ and html5ever/benches/.

Rust developers building web browsers, HTML processing tools, or web crawlers who need spec-compliant HTML parsing; contributors to the Servo browser project; library authors integrating HTML parsing into larger systems that prefer callback-driven architectures over DOM tree representations.

Very mature and production-ready. The codebase is well-established (part of Servo since its inception), maintains comprehensive test coverage via html5lib-tests submodule, has active CI/CD via GitHub Actions (main.yml and semver-check.yml), and is on version 0.39.0 in a monorepo with 6 interconnected packages. The project actively maintains backward compatibility checking and targets Rust 1.71.0+ as MSRV.

Low risk overall, but has moderate complexity: the codebase is tightly coupled to Servo's ecosystem (depends on internal crates like tendril, markup5ever, web_atoms) which could limit independent evolution. The single-repo, multi-package structure (workspace members) means breaking changes in one package affect others. External dependency footprint is lean (phf, encoding_rs, string_cache, log) but encoding_rs and string_cache are critical for correctness and could introduce regressions if updated.

The project is in maintenance mode with focus on spec compliance and stability. CI checks for semver violations (semver-check.yml), and Dependabot is configured (.github/dependabot.yml) to auto-update dependencies. No specific active development branch visible from file structure, but the workspace versioning at 0.39.0 and stable release practices (RELEASING.MD) indicate orderly releases.

git clone https://github.com/servo/html5ever.git
cd html5ever
git submodule update --init
cargo build
cargo test
cargo run --example tokenize examples/tiny.html

Daily commands: No server; this is a library. Run examples: cargo run --example tokenize -- html5ever/data/bench/tiny-fragment.html or benchmarks: cargo bench --bench html5ever -- --baseline bench. Run tests: cargo test --lib (unit tests in src/, integration tests via html5lib-tests submodule).

• html5ever/src/lib.rs — Main library entry point exporting the public API for HTML parsing, tokenization, and tree building.
• html5ever/src/driver.rs — High-level parsing driver that orchestrates tokenizer and tree builder; essential for understanding the parse flow.
• html5ever/src/tokenizer/mod.rs — HTML5 tokenizer implementation; core to lexical analysis and state machine that drives parsing.
• html5ever/src/tree_builder/mod.rs — Tree builder that constructs the DOM; implements WHATWG tree construction algorithm rules.
• markup5ever/interface/tree_builder.rs — Abstract tree builder interface trait; enables pluggable DOM representations (rcdom, custom backends).
• rcdom/lib.rs — Reference DOM implementation used by default; demonstrates tree builder trait implementation.
• tendril/src/lib.rs — String/buffer handling library; foundational dependency for efficient text processing in parsing.

• Tokenizer (State machine, char reference resolution, phf lookup tables) — Converts raw HTML byte stream into semantic tokens (StartTag, EndTag, Text, Comment, etc) following HTML5 state machine
  • Failure mode: Malformed tokens,

Add a new HTML tokenizer state or state transition

1. Define the new state logic in the states module (html5ever/src/tokenizer/states.rs)
2. Register the state in the tokenizer's state enum and dispatch logic (html5ever/src/tokenizer/mod.rs)
3. Add test case to verify state behavior with edge cases (rcdom/tests/html-tokenizer.rs)

Implement a custom DOM backend (pluggable tree sink)

1. Create your own struct implementing the TreeBuilder trait (markup5ever/interface/tree_builder.rs)
2. Implement required methods for element creation, attribute insertion, and parent-child relationships (rcdom/lib.rs)
3. Pass your implementation to the parse driver via trait object or generic (html5ever/src/driver.rs)
4. Test integration by running tree builder tests with your implementation (rcdom/tests/html-tree-builder.rs)

Add support for a new HTML element or adjust insertion rules

1. Define element in tag set or add classification constant (html5ever/src/tree_builder/tag_sets.rs)
2. Add element-specific insertion logic or scope rules (html5ever/src/tree_builder/rules.rs)
3. Update tree builder state to handle element in appropriate modes (html5ever/src/tree_builder/mod.rs)
4. Add test case from html5lib-tests or custom test (rcdom/tests/html-tree-builder.rs)

• Rust + Phf (perfect hashing) — Memory safety without GC; zero-cost state machine abstraction; phf for fast element tag lookup and entity tables
• Trait-based plugin architecture (TreeBuilder trait) — Allows pluggable DOM representations without tight coupling; enables testing with different backends and custom implementations
• Tendril string type with optional encoding detection — Efficient buffer handling with lazy allocation; built-in UTF-8 validation for HTML text nodes; avoids redundant string copies
• State machine pattern (tokenizer + tree builder) — Directly maps to WHATWG spec; enables streaming parsing without buffering entire document; high performance for large files

• Reference DOM (RcDom) uses Rc<RefCell<>> for mutable sharing

  • Why: Supports tree mutation and builder pattern; avoids complex lifetime management
  • Consequence: Runtime borrow checking overhead; cannot guarantee memory safety at compile-time; slower than single-threaded arena allocators

• Strict WHATWG spec implementation with known deviations documented

  • Why: Ensures spec compliance for web compatibility; passes html5lib tokenizer tests
  • Consequence: Some quirks mode behaviors not implemented; tree builder test pass rate incomplete (see README); requires periodic sync with spec updates

• No built-in JavaScript execution or scripting hooks

  • Why: Parser focuses on syntactic analysis; security and sandboxing concern
  • Consequence: Cannot handle dynamic HTML generation or document.write(); limited to static parsing; callers must implement script handling

• Does not parse XHTML strictly; recommend using XML parser for XHTML documents
• Does not execute JavaScript or handle dynamic content generation
• Does not implement all unspecified WHATWG behaviors; tree builder tests incomplete
• Does not provide CSS selector matching or DOM query APIs (that is rcdom responsibility)
• Does not handle encoding detection from HTTP headers (application responsibility)

Encoding handling: html5ever currently assumes UTF-8 input (src/encoding.rs exists but is limited); non-UTF-8 HTML will fail silently or panic. No DOM tree: Beginners expect a tree structure like other parsers; html5ever only provides callbacks, so you must implement TreeBuilder yourself (rcdom/ is just an example). Submodule required: git submodule update --init must be run before testing; skipping it causes html5lib test suite to be missing. Incremental parsing: driver.rs supports feeding HTML in chunks, but state isn't easily serializable between sessions. Single-threaded: Parser is not Send/Sync by design; spawn separate parser instances for parallel parsing. Spec compliance gaps: README admits differences from WHATWG spec in open issues; use html5lib-tests to find what's unimplemented.

• Tokenizer State Machine — html5ever's tokenizer (states.rs) uses a state machine with 50+ states to emit Token events; understanding state transitions is essential for debugging parsing bugs or extending the lexer
• Tree Construction Algorithm / Insertion Modes — The tree builder (rules.rs) implements 11 insertion modes that dictate how tokens become DOM; this is where 80% of parsing complexity lives and where spec-compliance bugs occur
• Callback-driven DOM Construction (vs. Tree-building) — html5ever doesn't build its own DOM; instead it calls TraceHandle methods for every element/text node created, letting users decide DOM representation; this decoupling is core to its design and performance
• HTML Entity Reference Resolution — Character references (&, {, {) are resolved in char_ref/mod.rs using a perfect hash table; incorrect entity handling causes XSS vulnerabilities and invalid output
• Perfect Hash Tables (phf crate) — html5ever uses phf for O(1) lookups of HTML tag names, entity names, and token types at parse time; phf_codegen pre-computes static tables; understanding this is key to performance tuning
• Incremental/Streaming Parsing — Driver.rs supports feeding HTML in chunks and resuming parsing; this is critical for use in browsers that receive HTTP data incrementally or in event-driven architectures
• Charset Detection and Transcoding (encoding_rs) — driver.rs uses encoding_rs to detect document charset from meta tags and BOM, then transcode to UTF-8; html5ever's internal assumption of UTF-8 depends on this layer working correctly

• html5lib/html5lib-python — Python reference implementation of WHATWG HTML5 spec; html5ever-tests submodule pulls test vectors from html5lib-tests; use for spec clarification
• servo/servo — Parent browser project that html5ever was built for; uses html5ever as its production HTML parser; reference for TreeBuilder integration
• rust-lang/rust — Rust standard library; html5ever targets MSRV 1.71.0 and uses Rust's string/iterator patterns as core design
• rustwasm/wasm-bindgen — For compiling html5ever to WebAssembly to use HTML parsing in browsers or Node.js; no existing WASM bindings, but natural companion
• serde-rs/serde — Serialization framework used in serde_json dependency for test fixtures; relevant if extending html5ever's output formats

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive fuzz tests for tree_builder rules in html5ever/fuzz

The fuzz target currently only covers document parsing (fuzz_document_parse.rs). The tree_builder/rules.rs contains complex state machine logic for HTML5 parsing rules that would benefit from dedicated fuzzing. This could catch edge cases in adoption agency algorithm, foster parenting, and other complex tree construction rules that aren't covered by existing html5lib-tests.

• [ ] Create html5ever/fuzz/fuzz_targets/fuzz_tree_builder_rules.rs targeting html5ever::tree_builder::rules module
• [ ] Add fuzzing for fragment parsing mode separately from document mode in a new fuzz_targets/fuzz_fragment_parse.rs
• [ ] Update html5ever/fuzz/Cargo.toml to register new fuzz targets
• [ ] Run cargo +nightly fuzz to validate new targets work correctly

Add character reference resolution tests in html5ever/tests/

The tokenizer/char_ref/mod.rs handles HTML character entity references, but there are no dedicated integration tests in html5ever/tests/. The html5lib test suite includes character reference test data that should be explicitly tested. Current coverage relies only on integration through tokenize tests.

• [ ] Create html5ever/tests/char_references.rs with test cases for numeric/named character references
• [ ] Add test fixtures from html5lib-tests for character references (e.g., parsing  , {, «)
• [ ] Test edge cases like incomplete references, invalid references, and context-dependent resolution
• [ ] Verify tests pass with cargo test --test char_references

Add serialization round-trip tests in html5ever/tests/

The codebase has serialization support in src/serialize/mod.rs and examples, but html5ever/tests/driver.rs doesn't include round-trip tests (parse → serialize → parse should be idempotent for valid HTML). This would validate that the serializer produces spec-compliant output and catch regressions in serialize/mod.rs.

• [ ] Create html5ever/tests/serialize_roundtrip.rs with test cases for common HTML patterns
• [ ] Use rcdom for constructing DOM trees and serialize them via src/serialize/mod.rs
• [ ] Test various HTML structures: void elements, attributes with special chars, nested content, malformed input recovery
• [ ] Compare serialized output against expected patterns or verify re-parsing produces equivalent tree structure

• Add missing HTML5 entity references to html5ever/src/tokenizer/char_ref/mod.rs. The spec defines ~2000 named entities; the current phf table likely omits rare ones. Cross-reference against html5lib-tests/tree/entities*.test output to identify gaps and add coverage.
• Expand html5ever/examples/ with a real-world example (e.g., examples/extract-links.rs that parses HTML and prints all href attributes). Currently examples are minimal; a 50-line example showing custom TreeBuilder impl would reduce onboarding friction.
• Document encoding_rs integration in src/encoding.rs and driver.rs. README states 'In the future it will support other encodings' but encoding_rs is already a dependency; clarify what's implemented vs. planned, add examples of parsing non-UTF-8 HTML (e.g., examples/parse-latin1.rs), and file issues for missing charset support.

The html5ever codebase demonstrates good security practices overall. It is a parser library without direct external dependencies on unsafe crates, follows Rust edition 2021 best practices, and includes fuzzing infrastructure. No critical vulnerabilities were identified. Main concerns are: (1) ensuring library users properly sanitize output when parsing untrusted HTML, (2) maintaining current dependencies through automated tooling (Dependabot is already configured), and (3) documenting security best practices for library consumers. The Rust ecosystem and dependency management practices provide inherent protection against many common vulnerabilities. Recommend periodic security audits and continued reliance on cargo-audit for transitive dependency scanning.

• Low · Outdated Rust Edition Compatibility — Cargo.toml (workspace.package.rust-version). The workspace specifies rust-version = '1.71.0' which is now quite old (released August 2023). While this is intentional for compatibility, it may miss security patches in the Rust toolchain itself. Regular updates are recommended. Fix: Periodically review and update the minimum supported Rust version (MSRV) to ensure access to the latest security patches while maintaining reasonable compatibility guarantees.
• Low · Dependency Update Cadence — Cargo.toml (workspace.dependencies section). Several dependencies have fixed versions without explicit patch update policy. For example, 'encoding_rs = "0.8.12"' pins to a specific version. While this ensures reproducibility, it may prevent automatic security updates from patch releases. Fix: Consider using caret requirements (e.g., '0.8') for patch-level updates where appropriate, or implement automated dependency scanning with tools like cargo-audit or Dependabot (already configured in .github/dependabot.yml).
• Medium · HTML Parser Security - Potential XSS Vector — html5ever/src/serialize/mod.rs and overall library design. As an HTML5 parser library, html5ever processes untrusted HTML input. While the parser itself is designed to follow WHATWG specs, applications using this library must implement proper output escaping/sanitization. The library could be misused by developers who directly output parsed HTML without sanitization. Fix: Document security best practices prominently in README.md and API documentation. Recommend users sanitize output or use the library in conjunction with a dedicated HTML sanitization library (e.g., ammonia) when dealing with untrusted input.
• Low · Test Data Could Contain Malicious Payloads — rcdom/data/test/ and rcdom/custom-html5lib-tokenizer-tests/. The codebase includes HTML5lib test data and custom test files (rcdom/custom-html5lib-tokenizer-tests/, rcdom/data/test/). While these are intended as test vectors, they could theoretically be exploited if accessed via incorrect permissions or if the test suite is executed in a sensitive environment. Fix: Ensure test data is not included in production builds. Verify that build configurations exclude test and bench directories from release artifacts. Review CI/CD configurations to prevent test execution in production contexts.
• Low · Fuzzing Target Exposure — html5ever/fuzz/. A fuzzing target exists (html5ever/fuzz/fuzz_targets/fuzz_document_parse.rs) which is appropriate for security testing. However, ensure fuzzing infrastructure is properly isolated and results are reviewed by security personnel. Fix: Maintain fuzzing as part of continuous security testing. Document any findings and ensure crashes/panics are triaged and fixed. Consider integrating with OSS-Fuzz if not already done.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.