If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/sparklemotion/mechanize shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across all four use cases

• Last commit 3mo ago
• 12 active contributors
• MIT licensed
• CI configured
• Tests present
• ⚠ Single-maintainer risk — top contributor 80% of recent commits

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live sparklemotion/mechanize repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/sparklemotion/mechanize.

What it runs against: a local clone of sparklemotion/mechanize — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in sparklemotion/mechanize | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 108 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of sparklemotion/mechanize. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/sparklemotion/mechanize.git
#   cd mechanize
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of sparklemotion/mechanize and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "sparklemotion/mechanize(\\.git)?\\b" \\
  && ok "origin remote is sparklemotion/mechanize" \\
  || miss "origin remote is not sparklemotion/mechanize (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "lib/mechanize.rb" \\
  && ok "lib/mechanize.rb" \\
  || miss "missing critical file: lib/mechanize.rb"
test -f "lib/mechanize/http/agent.rb" \\
  && ok "lib/mechanize/http/agent.rb" \\
  || miss "missing critical file: lib/mechanize/http/agent.rb"
test -f "lib/mechanize/page.rb" \\
  && ok "lib/mechanize/page.rb" \\
  || miss "missing critical file: lib/mechanize/page.rb"
test -f "lib/mechanize/form.rb" \\
  && ok "lib/mechanize/form.rb" \\
  || miss "missing critical file: lib/mechanize/form.rb"
test -f "lib/mechanize/cookie_jar.rb" \\
  && ok "lib/mechanize/cookie_jar.rb" \\
  || miss "missing critical file: lib/mechanize/cookie_jar.rb"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 108 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~78d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/sparklemotion/mechanize"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Mechanize is a Ruby library that automates web interaction by wrapping HTTP requests with stateful session management—it automatically handles cookies, follows redirects, parses HTML forms with Nokogiri, and manages browser history. It lets developers script website interactions (login, form submission, link following, file upload) as if driving a browser programmatically. Monolithic structure: lib/mechanize.rb is the entry point, lib/mechanize/ contains core classes (form parsing, cookies, HTTP handling), lib/mechanize/form/ isolates form field types (text, checkbox, submit, file_upload), lib/mechanize/http/ wraps request/response logic. Examples/ folder contains runnable scripts (flickr_upload.rb, spider.rb, wikipedia_links.rb) demonstrating real-world patterns.

Ruby developers and automation engineers who need to scrape websites, automate data entry, test web applications, or integrate with legacy web services without maintaining a full Selenium/Capybara stack. Typical users: web scrapers, integration testers, API proxy builders.

Production-ready and actively maintained. The project has CI/CD via GitHub Actions (ci.yml, upstream.yml), comprehensive test coverage, semantic versioning in CHANGELOG.md, and dependencies pinned in Gemfile. Last activity visible in GitHub workflows suggests ongoing maintenance, though commit frequency data is not provided.

Moderate risk: depends on 10 external gems (nokogiri, net-http-persistent, http-cookie, etc.), creating a wide attack surface if any become unmaintained. No single-maintainer risk evident (multiple authors listed). Main risk is Nokogiri version constraints and net-http-persistent compatibility with newer Ruby versions (requires Ruby ≥2.6). No visible breaking-change warnings in recent CHANGELOG.

Dependabot is actively monitoring dependencies (.github/dependabot.yml), CI workflows test against multiple Ruby versions (ci.yml), and upstream.yml likely tracks compatibility with newer gem releases. Specific pending changes not visible from file list, but the .autotest suggests active TDD workflow.

git clone https://github.com/sparklemotion/mechanize.git
cd mechanize
bundle install
bundle exec rake test

Daily commands:

bundle exec rake test          # Run full test suite
bundle exec ruby examples/spider.rb       # Run example scraper
bundle exec ruby examples/mech-dump.rb    # Inspect page structure

• lib/mechanize.rb — Main entry point and class definition; all mechanize functionality flows through this file as the primary public API.
• lib/mechanize/http/agent.rb — Core HTTP request/response handling; all network communication and protocol logic depends on this agent layer.
• lib/mechanize/page.rb — Represents parsed HTML responses; critical abstraction for page navigation, form extraction, and link following.
• lib/mechanize/form.rb — Form parsing and submission logic; essential for automated form interaction, a primary use case.
• lib/mechanize/cookie_jar.rb — Cookie storage and management; core for maintaining session state across requests.
• lib/mechanize/history.rb — Navigation history tracking; maintains state for back/forward and visit tracking functionality.
• lib/mechanize/parser.rb — Pluggable parser system; defines how responses are parsed into page objects based on content type.

Add Support for a New HTML Element Type

1. Create a new element class in lib/mechanize/page/ (e.g., lib/mechanize/page/custom_element.rb) extending Mechanize::Page::Base (lib/mechanize/page/custom_element.rb)
2. Implement element-specific methods (e.g., click, value accessor, attributes) (lib/mechanize/page/custom_element.rb)
3. Register the element type in lib/mechanize/page.rb by adding it to the parser via css/xpath selectors in the page initialization (lib/mechanize/page.rb)
4. Add integration test using lib/mechanize/test_case/servlets.rb or create a new test servlet returning HTML with the new element (lib/mechanize/test_case/servlets.rb)

Add a New Form Field Type

1. Create new field class in lib/mechanize/form/ extending lib/mechanize/form/field.rb (lib/mechanize/form/custom_field.rb)
2. Implement field-specific behavior (value getter/setter, serialization for submission) (lib/mechanize/form/custom_field.rb)
3. Register the field in lib/mechanize/form.rb's field detection logic (typically in a parse or build method based on HTML input type) (lib/mechanize/form.rb)
4. Write test in test suite verifying field extraction and form submission with the new field type (test/mechanize/test_form.rb)

Extend HTTP Authentication (Add New Auth Scheme)

1. Create new auth challenge handler in lib/mechanize/http/ extending auth logic (similar to net-http-digest_auth integration) (lib/mechanize/http/custom_auth.rb)
2. Register handler in lib/mechanize/http/agent.rb in the request/response cycle where auth challenges are detected (lib/mechanize/http/agent.rb)
3. Create a corresponding test servlet in lib/mechanize/test_case/ that returns appropriate auth challenge headers (lib/mechanize/test_case/custom_auth_servlet.rb)
4. Add integration test verifying auth flow: challenge → credential submission → authenticated request (test/mechanize/test_http_auth.rb)

Add Support for a New Content Type/Parser

1. Create parser class inheriting from lib/mechanize/parser.rb or implementing required interface (lib/mechanize/custom_parser.rb)
2. Register parser in lib/mechanize/pluggable_parsers.rb via add method with content type(s) it handles (lib/mechanize/pluggable_parsers.rb)
3. Implement parse method returning a page object (or subclass) with custom accessors for parsed data (lib/mechanize/custom_parser.rb)
4. Create test servlet in lib/mechanize/test_case/ serving the new content type and verify parsing produces expected object structure (lib/mechanize/test_case/custom_content_servlet.rb)

• Nokogiri — Industry-standard Ruby HTML/XML parser; provides robust DOM navigation via CSS selectors and XPath, essential for reliable page element extraction.
• net-http-persistent — HTTP connection pooling and reuse; critical for efficient multi-request automation, reduces latency and resource overhead.
• http-cookie — Standards-compliant cookie jar implementation; handles cookie domain/path matching and expiration per RFC 6265.
• WEBrick — Pure-Ruby HTTP server for testing; enables integration tests without external test server dependencies.
• net-http-digest_auth & rubyntlm — Authentication scheme libraries; Mechanize delegates auth handling to battle-tested gems for Digest and NTLM support.

• Automatic redirect following enabled by default
  • Why: Reduces boilerplate for common use case
  • Consequence: undefined

Form submission: File uploads require lib/mechanize/form/file_upload.rb and multipart encoding—easy to miss in nested forms. Cookie jar: domain matching is strict (domain_name gem rules); cookies from 'example.com' won't apply to 'subdomain.example.com' without explicit Domain attribute. Redirect loops: Mechanize follows redirects by default (configurable via agent.redirect_ok = false) but can hang on malformed Location headers. Nokogiri version: HTML parsing is fragile if Nokogiri is pinned to old libxml2. WebRobots: robots.txt compliance is optional but enabled by default—disable via agent.robots = false or face 403-like behavior on crawlers.

• Cookie Domain Matching (RFC 6265) — Mechanize's cookie_jar.rb must correctly match cookies to domains; misunderstanding suffix rules breaks session authentication across subdomains
• HTTP Redirect Following — Mechanize auto-follows 3xx redirects; understanding Location header resolution and redirect chains prevents infinite loops and unexpected behavior in scrapers
• HTML Form Serialization (application/x-www-form-urlencoded & multipart/form-data) — Form fields must be correctly encoded for submission; Mechanize's form.rb must handle both encodings, and file_upload.rb requires multipart awareness
• CSS/XPath Selectors for DOM Traversal — element_matcher.rb uses XPath to find form fields and links; understanding CSS-to-XPath conversion prevents selector failures on real websites
• HTTP Persistent Connections (Keep-Alive) — Mechanize wraps net-http-persistent to reuse TCP connections across multiple requests; understanding this improves scraper performance and reduces server load
• User-Agent Spoofing & robots.txt Compliance — Mechanize allows configurable User-Agent headers and respects robots.txt by default (via webrobots gem); ethical scraping requires understanding these levers
• Content Negotiation & Content-Type Handling — Mechanize must parse responses based on Content-Type header (HTML vs JSON vs binary); content_type_error.rb prevents mishandling of non-HTML responses

• jnunemaker/httparty — Simpler HTTP client without form/cookie/history features—for cases where Mechanize is overkill
• tomas/http_client — Lower-level HTTP abstraction; Mechanize wraps similar logic but adds stateful session handling on top
• teamcapybara/capybara — Browser automation framework using Selenium/Puppeteer; heavier than Mechanize but required for JavaScript-heavy sites
• ankane/ruby-rets — Real estate data scraper built on Mechanize patterns, shows production use case for form automation
• ruby/net-http-persistent — Upstream dependency providing connection pooling; Mechanize depends on this for efficient multi-request sessions

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive test coverage for lib/mechanize/form/* field types

The form directory contains many specialized field types (CheckBox, RadioButton, SelectList, FileUpload, etc.) but there's no visible test directory structure in the file listing. These field classes handle user input parsing and serialization - critical for form automation. Adding unit tests for edge cases (disabled fields, empty values, multi-select, file upload validation) would improve reliability and prevent regressions.

• [ ] Create test/mechanize/form/ directory structure mirroring lib/mechanize/form/
• [ ] Add test_check_box.rb covering checked/unchecked states and serialization
• [ ] Add test_select_list.rb covering single/multi-select, option ordering, and disabled options
• [ ] Add test_file_upload.rb covering file path validation and MIME type handling
• [ ] Add test_radio_button.rb covering mutual exclusivity within groups

Add CI workflow for testing against multiple Nokogiri versions

Mechanize depends on Nokogiri for HTML parsing, which is a critical dependency. The current .github/workflows/ci.yml likely tests against a single Nokogiri version. Since Nokogiri has major version breaks and different capabilities, a matrix workflow testing against Nokogiri 1.13.x, 1.14.x, and 1.15.x would catch compatibility issues early and help users know which Mechanize versions work with their Nokogiri version.

• [ ] Update .github/workflows/ci.yml to add a matrix strategy with nokogiri versions
• [ ] Configure Gemfile or test environment to allow pinning Nokogiri versions
• [ ] Test that lib/mechanize/page.rb and lib/mechanize/parser.rb work correctly with each version
• [ ] Document in README.md which Mechanize versions support which Nokogiri versions

Add integration tests for HTTP authentication in lib/mechanize/http/auth_*

The http/auth_* files (auth_challenge.rb, auth_realm.rb, auth_store.rb) handle digest and NTLM authentication - complex stateful interactions that benefit from integration tests. Currently, these are likely only tested indirectly. Adding tests that mock HTTP 401 responses and verify credential handling would ensure authentication workflows remain robust across Ruby versions.

• [ ] Create test/mechanize/http/test_auth_integration.rb for end-to-end auth flows
• [ ] Add test cases for Digest-Auth challenge-response cycles using lib/mechanize/http/www_authenticate_parser.rb
• [ ] Add test cases for NTLM authentication state transitions (via rubyntlm dependency)
• [ ] Add tests for auth_store.rb tracking multiple realms and credentials
• [ ] Test that stale=true in Digest-Auth challenges trigger re-authentication per RFC 7616

• Add support for HTML5 <input type='email'> and <input type='url'> validation in lib/mechanize/form/text.rb—currently only generic text field exists; new contributors should check RFC 5321/3986 compliance.
• Extend lib/mechanize/element_matcher.rb to support CSS4 pseudo-class selectors (:nth-child, :has) by leveraging Nokogiri's newer XPath support—currently limited to basic XPath expressions.
• Write integration tests for lib/mechanize/cookie_jar.rb against real RFC 6265 edge cases (SameSite attribute, Secure flag with HTTPS downgrade, third-party cookie blocking)—test coverage likely gaps here.

• Medium · Potential XSS via HTML/XML Parsing — lib/mechanize/page.rb, lib/mechanize/parser.rb, lib/mechanize/pluggable_parsers.rb. Mechanize uses Nokogiri for HTML/XML parsing in web scraping. While Nokogiri itself is secure, the library's purpose of parsing untrusted web content could lead to XSS vulnerabilities if parsed content is used without proper sanitization. The Page class and related parsers extract links, forms, and other elements from remote sites. Fix: Ensure all user-supplied data derived from parsed HTML is properly escaped before use in any output context. Implement output encoding based on context (HTML, JavaScript, URL, CSS). Consider using a dedicated sanitization library if displaying parsed content.
• Medium · HTTP Authentication Credentials in Memory — lib/mechanize/http/auth_store.rb, lib/mechanize/http/agent.rb. The library stores HTTP authentication credentials (Basic, Digest, NTLM) in memory via http/auth_store.rb. These credentials could potentially be exposed if memory is dumped or if there's a vulnerability allowing memory inspection. Fix: Implement secure credential storage with encryption at rest. Consider using OS-level credential storage (Keychain, Windows Credential Manager). Clear credentials from memory after use when possible. Add warnings in documentation about handling sensitive credentials.
• Medium · Insecure Cookie Storage — lib/mechanize/cookie_jar.rb, lib/mechanize/cookie.rb. The cookie_jar.rb stores cookies in memory without encryption. Session cookies and authentication tokens are stored as plaintext in memory, posing a risk if the process memory is compromised. Fix: Document the security implications of cookie storage. Consider adding optional encryption for stored cookies. Implement secure defaults for sensitive cookies. Provide options to restrict cookie persistence to non-sensitive cookies only.
• Medium · Insufficient Input Validation on File Operations — lib/mechanize/file_saver.rb, lib/mechanize/directory_saver.rb, lib/mechanize/http/content_disposition_parser.rb. The file_saver.rb and directory_saver.rb handle file downloads and savings. Potential path traversal vulnerabilities could exist if filenames from HTTP Content-Disposition headers are not properly validated. Fix: Implement strict filename validation to prevent path traversal attacks. Sanitize filenames from Content-Disposition headers. Use allowlist-based validation for safe characters. Consider using UUID-based filenames. Validate that resolved paths remain within intended directories.
• Low · Dependency on External HTTP Libraries — lib/mechanize.rb, lib/mechanize/http/agent.rb. Mechanize depends on net-http-persistent for connection pooling. Security patches in underlying HTTP libraries must be monitored. The library wraps Ruby's Net::HTTP which has had security issues historically. Fix: Keep dependencies updated regularly, especially net-http-persistent and nokogiri. Monitor security advisories for Ruby's Net::HTTP. Consider using bundler-audit or similar tools in CI/CD pipeline to detect vulnerable dependencies automatically.
• Low · No Rate Limiting or Request Throttling — lib/mechanize/http/agent.rb. The library doesn't implement built-in rate limiting or request throttling. Users could inadvertently cause DoS attacks or violate terms of service of target websites. Fix: Document best practices for responsible web scraping. Consider adding optional built-in rate limiting features. Implement delay mechanisms between requests. Add warnings about robots.txt compliance and legal considerations.
• Low · Robots.txt Bypass Possible — lib/mechanize/robots_disallowed_error.rb. While the library includes robots.txt support via webrobots gem, users can disable it entirely. The enforcement is opt-in rather than default-secure. Fix: Make robots.txt compliance a default-enabled feature. Require explicit opt-out rather than opt-in. Add logging/warnings when robots.txt is disabled. Document the legal and ethical implications.
• Low · Missing HTTPS Enforcement — lib/mechanize/http/agent.rb. The library does not enforce HTTPS by default. Users could be vulnerable to man-in-the-middle attacks if scripts accidentally or intentionally use HTTP for sensitive operations. Fix: undefined

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.