If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/projectdiscovery/httpx shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across the board

• Last commit 1d ago
• 9 active contributors
• Distributed ownership (top contributor 33% of recent commits)
• MIT licensed
• CI configured
• Tests present

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live projectdiscovery/httpx repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/projectdiscovery/httpx.

What it runs against: a local clone of projectdiscovery/httpx — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in projectdiscovery/httpx | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch dev exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 31 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of projectdiscovery/httpx. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/projectdiscovery/httpx.git
#   cd httpx
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of projectdiscovery/httpx and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "projectdiscovery/httpx(\\.git)?\\b" \\
  && ok "origin remote is projectdiscovery/httpx" \\
  || miss "origin remote is not projectdiscovery/httpx (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify dev >/dev/null 2>&1 \\
  && ok "default branch dev exists" \\
  || miss "default branch dev no longer exists"

# 4. Critical files exist
test -f "cmd/httpx/httpx.go" \\
  && ok "cmd/httpx/httpx.go" \\
  || miss "missing critical file: cmd/httpx/httpx.go"
test -f "common/httpx/httpx.go" \\
  && ok "common/httpx/httpx.go" \\
  || miss "missing critical file: common/httpx/httpx.go"
test -f "common/httpx/option.go" \\
  && ok "common/httpx/option.go" \\
  || miss "missing critical file: common/httpx/option.go"
test -f "common/httpx/response.go" \\
  && ok "common/httpx/response.go" \\
  || miss "missing critical file: common/httpx/response.go"
test -f "common/authprovider/interface.go" \\
  && ok "common/authprovider/interface.go" \\
  || miss "missing critical file: common/authprovider/interface.go"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 31 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~1d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/projectdiscovery/httpx"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

httpx is a fast, multi-threaded HTTP reconnaissance toolkit built in Go that probes multiple HTTP-based signals (status codes, titles, headers, TLS certs, favicon hashes, CSP, web servers, etc.) using the projectdiscovery/retryablehttp library for reliability. It accepts hosts, URLs, or CIDR ranges as input and performs intelligent fallback from HTTPS to HTTP while handling WAFs via retries and backoffs. Monolithic CLI tool architecture: cmd/httpx/httpx.go is the main entry point wrapping the core library; cmd/integration-test and cmd/functional-test validate behavior; common/ packages auth strategies (authprovider/authx), custom extraction (customextract), and headers; individual probe logic likely distributed across unnamed internal packages referenced by main httpx logic.

Security researchers, penetration testers, and bug bounty hunters who need to quickly fingerprint web services at scale across large IP ranges or domain lists, gathering response metadata without manual HTTP request crafting.

Production-ready and actively maintained. The codebase shows 369K lines of Go with comprehensive GitHub Actions CI (build-test, codeql-analysis, functional-test, release workflows), a goreleaser configuration for binary releases, and Docker support. Regular releases and automated dependency management indicate sustained development.

Low-to-moderate risk. Dependencies on 15+ projectdiscovery ecosystem packages and external libraries (go-rod for browser automation, jarm-go for TLS fingerprinting) create some maintenance surface, but the project has automated dependency updates (dependabot) and multi-stage testing. The large feature surface (30+ probe types) means potential regressions; functional-test and integration-test suites exist to catch them.

Active development evident from release workflow automation, CodeQL analysis setup, and dependency auto-merge pipelines. Functional and integration tests suggest recent feature additions. Go version is 1.26, indicating modern stdlib usage.

git clone https://github.com/projectdiscovery/httpx.git
cd httpx
go build -o httpx ./cmd/httpx
./httpx -h

Daily commands: For CLI: go run ./cmd/httpx/httpx.go -u http://example.com. For tests: go test ./... or make test (if Makefile target exists). For integration tests: go run ./cmd/integration-test/integration-test.go.

• cmd/httpx/httpx.go — Main CLI entry point that orchestrates the httpx tool, defines flags, and coordinates the HTTP probing workflow.
• common/httpx/httpx.go — Core HTTP client abstraction and request/response handling logic; all probe operations depend on this.
• common/httpx/option.go — Configuration options for the HTTP client; understanding these is essential for customizing behavior.
• common/httpx/response.go — Response parsing and metadata extraction; defines how HTTP responses are processed and analyzed.
• common/authprovider/interface.go — Authentication provider abstraction that must be implemented for any auth mechanism support.
• go.mod — Dependency list including retryablehttp and probe-related libraries; critical for understanding external integrations.

Add a New HTTP Header Extraction Feature

1. Create a new file in common/httpx/ (e.g., new_extractor.go) with a function that takes *http.Response and returns the extracted value (common/httpx/new_extractor.go)
2. Add a field to the Response struct in common/httpx/response.go to hold the extracted data (common/httpx/response.go)
3. Call your extraction function from httpx.Do() or httpx.makeHTTPRequest() in common/httpx/httpx.go after response is received (common/httpx/httpx.go)
4. Add CLI flags to cmd/httpx/httpx.go to enable/configure this extraction feature (cmd/httpx/httpx.go)
5. Write tests in common/httpx/new_extractor_test.go following the pattern of existing tests (e.g., httpx_test.go) (common/httpx/new_extractor_test.go)

Add a New Authentication Method

1. Create a new strategy file in common/authprovider/authx/ (e.g., oauth2_auth.go) implementing the Auth interface (common/authprovider/authx/oauth2_auth.go)
2. Register the strategy in common/authprovider/authx/strategy.go's constructor or NewAuthStrategy function (common/authprovider/authx/strategy.go)
3. Add the strategy type to the options in common/httpx/option.go under AuthProvider configuration (common/httpx/option.go)
4. Add CLI flags for the new auth method in cmd/httpx/httpx.go and wire them to the Options (cmd/httpx/httpx.go)
5. Test the integration using common/authprovider/authx/strategy_test.go as a template (common/authprovider/authx/oauth2_auth_test.go)

Add a New Input Format Parser

1. Create a parser function in a new file under common/inputformats/ (e.g., swagger.go) that converts the format to []model.Request (common/inputformats/swagger.go)
2. Register the parser in common/inputformats/formats.go in the parseFormat or Parse function (common/inputformats/formats.go)
3. Add logic to detect the new format in formats.go's DetectFormat or similar function (common/inputformats/formats.go)
4. Add integration tests in common/inputformats/formats_test.go with sample input files (common/inputformats/formats_test.go)
5. Update CLI to accept the new format in cmd/httpx/httpx.go if needed (via -input-format flag or auto-detection) (cmd/httpx/httpx.go)

• Go with goroutines — Enables fast, concurrent HTTP probing across thousands of targets with low memory overhead; ideal for network I/O-bound workloads.
• retryablehttp library — Provides automatic retry logic and fault tolerance for unreliable network conditions without custom implementation.
• goquery (jQuery-like parsing) — Simplifies HTML parsing for title extraction and metadata discovery without heavy DOM overhead.
• rod (headless browser automation) — Enables JavaScript rendering and shadow DOM exploration for modern web applications.
• DNS library (miekg/dns) — Provides custom DNS resolution and validation for virtual host and domain enumeration.

• Modular auth provider abstraction vs. built-in auth

  • Why: Allows pluggable auth strategies (basic, bearer, cookies, headers, query) without coupling to a specific mechanism.
  • Consequence: Slightly more code but easier to add new auth types without modifying core HTTP engine.

• Streaming response processing vs. buffering full response

  • Why: Enables memory-efficient handling of large responses and parallel analysis.
  • Consequence: Some metadata (like title extraction) requires buffering the body, creating a trade-off between memory and speed.

• Multiple input format support (URLs, Burp, raw requests)

  • Why: Maximizes integration with existing security tools and workflows.
  • Consequence: Increased complexity in input parsing and validation logic.

• Content hashing (MD5, MMIH3, SimHash) for duplicate detection

  • Why: Reduces noise in large-scale probing by identifying duplicate or similar page content.
  • Consequence: CPU overhead on every response; can be disabled if speed is prioritized over deduplication.

• Does not provide a persistent database or long-term result storage; focuses on probe-time analysis and output streaming.
• Does not implement advanced fuzzing or payload generation; designed for web discovery and reconnaissance, not exploitation.
• Not a full vulnerability scanner; does not perform automated remediation or detailed security assessment.
• Does not handle complex multi-step authentication flows like OAuth2 intricate handshakes; limited to simple token/header-based auth.
• Linux/Windows command-line tool; not a web UI or dashboard

No explicit environment variable requirements visible, but retryablehttp and fastdialer may require network policy / firewall rule setup. TLS fingerprinting via hdm/jarm-go and tlsx has strict version compatibility. Browser automation via go-rod requires a Chrome/Chromium binary at runtime or bundled. Custom extraction via DSL (projectdiscovery/dsl) has its own expression syntax not documented in httpx README; refer to dsl package docs. Functional tests in cmd/functional-test/testcases.txt hint at undocumented or edge-case features.

• Retryable HTTP with Exponential Backoff — httpx's resilience against WAF blocks and rate limiting depends on the retryablehttp library's backoff strategy; understanding retry counts, timeouts, and backoff curves is essential for tuning performance vs. reliability
• TLS Fingerprinting (JARM/JA3) — httpx uses JARM (via hdm/jarm-go) and TLS cert parsing to identify server versions and misconfigurations; this is a key fingerprinting probe that doesn't require an HTTP response
• HTTP/2 Detection and Protocol Negotiation — httpx can probe HTTP/2 support and HTTP pipelining; requires understanding of TLS ALPN negotiation and HTTP frame parsing
• Host Headers and Virtual Hosting — httpx's virtual host probe feature manipulates Host headers to detect vhost misconfiguration; critical for finding hidden services on shared infrastructure
• CDN Detection via IP/CIDR Inference — httpx integrates projectdiscovery/cdncheck to identify whether a response is from a CDN; requires understanding CIDR input handling and IP-based service mapping
• Content Hashing (Favicon, Simhash, Murmurhash) — httpx computes favicon hashes (corona10/goimagehash), simhash for page similarity (mfonda/simhash), and murmur3 for fast deduplication; useful for grouping similar responses
• DSL-based Dynamic Extraction — projectdiscovery/dsl allows users to define custom extraction rules via expressions (similar to Nuclei); understanding DSL syntax unlocks advanced response parsing without code changes

• projectdiscovery/nuclei — Higher-level vulnerability scanning engine that uses httpx for HTTP probing under the hood; complementary tool in the same ecosystem
• projectdiscovery/retryablehttp-go — Core HTTP library powering httpx; handles retries, backoffs, and WAF evasion logic
• projectdiscovery/tlsx — TLS fingerprinting and certificate parsing library; integrated into httpx for TLS-based probes
• projectdiscovery/wappalyzergo — Technology fingerprinting library used by httpx to identify web servers, frameworks, and applications
• aquasecurity/nuclei-templates — Community templates for nuclei that often rely on httpx probes; shows real-world usage patterns

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for common/authprovider/authx authentication strategies

The authprovider/authx module has multiple authentication implementations (basic_auth.go, bearer_auth.go, cookies_auth.go, headers_auth.go, query_auth.go) with only strategy_test.go and file_test.go present. Each auth type lacks dedicated unit tests for edge cases, invalid inputs, and integration scenarios. This is critical for a security-sensitive HTTP toolkit.

• [ ] Create common/authprovider/authx/basic_auth_test.go with tests for base64 encoding, special characters, empty credentials
• [ ] Create common/authprovider/authx/bearer_auth_test.go with tests for token formats, expiration handling, malformed tokens
• [ ] Create common/authprovider/authx/cookies_auth_test.go with tests for cookie parsing, expiration, secure/httponly flags
• [ ] Create common/authprovider/authx/headers_auth_test.go with tests for header injection, case sensitivity, multiple values
• [ ] Create common/authprovider/authx/query_auth_test.go with tests for URL encoding, parameter conflicts, injection attempts
• [ ] Run coverage analysis to ensure >80% coverage for the authx module

Add GitHub Actions workflow for integration tests with real HTTP servers

The repo has cmd/integration-test/ (http.go, integration-test.go, library.go) and cmd/functional-test/ but no CI workflow to run them automatically. Current workflows (build-test.yml, functional-test.yml, release-test.yml) don't show dedicated integration test execution. This ensures httpx works correctly with real-world scenarios before releases.

• [ ] Create .github/workflows/integration-test.yml that spins up test HTTP servers (from cmd/integration-test/http.go)
• [ ] Run cmd/integration-test/integration-test.go and cmd/integration-test/library.go against live endpoints
• [ ] Configure the workflow to run on PR, push to main, and scheduled daily
• [ ] Add test result reporting and failure notifications
• [ ] Document expected behavior in TESTING.md if not present

Add missing documentation for custom extraction and header modules in common/

The common/customextract/doc.go, common/customheader/doc.go, common/customlist/doc.go, common/customports/doc.go, common/hashes/doc.go, and common/fileutil/doc.go files exist but likely have minimal or placeholder documentation. These are core utility modules for httpx users but lack examples in README or dedicated USAGE guides.

• [ ] Expand common/customextract/doc.go with examples of custom extraction patterns and supported syntaxes
• [ ] Expand common/customheader/doc.go with examples of header injection, multi-value headers, and special characters
• [ ] Expand common/customlist/doc.go with examples of list file formats and pagination handling
• [ ] Expand common/customports/doc.go with port range syntax and protocol-specific defaults
• [ ] Create docs/CUSTOM_EXTRACTION.md with comprehensive examples linking to the above modules
• [ ] Add cross-references to README.md's Features section

• Add unit tests for common/authprovider/authx/ strategies; file_test.go and strategy_test.go exist but may lack full coverage (check for custom auth type tests).
• Document the DSL expression syntax supported by common/customextract in a guide under docs/ or as code comments, since it's exposed via CLI flags but not explained in README.
• Add examples in cmd/integration-test/library.go or a new examples/ directory showing how to use httpx programmatically: e.g., 'how to perform a TLS cert probe only' or 'how to set custom auth headers'.

Failed to generate security analysis.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.