GO — Healthy across the board

• Last commit 1d ago
• 24+ active contributors
• Distributed ownership (top contributor 32% of recent commits)
• MIT licensed
• CI configured
• Tests present

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

Sherlock is a Python-based username search engine that hunts for accounts across 400+ social networks and websites simultaneously. Given one or more usernames, it queries a curated database of site probes (defined in sherlock_project/resources/data.json) to identify where those usernames exist, aggregating results into text/CSV/XLSX reports. It solves the problem of manual, tedious cross-platform username lookups for OSINT, account verification, and threat hunting workflows. Single-package structure: sherlock_project/ is the main module with sherlock.py (core search engine), sites.py (site/probe management), result.py (result representation), and notify.py (notification integration). sherlock_project/resources/data.json is the canonical probe database. CLI entry point via __main__.py. Tests live in tests/ with conftest fixtures. Docker support via Dockerfile and AppriseActor support via .actor/ for cloud execution.

Security researchers, OSINT practitioners, and law enforcement who need to quickly map a username across multiple social platforms; also used by individuals verifying username availability before registration. Contributors include open-source developers maintaining the 400+ site probe definitions and adding new platform detection logic.

Actively maintained and production-ready. The project has significant GitHub stars (in the thousands based on description prominence), maintains comprehensive test coverage (tests/ directory with regression/validation workflows), uses GitHub Actions CI (exclusions.yml, regression.yml, validate_modified_targets.yml), and shows regular updates to the site list. Version 0.16.0 indicates stable iteration cycles.

Low-to-moderate risk for stable use, but several caveats: (1) Heavy reliance on external site structure stability—when sites change HTML/API signatures, probe definitions in data.json require manual updates, (2) No visible lock file in the repo structure suggests dependency pinning may be loose, (3) Rate-limiting and proxy support exist but anti-scraping measures on target sites could cause false negatives, (4) The site probe schema (data.schema.json) is complex and error-prone to maintain at scale.

Active maintenance of the site probe database—GitHub Actions workflows (update-site-list.yml, validate_modified_targets.yml) automate site list updates and validation. The repository handles false-positive/false-negative reports via issue templates. Development tooling in devel/ (e.g., site-list.py, summarize_site_validation.py) suggests ongoing refinement of probe accuracy and coverage.

git clone https://github.com/sherlock-project/sherlock.git
cd sherlock
pipx install -e .
sherlock --help
sherlock testuser

Alternatively, use Docker: docker run -it --rm sherlock/sherlock testuser or install via pipx install sherlock-project.

Daily commands:

python -m sherlock_project username1 username2

Or after pip install: sherlock username1 username2. Use --help for flags: --csv, --xlsx, --json, --site SITE_NAME, --timeout TIMEOUT, --proxy PROXY_URL, etc.

• sherlock_project/sherlock.py — Core orchestration engine that manages username searches across all sites; implements the main async request logic and result aggregation
• sherlock_project/resources/data.json — Central repository of 400+ social network site definitions; every search depends on this data being current and properly formatted
• sherlock_project/sites.py — Site abstraction layer that parses data.json and provides site-matching logic; mediates between core search logic and network definitions
• sherlock_project/__main__.py — CLI entry point that handles argument parsing and user interaction; defines how the tool is invoked and configuration options
• sherlock_project/result.py — Result aggregation and formatting; handles output serialization (JSON, CSV) and determines what findings users see
• sherlock_project/resources/data.schema.json — JSON Schema that validates all site definitions; prevents malformed site entries and ensures consistency

• sherlock.py (Orchestrator) (asyncio, aiohttp, httpx, regex) — Manages async HTTP client pool, rate limiting, timeout handling, and result collection; coordinates search across all sites
  • Failure mode: Network timeout → marks site as 'error'; bad regex → false positives/negatives; rate-limit hit → slow searches or IP blocks
• sites.py (Site Registry) (JSON, regex, URL templating) — Parses data.json and provides site objects; handles site filtering, URL templating, and detection rule evaluation
  • Failure mode: Malformed data.json → parsing exception; bad URL template → 404s on all probes; regex engine failure → crash on special chars
• result.py (Output Layer) (JSON, CSV, dataclasses) — Aggregates probe results and serializes to JSON, CSV, or human-readable tables; tracks found/not-found/errors
  • Failure mode: Large result set → memory exhaustion; missing serialization logic → output format corruption
• **** — undefined

Add a New Social Network Site

1. Add a new site entry to sherlock_project/resources/data.json with username_claimed, username_unclaimed, match detection logic, and HTTP request parameters (sherlock_project/resources/data.json)
2. Verify the entry conforms to the schema by running validation tests against data.schema.json (sherlock_project/resources/data.schema.json)
3. Add a test case to tests/test_probes.py with a known username and expected detection result (tests/test_probes.py)
4. Run tests locally: pytest tests/test_probes.py and tests/test_validate_targets.py to ensure consistency (pytest.ini)

Modify Search Algorithm or HTTP Behavior

1. Edit the main search loop and request/response handling in sherlock_project/sherlock.py (sherlock_project/sherlock.py)
2. Update site matching logic if needed in sherlock_project/sites.py to reflect new detection rules (sherlock_project/sites.py)
3. Run regression tests to ensure existing sites still work: pytest tests/test_probes.py (tests/test_probes.py)

Add a New Output Format or Notification Channel

1. Extend the Result class in sherlock_project/result.py to support new serialization format (sherlock_project/result.py)
2. Add notification handler to sherlock_project/notify.py (e.g., Slack, Telegram, webhook) (sherlock_project/notify.py)
3. Update CLI argument parser in sherlock_project/main.py to accept new output/notification options (sherlock_project/__main__.py)
4. Add end-to-end test in tests/test_ux.py to verify new format or notification works (tests/test_ux.py)

• Python + asyncio — I/O-bound workload (400+ parallel HTTP requests) requires async concurrency; Python ecosystem is mature for web scraping and username enumeration
• JSON-based site definitions (data.json) — Declarative, schema-validated, human-editable format enables community contributions without code changes; simple parsing avoids complex ORM overhead
• pytest + GitHub Actions workflows — Regression testing and validation automation catch false positives/negatives from site changes; CI gates prevent shipping broken site definitions
• Docker — Isolates dependency versions and avoids 'works on my machine' issues; simplifies deployment across ParrotOS, Ubuntu, and generic Linux

• Site definitions entirely declarative (data.json) rather than code-based plugins

  • Why: Lowers barrier to community contributions; no need to understand Python internals
  • Consequence: Complex site logic (pagination, JavaScript rendering, multi-step auth) is harder to express; limited to regex/text matching

• Sequential-like CLI (single username per invocation) rather than batch mode

  • Why: Simpler UX, easier integration with shell scripts, matches tool philosophy ('hunt one username')
  • Consequence: Users searching 100 usernames must invoke tool 100 times; no native batch optimization

• No persistent database or caching between runs

  • Why: Zero infrastructure requirements; tool remains standalone and portable
  • Consequence: Redundant HTTP requests across runs; no historical trend analysis; results are ephemeral

• Regex-based result detection rather than trained ML classifier

  • Why: Deterministic, auditable, no training data needed, works offline
  • Consequence: Fragile to site layout changes; high false-positive rate on sites with weak detection rules; requires manual regex tuning

• Does not crawl or scrape user profiles (read-only account existence detection only)
• Does not perform active exploitation or break authentication mechanisms
• Does not handle multiple-factor authentication or account takeover workflows
• Does not maintain a persistent database or historical records across invocations
• Does not provide real-time monitoring; only performs point-in-time searches
• Does not run on Windows natively (requires WSL or Docker)
• Does not support JavaScript-rendered content; only static HTML/API responses

1. Probe schema fragility: Each site entry in data.json requires precise 'errorType' (status code, response text, redirect, etc.) and matching logic—small changes to a site's HTML/API will break detection silently (false negatives). 2. Rate limiting: No built-in rate limiting; querying 400+ sites rapidly may trigger IP blocks; --timeout and --proxy are user-controlled but not automated. 3. Unicode handling: tests/test_unicode.py exists, suggesting historical Unicode edge cases—some sites may not handle international characters properly. 4. Local mode (--local): Runs without network; requires pre-cached responses, behavior differs from live mode. 5. NSFW flag: --nsfw flag exists but behavior is opaque—some sites may be excluded by default. 6. Exclusion logic: --ignore-exclusions flag and exclusions.yml workflow suggest certain sites are dynamically excluded (e.g., high false-positive or unstable sites). 7. Dependency on jsonschema: Validation uses strict schema; probe additions must match schema exactly or they silently fail to load.

• Probe-based detection (response fingerprinting) — Sherlock doesn't use APIs; it sends HTTP requests and fingerprints responses (status codes, text patterns, redirects) to infer username existence. Understanding error signatures and false-positive mitigation is central to adding sites.
• Asyncio concurrency patterns — Sherlock queries hundreds of sites in parallel using Python's asyncio. The search performance depends on async/await patterns and connection pooling in sherlock.py.
• JSON Schema validation — Probe definitions in data.json are validated against data.schema.json before use. Understanding schema structure is essential for adding or debugging new sites.
• Username enumeration / account discovery — Sherlock automates manual username enumeration across platforms. This is a core OSINT technique with privacy and legal implications worth understanding.
• HTTP status codes and redirect chains — Probes detect existence via 200 OK, 404 Not Found, 403 Forbidden, or redirects. Understanding HTTP semantics is critical for accurate site-specific error detection logic.
• Anti-scraping evasion (proxies, timeouts, rate limiting) — Sherlock supports --proxy, --timeout, and backoff strategies to avoid IP blocks. Understanding bot detection countermeasures is necessary for reliability at scale.
• Template string interpolation for dynamic URLs — Site probes use URL templates like https://example.com/{username} that are interpolated at runtime. The {?} wildcard feature supports fuzzy username matching (e.g., user_name, user-name).

• n0where/n0where-OSINT — Similar OSINT username search tool; comparison for feature parity and probe coverage
• webbreacher/Ghunt — Google-specific reconnaissance tool; complementary to Sherlock for deep dives into single platforms
• Josua89/Gitfinder — GitHub-specific username finder; demonstrates single-site probe pattern that Sherlock generalizes across 400+
• laramies/theHarvester — Email/domain reconnaissance tool often used alongside Sherlock in integrated OSINT workflows
• apify/apify-sdk-python — Sherlock's .actor/ directory uses Apify platform; this SDK is the underlying framework for cloud execution

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive integration tests for sherlock.py site lookup logic

The repo has basic unit tests (few_test_basic.py, test_probes.py) but lacks integration tests that validate the core sherlock.py lookup mechanism against real and mocked site responses. This would catch regressions when sites change their response formats and improve confidence in the 400+ site definitions in resources/data.json.

• [ ] Create tests/test_site_lookup_integration.py with fixtures for mocking HTTP responses
• [ ] Add parametrized tests for each site category (social media, code hosting, gaming, etc.) using samples from sherlock_project/resources/data.json
• [ ] Validate that sherlock.py correctly parses positive/negative indicators from data.json for at least 50+ representative sites
• [ ] Add tests for edge cases: rate limiting, timeouts, and malformed responses
• [ ] Reference existing test structure in tests/conftest.py for pytest fixtures

Create GitHub Actions workflow to validate data.json schema against live site samples

Currently .github/workflows/validate_modified_targets.yml exists but there's no automated validation that the JSON schema (sherlock_project/resources/data.schema.json) actually matches the structure used by the lookup logic in sherlock.py. This would prevent silent data corruption and ensure new site additions are valid before merge.

• [ ] Create .github/workflows/validate_data_schema.yml that runs on PR changes to sherlock_project/resources/data.json
• [ ] Use jsonschema library to validate data.json against data.schema.json
• [ ] Add a check that verifies all required fields (uri_check, presence_strs, absence_strs) are present for each site entry
• [ ] Include a step that spot-checks 10 random sites by making HEAD requests to verify URIs are syntactically valid URLs
• [ ] Reference the existing regex validation pattern from .github/workflows/validate_modified_targets.yml

Add performance benchmarking tests and CI workflow for concurrent site lookups

The repo supports concurrent requests (as evident from the CLI and actor.json) but lacks benchmarks to catch performance regressions. With 400+ sites, performance is critical for user experience. Adding benchmarks would ensure concurrent lookup speed is maintained across releases.

• [ ] Create tests/test_performance.py with pytest-benchmark fixtures to measure lookup times
• [ ] Benchmark sequential vs. concurrent lookups across 50-site and 400-site datasets using sherlock.py
• [ ] Add tests for memory usage during large batch operations (reference sherlock_project/main.py for batch handling)
• [ ] Create .github/workflows/performance.yml to run benchmarks on each PR and track results over time
• [ ] Store benchmark results as artifacts to enable comparison across commits

• Expand test coverage for sherlock_project/notify.py: File exists but has no visible tests in tests/. Add unit tests for notification integrations (email, webhook, etc.) to ensure notify functionality works reliably
• Document probe definition best practices in docs/: Contributors adding new sites in data.json need clear guidance on 'errorType' patterns, URL templating, and response matching. Create a guide referencing data.schema.json with 5-10 real examples
• Add performance benchmarks for concurrent site queries: No visible performance tests; add a pytest fixture that measures latency/throughput when querying 50+ sites concurrently with different timeout/proxy settings, help maintainers optimize asyncio logic

The Sherlock project shows generally good security practices as a Python CLI tool with proper project structure and GitHub security configurations. Main concerns are: (1) Docker build arguments lacking automation to ensure proper metadata, (2) missing visibility into dependency security status, (3) base image not pinned to digest for reproducibility, and (4) limited visibility into input validation patterns. The project includes security documentation (.github/SECURITY.md) and proper issue templates, which are positive indicators. No hardcoded secrets, injection vulnerabilities, or critical misconfigurations are apparent from the provided file structure. Recommend implementing automated dependency scanning and Docker image pinning for enhanced security posture.

• Medium · Docker Build Arguments Not Set — Dockerfile (lines with ARG VCS_REF and ARG VERSION_TAG). The Dockerfile contains ARG directives (VCS_REF and VERSION_TAG) with empty default values marked with '# CHANGE ME ON UPDATE' comments. These are critical for image labeling and version tracking. If not set during build, the image will lack proper metadata for tracking and auditing purposes. Fix: Implement a CI/CD pipeline that automatically sets these build arguments during image creation. Use git hooks or automated workflows to ensure VCS_REF and VERSION_TAG are always populated with correct values before building production images.
• Low · Incomplete Dependency Information — pyproject.toml, requirements files (not provided). The dependency/package file content was not provided in the analysis context. Unable to assess for known vulnerabilities in third-party packages (outdated versions, CVEs, etc.). The project uses external packages (indicated by pyproject.toml) that require verification. Fix: Regularly run pip audit or use tools like safety, bandit, and dependabot to identify vulnerable dependencies. Update all dependencies to their latest secure versions and maintain a Software Bill of Materials (SBOM).
• Low · Python 3.12-slim Base Image Selection — Dockerfile (FROM python:3.12-slim-bullseye). Using Debian slim images can have security gaps. While slim variants reduce attack surface by having fewer packages, they may miss security updates faster than full images. Additionally, no specific image digest is pinned, making builds non-reproducible. Fix: Pin the Docker image to a specific digest hash instead of using floating tags (e.g., python:3.12-slim-bullseye@sha256:...). Regularly scan base images with tools like Trivy and update to patched versions.
• Low · Missing Security Headers and Configuration Documentation — sherlock_project/ (main application code not fully visible). No visible security configuration for HTTP headers, CSP policies, or CORS settings in the codebase. While this is a CLI tool, the web documentation site and API endpoints (if any) may lack security hardening. Fix: If serving any HTTP endpoints, implement security headers (Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, etc.). Create and document security guidelines for any web-facing components.
• Low · No Input Validation Documentation Visible — sherlock_project/sherlock.py, sherlock_project/__main__.py. The application hunts usernames across social networks. While likely safe, username input validation and sanitization patterns are not visible in the provided file structure, which could be relevant if the tool accepts user input. Fix: Implement strict input validation for usernames. Ensure username inputs are validated against expected character sets and length limits. Sanitize any outputs before displaying to users.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/sherlock-project/sherlock shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live sherlock-project/sherlock repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/sherlock-project/sherlock.

What it runs against: a local clone of sherlock-project/sherlock — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in sherlock-project/sherlock | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 31 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of sherlock-project/sherlock. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/sherlock-project/sherlock.git
#   cd sherlock
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of sherlock-project/sherlock and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "sherlock-project/sherlock(\\.git)?\\b" \\
  && ok "origin remote is sherlock-project/sherlock" \\
  || miss "origin remote is not sherlock-project/sherlock (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "sherlock_project/sherlock.py" \\
  && ok "sherlock_project/sherlock.py" \\
  || miss "missing critical file: sherlock_project/sherlock.py"
test -f "sherlock_project/resources/data.json" \\
  && ok "sherlock_project/resources/data.json" \\
  || miss "missing critical file: sherlock_project/resources/data.json"
test -f "sherlock_project/sites.py" \\
  && ok "sherlock_project/sites.py" \\
  || miss "missing critical file: sherlock_project/sites.py"
test -f "sherlock_project/__main__.py" \\
  && ok "sherlock_project/__main__.py" \\
  || miss "missing critical file: sherlock_project/__main__.py"
test -f "sherlock_project/result.py" \\
  && ok "sherlock_project/result.py" \\
  || miss "missing critical file: sherlock_project/result.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 31 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~1d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/sherlock-project/sherlock"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.