If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/yangyangwithgnu/hardseed shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

WAIT — Stale — last commit 8y ago

• 3 active contributors
• GPL-2.0 licensed
• Tests present
• ⚠ Stale — last commit 8y ago
• ⚠ Small team — 3 contributors active in recent commits
• ⚠ Single-maintainer risk — top contributor 84% of recent commits
• ⚠ GPL-2.0 is copyleft — check downstream compatibility
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live yangyangwithgnu/hardseed repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/yangyangwithgnu/hardseed.

What it runs against: a local clone of yangyangwithgnu/hardseed — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in yangyangwithgnu/hardseed | Confirms the artifact applies here, not a fork | | 2 | License is still GPL-2.0 | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 2844 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of yangyangwithgnu/hardseed. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/yangyangwithgnu/hardseed.git
#   cd hardseed
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of yangyangwithgnu/hardseed and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "yangyangwithgnu/hardseed(\\.git)?\\b" \\
  && ok "origin remote is yangyangwithgnu/hardseed" \\
  || miss "origin remote is not yangyangwithgnu/hardseed (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(GPL-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"GPL-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is GPL-2.0" \\
  || miss "license drift — was GPL-2.0 at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "src/main.cpp" \\
  && ok "src/main.cpp" \\
  || miss "missing critical file: src/main.cpp"
test -f "src/lib/self/TopicsListWebpage.h" \\
  && ok "src/lib/self/TopicsListWebpage.h" \\
  || miss "missing critical file: src/lib/self/TopicsListWebpage.h"
test -f "src/lib/self/SeedWebpage.h" \\
  && ok "src/lib/self/SeedWebpage.h" \\
  || miss "missing critical file: src/lib/self/SeedWebpage.h"
test -f "src/lib/helper/Webpage.h" \\
  && ok "src/lib/helper/Webpage.h" \\
  || miss "missing critical file: src/lib/helper/Webpage.h"
test -f "config/portals_list.json" \\
  && ok "config/portals_list.json" \\
  || miss "missing critical file: config/portals_list.json"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 2844 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~2814d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/yangyangwithgnu/hardseed"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

hardseed is a C++ command-line torrent metadata and image crawler that automatically discovers, parses, and downloads seeds and associated images from Chinese adult forums (Aicheng, Caoliu, etc.). It combines HTTP webpage parsing, regex-based content extraction, and concurrent download management to harvest torrents and images while respecting proxy requirements for geographically-restricted access. Monolithic C++ application: src/lib/self/ contains forum-specific scrapers (Aicheng*, Caoliu*, Jandown*, Rmdown* classes) inheriting from base TopicsListWebpage and SeedWebpage; src/lib/helper/ provides utilities (Webpage.cpp for HTTP + parsing, CmdlineOption.cpp for CLI args, Time.cpp, Misc.cpp); src/lib/3rd/json11/ vendors a JSON parser. Main entry point is src/main.cpp. Config-driven portal URLs via config/portals_list.json.

Chinese users seeking automated bulk torrent and image collection from forum communities, who need programmatic access to dynamically-updated portal lists and can configure local proxy servers (goagent) for access. Contributors would be C++ developers familiar with string parsing, HTTP clients, and concurrent download pipelines.

Moderately mature but aging: last versioned release v0.2.14-1 from Feb 2016 (8+ years old). The project shows active maintenance through 2016 with incremental fixes (address changes, parsing corrections, Windows support), but no recent commits visible. Test coverage is minimal (only json11 test.cpp), and no CI/CD pipeline is present. Verdict: stable but unmaintained—suitable for learning but not production-critical systems.

High risk of breakage: the codebase depends entirely on scraping forum HTML structure (Aicheng, Caoliu) which changes frequently—v0.2.10-0.2.12 show repeated 'address changed' fixes. Single maintainer (yangyangwithgnu), no test suite beyond json11, and 8-year staleness mean regex patterns and URL parsing logic are likely obsolete. Forum APIs themselves may have been blocked or restructured, rendering the crawler non-functional. Dependencies on goagent proxy (external project) add fragility.

Project is dormant. Last activity (v0.2.14-1) was Feb 4, 2016—addressing Windows binary distribution and macOS build fixes. No ongoing PRs, issues, or roadmap visible. The repo exists as a static archive with no active development, bug tracking, or community engagement since 2016.

git clone https://github.com/yangyangwithgnu/hardseed.git
cd hardseed
cd build
cmake ..
make
./hardseed --help

Alternatively, use precompiled Windows binary from bin/build_4_win.7z. Requires CMake 3.x, C++11-capable compiler, libcurl (or equivalent HTTP library), and a working proxy (goagent recommended, configured via --proxy localhost:8087).

Daily commands:

cd build && cmake .. && make
./hardseed --portal aicheng --topics 10 --pictures --output /tmp/seeds --proxy localhost:8087 --timeout-download-picture 30

Key flags (inferred from v0.2.14 changelog): --portal (aicheng|caoliu), --topics (count), --pictures (enable image download), --proxy (local proxy addr:port), --timeout-download-picture (seconds, with adaptive retry logic as of v0.2.05+).

• src/main.cpp — Entry point of the application; orchestrates command-line parsing, portal selection, and seed harvesting workflow.
• src/lib/self/TopicsListWebpage.h — Base abstraction for topic list page parsing; all portal implementations (Aicheng, Caoliu) inherit from this interface.
• src/lib/self/SeedWebpage.h — Base abstraction for seed/torrent detail page parsing; defines contract for extracting magnet links and metadata.
• src/lib/helper/Webpage.h — Core HTTP client and HTML parsing utility; handles all network requests and DOM navigation for all portals.
• config/portals_list.json — Configuration registry for all supported portals (Aicheng, Caoliu, Jandown, Rmdown); defines scraping targets and metadata.
• src/lib/helper/CmdlineOption.h — Command-line argument parser; exposes user-facing options like --proxy, output formats, and portal selection.
• src/lib/3rd/json11/json11.hpp — Third-party JSON library; critical for parsing portals_list.json and serializing results.

Add Support for a New Portal

1. Create portal coordinator class inheriting from no base (leaf in hierarchy); e.g., src/lib/self/NewPortal.h/.cpp (src/lib/self/NewPortal.h)
2. Create topics-list parser inheriting from TopicsListWebpage; implement parse() to extract thread links using portal-specific CSS selectors (src/lib/self/NewPortalTopicsListWebpage.h)
3. Create topic parser inheriting from TopicWebpage; implement parse() to extract seed URLs from individual threads (src/lib/self/NewPortalTopicWebpage.h)
4. Create seed-detail parser inheriting from SeedWebpage; implement parse() to extract magnet links and metadata (src/lib/self/NewPortalSeedWebpage.h)
5. Register new portal in config/portals_list.json with base URL, pagination rules, CSS selectors, and output schema (config/portals_list.json)
6. In src/main.cpp, add conditional branch in portal factory to instantiate NewPortal when user selects it via CLI (src/main.cpp)

Modify Scraping Logic for an Existing Portal

1. Identify which layer needs changes: TopicsListWebpage (thread enumeration), TopicWebpage (seed link extraction), or SeedWebpage (metadata/magnet extraction) (src/lib/self/AichengTopicsListWebpage.h)
2. Update CSS selectors or DOM navigation logic in the parse() method of the target class (src/lib/self/AichengTopicWebpage.cpp)
3. If pagination rules change, update portals_list.json with new next-page selector or limit rules (config/portals_list.json)
4. Test by running: hardseed --portal aicheng --output json > results.json (src/main.cpp)

Add a New Command-Line Option

1. Define new option flag and handler in CmdlineOption.h enum or flag structure (src/lib/helper/CmdlineOption.h)
2. Implement parsing logic in CmdlineOption.cpp to extract and validate the option value (src/lib/helper/CmdlineOption.cpp)
3. In src/main.cpp, read the parsed option via cmdlineOpts.yourNewOption and pass to portal scraper or output formatter (src/main.cpp)

• libcurl + libxml2 — Minimal dependencies for HTTP requests and robust HTML/DOM parsing; critical for scraping multiple portals reliably.
• json11 (header-only) — Lightweight JSON parsing for portals_list.json config registry; no external dependencies beyond STL.
• C++11 — Efficient polymorphic design for portal scrapers; memory-safe string handling; cross-platform (Unix/Windows).
• CMake — Unified build system for Unix-like systems and Windows; simplifies dependency management (libcurl, libxml2, libiconv).
• Proxy support (via libcurl) — Essential for circumventing GFW and accessing Chinese portals from outside mainland China; user-specified via --proxy.

• Polymorphic base classes (TopicsListWebpage, SeedWebpage) with portal-specific subclasses

  • Why: Allows adding new portals without modifying core scraping logic; clean separation of concerns.
  • Consequence: Slight vtable overhead per request; requires developers to understand inheritance hierarchy when adding portals.

• Portal metadata in JSON config (portals_list.json) rather than hardcoded in C++

  • Why: Enables rapid portal updates without recompilation; users can edit selectors for portal DOM changes.
  • Consequence: Runtime JSON parsing overhead; requires careful selector validation to avoid scraper breakage.

• Sequential page fetching (topic list → individual topics → seed detail pages)

  • Why: Avoids overwhelming portal servers and getting IP-blocked; respects portal rate limits.
  • Consequence: Slower harvest for large topic counts; difficult to parallelize due to pagination state.

• Inline response caching (in Webpage.h) rather than external cache store

  • Why: No external dependencies; simple per-session caching of identical requests.
  • Consequence: Cache does not persist across runs; no distributed caching for multi-process harvests.

• Does not implement user authentication

Proxy requirement: hardseed is non-functional without an HTTP proxy (goagent or equivalent) configured via --proxy—it cannot reach target forums directly (geographically restricted). Forum URL brittleness: Hardcoded regex patterns and URL parsing in Aicheng*.cpp and Caoliu*.cpp break instantly if forum HTML structure changes; v0.2.10, v0.2.12 fixes show this happens frequently. Stale dependencies: goagent_out_of_box_yang (referenced in README) is a separate project and may be deprecated; GGC IP lists in proxy.ini can become invalid. Charset handling: No explicit mention of UTF-8 or encoding handling in helper utilities—may fail on non-ASCII characters in post titles. Image URL format: v0.2.04 shows URL escape issues with '%' characters; v0.2.03 mentions missing extensions—robust URL normalization is absent.

• Web Scraping & HTML Parsing — The entire hardseed application is built on regex-based HTML extraction from forum pages; understanding DOM traversal, regex patterns, and charset encoding is central to maintaining or extending scrapers.
• Template Method Pattern — hardseed uses TopicsListWebpage and SeedWebpage as abstract base classes with virtual methods overridden by Aicheng* and Caoliu* subclasses; this design allows adding new forums without duplicating core logic.
• HTTP Proxy Chaining & Tunneling — The --proxy flag and integration with goagent demonstrate how to route traffic through intermediate proxies to bypass geopolitical access restrictions; Webpage.cpp must handle proxy authentication and connection headers.
• Concurrent Download Management — v0.2.04 mentions '均速过低视为下载失败' (low-speed failure detection over 8s at <4KB/s) and adaptive retry logic—the image downloader implements a custom rate-limiting and backoff strategy beyond simple sequential fetching.
• URL Normalization & Escaping — v0.2.04 and v0.2.05 show bugs in % encoding and missing file extensions; proper URL RFC 3986 compliance and charset-aware escaping are non-obvious in hardseed and frequently broken.
• Pagination & State Management in Web Crawlers — AichengTopicsListWebpage and CaoliuTopicsListWebpage handle pagination (v0.2.13 fixed caoliu page-flipping bugs), maintaining page state and URL construction—essential for scraping large topic lists.
• Configuration-Driven Runtime Behavior — v0.2.12 introduced config/portals_list.json to externalize forum URLs, eliminating recompilation for address changes; this exemplifies the 'external configuration' pattern for dynamic systems.

• libtorrent/libtorrent — Complementary library for torrent parsing and magnet link handling; hardseed extracts metadata but does not parse .torrent files itself.
• yangyangwithgnu/goagent_out_of_box_yang — Direct dependency: the proxy configuration tool referenced throughout hardseed docs and required to access geographically-restricted forums.
• transmission/transmission — Industry-standard torrent client that could consume hardseed-scraped .torrent files or magnet links; shows the downstream use case.
• curl/curl — Likely underlying HTTP library used by Webpage.cpp for fetching forum pages; understanding libcurl is essential for maintaining the scraper.
• nlohmann/json — Modern JSON alternative to json11 (vendored in v0.2.12+); hardseed uses json11 for config/portals_list.json parsing.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add unit tests for src/lib/helper/Webpage.cpp web scraping utilities

The Webpage class is a critical dependency for all portal scrapers (Aicheng, Caoliu, Jandown, Rmdown). Currently there are no visible test files. Adding unit tests would catch regressions when portal HTML structures change and improve code reliability. This is especially important given the project's dependency on fragile web scraping.

• [ ] Create src/test/ directory structure with test framework (Google Test recommended for C++)
• [ ] Write test cases for Webpage.cpp methods: URL fetching, HTML parsing, and error handling
• [ ] Add mock HTTP responses for testing without live network calls
• [ ] Create build/CMakeLists.txt targets to compile and run tests
• [ ] Document test execution in README.md

Refactor src/lib/self/ portal scrapers into a plugin architecture

Currently there are 6+ portal-specific classes (Aicheng, Caoliu, Jandown, Rmdown) with parallel implementations of TopicWebpage and TopicsListWebpage variants. This violates DRY principles. A plugin system would reduce code duplication, make adding new portals easier, and improve maintainability by centralizing common scraping logic.

• [ ] Extract common interface from TopicsListWebpage.h and TopicWebpage.h into src/lib/self/PortalPlugin.h
• [ ] Create src/lib/self/PluginManager.cpp to dynamically load portal scrapers
• [ ] Refactor Aicheng*, Caoliu* classes to implement the plugin interface, reducing duplicate code
• [ ] Update config/portals_list.json schema to support plugin metadata (name, version, selector patterns)
• [ ] Update build/CMakeLists.txt to support optional portal compilation
• [ ] Update README with plugin development guide for contributors

Add GitHub Actions CI workflow for cross-platform builds and validation

The project has Windows build artifacts (bin/build_4_win.7z) and CMake configuration but no visible CI pipeline. Adding automated builds on Linux/Windows/macOS would catch platform-specific bugs early, validate dependency resolution, and improve contributor confidence. The json11 third-party library and curl/libxml2 dependencies especially need validation across platforms.

• [ ] Create .github/workflows/build.yml with matrix strategy for Linux (gcc), macOS (clang), and Windows (MSVC)
• [ ] Define dependency installation steps (libcurl, libxml2, CMake) for each platform in the workflow
• [ ] Configure build targets that compile src/main.cpp, src/lib/self/*.cpp, and src/lib/3rd/json11/json11.cpp
• [ ] Add validation step to test --help flag and basic config loading with config/portals_list.json
• [ ] Update README.md build section to reference CI status badge

• Add unit tests for Webpage.cpp HTTP parsing and regex extraction—currently only json11 has tests (test.cpp). Write test cases in src/test/ exercising sample HTML from real forum pages (create fixtures for Aicheng/Caoliu post structure).: No test coverage exists for the core HTML parsing logic; 8-year-old code is fragile without regression tests.
• Document the forum-scraper plugin architecture in DEVELOPMENT.md—explain how to add a new forum by subclassing TopicsListWebpage and SeedWebpage, with a worked example for a hypothetical 'NewForum'. Include regex pattern best practices.: README is in Chinese and sparse on architecture; new contributors cannot easily extend to additional forums.
• Audit and modernize Webpage.cpp for charset encoding (UTF-8 detection and iconv conversion if needed) and add integration tests with live forum snapshots to catch URL parsing regressions before v0.2.15.: v0.2.04 and v0.2.05 show repeated URL encoding bugs; explicit UTF-8 handling and snapshot tests would prevent silent parser failures.

The codebase presents significant security concerns centered around its primary function of scraping adult websites for torrent/seed metadata. Critical issues include unvalidated proxy configuration, potential MITM attack vectors, unsafe handling of untrusted HTML data, and likely legal/compliance violations. The web scraping architecture lacks observable security hardening, input validation, and proper error handling. Additionally, third-party dependencies are vendored without clear version management. The application is suitable only for controlled environments with proper legal review and should not be deployed in production without substantial security improvements and legal clearance.

• High · Web Scraping of Adult Content Without Proper Safeguards — src/lib/self/ (Aicheng.cpp, Caoliu.cpp, JandownSeedWebpage.cpp, RmdownSeedWebpage.cpp). The codebase is designed to scrape torrent/seed data from adult websites (Aicheng, Caoliu, Jandown, Rmdown). This raises concerns about: 1) Legal compliance with terms of service of targeted websites, 2) Potential copyright infringement, 3) No apparent rate limiting or ethical scraping practices, 4) Potential for abuse in data collection Fix: Implement rate limiting, robots.txt compliance checking, proper user-agent handling, and ensure compliance with website terms of service and applicable laws
• High · Proxy Configuration Without Validation — src/lib/helper/CmdlineOption.cpp, src/lib/helper/Webpage.cpp. The README indicates use of proxy configuration via command-line option --proxy with user-supplied address and port. No validation of proxy endpoints is apparent, potentially allowing MITM attacks or connection hijacking Fix: Implement proxy URL validation, certificate pinning for HTTPS proxy connections, and sanitize user input for proxy parameters
• Medium · Hardcoded HTTP Requests Without Certificate Validation Evidence — src/lib/helper/Webpage.cpp, src/lib/helper/Webpage.h. Webpage.cpp handles HTTP/HTTPS requests for web scraping. No visible evidence of proper SSL/TLS certificate validation or validation of remote endpoints Fix: Implement strict certificate validation, use HTTPS only, consider certificate pinning for critical endpoints, and validate all redirects
• Medium · Potential HTML Parsing and Injection Risks — src/lib/self/TopicWebpage.cpp, src/lib/self/SeedWebpage.cpp, src/lib/self/AichengTopicWebpage.cpp, src/lib/self/CaoliuTopicWebpage.cpp. Multiple webpage parsing classes (TopicWebpage, SeedWebpage) extract data from untrusted HTML sources. Risk of injection attacks if parsed data is used without sanitization in downstream processing Fix: Implement HTML entity encoding, use safe parsing libraries, validate and sanitize all extracted data, implement input bounds checking
• Medium · Third-party JSON Library (json11) Version Unknown — src/lib/3rd/json11/. json11 library is vendored but version and patch status are unclear. Potential security issues in JSON parsing could exist if library is outdated Fix: Document the specific version of json11 being used, regularly update to latest version, perform security audit of JSON parsing logic
• Low · Python Configuration Files in Source Tree — src/.ycm_extra_conf.py, src/.ycm_extra_conf.pyc. .ycm_extra_conf.py and .ycm_extra_conf.pyc present in src/ directory. These files could contain development configuration that may leak system information or paths Fix: Remove IDE configuration files from version control, add to .gitignore, use project-level secure configuration instead
• Low · No Input Validation in Command-line Parsing — src/lib/helper/CmdlineOption.cpp, src/lib/helper/CmdlineOption.h. CmdlineOption.cpp processes user input without apparent bounds checking or validation of all parameters Fix: Implement comprehensive input validation for all command-line parameters, set reasonable limits on string lengths and values

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.