AVOID — Stale and unlicensed — last commit 2y ago

• 3 active contributors
• Tests present
• ⚠ Stale — last commit 2y ago
• ⚠ Small team — 3 contributors active in recent commits
• ⚠ Single-maintainer risk — top contributor 81% of recent commits
• ⚠ No license — legally unclear to depend on
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

A comprehensive educational Python web scraping repository demonstrating practical techniques for extracting data from Chinese platforms (Taobao, JD.com, Netease Music, Bilibili, 12306, TikTok, and fiction sites). It provides working implementations of common scraping patterns including login automation, dynamic content handling, subtitle conversion, and media downloads—organized as both standalone scripts and Scrapy projects. Monolithic collection of independent scrapers: top-level scripts (biqukan.py, 12306.py, Netease/Netease.py) for single-target examples, organized subdirectories for complex projects (bilibili/, cartoon/ with Scrapy framework, 2020/ containing latest tutorials with platform-specific folders). No shared library—each scraper reimplements common patterns like HTTP headers, parsing, and file I/O.

Python learners and junior developers building web scraping skills who need concrete, runnable examples targeting real websites. It's also useful for Chinese internet researchers studying e-commerce, streaming, and content platforms.

Actively maintained but entry-level in production quality. The repo has substantial content (257KB Python code) across multiple platforms, but the 2020 reorganization and lack of visible CI/test infrastructure suggest educational focus over robustness. Recent structure indicates active teaching (2020/ subdirectory with latest tutorials) rather than production-grade tooling.

High maintenance burden due to website scraping volatility—any target site's HTML structure or anti-bot measures will break scrapers without automatic testing or CI validation. Single maintainer (Jack-Cherish) with fragmented codebase (no shared utilities), and explicit legal warnings about Chinese crawler prosecution indicate jurisdictional risk. Minimal dependencies (beautifulsoup4, Pillow, imageio) reduce supply-chain risk but also mean no framework standardization across examples.

Primary activity in 2020/ directory with modular tutorials (api/, bilibili/, dmzj/, taobao/, xbqg/, zycjw/). The 2020 reorganization reflects shift toward cleaner examples. Original root-level scripts appear stable/stale. No visible PR activity or issue tracking in provided metadata—appears to be documentation-first project with periodic tutorial additions.

Clone and explore platform-specific scripts directly: git clone https://github.com/Jack-Cherish/python-spider.git && cd python-spider && pip install beautifulsoup4 Pillow imageio numpy. Then run a simple example: python3 biqukan.py (fiction download) or cd bilibili && python3 bilibili.py (Bilibili scraper). No virtual environment or install script evident—install dependencies globally or in a venv.

Daily commands: No global entry point. Run platform-specific scripts: python3 2020/api/api.py (API server example), python3 2020/bilibili/download.py (Bilibili downloader), python3 2020/taobao/taobao_login.py (Taobao login demo), or scrapy crawl comic_spider from cartoon/ directory (if Scrapy installed). Most scripts are designed as one-off runners, not services.

• README.md — Main documentation and disclaimer; entry point explaining the repository's educational purpose and legal warnings for web scraping activities.
• 2020/bilibili/download.py — Core modern spider implementation for Bilibili; represents the 2020 refactored architecture that all new spiders should follow.
• bilibili/bilibili.py — Legacy Bilibili spider implementation; demonstrates the pre-2020 approach and common patterns used across older scrapers.
• cartoon/cartoon/spiders/comic_spider.py — Scrapy-based spider framework; shows the repository's structured scraping pattern for large-scale data extraction.
• 2020/taobao/taobao_login.py — Session management and authentication patterns; critical for understanding how to handle login-protected platforms.
• douyin_pro.py — Advanced anti-detection techniques; demonstrates JavaScript obfuscation bypass and request header spoofing for modern platforms.

Add a New Platform Spider (Modern 2020 Pattern)

1. Create new directory in 2020/ folder: 2020/platform_name/ (2020/platform_name/spider.py)
2. Implement core scraper with requests session, headers spoofing, and pagination logic (reference 2020/taobao/taobao_login.py for session handling) (2020/platform_name/spider.py)
3. Add platform-specific utilities (e.g., video conversion, subtitle parsing) following 2020/bilibili/xml2ass.py pattern (2020/platform_name/utils.py)
4. Create README.md with usage examples, data format, and legal disclaimers (2020/platform_name/README.md)

Add Anti-Detection Protection to Existing Spider

1. Study the proxy rotation pattern in daili.py and integrate proxy middleware (daili.py)
2. Add CAPTCHA handling following geetest.py pattern if platform uses image-based CAPTCHAs (geetest.py)
3. For JavaScript-heavy platforms, implement rendering bypass following douyin_pro.py approach or integrate Selenium/Playwright (douyin_pro.py)
4. Add realistic request headers and delays matching human browsing patterns in your spider's request loop (2020/platform_name/spider.py)

Create a Scrapy-Based Enterprise Spider

1. Create new Scrapy project structure mirroring cartoon/ folder: project/project/spiders/, items.py, pipelines.py, settings.py (cartoon/cartoon/spiders/comic_spider.py)
2. Define item models in items.py for your target data (images, text, metadata) (cartoon/cartoon/items.py)
3. Implement spider class in spiders/your_spider.py inheriting from scrapy.Spider with parse() methods (cartoon/cartoon/spiders/comic_spider.py)
4. Configure middleware in settings.py for user agent rotation, proxy handling, and request delays (reference cartoon/cartoon/settings.py) (cartoon/cartoon/settings.py)
5. Implement pipelines.py for post-processing: image downloads, database inserts, validation (cartoon/cartoon/pipelines.py)

Add Video Download Capability

1. Extract video URL/metadata from platform HTML using BeautifulSoup or Selenium (2020/bilibili/download.py)
2. Use video_downloader utilities for format handling and ffmpeg integration (reference video_downloader/ modules) (2020/zycjw/video_download.py)
3. If platform has subtitles, convert formats using xml2ass.py pattern for ASS/SRT output (2020/bilibili/xml2ass.py)
4. Implement concurrent downloads with thread pools to improve throughput while respecting rate limits (downloader.py)

• Python 3 + requests/urllib — Lightweight, easy to learn HTTP scraping without framework overhead; suitable for quick platform-specific scripts.
• BeautifulSoup4 — Fast HTML parsing for extracting structured data from static pages; lower memory footprint than full DOM rendering.
• Scrapy Framework — Enterprise-grade framework for large-scale scraping projects; provides built-in middleware, pipelines, and distributed crawling support (cartoon/ project).
• Selenium/JavaScript execution — Required for JavaScript-heavy platforms (Douyin/TikTok, modern React sites); enables dynamic content rendering before parsing.
• imageio + Pillow + numpy — Image processing for downloaded content; format conversion, quality adjustment, QR code generation (video_downloader/MyQR/).

• Standalone scripts (biqukan.py, douyin.py) vs. Scrapy framework (cartoon/)

  • Why: Standalone scripts are faster to prototype and single-platform focused; Scrapy adds overhead but enables scaling to multi-platform crawling.
  • Consequence: Simple scripts break easily when platform HTML changes; Scrapy projects require more initial setup but are more maintainable at scale.

• HTTP client requests vs. headless browser (Selenium/Playwright)

  • Why: Pure HTTP is 10-100x faster but doesn't execute JavaScript; browsers handle JS but are slow and resource-intensive.
  • Consequence: Legacy platforms work with HTTP; modern platforms (TikTok, 12306) require browser automation, increasing complexity and memory usage.

• Proxy rotation (daili.py) + random delays vs. respecting robots.txt

  • Why: Anti-detection enables bypassing IP bans and rate limits; respecting robots.txt is ethical but limits data access.
  • Consequence: Anti-detection increases legal/ethical risk and arms race with platforms; respecting robots.txt allows only public data.

Target sites actively detect and block scrapers: Taobao/JD use JavaScript rendering (scripts don't import selenium by default—likely manual verification needed). Netease Music uses encryption tokens (see Netease/Netease.py) requiring reverse-engineering. 12306 and Bilibili have CAPTCHA and IP blocking. No proxies or delay logic in most scripts—rapid execution will trigger 403/429 responses. Fiction sites (biqukan, xbqg) frequently change domain/structure. xml2ass subtitle conversion requires FFmpeg binaries not listed in dependencies. No error handling patterns shown—scripts will crash mid-scrape without recovery. Chinese encoding (utf-8) assumed throughout but not explicit in file headers.

• HTML/CSS Selector-based Web Scraping — BeautifulSoup is the primary tool across this repo; understanding CSS selectors and DOM traversal is essential to modify any parser without breaking on site changes
• Session Management & Cookie Handling — Taobao and Netease examples require persistent login sessions; understanding requests.Session() and cookie jars is critical for authenticated scraping
• Rotating Proxy IP Pools — daili.py and rate-limiting strategies prevent IP bans; understanding proxy rotation patterns and lifecycle management is necessary for large-scale scraping
• JavaScript Rendering & Dynamic Content — Bilibili and Taobao load content via JavaScript; recognizing when static parsing fails and when to use Selenium/Playwright is a key debugging skill
• Scrapy Framework (Spiders, Pipelines, Middleware) — cartoon/ project uses Scrapy's architecture; understanding Item definitions, CrawlSpider rules, and output pipelines is needed to extend multi-site crawlers
• Anti-Bot Detection & Rate Limiting — Many target sites use CAPTCHA, IP blocking, and User-Agent filtering; this repo implicitly assumes manual verification or token generation, a hidden complexity not fully documented
• Encoding & Character Set Handling — Chinese content throughout (Taobao, JD, fiction sites); understanding UTF-8 encoding, BOM strips, and codec errors prevents silent data corruption

• requests/requests — Core HTTP library underlying all scrapers in this repo; essential to understand for debugging connection issues
• scrapy/scrapy — The cartoon/ Scrapy project is a direct use case; understanding Scrapy's architecture helps maintain the more complex spiders
• psf/beautifulsoup4 — HTML parsing library used throughout the repo; all BeautifulSoup-based scripts depend on its CSS selector and navigation APIs
• luminati-io/luminati-proxy — Companion for daili.py proxy pooling; users needing production-grade proxy rotation would evaluate this over manual implementation
• 0xHJK/music-dl — Alternative music scraper using similar patterns to Netease/Netease.py; shows parallel approach to streaming platform extraction

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Create a unified requirements.txt and add dependency pinning for Python version compatibility

The repo has scattered spider modules with inconsistent dependency management. The dependencies snippet shows only high-level packages (imageio, numpy, Pillow, beautifulsoup4) but individual scripts like douyin.py, bilibili/bilibili.py, and 2020/api/api.py likely have undocumented dependencies on requests, selenium, or other libraries. This makes onboarding difficult and creates reproducibility issues. A comprehensive requirements.txt with pinned versions would help new contributors run any spider without guessing dependencies.

• [ ] Analyze imports across all .py files in root, bilibili/, douyin/, 2020/, and cartoon/ directories
• [ ] Create a master requirements.txt with all identified packages and pinned versions (e.g., requests==2.28.1)
• [ ] Create optional requirements-dev.txt for testing tools (pytest, etc.)
• [ ] Update README.md with clear 'Installation' section referencing pip install -r requirements.txt
• [ ] Test that each major spider (douyin.py, bilibili/bilibili.py, 2020/api/api.py, Netease/Netease.py) can import successfully

Add unit tests for core spider modules with pytest fixtures

The repo has no visible test suite despite containing multiple spider modules. Files like bilibili/bilibili.py, douyin.py, Netease/Netease.py, and 2020/xbqg/xbqg_spider.py would benefit from at least basic unit tests to verify parsing logic and API calls work correctly. This catches regressions when target websites change their HTML structure or API endpoints.

• [ ] Create tests/ directory at repo root with init.py
• [ ] Add tests/test_bilibili.py with mocked requests to test bilibili/bilibili.py's page parsing and video ID extraction
• [ ] Add tests/test_netease.py with mocked API responses for Netease/Netease.py music metadata extraction
• [ ] Add tests/conftest.py with pytest fixtures for sample HTML/JSON responses from each platform
• [ ] Create a pytest.ini or setup.cfg with test configuration
• [ ] Update README.md with 'Running Tests' section (e.g., pytest tests/)

Consolidate duplicate spider code in 2020/ and root directories into a shared utils module

The repo has parallel implementations: root-level douyin.py vs 2020/api/api.py, bilibili/bilibili.py vs 2020/bilibili/download.py, and biqukan.py vs 2020/xbqg/xbqg_spider.py. These contain duplicated logic for HTTP requests, HTML parsing, and error handling. Extracting common patterns (header injection, HTML parsing helpers, download utilities) into a utils module reduces maintenance burden and makes the codebase more modular.

• [ ] Create spider_utils/ package at repo root with init.py, http_utils.py, parse_utils.py, and download_utils.py
• [ ] In http_utils.py, add a Session wrapper class with shared headers and retry logic used by bilibili.py and douyin.py
• [ ] In parse_utils.py, add helper functions for common BeautifulSoup patterns (extract_by_selector, extract_metadata, etc.)
• [ ] In download_utils.py, consolidate download logic from 2020/bilibili/download.py and 2020/zycjw/video_download.py
• [ ] Refactor douyin.py, bilibili/bilibili.py, and Netease/Netease.py to import and use spider_utils functions
• [ ] Update README.md with a 'Project Structure' section explaining the utils module

• Add retry logic and delay handling to root-level scripts: biqukan.py, shuaia.py, and baiduwenku.py lack request retries and rate-limiting delays. Create a shared utils.py with @retry decorator and requests.Session wrapper, then refactor existing scripts to use it.
• Add unit tests for BeautifulSoup parsing in biqukan.py and Netease.py: No tests directory exists. Mock HTML responses and verify chapter/track extraction works before running live. Would prevent silent failures when site HTML structure changes.
• Document 2020/ platform examples with working setup instructions: 2020/api/, 2020/taobao/, 2020/bilibili/ lack README files explaining prerequisites (dependencies, login credentials, API keys). Add step-by-step walkthroughs for each.

• High · Outdated and Vulnerable Dependencies — Dependencies specification (imageio, numpy, Pillow, beautifulsoup4). The project uses outdated versions of dependencies (Pillow, numpy, beautifulsoup4) without version pinning. These libraries have known vulnerabilities in older versions. Pillow has had multiple CVEs related to image processing, and beautifulsoup4 may have parsing vulnerabilities. Fix: Pin dependencies to specific secure versions in requirements.txt. Example: 'Pillow>=10.0.0', 'beautifulsoup4>=4.12.0'. Regularly audit dependencies using tools like 'pip-audit' or 'safety'.
• High · Web Scraping Without Rate Limiting or User-Agent Validation — 2020/taobao/taobao_login.py, bilibili/bilibili.py, douyin/douyin.py, and similar spider files. Multiple spider scripts (taobao_login.py, bilibili.py, douyin.py, etc.) perform web scraping without visible rate limiting, proper User-Agent headers, or respect for robots.txt. This could lead to IP blocking, legal issues, and potential DoS-like behavior against target servers. Fix: Implement rate limiting with delays between requests. Use proper User-Agent headers. Respect robots.txt and terms of service. Consider using legitimate APIs where available. Add request timeouts and retry logic with exponential backoff.
• High · Potential Credential Exposure in Web Scraping — 2020/taobao/taobao_login.py, 12306.py. File 'taobao_login.py' and other login-related scripts may handle credentials. If credentials are hardcoded, logged, or stored insecurely, this poses a significant security risk. The codebase shows login automation which could involve storing sensitive authentication data. Fix: Never hardcode credentials. Use environment variables or secure credential management systems. Avoid logging sensitive data. Use OAuth/API tokens instead of passwords where possible. Implement proper session management.
• Medium · Missing Input Validation and Sanitization — All spider files (xbqg/xbqg_spider.py, baiwan/baiwan.py, cartoon/cartoon/spiders/comic_spider.py, etc.). Web scraping scripts likely parse HTML/JSON responses without sufficient validation. This could lead to injection attacks if user input is not properly validated before being used in requests or if parsed data is used unsafely. Fix: Validate and sanitize all external input and parsed data. Use parameterized requests. Avoid string concatenation for URLs. Use libraries like 'validators' to validate URLs and data formats.
• Medium · Insecure File Operations — 2020/bilibili/download.py, video_downloader/downloader.py, baiwan/baiwan.py. The project includes file download functionality (download.py, video_downloader/). Without proper validation of file paths and names, this could lead to directory traversal attacks or arbitrary file writes. Fix: Validate and sanitize file paths using 'pathlib' or 'os.path.abspath()'. Implement whitelisting for allowed download directories. Verify file names don't contain path traversal sequences (../, .., etc.).
• Medium · Potential Code Injection via JavaScript Execution — douyin/fuck-byted-acrawler.js, douyin/douyin.py, douyin_pro.py. File 'douyin/fuck-byted-acrawler.js' suggests JavaScript execution for anti-crawling bypass. Dynamic code execution without proper sandboxing or validation could lead to code injection vulnerabilities. Fix: Avoid executing untrusted JavaScript code. If necessary, use a sandboxed JavaScript environment. Validate and sanitize any dynamic code. Consider using legitimate APIs instead.
• Medium · No HTTPS Verification — All files using requests library (implied from web scraping pattern). HTTP requests in spider scripts may not verify SSL/TLS certificates by default, making them vulnerable to man-in-the-middle (MITM) attacks. Fix: Ensure SSL/TLS certificate verification is enabled. Use 'requests.Session()' with 'verify=True' (default). Pin certificates for critical endpoints if needed.
• Low · undefined — undefined. undefined Fix: undefined

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/Jack-Cherish/python-spider shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live Jack-Cherish/python-spider repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/Jack-Cherish/python-spider.

What it runs against: a local clone of Jack-Cherish/python-spider — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in Jack-Cherish/python-spider | Confirms the artifact applies here, not a fork | | 2 | Default branch master exists | Catches branch renames | | 3 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 4 | Last commit ≤ 658 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of Jack-Cherish/python-spider. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/Jack-Cherish/python-spider.git
#   cd python-spider
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of Jack-Cherish/python-spider and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "Jack-Cherish/python-spider(\\.git)?\\b" \\
  && ok "origin remote is Jack-Cherish/python-spider" \\
  || miss "origin remote is not Jack-Cherish/python-spider (artifact may be from a fork)"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "2020/bilibili/download.py" \\
  && ok "2020/bilibili/download.py" \\
  || miss "missing critical file: 2020/bilibili/download.py"
test -f "bilibili/bilibili.py" \\
  && ok "bilibili/bilibili.py" \\
  || miss "missing critical file: bilibili/bilibili.py"
test -f "cartoon/cartoon/spiders/comic_spider.py" \\
  && ok "cartoon/cartoon/spiders/comic_spider.py" \\
  || miss "missing critical file: cartoon/cartoon/spiders/comic_spider.py"
test -f "2020/taobao/taobao_login.py" \\
  && ok "2020/taobao/taobao_login.py" \\
  || miss "missing critical file: 2020/taobao/taobao_login.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 658 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~628d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/Jack-Cherish/python-spider"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.