If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/JoeanAmier/TikTokDownloader shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

WAIT — Single-maintainer risk — review before adopting

• Last commit 4d ago
• 3 active contributors
• GPL-3.0 licensed
• CI configured
• Tests present
• ⚠ Small team — 3 contributors active in recent commits
• ⚠ Single-maintainer risk — top contributor 82% of recent commits
• ⚠ GPL-3.0 is copyleft — check downstream compatibility

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live JoeanAmier/TikTokDownloader repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/JoeanAmier/TikTokDownloader.

What it runs against: a local clone of JoeanAmier/TikTokDownloader — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in JoeanAmier/TikTokDownloader | Confirms the artifact applies here, not a fork | | 2 | License is still GPL-3.0 | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 34 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of JoeanAmier/TikTokDownloader. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/JoeanAmier/TikTokDownloader.git
#   cd TikTokDownloader
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of JoeanAmier/TikTokDownloader and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "JoeanAmier/TikTokDownloader(\\.git)?\\b" \\
  && ok "origin remote is JoeanAmier/TikTokDownloader" \\
  || miss "origin remote is not JoeanAmier/TikTokDownloader (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(GPL-3\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"GPL-3\\.0\"" package.json 2>/dev/null) \\
  && ok "license is GPL-3.0" \\
  || miss "license drift — was GPL-3.0 at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "main.py" \\
  && ok "main.py" \\
  || miss "missing critical file: main.py"
test -f "src/application/TikTokDownloader.py" \\
  && ok "src/application/TikTokDownloader.py" \\
  || miss "missing critical file: src/application/TikTokDownloader.py"
test -f "src/interface/template.py" \\
  && ok "src/interface/template.py" \\
  || miss "missing critical file: src/interface/template.py"
test -f "src/downloader/download.py" \\
  && ok "src/downloader/download.py" \\
  || miss "missing critical file: src/downloader/download.py"
test -f "src/config/settings.py" \\
  && ok "src/config/settings.py" \\
  || miss "missing critical file: src/config/settings.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 34 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~4d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/JoeanAmier/TikTokDownloader"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

DouK-Downloader (formerly TikTokDownloader) is a free, open-source data scraping and file download tool built on HTTPX that harvests videos, images, music, live streams, comments, and metadata from TikTok and Douyin (Chinese TikTok). It supports batch downloading from user accounts, collections, live streams, and searching across both platforms while persisting data to CSV/XLSX/SQLite formats. Monolithic structure: main.py serves as the entry point; core logic split between loose modules; locale/ contains i18n files (po/mo format) for Chinese/English; docs/ holds tutorials and screenshots; .github/workflows/ automates Docker builds, releases, and executable compilation; Dockerfile enables containerized deployment.

Content researchers, archivists, and digital marketers who need to bulk-download and analyze TikTok/Douyin content across multiple accounts, trends, and search queries without relying on official APIs. Also useful for developers building data pipelines that consume social media metadata.

Actively developed and moderately mature: the project has accumulated significant GitHub stars (evident from release badges and GitHub Actions CI/CD workflows), maintains multi-language support (locale files for en_US and zh_CN), and runs automated Docker image builds and executable releases. However, the README explicitly warns of upcoming major code restructuring for stability and maintainability, signaling the authors recognize technical debt.

Medium risk: the tool relies heavily on reverse-engineering platform APIs (TikTok/Douyin), making it vulnerable to breaking changes when platforms update their endpoints or authentication. Single-maintainer project (JoeanAmier) increases maintenance risk. Large dependency surface (httpx, fastapi, lxml, pydantic, openpyxl, gmssl, qrcode) with no visible test suite in the file listing raises regression concerns. Platform scraping legality varies by jurisdiction.

The project is in active refinement mode with multiple CI/CD workflows visible (Close_Stale_Issues, manually/automatically triggered Docker image and executable builds, release pipelines). The README roadmap mentions planned code restructuring for robustness. GitHub Actions are set up for both manual builds and automated releases triggered by tags.

git clone https://github.com/JoeanAmier/TikTokDownloader.git && cd TikTokDownloader && pip install -r requirements.txt && python main.py

Daily commands: python main.py — launches interactive CLI or Web UI (based on mode selection). For Docker: docker build -t douk-downloader . && docker run -p 5000:5000 douk-downloader. Web API accessible at localhost:5000 (inferred from FastAPI setup).

• main.py — Entry point for the entire application; initializes the TikTok/Douyin downloader in terminal, server, or monitor mode.
• src/application/TikTokDownloader.py — Core orchestrator managing download workflows, output paths, and plugin initialization; defines the primary downloader class used across all modes.
• src/interface/template.py — Abstract base class for all API interface implementations; establishes the contract for request/response handling and rate limiting.
• src/downloader/download.py — Handles file downloads with retry logic, streaming, and media type validation; critical for actual content retrieval.
• src/config/settings.py — Global configuration loader for credentials, paths, proxies, and feature flags; required by nearly every module.
• src/encrypt/aBogus.py — Implements TikTok's aBogus encryption algorithm; essential for API request signing and authentication.
• src/extract/extractor.py — Parses API responses and extracts metadata (video info, user data, comments); transforms raw API data into structured objects.

Add Support for a New Content Type (e.g., Reels, Playlists)

1. Create a new interface file in src/interface/ (e.g., src/interface/reels.py) that inherits from src/interface/template.py (src/interface/template.py)
2. Implement the required API endpoint methods and response parsing using src/extract/extractor.py patterns (src/extract/extractor.py)
3. Register the new interface in src/application/TikTokDownloader.py by adding it to the plugin initialization logic (src/application/TikTokDownloader.py)
4. Add menu option in src/application/main_terminal.py to allow users to select the new content type (src/application/main_terminal.py)
5. If API requires new encryption, add methods to src/encrypt/aBogus.py or create a new file in src/encrypt/ (src/encrypt/aBogus.py)

Add a New Export Format (e.g., Database, Cloud Storage)

1. Create new export handler in src/cli_edition/write.py or a new file with methods for serializing metadata (src/cli_edition/write.py)
2. Hook the handler into src/config/settings.py to read format preferences from config files (src/config/settings.py)
3. Call the handler from src/application/TikTokDownloader.py after download completion (src/application/TikTokDownloader.py)

Add a New API Signing Algorithm or Update Existing One

1. Update or create encryption file in src/encrypt/ (e.g., src/encrypt/aBogus.py for new TikTok signatures) (src/encrypt/aBogus.py)
2. Update src/interface/template.py to call the new signing method in the request pre-processing hook (src/interface/template.py)
3. Store any new constants or keys in src/config/parameter.py (src/config/parameter.py)
4. Test by running any interface client (e.g., src/interface/user.py) to verify signed requests succeed (src/interface/user.py)

• FastAPI — Provides async HTTP server for remote operations; allows web UI and scheduled tasks to coexist with CLI.
• httpx[socks] — Modern async HTTP client with SOCKS proxy support; necessary for bypassing geographic restrictions and rate limits.
• aiosqlite + aiofiles — Non-blocking I/O for database and file operations; enables concurrent downloads without threading overhead.
• lxml + Pydantic — Fast HTML/XML parsing and strict data validation; ensures API responses conform to expected schemas.
• openpyxl + emoji — Excel export with rich formatting; emoji support for metadata readability in output.
• gmssl — Chinese cryptography standard; required for some Douyin-specific API signing if applicable.

• Synchronous encryption algorithms (aBogus, xBogus) in async context

  • Why: TikTok's signature algorithms have no official async implementations and must run in-process.
  • Consequence: Encryption operations block the event loop briefly (~10ms); mitigated by offloading to thread pool in high-concurrency scenarios.

• Multiple API interface classes per platform (detail.py, detail_tiktok.py, etc.)

  • Why: TikTok and Douyin have divergent API contracts despite code reuse at the template level.
  • Consequence: Code duplication; mitigated by inheritance and shared utility functions in src/custom/.

• Cookie-based authentication instead of OAuth

  • Why: TikTok/Douyin do not expose OAuth endpoints for third-party access; manual cookie capture is only viable method.
  • Consequence: Users must extract cookies themselves and refresh them periodically; increases friction but preserves user privacy.

• Single-threaded downloader queue with retry logic

  • Why: Simplifies state management and rate-limit compliance; avoids complex locking.
  • Consequence: Large bulk downloads are slower; users can spawn multiple instances or use batch mode for parallelism.

• Does not handle user account creation or OAuth authentication.
• Does not upload or modify content on TikTok/Douyin; read-only downloader.
• Does not provide real

Authentication: Most features require valid TikTok/Douyin cookies extracted from browser (documented in docs/Cookie获取教程.md); missing/expired cookies will cause silent or cryptic failures. Platform version lock: reverse-engineered API contracts break when TikTok/Douyin update; no version pinning against platform APIs. Device ID requirement: Some endpoints require device_id (see device_id获取示例图.png); obtaining this is non-obvious. Proxy requirement: CN users may need proxy support (SOCKS config via httpx); misconfiguration hangs requests. No visible async context in main.py—confirm event loop management if extending async logic.

• Reverse-engineered API scraping — This tool doesn't use official TikTok/Douyin APIs; instead it mimics browser HTTP requests to undocumented endpoints, making it fragile but unrestricted—understanding request/response contracts and User-Agent spoofing is essential for debugging platform breakage.
• Cookie-based authentication without OAuth — TikTok/Douyin scraping requires browser-extracted session cookies rather than token-based auth; this introduces manual extraction friction and expiry management not typical of API-first tools.
• Async concurrent downloads with HTTPX — The tool uses aiofiles and aiosqlite for concurrent I/O; understanding asyncio context, connection pooling in HTTPX, and backpressure handling is critical for optimizing batch download performance without overwhelming servers.
• gettext i18n (PO/MO format) — Localization is done via GNU gettext files (locale/*.po/.mo) compiled by po_to_mo.py; contributors adding features must mark strings for translation using gettext patterns.
• Polyglot data export (CSV/XLSX/SQLite) — The tool persists scraped data in three formats simultaneously; understanding schema mapping, transaction isolation (SQLite), and lazy writing to avoid memory explosion during bulk harvests is essential.
• SOCKS proxy tunneling — CN users often require SOCKS proxies to bypass regional blocks; httpx[socks] dependency enables this, but misconfigured proxies fail silently in async contexts.
• FastAPI + Uvicorn Web UI architecture — The tool offers both CLI and Web API modes via FastAPI; understanding async route handlers, Pydantic request/response models, and CORS/proxy headers is needed to extend the Web UI.

• davidteather/TikTok-Api — Predecessor/inspiration for reverse-engineered TikTok API scraping; uses similar HTTPX patterns for unofficial API calls
• yt-dlp/yt-dlp — Similar bulk video downloader architecture; demonstrates pluggable extractors and robust manifest parsing (applicable to TikTok/Douyin video formats)
• inAmnesia/TikTok_bot — Alternative Douyin scraper; cross-reference for CN platform-specific quirks and cookie handling
• tmct/TikTok-Crawler — Competing TikTok scraper; reference for battle-tested API endpoint mapping and anti-bot workarounds
• movements-app/movements — Data visualization + download orchestration for social platforms; reference for combining FastAPI Web UI with bulk download pipelines

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add unit tests for src/custom/function.py and src/custom/internal.py

The repo has utility functions in src/custom/ but no visible test directory. Given the complexity of TikTok/Douyin data scraping and the presence of helper functions (function.py, internal.py, static.py), adding unit tests would improve reliability and catch regressions when the upstream APIs change. This is critical for a scraping tool.

• [ ] Create tests/ directory at repo root with init.py
• [ ] Add tests/test_custom_functions.py to test src/custom/function.py utilities
• [ ] Add tests/test_custom_internal.py to test src/custom/internal.py logic
• [ ] Add a GitHub Actions workflow in .github/workflows/Run_Tests.yml to run pytest on push/PR
• [ ] Update requirements.txt to include pytest and pytest-asyncio for async test support

Add comprehensive API documentation for src/application/main_server.py endpoints

The repo exposes FastAPI endpoints via main_server.py but there's no docs/API_Documentation.md file explaining available endpoints, request/response schemas, and usage examples. Given the WebAPI mode shown in screenshots, contributors and users need clear documentation of the REST API contract.

• [ ] Create docs/API_Documentation.md with sections for each FastAPI endpoint in src/application/main_server.py
• [ ] Document request parameters, response schemas (reference Pydantic models from src/config/settings.py if applicable)
• [ ] Include cURL/Python examples for common operations (download video, fetch user data, search)
• [ ] Add troubleshooting section for common errors
• [ ] Link API docs from README.md and README_EN.md

Refactor src/application into feature modules and add dependency injection pattern

The src/application/ directory has multiple monolithic files (TikTokDownloader.py, main_server.py, main_terminal.py, main_monitor.py) that likely share common logic. Refactoring into domain-specific modules (e.g., src/application/downloader/, src/application/api/) with a shared service layer would reduce code duplication and make the codebase more maintainable. This is especially important given the complexity of supporting both TikTok and Douyin.

• [ ] Create src/application/services/ directory for shared business logic (auth, API calls, data processing)
• [ ] Extract common scraping logic from TikTokDownloader.py into src/application/services/scraper.py
• [ ] Create src/application/handlers/ for separate protocol handlers (terminal, server, monitor)
• [ ] Update main.py and src/cli_edition/main_cli.py to use refactored modules
• [ ] Add integration tests in tests/test_application_integration.py to ensure refactoring doesn't break workflows

• Add unit tests for export modules (CSV/XLSX/SQLite writers in locale/po_to_mo.py and aiosqlite wrappers)—currently no test/ directory visible, making regressions likely.
• Enhance error messages in cookie validation—docs show cookie screenshots but code likely lacks clear feedback when cookies are invalid/expired; wrap httpx auth errors with actionable hints.
• Document the internal data model/schema for video/user/comment objects—README lists features but no schema docs visible; create schema.md defining Pydantic models used by export writers.

• High · Potential Credential Exposure in Configuration — src/config/, src/encrypt/. The project appears to handle sensitive data like cookies, device IDs, and authentication tokens based on file names (device_id.py, verifyFp.py, etc.) and documentation references to 'Cookie获取教程' (Cookie retrieval tutorial). No evidence of secure credential storage mechanisms (e.g., environment variables, secrets management) is visible in the provided structure. Fix: Implement secure credential management using environment variables, .env files with proper .gitignore rules, or a secrets management system. Never hardcode credentials or tokens in source code. Use libraries like python-dotenv and ensure secrets are excluded from version control.
• High · Outdated or Vulnerable Dependency: lxml==6.1.0 — requirements.txt (lxml==6.1.0). lxml version 6.1.0 may have known XML processing vulnerabilities (XXE - XML External Entity attacks) if not properly configured. The library is used for web scraping and data extraction, which increases exposure risk. Fix: Update lxml to the latest stable version (4.9.4 or higher). Ensure XML parsing is configured with external entity processing disabled to prevent XXE attacks. Use defusedxml as an additional safety layer for untrusted input.
• High · FastAPI Application Running on Exposed Port Without Security Headers — Dockerfile (EXPOSE 5555), src/application/main_server.py. The Docker configuration exposes port 5555 for the FastAPI web server without visible security headers configuration (CORS, CSP, X-Frame-Options, etc.). The application handles user data and downloads, which could be vulnerable to web-based attacks. Fix: Implement FastAPI middleware for security headers (use fastapi-cors, add middleware for HSTS, X-Frame-Options, X-Content-Type-Options). Bind the application to localhost or a specific IP rather than 0.0.0.0. Require authentication and implement rate limiting. Add HTTPS/TLS termination in production.
• High · Insecure Deserialization and Data Processing — src/extract/extractor.py, src/downloader/download.py, src/custom/function.py. The extractor module (src/extract/extractor.py) and downloader module likely process JSON responses from TikTok/Douyin APIs without visible validation. Pydantic is used but the extent of input validation is unclear. Malicious API responses could lead to injection attacks or remote code execution. Fix: Implement strict Pydantic models with comprehensive validation for all API responses. Use type hints and validation decorators. Never use pickle or eval() on untrusted data. Sanitize all extracted URLs and file paths before use.
• Medium · Missing Input Validation on File Operations — src/downloader/download.py. The downloader module performs file operations (writing downloads to VOLUME). Without visible path traversal protections, malformed filenames from API responses could allow directory traversal attacks (../../../ sequences in filenames). Fix: Implement strict path validation and sanitization. Use os.path.normpath() and verify paths resolve within the intended download directory. Whitelist allowed characters in filenames. Reject any paths containing '..' or absolute paths.
• Medium · Unencrypted Data Storage in SQLite Database — aiosqlite==0.22.1 dependency, database-related modules. The project uses aiosqlite for database operations. Cookie data, user IDs, and authentication tokens may be stored without encryption, making the database file a security risk if compromised. Fix: Implement database-level encryption (SQLCipher for SQLite) for sensitive data. Encrypt sensitive fields at the application level using cryptography library. Never store unencrypted authentication tokens or cookies. Implement proper access controls on database files.
• Medium · Insufficient HTTPX Configuration for External Requests — httpx[socks]==0.28.1 dependency. The httpx library with SOCKS support is used for making requests to external TikTok/Douyin APIs. There's no visible certificate verification, timeout, or connection pooling limits configuration, which could lead to MITM attacks or DoS vulnerabilities. Fix: Configure httpx with: proper SSL certificate verification (verify=True), reasonable connection timeouts, connection pool limits,

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.