If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/ssssssss-team/spider-flow shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

WAIT — Stale — last commit 3y ago

• 15 active contributors
• MIT licensed
• ⚠ Stale — last commit 3y ago
• ⚠ Concentrated ownership — top contributor handles 63% of recent commits
• ⚠ No CI workflows detected
• ⚠ No test directory detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live ssssssss-team/spider-flow repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/ssssssss-team/spider-flow.

What it runs against: a local clone of ssssssss-team/spider-flow — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in ssssssss-team/spider-flow | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 1088 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of ssssssss-team/spider-flow. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/ssssssss-team/spider-flow.git
#   cd spider-flow
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of ssssssss-team/spider-flow and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "ssssssss-team/spider-flow(\\.git)?\\b" \\
  && ok "origin remote is ssssssss-team/spider-flow" \\
  || miss "origin remote is not ssssssss-team/spider-flow (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "spider-flow-core/src/main/java/org/spiderflow/core/Spider.java" \\
  && ok "spider-flow-core/src/main/java/org/spiderflow/core/Spider.java" \\
  || miss "missing critical file: spider-flow-core/src/main/java/org/spiderflow/core/Spider.java"
test -f "spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/ProcessExecutor.java" \\
  && ok "spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/ProcessExecutor.java" \\
  || miss "missing critical file: spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/ProcessExecutor.java"
test -f "spider-flow-core/src/main/java/org/spiderflow/core/expression/DefaultExpressionEngine.java" \\
  && ok "spider-flow-core/src/main/java/org/spiderflow/core/expression/DefaultExpressionEngine.java" \\
  || miss "missing critical file: spider-flow-core/src/main/java/org/spiderflow/core/expression/DefaultExpressionEngine.java"
test -f "spider-flow-api/src/main/java/org/spiderflow/model/Shape.java" \\
  && ok "spider-flow-api/src/main/java/org/spiderflow/model/Shape.java" \\
  || miss "missing critical file: spider-flow-api/src/main/java/org/spiderflow/model/Shape.java"
test -f "spider-flow-api/src/main/java/org/spiderflow/executor/ShapeExecutor.java" \\
  && ok "spider-flow-api/src/main/java/org/spiderflow/executor/ShapeExecutor.java" \\
  || miss "missing critical file: spider-flow-api/src/main/java/org/spiderflow/executor/ShapeExecutor.java"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 1088 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~1058d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/ssssssss-team/spider-flow"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Spider-Flow is a no-code visual web scraping platform that allows users to define crawler workflows using a drag-and-drop flowchart interface instead of writing code. It supports XPath/JsonPath/CSS selectors, handles JS-rendered pages via Selenium plugin, manages proxies, auto-saves to databases/files, and executes via Spring Boot with multi-source SQL support. Maven multi-module monorepo: spider-flow-api/ is the core API module containing executors (executor/), expression engine (expression/), threading models (concurrent/), and context management (context/). The main application modules (not fully listed) build on this via plugin architecture with ShapeExecutor and FunctionExtension interfaces. Data persists to SQL via spiderflow.sql schema.

Data engineers, business analysts, and non-technical users who need to scrape web data at scale without writing code; also developers extending the platform via custom executors (ShapeExecutor) and functions (FunctionExecutor) in the plugin ecosystem.

Actively developed (v0.5.0 as of pom.xml), with a published demo site (demo.spiderflow.org), comprehensive feature set (7+ official plugins listed), and documentation site (spiderflow.org). The codebase shows production-ready architecture (Spring Boot 2.0.7, Quartz scheduling, WebSocket support), though test coverage is not visible in the file list.

Moderate risk: single-maintainer appearance (ssssssss-team), dependency on FastJSON 1.2.83 (older version with known CVEs), and Spring Boot 2.0.7 is EOL (released 2018). Plugin ecosystem relies on external repos (selenium, redis, mongodb plugins) which may not be actively maintained. No CI/CD configuration visible in provided files.

Version 0.5.0 is current; no specific PR/issue data provided, but the project maintains active documentation (changelog.html linked) and 7 official plugins in active maintenance across Redis, Selenium, MongoDB, OSS, OCR, email, and proxy pool integrations.

Clone and build: git clone https://gitee.com/ssssssss-team/spider-flow.git && cd spider-flow && mvn clean install. Then import spider-flow-api and run the main application (exact module depends on which variant you need; frontend is in spider-flow-vue repo). Database setup: load db/spiderflow.sql into MySQL/compatible DB and set spring.datasource properties.

Daily commands: Exact startup command not in provided snippets, but standard Spring Boot: mvn spring-boot:run from the main module, or java -jar spider-flow-xxx.jar after packaging. WebSocket requires websocket dependency (present in pom.xml). Access UI at http://localhost:8080 (frontend served by spider-flow-vue).

• spider-flow-core/src/main/java/org/spiderflow/core/Spider.java — Main Spider execution engine that orchestrates the entire flow execution lifecycle and is the entry point for all crawling operations.
• spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/ProcessExecutor.java — Core shape executor that processes workflow nodes; essential for understanding how individual steps in a spider flow are executed.
• spider-flow-core/src/main/java/org/spiderflow/core/expression/DefaultExpressionEngine.java — Expression evaluation engine used throughout the codebase to interpret dynamic expressions in workflow definitions.
• spider-flow-api/src/main/java/org/spiderflow/model/Shape.java — Domain model representing workflow nodes; fundamental to understanding the spider flow data structure and serialization.
• spider-flow-api/src/main/java/org/spiderflow/executor/ShapeExecutor.java — Abstract interface defining the contract for executing different workflow node types; key extension point for new functionality.
• spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/RequestExecutor.java — Executes HTTP requests in workflows; critical for web scraping functionality that is central to the spider-flow platform.
• pom.xml — Maven build configuration defining all project dependencies and module structure; necessary for setting up the development environment.

Add a New Shape Executor (Custom Workflow Node Type)

1. Create a new executor class implementing ShapeExecutor interface in spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/ (spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/YourCustomExecutor.java)
2. Implement execute(Shape shape, SpiderContext context) method with your custom logic, and register supported shape types via getName() (spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/YourCustomExecutor.java)
3. Optionally add annotations (@Comment, @Example, @Return) from spider-flow-api/src/main/java/org/spiderflow/annotation/ for documentation (spider-flow-api/src/main/java/org/spiderflow/annotation/Comment.java)
4. Register your executor in Spider.java or via plugin loading mechanism so it's picked up during workflow execution (spider-flow-core/src/main/java/org/spiderflow/core/Spider.java)

Add a New Built-in Function

1. Create a new FunctionExecutor in spider-flow-core/src/main/java/org/spiderflow/core/executor/function/ extending FunctionExecutor (spider-flow-core/src/main/java/org/spiderflow/core/executor/function/YourNewFunctionExecutor.java)
2. Implement getFunctions() to define available functions and execute(String functionName, List<Object> args) to implement logic (spider-flow-core/src/main/java/org/spiderflow/core/executor/function/YourNewFunctionExecutor.java)
3. Add method annotations with @Comment and @Return from spider-flow-api/src/main/java/org/spiderflow/annotation/ for inline documentation (spider-flow-api/src/main/java/org/spiderflow/annotation/Return.java)
4. Register your FunctionExecutor in Spider.java's executor initialization so functions are available in expression engine (spider-flow-core/src/main/java/org/spiderflow/core/Spider.java)

Add a Function Extension Method to Existing Types

1. Create a FunctionExtension class in spider-flow-core/src/main/java/org/spiderflow/core/executor/function/extension/ implementing FunctionExtension (spider-flow-core/src/main/java/org/spiderflow/core/executor/function/extension/YourTypeExtension.java)
2. Implement getType() returning the target class (e.g., String.class, List.class) and register public methods to be callable on instances (spider-flow-core/src/main/java/org/spiderflow/core/executor/function/extension/YourTypeExtension.java)
3. Add @Comment annotations to your extension methods for user-facing documentation (spider-flow-api/src/main/java/org/spiderflow/annotation/Comment.java)
4. Register your extension in DefaultExpressionEngine.java (or the expression engine factory) so instance methods are resolved (spider-flow-core/src/main/java/org/spiderflow/core/expression/DefaultExpressionEngine.java)

Create a Plugin Package to Bundle Custom Functions & Executors

1. Create a PluginConfig-annotated class defining your plugin (name, version, functions, executors) in spider-flow-api/src/main/java/org/spiderflow/executor/ (spider-flow-api/src/main/java/org/spiderflow/executor/PluginConfig.java)
2. Bundle FunctionExecutor and ShapeExecutor implementations alongside PluginConfig in a JAR module (spider-flow-core/src/main/java/org/spiderflow/core/executor/function/Base64FunctionExecutor.java)
3. Drop the plugin JAR into spider-flow's plugin directory; the plugin loader will auto-discover your config and register executors/functions (spider-flow-api/src/main/java/org/spiderflow/executor/PluginConfig.java)

FastJSON deserialization security (version 1.2.83 is known to have CVEs—verify if patched in build or runtime). Spring Boot 2.0.7 is EOL; ensure JDK 8 compatibility if upgrading. Thread context via ThreadLocal (SpiderContextHolder)—beware of context leaks in async/thread-pool scenarios. Database schema (spiderflow.sql) must be pre-initialized; no auto-migration visible. WebSocket connections for real-time logs require proper cleanup on client disconnect. Cookie management (CookieContext) may conflict with upstream proxy/load balancer cookies.

• Expression Engine / Dynamic Evaluation — Core 'no-code' feature enabling users to write conditions, transformations, and variable bindings in a safe, sandboxed scripting language without full code access
• ThreadLocal-based Context Management — SpiderContext is bound to each thread via ThreadLocal so multiple concurrent flows don't interfere; critical for thread-safe state isolation in SpiderFlowThreadPoolExecutor
• Plugin/SPI Architecture — ShapeExecutor and FunctionExtension are Service Provider Interfaces allowing runtime injection of new node types and functions; enables ecosystem growth without core changes
• [XPath / JsonPath / CSS Selectors](https://www.w3.org/TR/xpath/ and https://jsonpath.com/) — Multi-format extraction syntax lets users parse HTML, XML, and JSON in a single flow without language-specific parsers; spider-flow abstracts complexity via UI
• Quartz Job Scheduling — Integrates Spring Boot Quartz for cron-based recurring crawls; enables unattended scheduled data collection without external schedulers
• WebSocket for Real-time Logging — Browser receives live flow execution logs via bidirectional WebSocket; required for the 'test & debug' UI feature shown in screenshots
• Multi-datasource / JDBC Abstraction — Supports parallel reads/writes to multiple SQL databases in one flow (via Druid connection pooling); common in ETL and data aggregation scenarios

• ssssssss-team/spider-flow-vue — Official frontend UI for spider-flow; handles drag-and-drop flow design, testing, and monitoring
• ssssssss-team/spider-flow-selenium — Official plugin enabling Selenium-based JS rendering and dynamic page crawling within spider-flow workflows
• ssssssss-team/magic-api — Sister project from same team; uses similar plugin architecture and expression engine for API auto-mapping from XML
• PuerkitoBio/goquery — Go equivalent scraper library (CSS/jQuery selectors); comparison for web-scraping approach differences
• scrapy/scrapy — Python's industry-standard web scraping framework; spider-flow is a no-code/visual alternative targeting non-developers

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for FunctionExecutor implementations

The spider-flow-core module has multiple FunctionExecutor implementations (Base64FunctionExecutor, DateFunctionExecutor, ExtractFunctionExecutor, FileFunctionExecutor, JsonFunctionExecutor, ListFunctionExecutor, MD5FunctionExecutor, RandomFunctionExecutor) but there's no visible test directory structure. These are critical components that handle data transformation and need robust test coverage to prevent regressions.

• [ ] Create spider-flow-core/src/test/java/org/spiderflow/core/executor/function/ directory structure
• [ ] Add unit tests for each FunctionExecutor class covering normal cases, edge cases, and error handling
• [ ] Test Base64FunctionExecutor with various string encodings and invalid inputs
• [ ] Test DateFunctionExecutor with multiple date formats and timezone scenarios
• [ ] Test ExtractFunctionExecutor regex patterns and group extraction logic
• [ ] Add integration tests in spider-flow-api/src/test for FunctionExtension plugin discovery

Add unit tests for concurrent execution strategies and SpiderFlowThreadPoolExecutor

The spider-flow-api/src/main/java/org/spiderflow/concurrent/ directory contains multiple thread submission strategies (ChildPriorThreadSubmitStrategy, LinkedThreadSubmitStrategy, ParentPriorThreadSubmitStrategy, RandomThreadSubmitStrategy) and SpiderFlowThreadPoolExecutor. These are critical for handling concurrent spider flows but appear to lack test coverage. Testing these is essential to ensure thread-safety and correct task scheduling.

• [ ] Create spider-flow-api/src/test/java/org/spiderflow/concurrent/ directory
• [ ] Add unit tests for each ThreadSubmitStrategy implementation verifying correct task ordering
• [ ] Test SpiderFlowThreadPoolExecutor with various scenarios: normal execution, task rejection, thread interruption
• [ ] Test SpiderFutureTask for proper future result handling and cancellation
• [ ] Add tests for SpiderContextHolder thread-local variable isolation across concurrent operations
• [ ] Create integration tests verifying all strategy types work correctly with actual thread pools

Implement and document expression evaluation security sandbox tests

The ExpressionEngine.java and DynamicMethod.java suggest dynamic code evaluation capabilities. Without visible security tests, there's a risk of expression injection vulnerabilities. This is critical for a web-based spider platform where user-defined expressions could be malicious.

• [ ] Create spider-flow-api/src/test/java/org/spiderflow/expression/ directory
• [ ] Add security tests preventing code injection through expression evaluation (e.g., System.exit(), Runtime.exec())
• [ ] Test DynamicMethod invocation with attempted unsafe method calls
• [ ] Document in docs/ or README.md the security model and any expression evaluation restrictions
• [ ] Add tests for ExpressionEngine with complex nested expressions and error scenarios
• [ ] Create a security guidelines document for contributors regarding expression handling in spider-flow-core/src/main/java/org/spiderflow/core/executor/

• Add comprehensive unit tests for SpiderFlowThreadPoolExecutor thread strategies (ChildPriorThreadSubmitStrategy, etc.) in spider-flow-api/src/main/java/org/spiderflow/concurrent/—currently no test files visible in the provided structure.
• Document the FunctionExtension and DynamicMethod interface contracts with concrete examples in README or in-code Javadoc; users cannot easily discover how to write custom functions.
• Create a migration guide for upgrading Spring Boot from 2.0.7 to 2.7.x LTS and FastJSON to 2.x with security patches, including CI test matrix.

• High · Outdated FastJSON Dependency with Known Vulnerabilities — pom.xml - alibaba.fastjson.version 1.2.83. FastJSON version 1.2.83 has multiple known CVEs including deserialization vulnerabilities (CVE-2020-1938, CVE-2022-24765, CVE-2022-45787). This version is significantly outdated and vulnerable to arbitrary code execution through malicious JSON input. Fix: Update FastJSON to version 2.0.25 or later. FastJSON 2.x has addressed many security issues. Replace 1.2.83 with >=2.0.25
• High · Outdated Druid Database Connection Pool — pom.xml - alibaba.druid.version 1.1.16. Druid version 1.1.16 is outdated and may contain known security vulnerabilities. Current versions include important security patches. Fix: Update to Druid 1.2.13 or later to patch known vulnerabilities
• High · SQL Injection Risk in ExecuteSQLExecutor — spider-flow-core/src/main/java/org/spiderflow/core/executor/shape/ExecuteSQLExecutor.java. The ExecuteSQLExecutor component executes raw SQL queries based on flow definitions. Without proper parameterization and input validation, this creates direct SQL injection vulnerabilities when user-controlled data is used in SQL queries. Fix: Implement parameterized queries/prepared statements for all SQL execution. Validate and sanitize all input used in SQL queries. Consider using ORM frameworks for complex queries.
• High · Expression Language Injection via ExpressionEngine — spider-flow-api/src/main/java/org/spiderflow/ExpressionEngine.java, spider-flow-api/src/main/java/org/spiderflow/expression/DynamicMethod.java. The ExpressionEngine and DynamicMethod components allow arbitrary expression evaluation. This could allow code injection if user input flows into expression evaluation without proper validation. Fix: Implement strict input validation and sanitization for all expressions. Use allowlists for permitted functions. Consider sandboxing expression evaluation or using restricted expression evaluators.
• High · Insecure Spring Boot Version — pom.xml - spring-boot-starter-parent 2.0.7.RELEASE. Spring Boot 2.0.7.RELEASE is significantly outdated (released in 2018) and contains multiple known CVEs affecting web framework, logging, and other components. Fix: Update to Spring Boot 2.7.x (latest 2.x) or migrate to Spring Boot 3.x with appropriate Java 17+ compatibility changes
• High · Potential XSS Vulnerabilities in Web UI — spider-flow-web components. The spider-flow platform includes a web UI (spider-flow-web) that processes and displays web scraped content. Without proper output encoding, XSS vulnerabilities could occur when displaying user-defined flow results or scraped HTML. Fix: Implement proper output encoding/escaping for all user-controlled and dynamic content. Use templating engines with auto-escaping. Implement Content Security Policy headers.
• Medium · Exposed Web Port in Docker — Dockerfile - EXPOSE 8088. Port 8088 is exposed without documented access controls. The Docker container runs Java with default configurations, which may lack security hardening. Fix: Implement authentication/authorization for the web interface. Use a reverse proxy with WAF. Run container with minimal privileges. Document security requirements clearly.
• Medium · Outdated JSoup Dependency — pom.xml - jsoup.version 1.11.3. JSoup 1.11.3 is outdated and may contain HTML parsing vulnerabilities. Current version is 1.15.x+ Fix: Update JSoup to 1.15.3 or later to patch known vulnerabilities in HTML/XML parsing
• Medium · Weak Random Number Generation — undefined. RandomFunctionExecutor may use weak random number generation. If used for security-sensitive operations (tokens, seeds), it could be predict Fix: undefined

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.