If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/WeiYe-Jing/datax-web shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

WAIT — Stale — last commit 2y ago

• 2 active contributors
• MIT licensed
• Tests present
• ⚠ Stale — last commit 2y ago
• ⚠ Small team — 2 contributors active in recent commits
• ⚠ Concentrated ownership — top contributor handles 77% of recent commits
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live WeiYe-Jing/datax-web repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/WeiYe-Jing/datax-web.

What it runs against: a local clone of WeiYe-Jing/datax-web — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in WeiYe-Jing/datax-web | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 735 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of WeiYe-Jing/datax-web. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/WeiYe-Jing/datax-web.git
#   cd datax-web
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of WeiYe-Jing/datax-web and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "WeiYe-Jing/datax-web(\\.git)?\\b" \\
  && ok "origin remote is WeiYe-Jing/datax-web" \\
  || miss "origin remote is not WeiYe-Jing/datax-web (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java" \\
  && ok "datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java" \\
  || miss "missing critical file: datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java"
test -f "datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java" \\
  && ok "datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java" \\
  || miss "missing critical file: datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java"
test -f "datax-admin/src/main/java/com/wugui/datax/admin/core/trigger/JobTrigger.java" \\
  && ok "datax-admin/src/main/java/com/wugui/datax/admin/core/trigger/JobTrigger.java" \\
  || miss "missing critical file: datax-admin/src/main/java/com/wugui/datax/admin/core/trigger/JobTrigger.java"
test -f "datax-admin/src/main/java/com/wugui/datax/admin/controller/JobInfoController.java" \\
  && ok "datax-admin/src/main/java/com/wugui/datax/admin/controller/JobInfoController.java" \\
  || miss "missing critical file: datax-admin/src/main/java/com/wugui/datax/admin/controller/JobInfoController.java"
test -f "datax-admin/src/main/java/com/wugui/datax/admin/entity/JobInfo.java" \\
  && ok "datax-admin/src/main/java/com/wugui/datax/admin/entity/JobInfo.java" \\
  || miss "missing critical file: datax-admin/src/main/java/com/wugui/datax/admin/entity/JobInfo.java"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 735 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~705d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/WeiYe-Jing/datax-web"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

DataX-Web is a distributed data synchronization platform built on Apache DataX that provides a web UI for non-technical users to create ETL jobs by selecting source/target datasources without writing JSON configs. It supports RDBMS, Hive, HBase, ClickHouse, and MongoDB sources, executes jobs across distributed executor clusters, and offers real-time log viewing, incremental sync via cron/autoincrement, and integrated scheduling via a custom xxl-job fork. Monolithic Spring Boot application: datax-admin/ contains the central scheduler (port 8080), controllers in controller/, core scheduling logic in core/cron/ and core/handler/, MyBatis entity mappings for Jobs/Datasources/Logs, Swagger API docs. Separate datax-executor module (not shown in top files) deploys as worker nodes registering with admin via JobRegistryController. Shell startup scripts in bin/ orchestrate the whole stack.

Data engineers and DBAs who need to build and schedule DataX sync tasks without hand-coding JSON configs; operations teams monitoring distributed ETL execution; organizations performing bulk table migrations between databases or data warehouses.

Actively maintained (v2.1.2 released, 898KB Java codebase with 40+ documented features). No visible GitHub CI/CD pipeline in file list, but installation scripts and database schema present suggest production deployments exist. Community-driven (QQ group 776939467 listed) but single-maintainer risk apparent.

Tight coupling to Python 2.7 (requires manual Python 3 migration per docs); custom xxl-job fork introduces maintenance burden; no test directory visible in top 60 files suggests weak test coverage; reliance on external DataX binary distribution for actual execution; potential security concern with datasource credential encryption (recent fix in v40).

Recent changes include encrypted datasource credentials (feature 40), executor CPU/memory/load monitoring dashboards (features 35-36), RDBMS bulk task creation (feature 33), ClickHouse JSON builder support (feature 34), and script-type task kill functionality (feature 38). Issue tracker active with bug_report and feature_request templates present.

git clone https://github.com/WeiYe-Jing/datax-web.git && cd datax-web && mvn clean package -DskipTests -P release. Then: ./bin/install.sh (initializes MySQL schema from bin/db/datax_web.sql), followed by ./bin/start-all.sh to launch admin + executor. Requires Java 8.201+, MySQL 5.7+, Python 2.7, and separate DataX binary.

Daily commands: ./bin/start-all.sh launches both admin (http://localhost:8080) and executor. Or run ./bin/start.sh to start admin only, then separately run executor. Logs stream to $DATAX_HOME/logs/. Stop via ./bin/stop-all.sh.

• datax-admin/src/main/java/com/wugui/datax/admin/DataXAdminApplication.java — Entry point for the DataX Web admin application; all Spring Boot initialization happens here
• datax-admin/src/main/java/com/wugui/datax/admin/core/scheduler/JobScheduler.java — Core scheduling engine that orchestrates job execution and cron-based triggering; mission-critical for data sync workflow
• datax-admin/src/main/java/com/wugui/datax/admin/core/trigger/JobTrigger.java — Handles job triggering logic and communicates with executors; essential for the trigger→executor pipeline
• datax-admin/src/main/java/com/wugui/datax/admin/controller/JobInfoController.java — REST API controller for job CRUD operations; main interface for UI and external clients to manage jobs
• datax-admin/src/main/java/com/wugui/datax/admin/entity/JobInfo.java — Core domain model representing a DataX job; all job metadata and configuration persists through this entity
• datax-admin/src/main/java/com/wugui/datax/admin/core/handler/AESEncryptHandler.java — Encryption handler for datasource credentials; ensures sensitive data like passwords are never stored in plaintext
• datax-admin/src/main/java/com/wugui/datax/admin/core/route/ExecutorRouter.java — Executor routing abstraction that selects which executor node runs a job; supports multiple routing strategies

Add a new REST API endpoint for job management

1. Create a new controller method in JobInfoController or a new @RestController class (datax-admin/src/main/java/com/wugui/datax/admin/controller/JobInfoController.java)
2. Implement the service logic in a @Service class (follow existing pattern with Service + ServiceImpl) (datax-admin/src/main/java/com/wugui/datax/admin/entity/JobInfo.java)
3. If storing/retrieving data, create or update a MyBatis Plus Mapper interface extending BaseMapper (datax-admin/src/main/java/com/wugui/datax/admin/config/MybatisPlusConfig.java)
4. Add database migration script to bin/db/datax_web.sql if new tables are needed (bin/db/datax_web.sql)

Add a new executor routing strategy

1. Add new enum value to ExecutorRouteStrategyEnum (datax-admin/src/main/java/com/wugui/datax/admin/core/route/ExecutorRouteStrategyEnum.java)
2. Create new strategy class in core/route/strategy/ implementing ExecutorRouteStrategy interface (datax-admin/src/main/java/com/wugui/datax/admin/core/route/strategy/ExecutorRouteConsistentHash.java)
3. Register strategy in ExecutorRouter.getRouter() switch/if statement (datax-admin/src/main/java/com/wugui/datax/admin/core/route/ExecutorRouter.java)

Add support for a new datasource type

1. Create new Reader and Writer DTO classes (e.g., ElasticsearchReaderDto, ElasticsearchWriterDto) (datax-admin/src/main/java/com/wugui/datax/admin/dto/RdbmsReaderDto.java)
2. Add builder logic in DataXJsonBuildDto or create a new builder service (datax-admin/src/main/java/com/wugui/datax/admin/dto/DataXJsonBuildDto.java)
3. Update JobDatasourceController to validate and handle the new datasource type (datax-admin/src/main/java/com/wugui/datax/admin/controller/JobDatasourceController.java)
4. Add datasource type option to the frontend and update MetadataController if column discovery is needed (datax-admin/src/main/java/com/wugui/datax/admin/controller/Met)

Must manually initialize MySQL schema via bin/db/datax_web.sql before startup or app will crash. DataX binary distribution must be on executor classpath or in $DATAX_HOME/bin/ (env var checked first per feature 22). Python 2.7 hardcoded in executor spawn logic—upgrading requires modifying datax/bin/*.py scripts. Executor registration happens via HTTP polling in JobRegistryController; if firewall blocks executor → admin traffic, jobs hang silently. Cron expressions use custom parser (CronExpression.java), not standard Quartz—some cron syntax may differ. Token auth for Swagger API (feature 30) requires specific header but not documented in README.

• Distributed Task Scheduling with Executor Registration — datax-web decouples the scheduler (admin) from execution (executors) via HTTP registration and heartbeat polling; understanding how executors auto-discover and receive job configs is critical to scaling and debugging deployment issues
• Incremental Data Sync via Cron + Watermark — Feature 17 enables incremental syncs using cron triggers and auto-advancing timestamp/PK watermarks; this is how datax-web avoids re-syncing entire tables on every run and implements near-real-time ETL
• DataX JSON Config Generation from Web Forms — DataxJsonController dynamically builds DataX reader/writer configs (JSON) based on selected datasources; understanding this mapping is essential for adding new datasource types or custom transformations
• Custom Cron Expression Parser — CronExpression.java implements a non-standard cron parser for job scheduling; deviations from Quartz syntax can cause subtle scheduling bugs if not understood
• MyBatis-Plus ORM with Auto Field Population — Feature 29 auto-fills audit fields (created_time, modified_time, created_by) on insert/update via MyBatis interceptors; requires understanding MyBatis plugin lifecycle to debug persistence issues
• Datasource Credential Encryption — Feature 40 encrypts stored database passwords; understanding the cipher (likely AES based on 'algorithm' mention) is needed to avoid plaintext leaks and implement credential rotation
• Executor Routing Policies (First, Last, Round-Robin, Consistent Hash, LRU, LFU, Failover, Busy-Transfer) — Feature 9 lists 8 routing strategies for distributing jobs across executor clusters; choosing the right one affects load balancing, latency, and failover behavior—critical for multi-node deployments

• apache/incubator-datax — The upstream DataX project that datax-web wraps; provides the actual ETL execution engine and reader/writer plugins
• xuxueli/xxl-job — The scheduler framework that datax-web's admin module is based on (forked and customized); handles distributed job dispatch, routing, and retry logic
• alibaba/DataX — Alibaba's open-source DataX; the canonical reader/writer plugin ecosystem that datax-web builds upon (same as apache/incubator-datax)
• azkaban/azkaban — Alternative distributed workflow/ETL scheduler; similar use case but different architecture—useful for comparing scheduling patterns
• getdbt/dbt-core — Modern alternative for data transformation and orchestration; addresses similar problems for analytics eng teams but with code-first approach vs UI-first

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for ExecutorRouter and routing strategy classes

The routing strategy classes (ExecutorRouteBusyover, ExecutorRouteConsistentHash, ExecutorRouteFailover, etc.) in datax-admin/src/main/java/com/wugui/datax/admin/core/route/strategy/ lack test coverage. These are critical components that determine how jobs are distributed across executor nodes. Adding unit tests would ensure routing strategies work correctly under various load conditions and failure scenarios, preventing production issues with task distribution.

• [ ] Create datax-admin/src/test/java/com/wugui/datax/admin/core/route/strategy/ directory structure
• [ ] Add unit tests for ExecutorRouteConsistentHash with hash collision and node removal scenarios
• [ ] Add unit tests for ExecutorRouteLRU and ExecutorRouteLFU with cache eviction edge cases
• [ ] Add unit tests for ExecutorRouteFailover with offline node handling
• [ ] Add integration test for ExecutorRouter selection logic with mocked executor registry

Add unit tests for AESEncryptHandler and security-sensitive operations

The AESEncryptHandler class (datax-admin/src/main/java/com/wugui/datax/admin/core/handler/AESEncryptHandler.java) handles encryption of datasource credentials, which is critical for security. The README explicitly mentions 'data source information encryption' as a feature, but there are no visible tests. Missing tests for encryption/decryption could lead to datasource credential leakage or data loss if encryption logic breaks.

• [ ] Create datax-admin/src/test/java/com/wugui/datax/admin/core/handler/ directory
• [ ] Add unit tests for AESEncryptHandler encrypt/decrypt round-trip with various input types
• [ ] Add tests for AESEncryptHandler with null and empty string inputs
• [ ] Add tests for MybatisMetaObjectHandler to verify encryption is applied during persistence
• [ ] Add security test to ensure encrypted values are never logged in plaintext

Add GitHub Actions workflow for automated testing and Maven build validation

The repository has issue and feature request templates but no CI/CD pipeline visible in .github/workflows/. Given the Java/Maven-based project with multiple modules (datax-admin, datax-executor, datax-alert, etc.), automated testing and build validation on pull requests is essential to maintain code quality. This prevents breaking changes and ensures all PRs pass compilation.

• [ ] Create .github/workflows/maven-test.yml workflow file
• [ ] Configure workflow to run 'mvn clean test' on push and pull requests for Java 1.8+ compatibility
• [ ] Add Maven build step to validate all modules compile (datax-admin, datax-executor, datax-alert)
• [ ] Configure workflow to fail if tests don't meet coverage thresholds for critical security classes
• [ ] Add workflow status badge to README.md

• Add unit tests for CronExpression.java: currently no test coverage visible for the custom cron parser, which is critical path for scheduling reliability. Start with test cases for common cron formats (daily, hourly, incremental sync intervals).
• Document executor registration flow: create a sequence diagram or wiki page explaining how executors auto-register with admin, heartbeat, and receive job dispatch—currently only visible in code via JobRegistryController, making onboarding hard.
• Add Python 3 support toggle: wrap the Python 2.7 → 3 conversion (mentioned in docs/datax-web/datax-python3/) into a config flag in env.properties so users can choose at runtime without manual file replacement.

• High · Potential SQL Injection in Dynamic Query Construction — datax-admin/src/main/java/com/wugui/datax/admin/controller/ (JobInfoController, JobDatasourceController, MetadataController). The codebase contains multiple controllers (JobInfoController, JobDatasourceController, MetadataController) that likely construct SQL queries dynamically. Without visible parameterized query implementations in the file structure, there is risk of SQL injection attacks when processing user input for data source queries and metadata operations. Fix: Ensure all database queries use prepared statements with parameterized queries. Validate and sanitize all user inputs. Use ORM frameworks (MyBatis Plus is referenced) consistently with proper parameter binding.
• High · Weak Encryption Implementation (AES) — datax-admin/src/main/java/com/wugui/datax/admin/core/handler/AESEncryptHandler.java. The codebase uses a custom AESEncryptHandler for data source credential encryption. Without visibility into the implementation, there are common risks: hardcoded encryption keys, weak key derivation, improper IV handling, or use of ECB mode instead of secure modes like CBC/GCM. Fix: Review AES implementation to ensure: (1) Keys are externalized and never hardcoded, (2) Use authenticated encryption (AES-GCM), (3) Generate random IVs for each encryption, (4) Use PBKDF2 or similar for key derivation from passwords, (5) Implement proper key rotation policies.
• High · Missing or Inadequate Authentication/Authorization — datax-admin/src/main/java/com/wugui/datax/admin/config/SecurityConfig.java. SecurityConfig.java exists but the configuration details are not visible. DataX Web handles sensitive operations (datasource credentials, job execution, process killing) that require strong authentication and fine-grained authorization. Risk of unauthorized access to sensitive operations. Fix: Implement comprehensive Spring Security configuration with: (1) Strong authentication mechanism (OAuth2/JWT), (2) Role-based access control (RBAC) for all endpoints, (3) Protect all admin endpoints, (4) Implement principle of least privilege for job execution, (5) Regular security audit of permission assignments.
• High · Potential Cross-Site Scripting (XSS) in Dynamic Content — datax-admin/src/main/java/com/wugui/datax/admin/controller/ (DataxJsonController, JobLogController, IndexController). Controllers serve dynamic content (DataxJsonController, IndexController) and job logs. Without visible output encoding, user-controlled data in DataX configurations, job parameters, and log outputs could lead to XSS attacks if rendered without proper HTML encoding. Fix: Implement output encoding for all dynamic content: (1) Use Spring's built-in HTML escaping in templates, (2) Apply proper content-type headers (text/html; charset=UTF-8), (3) Implement Content Security Policy (CSP) headers, (4) Validate and sanitize all user inputs, (5) Use templating engines with auto-escaping enabled.
• High · Command Injection Risk in Process Execution — datax-admin/src/main/java/com/wugui/datax/admin/core/kill/KillJob.java. KillJob.java suggests the application executes system commands to terminate processes. Without visible input validation, there is significant risk of command injection attacks if job IDs or parameters are not properly escaped. Fix: Avoid shell command execution where possible. If unavoidable: (1) Use ProcessBuilder instead of Runtime.exec() with shell strings, (2) Never concatenate user input into commands, (3) Use allowlist validation for process identifiers, (4) Run with minimal required privileges, (5) Implement strict input validation and sanitization.
• Medium · Potential Insecure Deserialization — datax-admin/src/main/java/com/wugui/datax/admin/core/util/JacksonUtil.java. JacksonUtil.java suggests JSON deserialization of potentially untrusted data from API calls and database records. Unsafe deserialization could lead to Remote Code Execution if Jackson is configured with unsafe type handling. Fix: Ensure Jackson security best practices: (1) Disable default typing if not required, (2) Use JsonTypeInfo with explicit type handling, (3) Implement custom deserializers with strict validation,

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.