If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/alibaba/DataX shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

WAIT — Slowing — last commit 10mo ago

• Last commit 10mo ago
• 25+ active contributors
• Distributed ownership (top contributor 16% of recent commits)
• Other licensed
• Tests present
• ⚠ Slowing — last commit 10mo ago
• ⚠ Non-standard license (Other) — review terms
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live alibaba/DataX repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/alibaba/DataX.

What it runs against: a local clone of alibaba/DataX — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in alibaba/DataX | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 341 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of alibaba/DataX. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/alibaba/DataX.git
#   cd DataX
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of alibaba/DataX and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "alibaba/DataX(\\.git)?\\b" \\
  && ok "origin remote is alibaba/DataX" \\
  || miss "origin remote is not alibaba/DataX (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "adbmysqlwriter/src/main/java/com/alibaba/datax/plugin/writer/adbmysqlwriter/AdbMysqlWriter.java" \\
  && ok "adbmysqlwriter/src/main/java/com/alibaba/datax/plugin/writer/adbmysqlwriter/AdbMysqlWriter.java" \\
  || miss "missing critical file: adbmysqlwriter/src/main/java/com/alibaba/datax/plugin/writer/adbmysqlwriter/AdbMysqlWriter.java"
test -f "adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/AdbpgWriter.java" \\
  && ok "adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/AdbpgWriter.java" \\
  || miss "missing critical file: adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/AdbpgWriter.java"
test -f "adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/AdsWriter.java" \\
  && ok "adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/AdsWriter.java" \\
  || miss "missing critical file: adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/AdsWriter.java"
test -f "cassandrareader/src/main/java/com/alibaba/datax/plugin/reader/cassandrareader/CassandraReader.java" \\
  && ok "cassandrareader/src/main/java/com/alibaba/datax/plugin/reader/cassandrareader/CassandraReader.java" \\
  || miss "missing critical file: cassandrareader/src/main/java/com/alibaba/datax/plugin/reader/cassandrareader/CassandraReader.java"
test -f "adbmysqlwriter/pom.xml" \\
  && ok "adbmysqlwriter/pom.xml" \\
  || miss "missing critical file: adbmysqlwriter/pom.xml"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 341 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~311d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/alibaba/DataX"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

DataX is Alibaba's open-source data integration framework that synchronizes data between 50+ heterogeneous data sources (MySQL, Oracle, PostgreSQL, HDFS, Hive, MaxCompute, HBase, etc.) through a Reader/Writer plugin architecture. It powers DataWorks and handles trillions of data rows daily in production, abstracting source-specific logic behind standardized plugin interfaces for cross-database ETL workloads. Monorepo structure: root pom.xml references datasource-specific submodules (adbmysqlwriter/, adbpgwriter/, adswriter/, etc.). Each plugin follows {name}reader/ or {name}writer/ layout with src/main/java/com/alibaba/datax/plugin/{reader|writer}/{name}/ containing core plugin class, src/main/resources/plugin.json for metadata, and doc/{name}.md for user docs. Shared utilities live in plugin-rdbms-util and datax-common dependencies.

Data engineers and platform teams building enterprise data pipelines who need to migrate, synchronize, or integrate data across incompatible systems (e.g., on-prem Oracle to cloud MaxCompute, MySQL to HDFS). Contributors extending DataX add support for new datasources by implementing Reader/Writer plugins following the established pattern.

Production-mature: deployed at Alibaba Group scale (3000+ customers, 30+ trillion records/day), with established plugin ecosystem covering major RDBMSes, data warehouses, and NoSQL systems. Java codebase (3.9MB) with Maven-based build and assembly packaging indicates enterprise-grade infrastructure. No recent activity visible in provided data, but the stable plugin structure and wide datasource coverage (50+) suggest maintenance mode rather than active development.

Low risk for existing datasources, moderate risk for custom extensions: dependencies on specific JDBC drivers (mysql-connector-java 5.1.40 is outdated—released 2015), heterogeneous plugin quality across 50+ datasources, and potential for breaking changes in plugin API across versions. Monolithic approach concentrates all datasource logic in single repo, making upgrades risky if your plugin depends on unversioned internal APIs.

No specific recent commits, PRs, or milestones are visible in the provided file snapshot. The stable plugin ecosystem and frozen version (0.0.1-SNAPSHOT in pom.xml) suggest the project is in maintenance mode, with community contributions through pull requests rather than active Alibaba development.

git clone https://github.com/alibaba/DataX.git
cd DataX
mvn clean package -DskipTests
tar -xzf target/datax/datax.tar.gz
cd datax && ./bin/datax.py job.json

Refer to userGuid.md and downloaded 202308 distribution tarball for configuration templates.

Daily commands: Build: mvn clean package -DskipTests in repo root. Run job: ./bin/datax.py config/job.json (requires Python 2.7+ per Shell scripts). Configuration is JSON-based; each datasource has plugin_job_template.json in src/main/resources/ as reference template.

• adbmysqlwriter/src/main/java/com/alibaba/datax/plugin/writer/adbmysqlwriter/AdbMysqlWriter.java — Entry point for ADB MySQL writer plugin; implements the core write logic and job/task configuration
• adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/AdbpgWriter.java — Entry point for ADB PostgreSQL writer plugin; demonstrates the writer plugin pattern used across DataX
• adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/AdsWriter.java — Core ADS writer implementation; shows advanced patterns with ODPS load and insert proxies
• cassandrareader/src/main/java/com/alibaba/datax/plugin/reader/cassandrareader/CassandraReader.java — Reader plugin pattern reference; demonstrates how DataX abstracts source data reading
• adbmysqlwriter/pom.xml — Maven configuration showing dependency structure; datax-common is the shared framework dependency
• README.md — Project overview explaining DataX's architecture as a pluggable framework with Reader/Writer abstractions
• adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/copy/AdbProxy.java — Abstract proxy pattern used for database operations; critical for understanding writer plugin extensibility

Add a New Database Writer Plugin

1. Create a new plugin module directory (e.g., mynewdbwriter/) with standard Maven pom.xml structure inheriting from datax-all parent (adbmysqlwriter/pom.xml)
2. Implement main writer class extending DataX's Writer framework, following the pattern in AdbMysqlWriter or AdsWriter (adbmysqlwriter/src/main/java/com/alibaba/datax/plugin/writer/adbmysqlwriter/AdbMysqlWriter.java)
3. Create database-specific proxy classes for operations (INSERT, COPY, LOAD) by extending AdbProxy pattern (adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/copy/AdbProxy.java)
4. Implement utility classes for connection management, type mapping, and validation in util/ package (adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/util/Adb4pgUtil.java)
5. Register plugin by creating plugin.json with plugin name and version in src/main/resources/ (adbmysqlwriter/src/main/resources/plugin.json)
6. Create plugin_job_template.json with default configuration parameters users will customize (adbmysqlwriter/src/main/resources/plugin_job_template.json)

Implement Dual-Mode Write Strategy (INSERT vs COPY/LOAD)

1. Create abstract base proxy class defining common interface for database operations (adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/insert/AdsProxy.java)
2. Implement concrete proxy for bulk COPY/LOAD operations for better performance (adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/copy/Adb4pgClientProxy.java)
3. Implement concrete proxy for standard INSERT operations as fallback or selective writes (adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/insert/AdsInsertProxy.java)
4. In main writer class, create factory/selector logic to choose proxy based on configuration or data characteristics (adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/AdsWriter.java)

Add Support for Cross-Database Schema Mapping

1. Create classes representing source and target table schemas and metadata (adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/ads/TableInfo.java)
2. Define column metadata and source data type constants in separate classes (adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/ads/ColumnInfo.java)
3. Create helper class to extract metadata from target database and build TableInfo (adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/load/TableMetaHelper.java)
4. Implement type mapping utility to convert between source and target database data types (adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/util/AdsUtil.java)
5. Call metadata helper and type mapper in writer initialization to validate schema compatibility (adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/AdsWriter.java)

Add Localization for Reader/Writer Error Messages

1. Create LocalStrings.properties base file with error messages and UI strings (cassandrareader/src/main/java/com/alibaba/datax/plugin/reader/cassandrareader/LocalStrings.properties)
2. Create locale-specific properties files (en_US, zh_CN, ja_JP) with translations (cassandrareader/src/main/java/com/alibaba/datax/plugin/reader/cassandrareader/LocalStrings_en_US.properties)
3. In main plugin class or error code enums, load ResourceBundle from LocalStrings properties files (cassandrareader/src/main/java/com/alibaba/datax/plugin/reader/cassandrareader/CassandraReaderErrorCode.java)

1. Outdated JDBC drivers: mysql-connector-java 5.1.40 (2015) has security vulnerabilities; upgrading may break compatibility with pinned driver versions in legacy plugins. 2. Python wrapper dependency: datax.py requires Python 2.7+, not bundled—must be installed separately on execution servers. 3. Plugin API instability: BaseDataxPlugin interface in datax-common is unversioned; breaking changes propagate to all plugins, necessitating bulk updates across 50+ modules. 4. Monolithic versioning: single 0.0.1-SNAPSHOT version for all plugins prevents independent plugin releases; bug fixes in one datasource require full repo rebuild. 5. Assembly packaging: src/main/assembly/package.xml creates fat JARs per plugin, making classpath conflicts likely if multiple datasources loaded simultaneously. 6. No plugin isolation: shared dependencies (datax-common) version-locked across all plugins—incompatible transitive dependencies can cause silent failures.

• Reader/Writer Plugin Architecture — Core abstraction in DataX—every datasource implements Reader to extract records and Writer to load them, decoupling source/target logic and enabling arbitrary N×M datasource combinations without cross-dependencies.
• SPI (Service Provider Interface) Pattern — DataX discovers and loads plugins dynamically via plugin.json metadata and reflection, allowing new datasource plugins to be dropped into the monorepo without core framework changes.
• Bulk Batch Data Transfer — DataX optimizes throughput by transferring data in configurable batches (see batchSize in plugin configs), amortizing network/IO overhead for trillion-record datasets rather than row-by-row processing.
• JDBC Connection Pooling & Proxy Patterns — Plugins like adbpgwriter use proxy objects (Adb4pgClientProxy.java, AdbProxy.java) to manage connection lifecycles, retry logic, and protocol optimization (e.g., PostgreSQL COPY fast-path) transparently.
• Heterogeneous Data Type Mapping — DataX's Record abstraction and Column types in datax-common handle semantic mismatches (e.g., Oracle NUMBER → MySQL DECIMAL, Hive STRING → HBase bytes) across 50+ datasources with different type systems.
• Monorepo Dependency Management — Single Maven reactor (parent pom.xml) coordinates builds across 50+ plugin modules, controlling transitive dependency conflicts and version alignment—critical for stability but complicates independent plugin releases.
• Configuration-Driven Job Specification (JSON Schema) — DataX jobs are declarative JSON documents (see plugin_job_template.json files) defining source, target, mappings, and tuning parameters—enabling non-programmers to compose data pipelines without code.

• apache/sqoop — Predecessor bulk data transfer tool between RDBMS and Hadoop; DataX is spiritual successor with cleaner plugin architecture and broader datasource support.
• alibaba/DataWorks — Commercial Alibaba Cloud product built on DataX framework, adding UI, scheduling, and monitoring for enterprise data integration workflows.
• dbt-labs/dbt — Alternative modern approach to data transformation pipelines, focusing on SQL-first ELT; serves similar teams but uses different paradigm (transformation over movement).
• apache/airflow — Orchestration platform frequently paired with DataX for job scheduling, error handling, and cross-system data workflow coordination.
• getdbt/transformers — Community repository of dbt adapters mirroring DataX's multi-datasource plugin ecosystem, showing parallel evolution in data integration tooling.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add integration tests for adbmysqlwriter and adbpgwriter with MySQL 5.1.40 compatibility

The adbmysqlwriter module depends on mysql-connector-java 5.1.40, which is quite old (released 2015). There are no visible test files in adbmysqlwriter/src/test or adbpgwriter/src/test directories. Adding integration tests would validate that the writers correctly handle data types, batch inserts, and error scenarios with modern MySQL/PostgreSQL versions, while also revealing any compatibility issues with the outdated JDBC driver.

• [ ] Create adbmysqlwriter/src/test/java/com/alibaba/datax/plugin/writer/adbmysqlwriter/ directory structure
• [ ] Add AdbMysqlWriterTest.java with tests for column type mapping (ColumnDataType.java patterns from adswriter)
• [ ] Add AdbpgWriterTest.java testing the copy protocol in Adb4pgClientProxy.java
• [ ] Create TestContainers-based integration tests using real MySQL 5.7+ and PostgreSQL instances
• [ ] Add test cases for batch insert operations, null handling, and error recovery

Refactor duplicate code between adswriter, adbmysqlwriter, and adbpgwriter into shared utilities

Examining the file structure, adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/ads/ contains ColumnDataType.java, ColumnInfo.java, and TableInfo.java that appear to solve similar problems as the adbpgwriter/src/main/java/com/alibaba/datax/plugin/writer/adbpgwriter/util/ classes. These three writer modules likely share RDBMS connection logic, column type conversion, and table metadata handling. Creating a shared utility module (plugin-writer-common or similar) would reduce maintenance burden and inconsistencies.

• [ ] Analyze adswriter/ads/ColumnDataType.java and adbpgwriter/util/Adb4pgUtil.java for overlapping type mapping logic
• [ ] Create a new module: plugin-writer-rdbms-common with shared classes for ColumnInfo, ColumnDataType, TableInfo
• [ ] Move AdsProxy.java, AdsInsertProxy.java patterns into a common BaseRdbmsWriterProxy interface
• [ ] Update pom.xml in adbmysqlwriter, adbpgwriter, adswriter to depend on plugin-writer-rdbms-common
• [ ] Add integration tests in plugin-writer-rdbms-common to verify type mappings work across all writers

Add configuration validation and plugin.json schema documentation for all writer modules

Each writer module has plugin_job_template.json and plugin.json files (e.g., adbmysqlwriter/src/main/resources/plugin.json) but there is no visible validation logic or schema documentation. This makes it difficult for new contributors to understand required fields, constraints, and defaults. Adding JSON Schema validation in the plugin loading phase and documenting each writer's configuration would prevent runtime errors from misconfiguration.

• [ ] Create adbmysqlwriter/src/main/resources/adbmysqlwriter-schema.json defining the plugin.json structure with required fields, types, and constraints
• [ ] Create corresponding schema files for adbpgwriter and adswriter
• [ ] Add ConfigValidator.java in datax-common or plugin-rdbms-util to validate job configs against schema at plugin initialization
• [ ] Update adbmysqlwriter/doc/adbmysqlwriter.md with Configuration section documenting each field from plugin_job_template.json
• [ ] Add unit tests validating ConfigValidator against valid and invalid configurations

• Add integration tests for adbmysqlwriter/AdbMysqlWriter.java and adbpgwriter/AdbpgWriter.java: neither module contains visible src/test/ directories. Contribute parametrized JUnit 4 tests covering connection failures, schema mismatches, and batch edge cases.
• Update mysql-connector-java from 5.1.40 (2015) to 8.0.33 (current) and adbmysqlwriter/pom.xml accordingly, then test against MySQL 5.7+ to validate compatibility. Document any breaking changes in adbmysqlwriter/doc/adbmysqlwriter.md.
• Extract shared JDBC retry logic visible in adbpgwriter/copy/Adb4pgClientProxy.java and adbmysqlwriter into a reusable utility in plugin-rdbms-util/src/main/java/, reducing code duplication across 10+ RDBMS plugins.

• High · Outdated MySQL JDBC Driver with Known Vulnerabilities — adbmysqlwriter/pom.xml. The adbmysqlwriter module uses mysql-connector-java version 5.1.40, which is severely outdated (released in 2016). This version contains multiple known CVEs including CVE-2015-4740, CVE-2015-2951, and others related to authentication bypass, man-in-the-middle attacks, and code execution. Fix: Upgrade to mysql-connector-java 8.0.33 or later. Alternatively, use mysql-connector-j (the newer official driver) version 8.2.0+. Test thoroughly after upgrading to ensure compatibility.
• High · SQL Injection Risk in Database Writer Plugins — adbmysqlwriter, adbpgwriter, adswriter modules - particularly insert/load utilities. The codebase contains multiple database writer plugins (adbmysqlwriter, adbpgwriter, adswriter) that interact with databases. While not directly visible in the provided snippets, the presence of classes like 'AdsInsertUtil.java', 'AdsInsertProxy.java', and dynamic table/column handling suggests potential for SQL injection if user inputs or configuration values are not properly parameterized. Fix: Conduct thorough code review of all SQL construction code. Ensure all SQL queries use prepared statements with parameterized queries. Never concatenate user inputs directly into SQL strings. Implement input validation and sanitization for table names and column names where dynamic SQL is necessary.
• Medium · Potential Credential Exposure in Configuration Files — adbmysqlwriter/src/main/resources/, adbpgwriter/src/main/resources/, adswriter/src/main/resources/ and corresponding util/Key.java files. The presence of configuration files (plugin.json, plugin_job_template.json) and utility classes with constants (Key.java, Constant.java) suggest that database credentials may be stored or transmitted. No encryption mechanism is evident from the file structure for sensitive configuration like database passwords, connection strings, or API keys. Fix: Implement secure credential management using environment variables or dedicated secret management systems. Never hardcode credentials in configuration files. Encrypt sensitive data at rest. Use OAuth/IAM integration where possible instead of password-based authentication.
• Medium · Logging of Sensitive Information — adbmysqlwriter, adbpgwriter, adswriter modules - logging configuration. The project uses logback for logging (logback-classic dependency). Database writers often log connection details, query execution, and error messages. Without proper configuration, this may expose sensitive information like passwords, connection strings, or data being transferred. Fix: Review all logging statements and logback configuration. Implement log filtering to mask sensitive data (passwords, credentials, personal data). Set appropriate log levels for production (WARN/ERROR). Never log full SQL queries with sensitive data. Ensure log files are properly secured with restricted access permissions.
• Medium · Missing Input Validation on Plugin Configuration — adbmysqlwriter/src/main/resources/plugin_job_template.json and corresponding writer classes. Plugin configuration files (plugin_job_template.json) and the plugin framework suggest user-configurable parameters. Without visible input validation, there's risk of injection attacks through malformed configuration, particularly for parameters that influence SQL generation or command execution. Fix: Implement comprehensive input validation for all configuration parameters. Validate table names, column names, and connection parameters against whitelists. Implement bounds checking for numeric parameters. Reject or sanitize any potentially dangerous characters in configuration values.
• Low · Missing Dependency Version Management for Transitive Dependencies — adbmysqlwriter/pom.xml (and other module poms). While direct dependencies are specified, transitive dependencies inherited from datax-common and plugin-rdbms-util are not explicitly managed or pinned. This could lead to unexpected security issues from transitive dependencies with known vulnerabilities. Fix: Use Maven dependency management to explicitly declare and pin versions of critical transitive dependencies. Run regular dependency audits using tools like OWASP Dependency-Check or GitHub's Dependabot. Consider using Maven's dependency lock files.
• Low · No Visible HTTPS Enforcement for Remote Data Transfers — undefined. The plugins appear to handle connections to various database systems and remote services (ADS, ADB, etc.). Without visible enforcement of SSL/TLS validation, connections could be vulnerable to Fix: undefined

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.