If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/apache/flink-cdc shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across the board

• Last commit 5d ago
• 27+ active contributors
• Distributed ownership (top contributor 18% of recent commits)
• Apache-2.0 licensed
• CI configured
• Tests present

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live apache/flink-cdc repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/apache/flink-cdc.

What it runs against: a local clone of apache/flink-cdc — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in apache/flink-cdc | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 35 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of apache/flink-cdc. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/apache/flink-cdc.git
#   cd flink-cdc
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of apache/flink-cdc and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "apache/flink-cdc(\\.git)?\\b" \\
  && ok "origin remote is apache/flink-cdc" \\
  || miss "origin remote is not apache/flink-cdc (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "pom.xml" \\
  && ok "pom.xml" \\
  || miss "missing critical file: pom.xml"
test -f ".github/workflows/flink_cdc_ci.yml" \\
  && ok ".github/workflows/flink_cdc_ci.yml" \\
  || miss "missing critical file: .github/workflows/flink_cdc_ci.yml"
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "docs/content.zh/docs/get-started/introduction.md" \\
  && ok "docs/content.zh/docs/get-started/introduction.md" \\
  || miss "missing critical file: docs/content.zh/docs/get-started/introduction.md"
test -f ".github/workflows/modules.py" \\
  && ok ".github/workflows/modules.py" \\
  || miss "missing critical file: .github/workflows/modules.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 35 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~5d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/apache/flink-cdc"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Flink CDC is a distributed, real-time change data capture (CDC) streaming tool built on Apache Flink that synchronizes data from multiple databases (MySQL, PostgreSQL, Oracle, MongoDB, DB2, etc.) to various sinks (Doris, Kafka, etc.). It handles full database syncs, sharded table consolidation, schema evolution, and on-the-fly data transformation via declarative YAML pipelines without requiring code. Modular monorepo structured as: flink-cdc-cli (entry point, YAML parsing via snakeyaml), flink-cdc-common (shared types/interfaces), flink-cdc-composer (pipeline orchestration), then connector modules (flink-connector-mysql-cdc, -postgres-cdc, etc.) each with source implementations. Docs in docs/content.zh and docs/content (Hugo-based site). CI orchestrated through .github/workflows/ with module-specific pytest logic (modules.py).

Data engineers and platform teams building real-time data pipelines who need to replicate heterogeneous database changes to data warehouses or lakes; DevOps/SRE teams managing multi-database synchronization at scale; anyone using Apache Flink who wants zero-code CDC capabilities via the YAML Pipeline API.

Production-ready: Apache top-level project with active governance (ASF, LICENSE/NOTICE present), comprehensive CI/CD setup (.github/workflows/ includes nightly builds, PR checks, docs deployment), 14M+ lines of Java indicating substantial codebase. Evidence of active development through versioned releases and multi-language connector ecosystem. Safe for production use with proper Flink infrastructure.

Low risk overall but monitor: large monorepo (14M LOC Java) means slow test cycles and potential merge conflicts; dependency on Apache Flink version compatibility (check flink_cdc_base.yml workflows for version pins); sheer connector breadth (MySQL, Postgres, Oracle, MongoDB, DB2, etc.) means test coverage must be deep per connector. ASF governance mitigates abandonment risk.

Active development: YAML Pipeline API is primary UX focus (see README prominence); schema evolution and routing/transformation features are being hardened (mentioned in docs); connector ecosystem expanding (Oracle, Postgres, TiDB, OceanBase visible); nightly CI runs suggest continuous regression testing. No specific PR/milestone data in file list, check GitHub Actions artifacts.

git clone https://github.com/apache/flink-cdc.git
cd flink-cdc
mvn clean install -DskipTests  # parent POM at root, Java 11+ required
# CLI available at flink-cdc-cli/target/flink-cdc-*.tar.gz after build

Daily commands:

# Build the distribution
mvn clean package -DskipTests -pl flink-cdc-cli -am

# Extract and run CLI
tar xzf flink-cdc-cli/target/flink-cdc-*.tar.gz
cd flink-cdc
./bin/flink-cdc.sh /path/to/pipeline.yaml

# Requires Apache Flink cluster or session mode to be running
# E.g., flink-1.18.x or newer with appropriate memory/slots

• pom.xml — Root Maven POM defining all dependencies, build profiles, and modules for the entire Flink CDC ecosystem across connectors and core packages.
• .github/workflows/flink_cdc_ci.yml — Primary CI/CD pipeline that validates builds, tests, and integration across all modules on every commit to master.
• README.md — Project overview, quickstart links, and feature summary that every contributor must understand before working on connectors or core functionality.
• docs/content.zh/docs/get-started/introduction.md — Foundational documentation explaining CDC concepts, architecture, and Flink CDC's role in the data pipeline ecosystem.
• .github/workflows/modules.py — Build automation script that orchestrates multi-module compilation and test execution across the distributed connector infrastructure.
• Dockerfile — Container definition for Flink CDC runtime deployment, essential for understanding deployment topology and dependency versions.
• LICENSE — Apache 2.0 license file required for legal compliance verification in all contributions to this ASF project.

Add Support for a New Database Connector

1. Create a new Maven module in the appropriate source directory (e.g., flink-cdc-connectors/flink-connector-[db-name]) with its own pom.xml inheriting from the parent (pom.xml)
2. Implement the SourceFunction or SourceProvider interface following patterns in existing connectors (MySQL, PostgreSQL, Oracle) (docs/content.zh/docs/connectors/flink-sources/_index.md)
3. Add changelog capture logic, schema extraction, and event deserialization specific to your database protocol (docs/content.zh/docs/core-concept/schema-evolution.md)
4. Create user documentation in docs/content.zh/docs/connectors/flink-sources/[db-name]-cdc.md with configuration options and examples (docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md)
5. Add integration tests to the module and register in .github/workflows/flink_cdc_ci.yml for automated validation (.github/workflows/flink_cdc_ci.yml)
6. Update the root pom.xml to include your new module in the build and ensure dependency versions align with parent BOM (pom.xml)

Add a New Sink Connector

1. Create a new Maven module in flink-cdc-connectors/flink-connector-[sink-name]-pipeline-connector with parent POM reference (pom.xml)
2. Implement the Sink interface (or DataStreamSink) with write logic, batch/async flushing, and error handling (docs/content.zh/docs/connectors/pipeline-connectors/_index.md)
3. Handle schema mapping and type conversion based on target system capabilities (docs/content.zh/docs/core-concept/type-mappings.md)
4. Write user documentation in docs/content.zh/docs/connectors/pipeline-connectors/[sink-name].md with examples and configuration reference (docs/content.zh/docs/connectors/pipeline-connectors/kafka.md)
5. Ensure integration test coverage and add to CI workflow matrix in .github/workflows/flink_cdc_ci.yml (.github/workflows/flink_cdc_ci.yml)

Add a Tutorial or Quickstart

1. Create a new markdown file in docs/content.zh/docs/get-started/quickstart-for-[version]/[scenario]-tutorial.md or docs/content.zh/docs/connectors/flink-sources/tutorials/[db]-tutorial.md (docs/content.zh/docs/get-started/quickstart-for-1.20/mysql-to-kafka.md)
2. Include step-by-step setup instructions, SQL/code examples, and sample output for users to replicate (docs/content.zh/docs/get-started/quickstart-for-1.20/_index.md)
3. Link the new tutorial from the relevant index page so it appears in navigation and documentation site (docs/content.zh/docs/connectors/flink-sources/tutorials/_index.md)
4. Verify documentation builds correctly by checking build_docs.yml workflow and testing locally with Hugo (.github/workflows/build_docs.yml)

• Apache Flink — Distributed streaming framework with built-in checkpointing, state management, and exactly-once semantics; essential for reliable CDC with low latency across petabyte-scale data
• Java / Maven — Primary development language and build tool enabling JVM deployment flexibility, wide ecosystem of database drivers, and compatibility with Flink's Java API
• YAML-based Pipeline DSL — Declarative configuration language allowing non-engineers to define multi-connector data pipelines without coding, reducing operational complexity
• Docker / Kubernetes — Containerized deployment model enables portability, resource isolation, and orchestration at scale using industry-standard cloud infrastructure
• Hugo Documentation Engine — Static site generator for fast, searchable documentation deployment; minimal dependencies for user-accessible API and connector reference

• Multi-module Maven architecture vs. monolithic JAR

  • Why: Allows independent evolution of each connector (MySQL, PostgreSQL, Oracle, etc.) and sink implementations without tight coupling
  • Consequence: Increased build complexity and dependency management overhead; slower per-module iteration for cross-connector features

• Schema evolution via DDL detection vs. schema registry

  • Why: Detects schema changes directly from database WAL/logs without external dependency, ensuring Flink CDC remains self-contained
  • Consequence: More complex parsing logic per database type; eventual consistency for downstream schema validation vs. registry-enforced schema contracts

• Batch writes with micro-batching vs. record-by-record

  • Why: Reduces latency variation and improves throughput for bulk sink operations (Kafka, Iceberg, Paimon)
  • Consequence: Higher memory consumption during batch accumulation; added complexity in handling partial-batch failures

1. Flink version lock: flink-cdc-base.yml will pin to specific Flink version; building against mismatched Flink can cause classloader hell. 2. Database prerequisites: MySQL connector requires binlog enabled (log_bin=ON, binlog_format=ROW), Postgres requires logical replication (wal_level=logical). 3. Stateful snapshot restore: CDC relies on Flink checkpoints; incomplete checkpoint barriers can cause duplicate events on failover. 4. YAML schema validation: snakeyaml 2.6 uses stricter parsing; unquoted colons or tabs in values will fail silently. 5. Slow test suite: integration tests spin up real database containers (testcontainers), so full mvn test runs 30+ min per connector.

• Change Data Capture (CDC) — Core concept of Flink CDC; understanding binlog/WAL tailing and changelog streams is essential to using this tool effectively
• Exactly-once semantics — Flink CDC guarantees at-least-once by default, but achieves exactly-once via distributed snapshots; critical for financial/dedup use cases
• Write-Ahead Logging (WAL) — PostgreSQL and Oracle use WAL for CDC; Flink CDC reads WAL LSNs to track replication position
• Binlog — MySQL/TiDB replication log; Flink CDC's MySQL connector parses binlog events to capture DML changes
• Schema Evolution — Flink CDC handles ALTER TABLE tracking and downstream schema adaptation; non-trivial in distributed systems
• Event routing / table sharding — Flink CDC's route feature consolidates sharded tables (e.g., orders_001, orders_002) into a single sink table; common in horizontally partitioned databases
• Distributed checkpointing — Flink's Chandy-Lamport algorithm ensures consistent snapshots across source/sink; failure recovery depends on checkpoint correctness

• apache/flink — Parent runtime; Flink CDC is built as a connector ecosystem on top of Flink's DataStream API and checkpoint mechanism
• debezium/debezium — Canonical CDC source library (Kafka Connector ecosystem); Flink CDC integrates some Debezium connectors but competes with different UX/deployment model
• airbnb/data-egress — Similar real-time data integration pattern; architectural reference for multi-sink routing
• apache/flink-kubernetes-operator — Companion project; Flink CDC pipelines typically deployed via this Kubernetes operator for prod orchestration
• alibaba/Canal — Predecessor CDC concept for MySQL; Flink CDC improves on it with Flink's distributed engine and multi-database support

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive integration tests for flink-cdc-cli YAML parsing and validation

The flink-cdc-cli module depends on snakeyaml-engine for YAML parsing but there are no visible test files in the provided structure for validating YAML configuration parsing, error handling, and schema validation. This is critical for a CDC tool where configuration errors directly impact data pipelines. New contributors can add tests covering malformed YAML, missing required fields, invalid connector types, and edge cases.

• [ ] Create flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/parser/ directory
• [ ] Add YamlConfigParserTest.java with tests for valid/invalid YAML structures
• [ ] Add YamlSchemaValidationTest.java testing required vs optional fields per connector type (MySQL, PostgreSQL, MongoDB, etc.)
• [ ] Add integration tests in flink-cdc-cli/src/test/java/org/apache/flink/cdc/cli/ConfigLoaderIntegrationTest.java
• [ ] Reference existing connector documentation in docs/content.zh/docs/connectors/flink-sources/ to ensure tests cover all supported databases

Implement GitHub Actions workflow for testing connector-specific documentation examples

The repo has extensive documentation for multiple connectors (MySQL, PostgreSQL, MongoDB, Oracle, DB2, etc.) in docs/content.zh/docs/connectors/flink-sources/ with tutorials, but there's no visible CI workflow that validates YAML examples embedded in documentation actually parse correctly. This prevents documentation drift and broken examples from merging.

• [ ] Create .github/workflows/validate_doc_examples.yml workflow
• [ ] Add script to extract YAML code blocks from .md files matching pattern yaml...
• [ ] Integrate flink-cdc-cli validation against extracted examples for each connector (mysql-cdc.md, postgres-cdc.md, mongodb-cdc.md, etc.)
• [ ] Run on pull requests modifying docs/content.zh/docs/connectors/
• [ ] Add job that validates schema compatibility against flink-cdc-common definitions

Add end-to-end test suite for pipeline connectors with mock sources

The repository documents multiple pipeline connectors (Doris, Elasticsearch, Hudi, Iceberg, Kafka, MaxCompute, MySQL) in docs/content.zh/docs/connectors/pipeline-connectors/ but there's no visible integration test directory validating that data flows correctly through each sink connector. This is high-value because pipeline correctness is critical for production use.

• [ ] Create flink-cdc-connectors/flink-connector-*/src/test/java/integration/ test directories for each pipeline connector
• [ ] Implement abstract PipelineConnectorE2ETest.java base class with CDC record generation and validation
• [ ] Add concrete test classes (DorisE2ETest, ElasticsearchE2ETest, IcebergE2ETest, KafkaE2ETest, etc.)
• [ ] Use testcontainers for containerized sink systems (Docker for Elasticsearch, Kafka, etc.)
• [ ] Validate schema mappings, type conversions, and CDC operation handling (INSERT/UPDATE/DELETE) for each connector

• Add integration test for schema evolution on Oracle connector (flink-connector-oracle-cdc/ lacks full test coverage of ALTER TABLE scenarios; see flink-connector-mysql-cdc/src/test for reference pattern)
• Write end-to-end YAML example docs for multi-table routing with MongoDB source (docs/content.zh/docs/connectors/flink-sources/mongodb-cdc.md has no routing examples; add 2-3 in docs/content/en/)
• Implement missing --dry-run flag in flink-cdc-cli to validate YAML without submitting to Flink (flink-cdc-cli/src/main/java/com/ververica/cdc/cli currently has no validation mode)

• High · SnakeYAML Deserialization Vulnerability — pom.xml (snakeyaml-engine 2.6 dependency) and Dockerfile (PIPELINE_DEFINITION_FILE). SnakeYAML version 2.6 is being used as a YAML parser. While 2.6 is relatively recent, SnakeYAML has a history of unsafe deserialization vulnerabilities (CVE-2013-4152, CVE-2017-18640). The use of snakeyaml-engine for parsing YAML configuration files (particularly PIPELINE_DEFINITION_FILE in Docker) could be susceptible to arbitrary code execution if untrusted YAML is processed without proper validation. Fix: 1) Ensure YAML parsing uses safe mode with explicit typing disabled. 2) Validate and sanitize all YAML input before parsing. 3) Monitor CVE databases for SnakeYAML vulnerabilities. 4) Consider using a safer YAML parser alternative if processing untrusted input. 5) Implement input schema validation.
• High · Incomplete Docker Base Image Configuration — Dockerfile (line 1: FROM flink). The Dockerfile uses 'FROM flink' without specifying a version tag. This leads to unpredictable base image pulls, potentially introducing vulnerable or unexpected versions of the Flink runtime. The image could be modified or contain security vulnerabilities without explicit versioning. Fix: Specify explicit version tags: 'FROM flink:1.x.x' or 'FROM flink:latest' with explicit version pinning in CI/CD. Use image scanning tools to validate the base image before deployment.
• High · Missing USER Directive in Dockerfile — Dockerfile (missing USER directive). The Dockerfile does not create or switch to a non-root user. All processes will run as root, violating container security best practices and increasing the attack surface significantly. Fix: Add a non-root user and switch to it before running the application: 'RUN useradd -m -u 1000 flinkcdc && USER flinkcdc'. Ensure proper file permissions are set for the CDC application.
• Medium · Unvalidated Command-Line Input (commons-cli) — pom.xml (commons-cli 1.7.0 dependency). Apache Commons CLI 1.7.0 is used for CLI argument parsing. While this is a stable version, the integration should ensure proper input validation. The CLI tool may process untrusted arguments that could lead to injection attacks or unexpected behavior. Fix: 1) Implement strict input validation for all CLI arguments. 2) Use whitelist-based validation for file paths and configuration options. 3) Escape special characters in user-provided inputs. 4) Add argument length limits to prevent buffer overflow attacks.
• Medium · Missing Docker Network Isolation — Dockerfile (infrastructure configuration). The Dockerfile does not configure network policies or expose restrictions. The container may communicate with unexpected services, creating lateral movement risks. Fix: 1) Explicitly document exposed ports (if any) with EXPOSE directive. 2) Implement network policies in Kubernetes manifests (documented in docs/content.zh/docs/deployment/kubernetes.md). 3) Use private networks for inter-service communication. 4) Restrict egress traffic to necessary endpoints only.
• Medium · Potential SQL Injection in CDC Connectors — Connector implementations (referenced in docs: mysql-cdc.md, postgres-cdc.md, oracle-cdc.md, sqlserver-cdc.md, db2-cdc.md). Flink CDC supports multiple database connectors (MySQL, PostgreSQL, Oracle, SQL Server, DB2). The codebase likely constructs SQL queries based on user-provided configurations. Without code inspection, potential SQL injection risks exist in schema discovery, table selection, and query construction. Fix: 1) Use parameterized queries and prepared statements for all database operations. 2) Implement input validation for schema/table names using whitelisting. 3) Apply principle of least privilege to database connections. 4) Add security testing for SQL injection vectors in connector tests.
• Medium · No Explicit Dependency Vulnerability Management — pom.xml (parent POM reference). The pom.xml does not specify dependency version ranges or use dependency-check plugin for known vulnerability scanning. Transitive dependencies may contain unpatched vulnerabilities. Fix:

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.