If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/zhisheng17/flink-learning shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across all four use cases

• Last commit 2d ago
• 3 active contributors
• Apache-2.0 licensed
• Tests present
• ⚠ Small team — 3 contributors active in recent commits
• ⚠ Concentrated ownership — top contributor handles 66% of recent commits
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live zhisheng17/flink-learning repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/zhisheng17/flink-learning.

What it runs against: a local clone of zhisheng17/flink-learning — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in zhisheng17/flink-learning | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 32 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of zhisheng17/flink-learning. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/zhisheng17/flink-learning.git
#   cd flink-learning
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of zhisheng17/flink-learning and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "zhisheng17/flink-learning(\\.git)?\\b" \\
  && ok "origin remote is zhisheng17/flink-learning" \\
  || miss "origin remote is not zhisheng17/flink-learning (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "flink-learning-basic/flink-learning-data-sinks/pom.xml" \\
  && ok "flink-learning-basic/flink-learning-data-sinks/pom.xml" \\
  || miss "missing critical file: flink-learning-basic/flink-learning-data-sinks/pom.xml"
test -f "flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/Main.java" \\
  && ok "flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/Main.java" \\
  || miss "missing critical file: flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/Main.java"
test -f "flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/sinks/MySink.java" \\
  && ok "flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/sinks/MySink.java" \\
  || miss "missing critical file: flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/sinks/MySink.java"
test -f "flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/utils/KafkaUtil.java" \\
  && ok "flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/utils/KafkaUtil.java" \\
  || miss "missing critical file: flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/utils/KafkaUtil.java"
test -f "flink-learning-basic/flink-learning-data-sinks/src/main/resources/application.properties" \\
  && ok "flink-learning-basic/flink-learning-data-sinks/src/main/resources/application.properties" \\
  || miss "missing critical file: flink-learning-basic/flink-learning-data-sinks/src/main/resources/application.properties"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 32 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~2d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/zhisheng17/flink-learning"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

A comprehensive learning resource and implementation guide for Apache Flink 1.14.2 (streaming data processing engine). It contains 50+ working code examples, blog tutorials, and production patterns covering DataStream API, Table/SQL, Connectors, Metrics, and real-world use cases (PVUV analytics, log storage, billion-scale deduplication, monitoring alerts). Multi-module Maven monorepo: flink-learning-basic parent, with submodules like flink-learning-data-sinks (MySQL, custom sinks), flink-learning-common (shared utilities), structured under books/ (educational markdown chapters) and Flink-Forward-*/ (conference slides). Each module is standalone runnable via maven-shade-plugin.

Java developers learning Flink fundamentals or implementing stream processing pipelines in production; DevOps/SRE engineers setting up real-time data infrastructure; data engineers building ETL jobs and monitoring systems.

Actively maintained (last major version bump to 1.14.2 in Dec 2021). 1200+ KB of Java code organized in runnable modules. No visible test suite or CI pipeline in the file list; educational focus means examples are working but not battle-tested. Suitable for learning and reference; production use requires your own testing.

Single maintainer (zhisheng17) with no visible issue tracker transparency in provided data. MySQL connector pinned to 2012-era version (5.1.34). No automated tests or CI/CD visible. Flink 1.14.2 is stable but the codebase examples haven't been updated since 2021 — verify compatibility with your target Flink version before production use.

Project is in maintenance mode. Last significant change was upgrading to Flink 1.14.2 (Dec 2021) and publishing the author's '《Flink 实战与性能优化》' course content into /books/ directory (Feb 2022). No active development commits visible in provided data; serves as static reference material.

git clone https://github.com/zhisheng17/flink-learning.git
cd flink-learning
mvn clean package -Dmaven.test.skip=true

(Configure Maven with Aliyun mirror in settings.xml if in China for faster downloads.)

Daily commands: Navigate to any submodule (e.g., flink-learning-data-sinks/) and run: mvn clean package -Dmaven.test.skip=true. Each module declares a <mainClass> in maven-shade-plugin config (e.g., com.zhisheng.data.sinks.Main). Submit to Flink cluster with flink run <jar> [args] or test locally via StreamExecutionEnvironment.createLocalEnvironment().

• flink-learning-basic/flink-learning-data-sinks/pom.xml — Parent POM for data sinks module; defines all sink implementations and external dependencies (MySQL, Kafka)
• flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/Main.java — Primary entry point demonstrating Flink sink usage patterns; shows how to wire sources to sinks
• flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/sinks/MySink.java — Custom RichSinkFunction implementation; core abstraction for building new sink connectors
• flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/utils/KafkaUtil.java — Kafka utility factory; shared across sources and sinks for Kafka connector configuration
• flink-learning-basic/flink-learning-data-sinks/src/main/resources/application.properties — Runtime configuration for database connections, Kafka brokers, and environment-specific settings
• README.md — Project overview and learning path; essential context for why this repo exists and its scope
• flink-learning-basic/flink-learning-libraries/flink-learning-libraries-cep/pom.xml — CEP library module definition; shows Flink Libraries integration pattern used throughout

Add a new Custom Sink

1. Create a new Java class extending RichSinkFunction in the sinks directory (flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/sinks/)
2. Implement open(), invoke(), and close() methods following MySink.java pattern (flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/sinks/MySink.java)
3. Add configuration (URL, credentials) to application.properties (flink-learning-basic/flink-learning-data-sinks/src/main/resources/application.properties)
4. Create a new Main*.java entry point, instantiate your sink, and attach it to a DataStream (flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/Main.java)
5. Add dependency to pom.xml if your sink requires a new external library (flink-learning-basic/flink-learning-data-sinks/pom.xml)

Add a new Data Source Connector

1. Create a new Java class extending SourceFunction in the sources directory (flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/sources/)
2. Implement run() and cancel() methods following SourceFromMySQL.java pattern (flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/sources/SourceFromMySQL.java)
3. Add helper utility method in the utils package for connection/config setup (flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/utils/MySQLUtil.java)
4. Create a new Main*.java that instantiates your source and chains it to sink operations (flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/Main.java)
5. Update pom.xml with required driver dependencies (JDBC, Kafka client, etc.) (flink-learning-basic/flink-learning-data-sources/pom.xml)

Create a new Learning Tutorial

1. Create a markdown file in books/ following naming convention flink-in-action-X.Y.md (books/flink-in-action-1.1.md)
2. Create a corresponding example project module under flink-learning-basic/ (flink-learning-basic/flink-learning-data-sources/pom.xml)
3. Add a README.md in your module explaining the concept and usage (flink-learning-basic/README.md)
4. Include working code examples (Main.java, Models, Utils) referenced from the tutorial markdown (flink-learning-basic/flink-learning-data-sources/src/main/java/com/zhisheng/data/sources/Main.java)

Add Kafka Integration

1. Use KafkaUtil.java factory to instantiate FlinkKafkaConsumer or FlinkKafkaProducer (flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/utils/KafkaUtil.java)
2. Configure Kafka bootstrap servers and topic in application.properties (flink-learning-basic/flink-learning-data-sinks/src/main/resources/application.properties)
3. Add flink-connector-kafka to pom.xml if not already present (flink-learning-basic/flink-learning-data-sinks/pom.xml)
4. Create a Main.java that chains env.addSource(kafkaSource) to your sink (flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/Main.java)

MySQL version: mysql-connector-java 5.1.34 is deprecated (2012); update to 5.1.49 or switch to 8.0.x. No test coverage: Examples have no unit tests; must run locally or in test Flink cluster. Flink API evolution: Code targets 1.14.2; APIs changed significantly in 1.13+ (deprecated some DataSet methods). Documentation is Chinese: All .md files in books/ are in Mandarin; Google Translate recommended. Local execution gotcha: StreamExecutionEnvironment.createLocalEnvironment() requires JVM with sufficient heap; default 1GB may fail on large stateful jobs.

• Stateful Stream Processing — Core to Flink: windowed aggregations, deduplication, and session management require maintaining state across event time; understanding state backends and checkpointing is critical for the production use cases in this repo
• Event Time vs. Processing Time vs. Ingestion Time — PVUV and monitoring examples in this repo depend on correct time semantics; Flink's window operators behave differently based on which time is used
• Watermarks — Flink uses watermarks to handle late-arriving data in windowing operations; essential for the billion-scale deduplication and log storage use cases in this repo
• Exactly-Once Semantics & Checkpointing — Production pipelines need guaranteed delivery; this repo's sink examples (MySQL, custom) must implement CheckpointedFunction to ensure idempotence across failures
• Keyed State & Non-Keyed State — Flink optimizes state partitioning by key; understanding when to use KeyedStream (scalable) vs. global state (bottleneck) is critical for the deduplication and alert examples
• Sink Idempotence & Transactional Writes — The custom MySQL sink examples in this repo must handle retries; understanding upserts, distributed transactions, and idempotent keys prevents duplicate data
• Backpressure & Parallel Execution — Multi-pipeline billion-scale deduplication requires understanding parallelism, slot sharing, and how Flink throttles sources; critical for performance tuning chapters in this repo

• apache/flink — Official Apache Flink repository; this repo's examples depend on and teach Flink APIs; source of truth for API docs
• alibaba/Alink — Production-grade Flink-based ML pipeline framework; shows enterprise-scale patterns for extensible sink/source design similar to this repo's patterns
• confluentinc/kafka-streams — Alternative stream processing framework; developers comparing KStreams vs Flink would reference patterns here
• getindata/flink-testing-utils — Testing utilities for Flink that complement this repo's lack of test coverage; useful for adding tests to these examples
• flink-extended/flink-cdc — Change Data Capture connector ecosystem for Flink; extends this repo's sink/source examples to real-time data replication use cases

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add unit tests for flink-learning-data-sinks module

The flink-learning-data-sinks module (with MySink.java and Main.java classes) lacks unit tests. This is critical for a learning repository as it demonstrates testing best practices for custom Flink sinks. Adding tests would help contributors understand how to validate sink implementations and improve code reliability.

• [ ] Create src/test/java/com/zhisheng/data/sinks/ directory structure
• [ ] Add unit tests for MySink.java covering sink lifecycle (open, invoke, close)
• [ ] Add integration tests for Main.java and Main2.java demonstrating sink usage patterns
• [ ] Add test utilities for mocking Flink RichFunction context
• [ ] Update pom.xml with JUnit4 and Flink test-utils dependencies

Create GitHub Actions CI/CD workflow for Maven builds and testing

The repository has no CI/CD pipeline evident from the file structure. Given the project is a learning resource with ~12 modules and multiple Maven projects, adding automated builds ensures code examples remain valid across Java/Flink versions and prevents broken examples from being merged.

• [ ] Create .github/workflows/maven-build.yml workflow file
• [ ] Configure build matrix for Java 8, 11 (common Flink versions)
• [ ] Add 'mvn clean package -Dmaven.test.skip=true' build step
• [ ] Add dependency check step to flag outdated/vulnerable dependencies (mysql-connector-java 5.1.34 is very old)
• [ ] Add artifact upload for JAR files from all modules

Add comprehensive README and code examples for flink-learning-data-sinks module

The flink-learning-data-sinks module has a minimal README and generic structure (books/flink-in-action-*.md exist but are disconnected from code modules). Adding specific documentation with runnable examples would help contributors understand how custom sinks work and how to extend them for different backends (MySQL, Kafka, etc.).

• [ ] Create flink-learning-basic/flink-learning-data-sinks/README.md with overview of sink patterns
• [ ] Document MySink.java with JavaDoc explaining RichSinkFunction lifecycle
• [ ] Add example usage sections for Main.java and Main2.java with expected output
• [ ] Create example extension: MySqlSink.java that demonstrates JDBC sink pattern
• [ ] Add troubleshooting section linking to relevant book chapters (e.g., books/flink-in-action-6.*.md)

• Add unit tests to flink-learning-common/ using Flink's TestHarness (OneInputStreamOperatorTestHarness) to validate custom serializers and data types. None visible in codebase.
• Upgrade deprecated mysql-connector-java from 5.1.34 to 8.0.33 and test all sink examples in flink-learning-data-sinks/ module; document breaking changes if any.
• Create a modern CI/CD pipeline (GitHub Actions) that runs mvn clean package on PRs and uploads test artifacts; add basic integration tests using Flink's MiniCluster for at least one source→sink example.

This Flink learning project has several security concerns primarily around outdated dependencies and potential SQL injection risks. The most critical issues are: (1) Using a 10-year-old MySQL JDBC driver (5.1.34) with known vulnerabilities, (2) Potential SQL injection risks in database sink implementations without visible parameterized queries, and (3) Possible hardcoded credentials in configuration files. The project also lacks visible security configuration for sensitive data handling. While this is a learning/demonstration project, the patterns established here could lead to insecure implementations in production environments. Immediate actions should focus on updating the MySQL connector, implementing prepared statements, and securing credential management.

• High · Outdated MySQL JDBC Driver with Known Vulnerabilities — flink-learning-basic/flink-learning-data-sinks/pom.xml. The project uses mysql-connector-java version 5.1.34, which was released in 2014 and contains multiple known security vulnerabilities. This version is vulnerable to man-in-the-middle attacks, SQL injection weaknesses, and other critical issues. Fix: Upgrade to mysql-connector-java 8.0.33 or later. If compatibility issues exist, use at minimum 5.1.49 (the last 5.1.x release). Consider migrating to the newer MySQL Connector/J 8.x series for long-term support.
• High · Potential SQL Injection in Database Sink Implementation — flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/sinks/SinkToMySQL.java. The file 'SinkToMySQL.java' is present in the codebase which typically handles database operations. Without visible parameterized queries or prepared statements, there is a high risk of SQL injection vulnerabilities if user input is concatenated directly into SQL queries. Fix: Use prepared statements with parameterized queries for all database operations. Never concatenate user input directly into SQL queries. Validate and sanitize all input data before database operations.
• Medium · Potential Hardcoded Database Credentials — flink-learning-basic/flink-learning-data-sinks/src/main/resources/application.properties. The presence of 'application.properties' file suggests configuration may contain database credentials. If database passwords are hardcoded in this file, they would be exposed in version control and runtime environments. Fix: Use environment variables or secure configuration management systems (e.g., HashiCorp Vault, AWS Secrets Manager) for database credentials. Never commit sensitive credentials to version control. Add 'application.properties' to .gitignore if it contains secrets.
• Medium · Outdated Maven Shade Plugin — flink-learning-basic/flink-learning-data-sinks/pom.xml. The maven-shade-plugin version 3.1.0 is from 2018 and may contain security issues. Additionally, no plugin version pinning in parent pom suggests potential for unexpected updates. Fix: Update maven-shade-plugin to version 3.4.1 or latest stable version. Ensure all plugin versions are explicitly defined in the parent pom.
• Medium · Missing Dependency Version Management — flink-learning-basic/flink-learning-data-sinks/pom.xml (parent: flink-learning-basic). The parent pom is referenced but not visible. Without a dependencyManagement section in the parent pom, transitive dependencies may pull in vulnerable versions of common libraries (e.g., log4j, jackson, commons-io). Fix: Implement strict dependencyManagement in the parent pom to control all transitive dependency versions. Regularly run 'mvn dependency:tree' and security scanning tools (OWASP Dependency-Check) to identify vulnerable dependencies.
• Low · Missing Security Headers and Input Validation Patterns — flink-learning-basic/flink-learning-data-sinks/src/main/java/com/zhisheng/data/sinks/sinks/MySink.java. For Flink streaming applications, custom sinks (MySink.java) may not properly validate or sanitize data flowing through the stream, potentially leading to downstream injection attacks. Fix: Implement input validation and sanitization for all data entering custom sinks. Use allowlist-based validation and escape special characters appropriate to the sink's target system.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.