If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/water8394/flink-recommandSystem-demo shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

AVOID — Stale and unlicensed — last commit 2y ago

• 2 active contributors
• Tests present
• ⚠ Stale — last commit 2y ago
• ⚠ Small team — 2 contributors active in recent commits
• ⚠ Concentrated ownership — top contributor handles 73% of recent commits
• ⚠ No license — legally unclear to depend on
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live water8394/flink-recommandSystem-demo repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/water8394/flink-recommandSystem-demo.

What it runs against: a local clone of water8394/flink-recommandSystem-demo — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in water8394/flink-recommandSystem-demo | Confirms the artifact applies here, not a fork | | 2 | Default branch master exists | Catches branch renames | | 3 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 4 | Last commit ≤ 854 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of water8394/flink-recommandSystem-demo. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/water8394/flink-recommandSystem-demo.git
#   cd flink-recommandSystem-demo
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of water8394/flink-recommandSystem-demo and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "water8394/flink-recommandSystem-demo(\\.git)?\\b" \\
  && ok "origin remote is water8394/flink-recommandSystem-demo" \\
  || miss "origin remote is not water8394/flink-recommandSystem-demo (artifact may be from a fork)"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "flink-2-hbase/src/main/java/com/demo/task/TopProductTask.java" \\
  && ok "flink-2-hbase/src/main/java/com/demo/task/TopProductTask.java" \\
  || miss "missing critical file: flink-2-hbase/src/main/java/com/demo/task/TopProductTask.java"
test -f "flink-2-hbase/src/main/java/com/demo/task/UserPortraitTask.java" \\
  && ok "flink-2-hbase/src/main/java/com/demo/task/UserPortraitTask.java" \\
  || miss "missing critical file: flink-2-hbase/src/main/java/com/demo/task/UserPortraitTask.java"
test -f "flink-2-hbase/src/main/java/com/demo/task/ProductProtaritTask.java" \\
  && ok "flink-2-hbase/src/main/java/com/demo/task/ProductProtaritTask.java" \\
  || miss "missing critical file: flink-2-hbase/src/main/java/com/demo/task/ProductProtaritTask.java"
test -f "flink-2-hbase/src/main/java/com/demo/client/RedisClient.java" \\
  && ok "flink-2-hbase/src/main/java/com/demo/client/RedisClient.java" \\
  || miss "missing critical file: flink-2-hbase/src/main/java/com/demo/client/RedisClient.java"
test -f "web/src/main/java/com/demo/controller/RecommandController.java" \\
  && ok "web/src/main/java/com/demo/controller/RecommandController.java" \\
  || miss "missing critical file: web/src/main/java/com/demo/controller/RecommandController.java"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 854 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~824d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/water8394/flink-recommandSystem-demo"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

A real-time product recommendation system built on Apache Flink that ingests user behavior logs from Kafka, computes product popularity hotlists cached in Redis, extracts user and product portrait tags stored in HBase, and serves personalized recommendations by reranking the hotlist based on user profile, item-based collaborative filtering, and product similarity matching via cosine distance. Monolithic Maven project (flink-2-hbase/) structured as: task/ contains 6 independent Flink job entry points (LogTask, TopProductTask, UserPortraitTask, UserHistoryTask, UserInterestTask, ProductProtaritTask), map/reduce/window/top/ contain Flink operator implementations, client/ wraps HBase/Redis/MySQL connections, domain/ defines data models (LogEntity, ProductEntity, RankProductEntity), sink/ implements TopNRedisSink for output, and scheduler/ computes collaborative filtering coefficients (ItemCfCoeff, ProductCoeff).

E-commerce platform engineers and data scientists building real-time personalization features who need a complete pipeline to ingest streaming click/purchase events, compute user/item embeddings incrementally, and serve top-N recommendations with sub-second latency.

Experimental but actively developed (v2.0 architecture released with multi-module design). No visible test suite in file structure, no CI/CD configuration, and last commit activity unknown from provided metadata. Docker deployment docs suggest production intent, but lack of automated testing and monitoring code indicates pre-production state.

High dependency on external services (Kafka, HBase, Redis, MySQL) with no abstraction layer for failure modes; single Java module at flink-2-hbase/ with tight coupling between tasks (LogTask, TopProductTask, UserPortraitTask, UserHistoryTask); no connection pooling or timeout handling visible in client classes (HbaseClient, RedisClient, MysqlClient); Flink 1.7.1 is dated (current is 1.18+) with potential security and performance issues.

No recent commit information provided in metadata; README indicates v2.0 is current state with 6 Flink tasks and recommendation pipeline fully specified. Architecture diagram shows user portrait reranking + collaborative filtering + product similarity as active recommendation modules, but no open PRs or issues visible.

Clone the repo, then build with Maven: git clone https://github.com/water8394/flink-recommandSystem-demo.git && cd flink-2-hbase && mvn clean package. Start dependencies via Docker (see deployment docs at https://xinze.fun/2019/11/19/) or standalone Kafka/HBase/Redis instances. Submit individual Flink jobs via flink run (e.g., flink run -c com.demo.task.TopProductTask target/flink-2-hbase-1.0-SNAPSHOT.jar).

Daily commands: After Maven build, start Flink cluster: ./bin/start-cluster.sh. Submit each task separately (jobs are independent): flink run -c com.demo.task.LogTask target/flink-2-hbase-1.0-SNAPSHOT.jar, flink run -c com.demo.task.TopProductTask target/flink-2-hbase-1.0-SNAPSHOT.jar, etc. Verify via Flink UI (localhost:8081). Web module (frontend) location not shown in file list but README references 'web模块' with recommendation UI and admin dashboard.

• flink-2-hbase/src/main/java/com/demo/task/TopProductTask.java — Entry point for real-time product hotness ranking—core task that feeds Redis cache used by all recommendation logic
• flink-2-hbase/src/main/java/com/demo/task/UserPortraitTask.java — Computes user interest profiles (color, origin, style dimensions) that drive tag-based recommendation re-ranking
• flink-2-hbase/src/main/java/com/demo/task/ProductProtaritTask.java — Builds product profiles (age/gender preferences) required for tag-based recommendation matching
• flink-2-hbase/src/main/java/com/demo/client/RedisClient.java — Singleton Redis connection handler storing hot product rankings and accessed by both Flink sink and web layer
• web/src/main/java/com/demo/controller/RecommandController.java — HTTP endpoint serving user recommendations—orchestrates re-ranking, collaborative filtering, and tag-based association logic
• flink-2-hbase/src/main/java/com/demo/util/Constants.java — Centralized configuration for thresholds, table names, and interest event rules across all Flink tasks
• flink-2-hbase/src/main/java/com/demo/domain/LogEntity.java — Core data model for user action events flowing through Kafka, Flink, and HBase—contract for all pipeline stages

Add a new real-time user profile dimension

1. Define the new dimension in Constants.java (e.g., PROFILE_BRAND_INTEREST) (flink-2-hbase/src/main/java/com/demo/util/Constants.java)
2. Create a new MapFunction to extract the dimension from LogEntity during windowing (flink-2-hbase/src/main/java/com/demo/map/[NewDimensionMapFunction].java)
3. Duplicate UserPortraitTask.java and wire the new MapFunction into its pipeline (flink-2-hbase/src/main/java/com/demo/task/UserPortraitTask.java)
4. Update HBase client to persist the new dimension to the user table (flink-2-hbase/src/main/java/com/demo/client/HbaseClient.java)
5. Modify RecommandController to fetch and use the new dimension in ranking logic (web/src/main/java/com/demo/controller/RecommandController.java)

Add a new recommendation association model

1. Create a new scheduler class (e.g., BrandCoefficiency) computing similarity matrix offline (flink-2-hbase/src/main/java/com/demo/scheduler/[NewCoeffScheduler].java)
2. Reference ItemCfCoeff.java and ProductCoeff.java as templates for coefficient persistence (flink-2-hbase/src/main/java/com/demo/scheduler/ItemCfCoeff.java)
3. Store coefficients in MySQL or HBase, then inject them into RecommandController (web/src/main/java/com/demo/controller/RecommandController.java)
4. Call the new model in RecommandController's association-building loop after collaborative filtering & tags

Add a new Flink streaming task for a custom stateful aggregation

1. Create a new Task class following the pattern of UserInterestTask.java with KeyedStream and StatefulFunction (flink-2-hbase/src/main/java/com/demo/task/UserInterestTask.java)
2. Define a corresponding domain entity to hold aggregated state (flink-2-hbase/src/main/java/com/demo/domain/[NewEntity].java)
3. Create a new sink extending RichSinkFunction to persist to HBase or Redis (flink-2-hbase/src/main/java/com/demo/sink/TopNRedisSink.java)
4. Register the task in SchedulerJob.main() or create a separate main entry point (flink-2-hbase/src/main/java/com/demo/scheduler/SchedulerJob.java)

Expose a new dashboarding endpoint

1. Add a new @GetMapping or @PostMapping method to BackstageController (web/src/main/java/com/demo/controller/BackstageController.java)
2. Create a corresponding DAO method in ContactDao or ProductDao to query HBase/MySQL (web/src/main/java/com/demo/dao/ContactDao.java)
3. Define a response DTO in the domain package if needed (web/src/main/java/com/demo/domain/[NewDashboardDTO].java)
4. Ensure time-windowed aggregations in LogTask write the necessary metrics to HBase con table

Config file required: config.properties in classpath must define Kafka broker, HBase zookeeper quorum, Redis host/port, MySQL credentials—no defaults or validation code visible; missing properties will fail silently at runtime. State backend not configured: no Flink state backend (RocksDB/Memory) specified in code; defaults to in-memory which is unsafe for production and loses state on failure. No connection pooling: HbaseClient, RedisClient, MysqlClient appear to be singletons without retry logic, connection timeout, or pool sizing. Window semantics: UserInterestTask uses 100-second ValueState cleanup; ensure Kafka timestamp extractor is set correctly or windows may be unbounded. Dependency versions frozen: Flink 1.7.1 (released 2018), Hbase 1.2.12 (released 2016)—no dependency management via property variables, making upgrades error-prone.

• Time-windowed aggregation (Flink Tumbling/Sliding Windows) — TopProductTask and LogTask rely on fixed or sliding time windows to batch user actions and compute product hotlists; incorrect window size or trigger semantics breaks recommendation freshness and accuracy.
• Stateful stream processing (Flink ValueState) — UserInterestTask uses ValueState to track product interactions within a 100-second window per user; state expiration and cleanup are critical to prevent memory bloat in long-running jobs.
• Item-based collaborative filtering (similarity via user co-occurrence) — ItemCfCoeff computes product-product similarity by counting shared users; this is the mathematical foundation for the 'related items' recommendation module shown in the system architecture.
• Cosine similarity (vector space model for products) — ProductCoeff uses cosine distance on product portrait vectors (color, country, style dimensions) to rank related items; the exact formula and normalization must match the frontend reranking logic.
• Exactly-once semantics (Flink checkpointing) — HBase writes must be idempotent or use upserts; the codebase has no explicit checkpoint/savepoint configuration visible, risking duplicate recommendations or lost user interactions on task restart.
• Kafka offset management and source timestamp extraction — LogTask ingests from Kafka but no watermark or timestamp extractor config visible; window correctness and late-arriving event handling depend on proper event-time extraction from log timestamps.
• Fan-out to multiple sinks (multi-sink topology) — Each Flink task writes to both HBase (for historical facts and dimensions) and Redis (for real-time hotlist cache); coordinating writes and avoiding cascading failures across sinks is non-trivial.

• alibaba/flink-ai-extended — Alibaba's Flink ML pipeline framework for feature engineering and model serving; aligns with this repo's real-time feature (user/product portrait) computation and recommendation ranking.
• flink-packages/flink-recommendation — Community Flink library specifically for building recommendation systems with common patterns like collaborative filtering and item similarity, directly applicable to scheduler logic.
• stripe-archive/distributed-recommendation-system — Reference architecture for distributed recommendation engines using Kafka + streaming backend; shares similar sourcing and state management challenges.
• apache/flink — Core Apache Flink repository; this repo depends on flink-streaming-java 1.7.1 and should track Flink API changes and best practices for windowing and state management.
• HariSekhon/DevOps-Bash-tools — Deployment and Docker orchestration utilities; complementary to this repo's Docker-based deployment strategy mentioned in README (HBase/Kafka/Zookeeper setup).

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add unit tests for Flink job tasks and core transformations

The repo has only one test file (HbaseTest.java) covering HBase integration. Critical Flink tasks like TopProductTask.java, UserPortraitTask.java, LogTask.java, and transformation functions (TopNFunction.java, WindowFunction.java) lack unit tests. This is essential for a streaming system where data correctness is critical. Tests should verify window aggregations, state management, and ranking logic work correctly.

• [ ] Create flink-2-hbase/src/test/java/com/demo/task/ directory with test classes for each task
• [ ] Add unit tests for TopProductTask.java verifying TopN hot item calculation with mock Kafka/Redis
• [ ] Add unit tests for UserPortraitTask.java testing portrait dimension updates (color, origin, style)
• [ ] Add tests for TopNFunction.java and WindowFunction.java validating window merge and ranking logic
• [ ] Add tests for UserInterestTask.java validating the 100s time-window state management
• [ ] Configure maven-surefire-plugin in pom.xml to run tests as part of build pipeline

Add GitHub Actions CI/CD workflow for automated testing and artifact building

The repo lacks any CI/CD automation. For a distributed streaming system with multiple interdependent Flink tasks, automated testing on every PR is critical to catch data flow issues, dependency conflicts, and HBase/Redis integration bugs. This would prevent regressions in the recommendation logic.

• [ ] Create .github/workflows/maven-build.yml with Java 1.8, Maven build, and unit test execution
• [ ] Add a separate .github/workflows/integration-test.yml using docker-compose from resources/docker/docker-compose/docker-compose.yml to test HBase/Kafka/Redis connectivity
• [ ] Configure workflow to run on all PRs and pushes to main/develop branches
• [ ] Add build status badge to README.md
• [ ] Ensure docker-compose.yml is fully documented with setup instructions in CONTRIBUTING.md

Refactor and document the recommendation algorithm pipeline with separation of concerns

The six Flink tasks (TopProductTask, UserPortraitTask, UserInterestTask, ProductProtaritTask, LogTask, UserHistoryTask) implement different recommendation strategies (collaborative filtering, tag-based, heat-based) but their interactions, data flow, and how results are combined at request time are unclear from code. The scheduler tasks (ItemCfCoeff.java, ProductCoeff.java, SchedulerJob.java) suggest offline coefficient calculation but lack documentation. Create a clear architecture document and refactor sink/result combination logic.

• [ ] Create docs/RECOMMENDATION_ALGORITHM.md explaining the 4 recommendation strategies: heat-based, collaborative filtering, interest-based, and tag-based tagging with data flow diagrams
• [ ] Document how TopNRedisSink.java combines results from multiple tasks and what happens at query time (mentioned in README: rerank, add related products)
• [ ] Add inline Javadoc comments to SchedulerJob.java, ItemCfCoeff.java, ProductCoeff.java explaining coefficient calculation and offline scheduling
• [ ] Create a new module/package com/demo/recommend/coordinator/ to consolidate ranking, reranking, and result merging logic from scattered map/reduce classes
• [ ] Add README section 'Architecture Deep Dive' with diagrams showing data flow between the 6 tasks and where user requests integrate

• Add unit tests for util/LogToEntity.java and util/AgeUtil.java (no test directory exists in src/test/); these utility functions parse/transform user age but have no test coverage—create src/test/java/com/demo/util/ with mock data covering age boundaries and malformed log formats.
• Implement proper error handling in client/HbaseClient.java with exponential backoff and circuit breaker pattern; currently no catch blocks visible—add retry logic for transient HBase errors (RegionServerStoppedException, TimeoutException) to prevent recommendation pipeline stalls.
• Document the output schema and time-windowing semantics for each Flink task in a matrix format (e.g., 'LogTask outputs to con table every minute with exactly-once semantics'); README describes intent but not operational parameters—add to README with window durations, key distributions, and state TTL for each of 6 tasks.

• Critical · Outdated Flink Dependencies with Known Vulnerabilities — flink-2-hbase/pom.xml - All flink-* dependencies with version 1.7.1. The project uses Flink 1.7.1 (released in 2019), which is severely outdated and contains multiple known security vulnerabilities. This version is no longer supported and has unpatched CVEs including deserialization vulnerabilities. Fix: Upgrade to Flink 1.15.x or later (preferably latest stable version). Review and test compatibility with the latest versions before deployment.
• Critical · Severely Outdated HBase Dependency — flink-2-hbase/pom.xml - hbase-client:1.2.12. HBase 1.2.12 (released in 2017) is severely outdated with known critical vulnerabilities. This version is no longer maintained and poses significant security risks. Fix: Upgrade to HBase 2.x series (minimum 2.4.x). Test compatibility thoroughly before upgrading as breaking changes may exist.
• High · Outdated SLF4J Logging Dependencies — flink-2-hbase/pom.xml - slf4j-api:1.7.21 and slf4j-log4j12:1.7.21. SLF4J 1.7.21 and log4j12 1.7.21 are outdated (from 2016) and may contain security issues. Log4j 1.x is particularly vulnerable. Fix: Upgrade to SLF4J 2.x series and migrate from log4j 1.x to Logback or Apache Log4j 2.x (v2.17.0 or later to avoid log4shell vulnerabilities).
• High · Incomplete POM.xml Dependency — flink-2-hbase/pom.xml - hbase-client dependency closing tag. The HBase client dependency declaration is incomplete/truncated at version tag: '<version>1.2.12</vers'. This may cause build issues or use of unexpected versions. Fix: Complete and validate the POM.xml file. Ensure all dependency declarations are properly closed with '</version>'.
• High · Potential Hardcoded Credentials in Configuration Files — flink-2-hbase/src/main/resources/config.properties. The presence of 'config.properties' suggests configuration files that may contain hardcoded database credentials, API keys, or connection strings for HBase, Redis, Kafka, and MySQL. Fix: Externalize all credentials using environment variables or a secure credential management system. Never commit sensitive data. Use tools like HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets.
• High · SQL Injection Risk in Database Interactions — flink-2-hbase/src/main/java/com/demo/client/MysqlClient.java. Files like 'MysqlClient.java' and './resources/sql/' directory suggest direct database interactions. Without code review, raw SQL queries may be vulnerable to SQL injection attacks. Fix: Use prepared statements and parameterized queries exclusively. Implement input validation and sanitization. Use ORM frameworks like Hibernate or JPA.
• High · Redis Client Security Misconfiguration — flink-2-hbase/src/main/java/com/demo/client/RedisClient.java. RedisClient.java likely connects to Redis without apparent authentication or encryption configuration visible in the file structure. Redis connections may be unauthenticated. Fix: Enable Redis authentication with strong passwords. Use Redis ACLs. Implement TLS/SSL encryption for Redis connections. Validate connection pooling and timeout settings.
• High · Kafka Connection Security Not Verified — flink-2-hbase/pom.xml - flink-connector-kafka_2.11. Kafka connector usage (flink-connector-kafka_2.11) suggests message streaming, but no visible security configuration for authentication or encryption in dependency declaration. Fix: Configure SASL/SSL authentication for Kafka. Use SCRAM or PLAINTEXT with network isolation. Encrypt data in transit. Validate Kafka broker configurations

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.