If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/pwxcoo/chinese-xinhua shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

WAIT — Stale — last commit 2y ago

• 3 active contributors
• MIT licensed
• ⚠ Stale — last commit 2y ago
• ⚠ Small team — 3 contributors active in recent commits
• ⚠ Single-maintainer risk — top contributor 87% of recent commits
• ⚠ No CI workflows detected
• ⚠ No test directory detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live pwxcoo/chinese-xinhua repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/pwxcoo/chinese-xinhua.

What it runs against: a local clone of pwxcoo/chinese-xinhua — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in pwxcoo/chinese-xinhua | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 893 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of pwxcoo/chinese-xinhua. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/pwxcoo/chinese-xinhua.git
#   cd chinese-xinhua
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of pwxcoo/chinese-xinhua and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "pwxcoo/chinese-xinhua(\\.git)?\\b" \\
  && ok "origin remote is pwxcoo/chinese-xinhua" \\
  || miss "origin remote is not pwxcoo/chinese-xinhua (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "data/idiom.json" \\
  && ok "data/idiom.json" \\
  || miss "missing critical file: data/idiom.json"
test -f "data/word.json" \\
  && ok "data/word.json" \\
  || miss "missing critical file: data/word.json"
test -f "data/ci.json" \\
  && ok "data/ci.json" \\
  || miss "missing critical file: data/ci.json"
test -f "data/xiehouyu.json" \\
  && ok "data/xiehouyu.json" \\
  || miss "missing critical file: data/xiehouyu.json"
test -f "scripts/chengyu.py" \\
  && ok "scripts/chengyu.py" \\
  || miss "missing critical file: scripts/chengyu.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 893 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~863d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/pwxcoo/chinese-xinhua"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

chinese-xinhua is a Chinese linguistic database and dictionary aggregator that collects 16,142 Chinese characters, 264,434 words/phrases (词语), 31,648 idioms (成语), and 14,032 witty sayings (歇后语) into structured JSON files. It solves the problem of having no readily available, open-source Chinese reference dataset for NLP projects, chengyu games, or language tools by providing pre-scraped and deduplicated data with pinyin, radical information, and explanations. Simple flat structure: data/ folder contains four JSON files (idiom.json, ci.json, word.json, xiehouyu.json) that are the end products. scripts/ folder holds independent Python scrapers (chengyu.py, ci.py, word.py, xiehouyu.py) used to generate those JSON files from web sources. archived/ holds deprecated data. The Jupyter notebook scripts/clean.ipynb suggests data cleaning was a manual, exploratory process.

Chinese NLP engineers and game developers who need authoritative linguistic datasets without building scrapers themselves; particularly useful for developers building chengyu-lianlong (成语接龙) word games, Chinese language learning tools, or semantic search systems that require canonical character/word definitions with pinyin romanization.

Moderately mature but dormant. The data is stable (last major changelog entry 2018-12-16), no active test suite visible, no CI/CD pipeline configured. The project successfully serves as a data archive with 7,455 lines of Python scraper logic, but shows no recent commits visible in the repo snapshot—treat it as a frozen dataset rather than actively developed software.

Low technical risk but high dependency on stale data. No external Python dependencies are enforced (scripts appear to use stdlib), and the data files are self-contained JSON. However, the content itself may be outdated (4+ years old) and the web scraping scripts (scripts/*.py) may break if source websites changed their structure. Single maintainer (pwxcoo) with no apparent contribution guidelines or issue triage process.

Nothing—this is a completed dataset archive. The last logged activity was 2018-12-16 (deduplication of idiom data and API deprecation). No open PRs, issues, or active development visible. The repo functions purely as a static data distribution point.

git clone https://github.com/pwxcoo/chinese-xinhua.git
cd chinese-xinhua
cat data/idiom.json | head -20

No build step or installation required—all data is immediately consumable JSON.

Daily commands: This is not a runnable application. To regenerate the data: python scripts/chengyu.py && python scripts/ci.py && python scripts/word.py && python scripts/xiehouyu.py (exact behavior depends on scripts and source availability). To consume the data: simply parse JSON files with any JSON library in any language.

• data/idiom.json — Core dataset containing 31,648 idioms with pinyin, explanations, examples, and abbreviations—the primary data source for idiom lookups.
• data/word.json — Core dataset containing 16,142 Chinese characters with stroke counts, radicals, pinyin, and detailed explanations—essential for character-based queries.
• data/ci.json — Core dataset containing 264,434 words/phrases with explanations—the largest dataset that powers word lookup functionality.
• data/xiehouyu.json — Core dataset containing 14,032 proverbs/idioms with complete and short forms—supports idiom pattern matching.
• scripts/chengyu.py — Data processing script for idioms; demonstrates how to parse, clean, and structure idiom data from raw sources.
• scripts/word.py — Data processing script for characters; shows how to extract and normalize character metadata.
• scripts/ci.py — Data processing script for words/phrases; illustrates bulk data transformation and CSV/JSON conversion patterns.

• data/idiom.json (JSON) — Store 31,648 idioms; each record contains word, pinyin, abbreviation, explanation, derivation, and example.
  • Failure mode: If corrupted or truncated, all idiom lookups fail; if not present, any consuming service cannot initialize.
• data/word.json (JSON) — Store 16,142 Chinese characters with stroke count, radicals, pinyin variants, and detailed glyph explanations.
  • Failure mode: Missing or malformed data breaks character decomposition and stroke-based search features.
• data/ci.json (JSON) — Store 264,434 words/phrases; largest dataset providing definitions and usage context.
  • Failure mode: If unavailable or incomplete, word-lookup services degrade significantly; this is the heaviest file (~40+ MB likely).
• data/xiehouyu.json (JSON) — Store 14,032 proverbs with full and short forms; enable idiom pattern matching and cultural reference lookups.
  • Failure mode: Absence breaks proverb-specific queries; data corruption causes mismatched form pairs.
• scripts/chengyu.py — Transform raw idi

Add a new dictionary dataset (e.g., phrases, slang)

1. Create a new Python processing script in scripts/ following the pattern of scripts/ci.py or scripts/word.py—define parsing logic for your raw data source. (scripts/mynewdata.py)
2. Run the script to generate a normalized JSON output with consistent field structure (e.g., 'term', 'explanation', 'example'). (scripts/mynewdata.py)
3. Save the output JSON to data/mynewdata.json, ensuring it matches the schema of existing datasets. (data/mynewdata.json)
4. Document the new dataset in README.md under the Database Introduction section, including field descriptions and record count. (README.md)

Improve data quality by adding missing fields (e.g., add 'example' to ci.json)

1. Open scripts/clean.ipynb to inspect current data structure and identify gaps. (scripts/clean.ipynb)
2. Modify scripts/ci.py (or relevant script) to include new field extraction logic from raw sources. (scripts/ci.py)
3. Run the updated script and validate output; compare old vs. new JSON structure. (scripts/ci.py)
4. Replace data/ci.json with the enhanced version and update README.md schema documentation. (data/ci.json)

Refactor data schema (e.g., rename fields, consolidate abbreviations)

1. Plan schema changes in scripts/README.md with rationale and backward-compatibility considerations. (scripts/README.md)
2. Update all affected Python scripts (scripts/chengyu.py, scripts/word.py, scripts/ci.py, scripts/xiehouyu.py) to emit the new field structure. (scripts/chengyu.py)
3. Run all scripts to regenerate data/*.json files with the new schema. (data/idiom.json)
4. Update README.md and archived/ (if needed) to document schema version history. (README.md)

• JSON data files (idiom.json, word.json, ci.json, xiehouyu.json) — Provides language-agnostic, human-readable storage for dictionaries; enables easy integration with APIs, CLIs, and offline lookups without database overhead.
• Python scripts (chengyu.py, word.py, ci.py, xiehouyu.py) — Chosen for rapid data transformation, text processing, and manipulation; standard in data engineering; low friction for adding new dictionary sources.
• CSV + JSON dual format (ci.csv and ci.json) — CSV for bulk imports/exports and spreadsheet workflows; JSON for API consumption and nested data structures.
• Jupyter Notebook (clean.ipynb) — Enables interactive data exploration, validation, and documentation of data cleaning decisions in a reproducible format.

• Flat JSON files instead of a relational database (SQLite, PostgreSQL)

  • Why: Simplicity: no setup required, version-controllable, fully portable, free from licensing constraints; ideal for a read-heavy dictionary with infrequent updates.
  • Consequence: Queries are O(n) scans; not suitable for high-throughput real-time APIs without an indexing layer; parallel access is limited to single-threaded sequential reads.

• Scripts generate and write JSON at build/import time rather than on-demand

  • Why: Separates data preparation from serving; immutable snapshots ensure consistency and reproducibility.
  • Consequence: Data updates require re-running scripts and redeploying JSON files; no dynamic ingestion or live updates.

• Manual abbreviation generation (addAbbreviation.py) rather than real-time computation

  • Why: Pre-computed abbreviations ensure consistency and can be manually curated for accuracy.
  • Consequence: Abbreviations are static; if pinyin rules change, manual regeneration is needed.

• Field structure varies slightly across datasets (idiom.json has 'derivation', ci.json does not)

  • Why: Reflects the nature of each dictionary type; idioms require etymology, words only need definition.
  • Consequence: Consuming code must handle heterogeneous schemas; no single unified data model.

• Real-time API server—this is a static data repository, not a running service.
• User authentication or access control—all data is public domain.
• Full-text search indexing or ranking—provides raw data only; search implementation is delegated to consuming applications.
• Multilingual translation—focused exclusively on Chinese-language dictionaries.
• Audio pronunciation or stroke animation—data includes only textual metadata.

Scripts in scripts/ have no error handling or logging—they will silently fail if upstream websites changed HTML structure or blocked scrapers. No pinyin normalization across files (idiom.json uses tone marks like 'ā bí dì yù', but ci.json has no pinyin at all)—expect schema inconsistencies. The 'abbreviation' field in idiom.json (e.g., 'abdy') appears to be pinyin initials but is only present in some records. archived/idiom-dirty.json exists but is not documented—unclear what 'dirty' means. Web scraping source URLs are not recorded in the files themselves, making it impossible to refresh data without reverse-engineering the original scripts.

• Pinyin (拼音) romanization and tone marks — The idiom.json uses tone-marked pinyin (ā, á, etc.) for pronunciation lookup; understanding how Mandarin tones map to Unicode diacritics is essential for building phonetic indexing or learning tools.
• Chinese radicals (部首) and stroke order (笔画) — word.json includes radical and stroke count fields used by traditional dictionary lookup methods and handwriting recognition—critical for character-level NLP systems.
• Chengyu (成语) idioms—four-character fixed expressions — This dataset's largest cultural-specific asset; chengyu are opaque multi-word units where individual characters don't compose meaning, so they require lookup tables rather than compositional parsing.
• Xiehouyu (歇后语) pun-based witticisms — A uniquely Chinese linguistic phenomenon: riddle-answer pairs that exploit homophony and metaphor; not translatable and requires cultural knowledge, making this dataset invaluable for generative models trained on Chinese text.
• Web scraping and ETL (extract, transform, load) patterns — scripts/*.py demonstrate practical web scraping pipelines for dataset creation; understanding how to sanitize, deduplicate, and serialize scraped data is key to maintaining or extending this project.
• Character encoding: GB2312, GBK, UTF-8 — Older Chinese datasets used GB2312/GBK; this repo uses UTF-8 JSON, but legacy Chinese NLP systems may still expect different encodings, requiring transcoding when integrating datasets.

• fxsjy/jieba — Leading Chinese text segmentation library; commonly paired with chinese-xinhua for tokenizing input before idiom/phrase lookup.
• mozillazg/pinyin — Python library for converting Chinese characters to pinyin—complements this dataset by providing programmatic pinyin generation for enriching or validating the pinyin field.
• sonya/chinese-zipf-corpus — Companion frequency/corpus dataset for modern Chinese; useful for weighting idiom/word lookup by actual usage frequency.
• anarki1989/seagull — A chengyu-lianlong game implementation that likely depends on or was inspired by datasets like this for word chain logic.
• yanyiwu/cppjieba — C++ variant of jieba segmenter; needed when integrating chinese-xinhua data into high-performance systems like search engines or real-time game servers.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add data validation and integrity tests for JSON files

The repo contains 4 critical data files (idiom.json, ci.json, word.json, xiehouyu.json) with specific schema requirements (e.g., idiom requires 'word', 'pinyin', 'explanation', 'abbreviation' fields). There are no visible tests to validate data integrity, detect missing fields, or catch malformed entries. This is critical since the README advertises exact counts (14032 idioms, 16142 characters, etc.) that should be verified.

• [ ] Create tests/test_data_integrity.py to validate each JSON file against its expected schema
• [ ] Add assertions for required fields in idiom.json (word, pinyin, explanation, derivation, example, abbreviation)
• [ ] Add assertions for ci.json structure (ci, explanation fields present and non-empty)
• [ ] Add assertions for word.json structure (word, pinyin, strokes, radicals, explanation fields)
• [ ] Add record count validation to match README claims (14032 idioms, 264434 words, etc.)
• [ ] Add CI workflow (.github/workflows/test.yml) to run these tests on pull requests

Create a data processing pipeline documentation and validation for scripts/

The scripts/ folder contains 5 Python scripts (chengyu.py, ci.py, word.py, xiehouyu.py, addAbbreviation.py) that generate the JSON files, but scripts/README.md is empty. There's no documented workflow for how data flows from source → processing → output, making it hard for contributors to understand how to update or maintain data. The archived/idiom-dirty.json suggests data cleaning happens but isn't documented.

• [ ] Document scripts/README.md with the data pipeline flow (which script generates which output file)
• [ ] Add input source descriptions for each script (e.g., where does chengyu.py pull idiom data from?)
• [ ] Document the purpose of addAbbreviation.py and its dependencies on other scripts
• [ ] Explain the role of clean.ipynb in the data cleaning pipeline and when it should be run
• [ ] Add a section explaining the archived/idiom-dirty.json and data quality standards

Add Python type hints and a linting configuration for scripts/

The scripts/ folder contains Python files without visible type hints or linting configuration (.pylintrc, pyproject.toml, or setup.py). Since these scripts are critical for data generation and contributors may modify them, adding type safety would prevent bugs and make the codebase more maintainable.

• [ ] Add type hints to scripts/chengyu.py, ci.py, word.py, xiehouyu.py, and addAbbreviation.py (function signatures and return types)
• [ ] Create pyproject.toml or setup.cfg with black, isort, and flake8 configuration
• [ ] Add a GitHub Actions workflow (.github/workflows/lint.yml) to check type hints with mypy on pull requests
• [ ] Document in scripts/README.md that contributors should follow these typing standards

• Add JSON schema validation: create a scripts/validate.py that verifies all data/ JSON files conform to expected schemas (idiom must have word, pinyin, explanation; ci must have ci, explanation; etc.). This catches data corruption and documents the intended structure.
• Document scraper data sources: edit scripts/README.md to list the original website URLs, last-verified dates, and HTML selectors used for each scraper (chengyu.py → url/selector, ci.py → url/selector, etc.) so future maintainers know which pages to check for structure changes.
• Reconcile pinyin across datasets: write scripts/normalize_pinyin.py to audit which JSON files contain pinyin (idiom.json has it, ci.json doesn't) and add tone-marked pinyin to ci.json using a library like pinyin or pypinyin, then update data/ci.json with the enriched output.

This is a Chinese dictionary database project with a relatively low security risk profile, as it primarily consists of static data files and data processing scripts rather than a production web application or service. No critical vulnerabilities were identified. The main concerns are: (1) Absence of dependency management files making it difficult to track and update vulnerable packages, (2) Lack of security documentation and vulnerability disclosure procedures, (3) Presence of archived potentially problematic data, and (4) No visible input validation documentation for data processing scripts. The project would benefit from implementing standard Python project practices (requirements.txt, setup.py), adding security documentation, and ensuring robust input validation in data processing pipelines.

• Low · No Dependency Management File Present — Root directory / Project root. The codebase lacks a standard dependency management file (requirements.txt for Python, package.json for Node.js, etc.). This makes it difficult to track and manage dependencies, potentially allowing vulnerable packages to be installed without detection. Fix: Add a requirements.txt (for Python scripts) or package.json (if Node.js is used) with pinned versions of all dependencies. Implement dependency scanning tools like Dependabot or Snyk.
• Low · No Security Policy Documentation — Root directory. There is no SECURITY.md file or security policy documented. This makes it unclear how security vulnerabilities should be reported or handled. Fix: Create a SECURITY.md file outlining vulnerability disclosure procedures and security best practices for contributors.
• Low · Archived Data Directory Present — archived/ directory. The 'archived/idiom-dirty.json' file suggests unvalidated or potentially problematic data may be retained. This could indicate data quality issues or incomplete sanitization. Fix: Review archived data for sensitive information or corrupted entries. Document why data was archived and establish a data retention policy. Consider removing if no longer needed.
• Low · Python Scripts Without Input Validation Documentation — scripts/ directory (all .py files). Multiple Python scripts (chengyu.py, ci.py, word.py, xiehouyu.py) are present but there is no documentation indicating whether they validate input from CSV/JSON sources. CSV injection and JSON parsing vulnerabilities could exist. Fix: Review all scripts for proper input validation. Ensure CSV and JSON parsing uses safe libraries and validates data types. Document security assumptions made during data processing.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.