If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/NLPchina/ansj_seg shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across all four use cases

• 10 active contributors
• Distributed ownership (top contributor 40% of recent commits)
• Apache-2.0 licensed
• CI configured
• Tests present
• ⚠ Stale — last commit 2y ago

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live NLPchina/ansj_seg repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/NLPchina/ansj_seg.

What it runs against: a local clone of NLPchina/ansj_seg — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in NLPchina/ansj_seg | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 931 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of NLPchina/ansj_seg. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/NLPchina/ansj_seg.git
#   cd ansj_seg
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of NLPchina/ansj_seg and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "NLPchina/ansj_seg(\\.git)?\\b" \\
  && ok "origin remote is NLPchina/ansj_seg" \\
  || miss "origin remote is not NLPchina/ansj_seg (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "library/default.dic" \\
  && ok "library/default.dic" \\
  || miss "missing critical file: library/default.dic"
test -f "plugin/ansj_lucene5_plugin/src/main/java/org/ansj/lucene5/AnsjAnalyzer.java" \\
  && ok "plugin/ansj_lucene5_plugin/src/main/java/org/ansj/lucene5/AnsjAnalyzer.java" \\
  || miss "missing critical file: plugin/ansj_lucene5_plugin/src/main/java/org/ansj/lucene5/AnsjAnalyzer.java"
test -f "plugin/ansj_lucene3_plugin/src/main/java/org/ansj/lucene3/AnsjAnalysis.java" \\
  && ok "plugin/ansj_lucene3_plugin/src/main/java/org/ansj/lucene3/AnsjAnalysis.java" \\
  || miss "missing critical file: plugin/ansj_lucene3_plugin/src/main/java/org/ansj/lucene3/AnsjAnalysis.java"
test -f "ansj_library.properties" \\
  && ok "ansj_library.properties" \\
  || miss "missing critical file: ansj_library.properties"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 931 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~901d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/NLPchina/ansj_seg"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Ansj is a Java implementation of n-Gram+CRF+HMM-based Chinese word segmentation achieving ~2 million characters/second throughput and 96%+ accuracy. It provides core NLP capabilities including Chinese tokenization, person name recognition, part-of-speech tagging, custom dictionary support, and keyword extraction—intended as a faster, more accurate alternative to the open-source ICT segmenter. Monorepo structure: core ansj_seg library at root with Maven POM; plugin/ subdirectory contains separate Maven modules for Lucene 3/4/5 integrations (ansj_lucene3_plugin, ansj_lucene4_plugin, ansj_lucene5_plugin). Library resources (dictionaries, models) in library/ directory; main segmentation logic expected in src/main/java/org/ansj/.

Chinese NLP engineers and Java backend developers building search, recommendation, or text analysis systems who need production-grade Chinese segmentation with custom vocabulary support and named entity recognition (especially person names), without relying on external C++ bindings.

Actively maintained but past peak activity. The project uses Maven, has Travis CI configured (.travis.yml), and the README documents versions up to 5.x with a published Maven artifact (org.ansj:ansj_seg:5.1.1). However, the large monolithic codebase (688K LOC Java) and presence of multiple Lucene plugin versions (3, 4, 5) suggest maintenance focus rather than active feature development. Production-ready for stable release versions.

Single-author project (NLPchina org, limited contributor visibility in this snapshot) with no visible test directory in the file list, raising concerns about regression coverage. The reliance on pre-built dictionary files (library/*.dic) loaded at runtime creates tight coupling to binary data; version mismatches could silently degrade accuracy. Dictionary format and model format changes appear undocumented, making upgrades risky without integration tests.

No commit timeline visible in provided data, but the README lists outstanding work: time recognition, IP/email/URL/postal code recognition features are incomplete; CRF model optimization and LSTM segmentation are planned but not started. The existence of 3+ Lucene plugin versions suggests the project maintains backward compatibility rather than cutting-edge features.

Clone: git clone https://github.com/NLPchina/ansj_seg.git && cd ansj_seg. Install: mvn clean install (requires Maven 3.x, Java 8+). Test basic segmentation: mvn test or write a simple Java class importing org.ansj.seg.* and calling ToAnalysis.parse("欢迎使用ansj").

Daily commands: mvn clean compile to compile. For Lucene plugin testing: cd plugin/ansj_lucene4_plugin && mvn test (see test classes like IndexTest.java). Core library has no standalone binary entry point; it's used as a Maven dependency in downstream projects.

• README.md — Entry point documentation describing the n-Gram+CRF+HMM segmentation algorithm, API usage, and core features (分词, 姓名识别, 词性标注)
• library/default.dic — Core dictionary file used by the segmentation engine; understand this format to customize tokenization behavior
• plugin/ansj_lucene5_plugin/src/main/java/org/ansj/lucene5/AnsjAnalyzer.java — Primary Lucene integration point; shows how the segmentation engine integrates with Lucene 5+ for indexing and search
• plugin/ansj_lucene3_plugin/src/main/java/org/ansj/lucene3/AnsjAnalysis.java — Reference implementation for Lucene 3 tokenizer integration; demonstrates the standard pattern for all version plugins
• ansj_library.properties — Configuration file that sets library paths and initialization; controls which dictionaries and models are loaded at runtime
• library/ambiguity.dic — Disambiguation rules dictionary; used by HMM module to resolve segmentation ambiguities
• library/regex.dic — Regular expression rules for tokenizing special patterns (numbers, URLs, punctuation); applied before dictionary matching

Add Support for a New Lucene Version

1. Create new plugin directory: plugin/ansj_lucene8_plugin following the structure of plugin/ansj_lucene7_plugin (plugin/ansj_lucene7_plugin/pom.xml)
2. Update <version> and <artifactId> in the new pom.xml; set lucene-core dependency to target version (e.g., 8.0.0) (plugin/ansj_lucene8_plugin/pom.xml)
3. Copy and adapt AnsjAnalyzer.java from lucene7 plugin, updating imports and Analyzer API calls for Lucene 8 compatibility (plugin/ansj_lucene8_plugin/src/main/java/org/ansj/lucene8/AnsjAnalyzer.java)
4. Reuse or adapt AnsjTokenizer.java from shared utilities if API unchanged; otherwise copy to new module (plugin/ansj_lucene8_plugin/src/main/java/org/ansj/lucene/util/AnsjTokenizer.java)
5. Add integration tests in new module mirroring tests from lucene7 (IndexTest, PhraseTest, HeightLightTest) (plugin/ansj_lucene8_plugin/src/test/java/org/ansj/ansj_lucene8_plug/IndexTest.java)

Extend Dictionary with Custom Stopwords or Domain Terms

1. Review the format of library/default.dic and library/stop.dic (one term per line, optional frequency/POS metadata) (library/default.dic)
2. Add your domain terms to library/default.dic (for segmentation) or library/stop.dic (to filter low-value tokens) (library/default.dic)
3. Update ansj_library.properties if you use custom dictionary paths instead of defaults (ansj_library.properties)
4. Restart the JVM; the segmentation engine will reload dictionaries on next initialization (README.md)

Customize Segmentation Rules (Regex Patterns & Ambiguity Handling)

1. Edit library/regex.dic to add or modify regex patterns for special token types (numbers, URLs, punctuation) (library/regex.dic)
2. Edit library/ambiguity.dic to add HMM disambiguation rules; format: word pairs that often conflict + resolution weight (library/ambiguity.dic)
3. Test changes using Lucene plugin tests (e.g., plugin/ansj_lucene5_plugin/src/test) to verify segmentation quality (plugin/ansj_lucene5_plugin/src/test/java/org/ansj/ansj_lucene5_plug/IndexAndTest.java)

Integrate ansj into a New Search Project (Non-Lucene)

1. Add Maven dependency on core ansj_seg library (see README.md for latest version 5.1.1) (README.md)
2. Call ToAnalysis.parse(String) to segment text; returns list of Token objects with word, POS, and name-entity tags (README.md)
3. For custom use cases, reference plugin/ansj_lucene5_plugin/src/main/java/org/ansj/lucene5/AnsjAnalyzer.java to understand the Token stream API (plugin/ansj_lucene5_plugin/src/main/java/org/ansj/lucene5/AnsjAnalyzer.java)
4. Place your custom dictionaries in a directory referenced by ansj_library.properties or override at runtime via configuration (ansj_library.properties)

• n-Gram + CRF + HMM — Combines statistical methods for robust Chinese segmentation without explicit rule engines; achieves 96%+ accuracy and 200M chars/sec throughput
• Multi-version Lucene plugins (3–7+) — Allows ansj to integrate with diverse Lucene deployments across major version families; each plugin adapts core Analyzer/Tokenizer API to specific Lucene releases
• Dictionary-based pre-processing (regex.dic, default.dic) — Fast pattern matching for known tokens (numbers, English, punctuation) before expensive n-Gram matching; improves overall segmentation speed and accuracy
• Maven multi-module build — Enables independent compilation and versioning of core library and each Lucene plugin; facilitates CI/CD and reduces dependency bloat for users

• Dictionary-driven + statistical model (hybrid approach)
  • Why: Pure dictionary-based segmentation is fast but rigid; pure statistical models are flexible but slow. Hybrid balances speed (~200M chars/sec) with accuracy (96%+)
  • Consequence: Users must maintain and update dictionaries for new domain terms; statistical model tuning requires expertise in C

1. Dictionary format is binary/proprietary (*.dic files); no documented schema for custom dictionary creation or merging. 2) The library loading happens at static initialization time; swapping dictionaries at runtime requires careful classloader isolation or full JVM restart. 3) CRF model files appear embedded or loaded from classpath but exact location/format not documented in README—adding custom models requires reverse-engineering existing ones. 4) Lucene plugin versions are independent; using Lucene3 plugin with Lucene4 index causes silent failures, not exceptions.

• Conditional Random Fields (CRF) — Core algorithm used in ansj for sequence labeling (word boundaries, POS tags). Understanding CRF scoring and inference is essential for debugging incorrect segmentations.
• Hidden Markov Model (HMM) — Paired with CRF in ansj's n-Gram+CRF+HMM architecture for probabilistic disambiguation when multiple valid segmentations exist. Affects person name recognition accuracy.
• n-Gram Language Model — First stage of ansj pipeline; uses character/word n-grams to pre-score segmentation candidates before CRF refinement. Dictionary vocabulary must cover high-frequency n-grams.
• Trie Data Structure — Inferred from dictionary-driven design; enables O(word_length) lookup for matching dictionary entries during tokenization. Custom dictionary addition requires understanding Trie construction.
• Viterbi Algorithm — Used by HMM layer to find optimal segmentation path through state lattice. Understanding Viterbi backtracking helps debug why certain rare character sequences are mishandled.
• Analyzer (Lucene concept) — ansj_lucene plugins implement Lucene's Analyzer interface; essential for integrating ansj into Elasticsearch, Solr, or standalone Lucene applications. Plugin architecture decouples core segmentation from indexing.
• Part-of-Speech Tagging — ansj outputs POS tags (v, n, nz, en, etc.) alongside tokens; required for downstream NLP tasks like entity recognition and syntax analysis. Tag set is implicit in output format.

• fxsjy/jieba — Alternative Chinese segmenter written in Python; comparable accuracy/speed but different algorithm (Trie-based HMM vs. n-Gram+CRF). Reference for cross-language feature parity.
• hankcs/HanLP — Java-based Chinese NLP library with segmentation, NER, and dependency parsing. More feature-rich but heavier than ansj; often used when multi-task NLP needed beyond segmentation.
• medcl/elasticsearch-analysis-ik — Elasticsearch plugin for Chinese analysis using IKAnalyzer. Competing solution in Elasticsearch ecosystem; ansj_lucene plugins target similar use case.
• NLPchina/nlp-lang — Companion library by same author (NLPchina) providing tokenization and lemmatization utilities. Likely shares dictionary/model infrastructure with ansj_seg.
• lanl/sorta — CRF framework written in Java. ansj's CRF model training likely uses similar sequence tagging patterns; reference for understanding the CRF layer in ansj's pipeline.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for Lucene plugin compatibility across versions

The repo maintains three separate Lucene plugins (3, 4, and 5) with duplicated code in AnsjTokenizer.java and PorterStemmer.java. There are minimal tests (only AnsjAnalysisTest.java, NearTest.java, TestFilter.java in lucene3_plugin, and AppTest.java, IndexTest.java, Test.java in lucene4_plugin). New contributors should add integration tests that verify each plugin version works correctly with its target Lucene version and that the tokenization output is consistent across versions.

• [ ] Create plugin/ansj_lucene3_plugin/src/test/java/org/ansj/lucene3/TokenizationConsistencyTest.java with tests comparing output across plugin versions
• [ ] Add plugin/ansj_lucene5_plugin/src/test/java/org/ansj/ansj_lucene5_plug/IndexTest.java (currently missing, unlike lucene4)
• [ ] Create a shared test base class in plugin/src/test for testing AnsjTokenizer.java behavior identically across all three plugin versions

Refactor duplicated Lucene plugin code into a shared module

The three Lucene plugin folders (lucene3, lucene4, lucene5) contain identical copies of AnsjTokenizer.java and PorterStemmer.java in plugin/ansj_*_plugin/src/main/java/org/ansj/lucene/util/. This violates DRY principles and makes maintenance harder. A contributor should extract these into a shared plugin-common module.

• [ ] Create plugin/ansj_lucene_common/pom.xml as a parent module containing shared utilities and a src/main/java/org/ansj/lucene/util/ directory with AnsjTokenizer.java and PorterStemmer.java
• [ ] Update plugin/pom.xml to define ansj_lucene_common as a module and have lucene3/4/5 plugins depend on it
• [ ] Remove the duplicate AnsjTokenizer.java and PorterStemmer.java from plugin/ansj_lucene3_plugin, plugin/ansj_lucene4_plugin, and plugin/ansj_lucene5_plugin, replacing with dependency references

Add documentation and tests for custom dictionary loading from ansj_library.properties

The ansj_library.properties file and library/ directory with .dic files (default.dic, stop.dic, synonyms.dic, ambiguity.dic, regex.dic) are referenced in the codebase but lack clear documentation on dictionary format, loading mechanism, and custom dictionary integration. New contributors should document this and add tests verifying dictionary loading works correctly.

• [ ] Create docs/DICTIONARY_FORMAT.md documenting the format of .dic files (default.dic, stop.dic, synonyms.dic, etc.) with examples
• [ ] Add a test file src/test/java/org/ansj/DictionaryLoadingTest.java that verifies each dictionary type loads correctly from library/ directory
• [ ] Update README.md with a section explaining how users can create and load custom dictionaries via ansj_library.properties, with a concrete example

• Add integration tests for all recognition modules (Time, Email, URL, IP, PostalCode classes mentioned in README as 'incomplete'). Start by creating ansj_seg/src/test/java/org/ansj/recognition/test/ with JUnit cases for edge cases like '2024年1月15日' for TimeRecognition.
• Document the dictionary format (library/*.dic structure) with a parsing example. Create docs/DICTIONARY_FORMAT.md explaining the binary format used in default.dic and provide a tool to export/import dictionaries from/to human-readable CSV or JSON.
• Fill the test gaps for plugin modules: plugin/ansj_lucene4_plugin/src/test/ has only 3 test files. Add tests for custom analyzer options, stopword filtering, and handling of rare CJK characters following the pattern in IndexTest.java.

The codebase has moderate security concerns, primarily driven by outdated and end-of-life dependencies (Lucene 3.x-6.x) that lack security patches. The core ansj_seg library appears to be a natural language processing tool with minimal external attack surface, but integration points (tokenizers, file loading) could benefit from enhanced input validation and access controls. The repository maintains multiple legacy plugin versions that should be consolidated. Dependencies should be updated immediately, and input validation should be implemented across all user-facing APIs. Overall security posture is below average due to technical debt from unmaintained dependency versions.

• High · Outdated Lucene Dependency (lucene-core 3.4.0) — plugin/ansj_lucene3_plugin/pom.xml. The ansj_lucene3_plugin uses lucene-core version 3.4.0, which was released in 2012 and contains multiple known security vulnerabilities and bugs. This version is no longer maintained and lacks security patches. Fix: Upgrade to the latest stable version of Lucene (currently 9.x). If Lucene 3 compatibility is required, migrate to Lucene 4+ as a minimum. Review all dependencies for known CVEs using tools like OWASP Dependency-Check.
• High · Multiple Outdated Plugin Dependencies — plugin/ansj_lucene3_plugin, plugin/ansj_lucene4_plugin, plugin/ansj_lucene5_plugin, plugin/ansj_lucene6_plugin. The repository contains multiple Lucene plugins targeting older versions (3, 4, 5, 6) with unmaintained dependencies. Lucene 3, 4, and 5 are end-of-life versions with potential security vulnerabilities and lack of security updates. Fix: Consolidate on a modern Lucene version (8.x or 9.x+). Remove deprecated plugin versions. Establish a dependency management policy to regularly update and audit third-party libraries.
• Medium · Potential Path Traversal in Dictionary Loading — library/ directory and associated loading mechanism. The library loads dictionary files from 'library/' directory (ambiguity.dic, default.dic, regex.dic, stop.dic, synonyms.dic). Without proper input validation on file paths, this could potentially be exploited for directory traversal attacks if user input influences file path construction. Fix: Implement strict path validation for all dictionary file loading. Use canonicalized paths and whitelist allowed dictionary locations. Avoid constructing file paths from user input. Validate that loaded files exist within expected directories.
• Medium · No Input Validation in Tokenizer/Analyzer — plugin/ansj_lucene*/src/main/java/org/ansj/lucene*/*.java. The codebase implements Chinese word segmentation and tokenization. Without visible input validation in the analyzer classes (AnsjAnalysis, AnsjIndexAnalysis, AnsjAnalyzer), there could be risks of denial-of-service (DoS) attacks through maliciously crafted input or resource exhaustion. Fix: Implement input length limits and validation. Add protection against pathologically complex inputs. Consider rate limiting for tokenization operations. Add resource usage monitoring and timeouts.
• Medium · Lack of Access Controls on Dictionary Files — library/ directory. Dictionary files in the 'library/' directory lack visibility into file permissions and access control mechanisms. If the application is deployed with improper file permissions, users could potentially read or modify dictionary content. Fix: Ensure dictionary files are read-only and owned by the application user. Set file permissions to 0644 (or more restrictive if possible). Verify that the application runs with minimal required privileges. Implement file integrity checking (checksums/signatures).
• Low · Missing Security Configuration Files — Root directory / configuration files. No visible security-related configuration files (security.properties, security headers configuration, or similar) are present in the codebase. This suggests lack of explicit security hardening configuration. Fix: Create security.properties or similar configuration file to explicitly define security policies. Document security configuration requirements. Add security headers configuration when integrating with web frameworks.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.