GO — Healthy across all four use cases

• Last commit 6mo ago
• 5 active contributors
• Apache-2.0 licensed
• CI configured
• ⚠ Slowing — last commit 6mo ago
• ⚠ Single-maintainer risk — top contributor 88% of recent commits
• ⚠ No test directory detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

HanLP is a production-grade multilingual NLP toolkit for Chinese and 130+ languages, implementing 10 joint tasks (tokenization, POS tagging, NER, dependency parsing, constituency parsing, semantic role labeling, etc.) with dual PyTorch/TensorFlow 2.x backends. It solves the problem of deploying state-of-the-art NLP models across morphologically-rich and analytic languages without reimplementing core algorithms. Monorepo structure: root Python package (hanlp/) contains task-specific modules (tokenization, tagging, parsing), docs/ holds 9 annotation standard specifications (docs/annotations/{pos,ner,dep,sdp,srl,constituency,tok}), .github/workflows manages CI, with notebook-based tutorials in plugins/. Models are remotely hosted with API reference in docs/api/.

NLP practitioners and production teams building Chinese language processing systems, researchers fine-tuning multilingual models, and developers deploying downstream applications (classification, QA, machine translation) that require accurate tokenization and syntactic/semantic analysis. Contributors range from academic researchers improving baselines to engineers adding language support.

Actively maintained production system: 2.1M+ Python LOC with Jupyter documentation, CI via GitHub Actions (.github/workflows/unit-tests.yml), and multiple pre-trained models across treebanks (CTB, UD, OntoNotes, MSRA). Last evidence of activity is recent unit test infrastructure and documentation coverage of 9+ annotation standards, indicating stability and ongoing iteration.

Primary risk is maintainer concentration (single author @hankcs visible in repo structure), though the project shows institutional backing via extensive documentation and corpus maintenance. Dual-engine dependency (PyTorch + TensorFlow 2.x) creates version compatibility surface. Large model download footprint (pre-trained weights likely gigabytes) may cause cold-start latency in production. Chinese-language issue tracker and community suggest potential friction for non-Mandarin contributors.

Active development on multi-task learning capabilities with recent unit test infrastructure setup (.github/workflows/unit-tests.yml). Documentation expansion across multiple annotation standards (PTB, UD, CTB, MSRA, OntoNotes schemas all documented). Model iteration indicated by versioned pretrained APIs and ongoing corpus maintenance for 130+ language support.

git clone https://github.com/hankcs/HanLP.git
cd HanLP
pip install hanlp
python -c "import hanlp; print(hanlp.tokenize('今天天气很好'))" # Chinese tokenization test

For Jupyter tutorials: jupyter notebook plugins/hanlp_demo/hanlp_demo/zh/tutorial.ipynb or use mybinder.org link in README.

Daily commands: Install via pip install hanlp then invoke Python API directly: hanlp.tokenize(), hanlp.pos(), hanlp.ner(), etc. For development: clone + pip install -e . for editable mode. No separate server start required for local usage; REST APIs documented separately in plugins/hanlp_demo/.

• README.md — Entry point documenting HanLP's 15+ NLP tasks, supported languages, and quick-start examples—essential for understanding project scope and capabilities.
• docs/api/hanlp/common/component.rst — Core abstract component interface that all NLP models (taggers, parsers, classifiers) inherit from—defines the contract for prediction and training.
• docs/api/hanlp/components/pipeline.md — Pipeline orchestration layer that chains multiple components together—critical for understanding how tasks are composed into end-to-end workflows.
• docs/api/hanlp/components/mtl/mtl.md — Multi-task learning framework that enables joint training across tokenization, POS tagging, NER, parsing, and SRL—architectural foundation for model efficiency.
• .github/workflows/unit-tests.yml — CI/CD pipeline defining test execution, coverage, and deployment—critical for understanding code quality gates and release process.
• docs/annotations/index.md — Master index of all supported annotation schemes (CTB, UD, Penn Treebank, SRL, SDP)—reference for understanding data format contracts across components.
• CITATION.cff — Citation metadata for academic contributions—required reading for proper attribution and understanding the research lineage.

Add a new sequence tagging task (e.g., chunking, aspect extraction)

1. Create task class in docs/api/hanlp/components/mtl/tasks/ extending base Task, define loss function, metrics, and decoding logic (docs/api/hanlp/components/mtl/tasks/task.md)
2. Register task in multi-task learning framework with proper data loaders and vocabulary definitions (docs/api/hanlp/components/mtl/mtl.md)
3. Add annotation scheme documentation in docs/annotations/ with label inventory and examples (docs/annotations/index.md)
4. Implement unit tests in .github/workflows to validate task integration with component base classes (.github/workflows/unit-tests.yml)

Add support for a new language or annotation standard

1. Create annotation scheme documentation under docs/annotations/{task_type}/ with label mapping and CoNLL format examples (docs/annotations/index.md)
2. Implement CoNLL-U format parser/serializer for the new scheme in common utilities (docs/api/common/conll.rst)
3. Add dataset loader in docs/api/hanlp/common/dataset.md with train/dev/test splits and vocabulary building (docs/api/hanlp/common/dataset.md)
4. Register in pipeline and multi-task learning framework to enable end-to-end training (docs/api/hanlp/components/pipeline.md)

Add a new neural architecture (e.g., Mamba, Flash Attention encoder)

1. Extend torch_component base class with new encoder implementation and forward pass (docs/api/hanlp/common/torch_component.md)
2. Create task-specific head classes (tagging, parsing, classification) that use the new encoder (docs/api/hanlp/components/mtl/tasks/task.md)
3. Add configurable parameters (hidden size, dropout, etc.) to component configuration mixin (docs/api/hanlp/common/configurable.rst)
4. Integrate into pipeline and add benchmark tests to validate performance across MTL tasks (.github/workflows/unit-tests.yml)

Create a new end-to-end NLP pipeline for a specific domain (e.g., biomedical, legal)

1. Compose task components (tokenizer, tagger, parser, NER) using pipeline orchestration layer (docs/api/hanlp/components/pipeline.md)
2. Optionally fine-tune multi-task model on domain-specific corpus using MTL framework (docs/api/hanlp/components/mtl/mtl.md)
3. Define custom annotation scheme if domain requires new label inventory (docs/annotations/index.md)
4. Package as reusable component extending base component interface (docs/api/hanlp/common/component.rst)

• Py — undefined

1. Model download on first use: Calling hanlp.tokenize() for first time auto-downloads gigabyte-scale pre-trained models to ~/.hanlp/; can fail silently on network loss or quota limits. 2) Dual-engine version lock: PyTorch and TensorFlow 2.x versions must be compatible with underlying model binaries; version skew causes cryptic deserialization errors. 3) Chinese-dominant documentation: Issue tracker, forums (bbs.hankcs.com), and tutorial notebooks heavily Mandarin; English-language bug reports may receive slow responses. 4) Task interdependence: Some multi-task models (e.g., tok+pos+ner) require sequential inference; calling individual tasks may use different model versions than joint inference. 5) No explicit dev requirements.txt: Package dependencies inferred from setup.py; development workflows not fully documented for contributors.

• Multi-task learning with shared encoders — HanLP's core architecture: one transformer encoder feeds task-specific decoders (POS, NER, SRL heads); understanding weight sharing vs. task-specific parameters is critical for model debugging and fine-tuning
• Universal Dependencies (UD) annotation scheme — HanLP implements UD for 100+ languages (docs/annotations/dep/ud.md); UD is the interlingua allowing cross-lingual transfer — knowing UD's 37 universal POS tags and 43 dependency relations is mandatory for contributing
• Semantic role labeling (SRL) PropBank vs. FrameNet — HanLP supports both PropBank (English, docs/annotations/srl/propbank.md) and semantic-role-based dependency parsing (SDP); the conceptual gap between predicate-argument structure and semantic frames explains model design choices
• Constituency vs. dependency tree duality — HanLP provides both CTB constituency (Chinese) and UD dependency parsers; understanding their complementary strengths (constituency for linguistic structure, dependency for semantic relations) informs task selection
• CONLL-X/CONLL-U tabular data format — Inferred from docs/api/common/conll.rst: HanLP ingests and outputs CoNLL formats (10-column tab-separated, 1 token/line); format parsing is non-trivial for handling multiword tokens and enhanced dependencies
• Transformer encoder fine-tuning vs. pre-training — HanLP's mminilmv2l6 models are distilled from larger transformers; understanding parameter freezing, learning rate scheduling, and domain adaptation is essential for retraining on custom treebanks
• Coreference resolution (指代消解) as downstream application — HanLP README lists coreference as a capability; it depends on prior NER and dependency parsing outputs — understanding task chaining prevents error cascade when modifying base tasks

• stanfordnlp/stanza — Multilingual NLP toolkit with similar multi-task architecture (tokenization→pos→ner→dependency), Stanford's UD-based, direct competitor in English/Chinese dual-language support
• pytorch/fairseq — Underlying PyTorch sequence model infrastructure that HanLP's constituency/semantic parsing tasks likely build upon
• google-research/bert — Pre-trained encoder foundation: HanLP's mminilmv2l6 models are fine-tuned variants of BERT-style transformers, making BERT checkpoint exploration useful for understanding ablations
• thunlp/OpenEntity — Companion resource from Tsinghua NLP lab for Chinese NER annotation and evaluation; shares similar OntoNotes/MSRA treebank standards
• yzhangcs/parser — Graph-based dependency parsing reference implementation that HanLP's SRL/SDP tasks abstractly follow

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for semantic role labeling (SRL) component

The repo has extensive NLP components including SRL (docs/annotations/srl/ shows cpb.md and propbank.md), but the GitHub Actions workflow (unit-tests.yml) appears minimal. SRL is a complex task with multiple annotation schemes (Chinese PropBank, English PropBank). Adding dedicated test cases would ensure quality across both schemes and catch regressions in semantic parsing.

• [ ] Create tests/srl/test_cpb.py with test cases for Chinese semantic role labeling using CPB dataset
• [ ] Create tests/srl/test_propbank.py for English PropBank SRL component
• [ ] Add integration tests verifying SRL works correctly with upstream components (tokenization, POS tagging, dependency parsing)
• [ ] Update .github/workflows/unit-tests.yml to explicitly run SRL tests with coverage reporting

Add missing documentation for semantic dependency parsing (SDP) task variations

The docs/annotations/sdp/ directory lists 5 SDP variants (dm.md, pas.md, psd.md, semeval16.md, index.md) but there's no corresponding docs/api/hanlp/components/mtl/tasks/sdp.md API documentation. The MTL (Multi-Task Learning) architecture supports SDP but lacks parallel documentation to constituency.md and dep.md that are already present.

• [ ] Create docs/api/hanlp/components/mtl/tasks/sdp.md documenting SDP task configuration and usage
• [ ] Document all 4 SDP schemes (DM, PAS, PSD, SemEval16) with their differences and appropriate use cases
• [ ] Add code examples showing how to load and use different SDP models via the MTL interface
• [ ] Link the annotation docs (docs/annotations/sdp/*.md) to the API documentation for consistency

Add cross-lingual NER test suite for multiple annotation standards

The repo documents 3 NER annotation schemes (docs/annotations/ner/: msra.md, ontonotes.md, pku.md) representing different standards (Chinese MSRA, English OntoNotes, Chinese PKU). However, there's no organized test suite validating model outputs across these standards. Adding comparative tests would ensure consistent quality across schemes.

• [ ] Create tests/ner/test_msra.py with unit tests for MSRA Chinese NER dataset and model validation
• [ ] Create tests/ner/test_ontonotes.py for OntoNotes English NER with entity type validation
• [ ] Create tests/ner/test_pku.py for PKU Chinese NER component
• [ ] Add tests/ner/test_ner_integration.py with cross-scheme comparison tests (same text, different annotation standards) to catch inconsistencies
• [ ] Update .github/workflows/unit-tests.yml to run all NER tests with per-scheme metrics reporting

• Add English tokenization validation tests to unit-tests.yml: CI pipeline exists but file structure suggests minimal language coverage testing — adding assertions for English UD_ONTONOTES_TOK model would ensure cross-lingual stability
• Document model download caching behavior in docs/api/common/configurable.rst: Hidden trap #1 above: users need explicit guidance on ~/.hanlp/ directory, cache invalidation, and offline-mode limitations
• Expand docs/annotations/index.md with comparison table of treebank coverage by language: Currently 9+ treebank specs are scattered; a matrix showing which treebanks cover which languages (Chinese/English/Japanese/Russian/etc.) would reduce user confusion

HanLP appears to be a legitimate open-source NLP project with reasonable security practices (GitHub Actions CI/CD, presence of LICENSE, multiple issue templates). However, the analysis is limited due to lack of actual code inspection and missing dependency file. Key recommendations: (1) Provide dependency files for vulnerability scanning, (2) Create SECURITY.md file, (3) Review GitHub Actions workflow security, (4) Implement robust input validation for text processing, (5) Use automated tools (pip-audit, SAST) in CI/CD pipeline. The project's focus on text processing and NLP tasks requires special attention to input validation and ReDoS vulnerabilities.

• Medium · Missing Dependency File Analysis — requirements.txt, setup.py, pyproject.toml, or poetry.lock. The dependency/package file content was not provided for analysis. This prevents identification of known vulnerable package versions, outdated libraries, or insecure dependencies that could be used in this NLP project. Fix: Provide dependency files for scanning. Use tools like 'pip-audit', 'safety', or 'OWASP Dependency-Check' to identify known vulnerabilities in dependencies.
• Low · No Secrets Detection in Visible Files — Repository root and config files. No .env files, credential files, or obvious hardcoded secrets were detected in the provided file structure. However, this is a static analysis of directory structure only. Fix: Implement pre-commit hooks using tools like 'git-secrets', 'detect-secrets', or 'TruffleHog' to prevent accidental credential commits.
• Low · GitHub Actions Workflow Security — .github/workflows/unit-tests.yml. The repository uses GitHub Actions for CI/CD (.github/workflows/unit-tests.yml). Without inspecting the workflow file content, potential risks include: insecure secret handling, outdated action versions, or insufficient access controls. Fix: Review workflow files for: (1) Use of secrets with restricted scope, (2) Pin action versions to specific commits (not branches), (3) Use of 'contents: read' permission by default, (4) Avoid running untrusted code in workflows.
• Low · Missing Security Documentation — Repository root. No SECURITY.md file found in the repository. This file is important for security researchers to responsibly disclose vulnerabilities. Fix: Create a SECURITY.md file documenting the vulnerability disclosure process and security contact information.
• Low · Input Validation in NLP Processing — hanlp/common/document.rst and core processing modules. As an NLP library processing text input, the codebase should implement robust input validation. Without code inspection, potential risks include: buffer overflows, ReDoS (Regular Expression Denial of Service), or processing of malicious input. Fix: Implement input size limits, regex complexity validation, and sanitization for user-provided text data. Test with fuzzing tools.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/hankcs/HanLP shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live hankcs/HanLP repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/hankcs/HanLP.

What it runs against: a local clone of hankcs/HanLP — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in hankcs/HanLP | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch doc-zh exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 204 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of hankcs/HanLP. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/hankcs/HanLP.git
#   cd HanLP
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of hankcs/HanLP and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "hankcs/HanLP(\\.git)?\\b" \\
  && ok "origin remote is hankcs/HanLP" \\
  || miss "origin remote is not hankcs/HanLP (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify doc-zh >/dev/null 2>&1 \\
  && ok "default branch doc-zh exists" \\
  || miss "default branch doc-zh no longer exists"

# 4. Critical files exist
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "docs/api/hanlp/common/component.rst" \\
  && ok "docs/api/hanlp/common/component.rst" \\
  || miss "missing critical file: docs/api/hanlp/common/component.rst"
test -f "docs/api/hanlp/components/pipeline.md" \\
  && ok "docs/api/hanlp/components/pipeline.md" \\
  || miss "missing critical file: docs/api/hanlp/components/pipeline.md"
test -f "docs/api/hanlp/components/mtl/mtl.md" \\
  && ok "docs/api/hanlp/components/mtl/mtl.md" \\
  || miss "missing critical file: docs/api/hanlp/components/mtl/mtl.md"
test -f ".github/workflows/unit-tests.yml" \\
  && ok ".github/workflows/unit-tests.yml" \\
  || miss "missing critical file: .github/workflows/unit-tests.yml"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 204 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~174d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/hankcs/HanLP"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.