GO — Healthy across all four use cases

• Last commit 4mo ago
• 20 active contributors
• Distributed ownership (top contributor 34% of recent commits)
• MIT licensed
• Tests present
• ⚠ Slowing — last commit 4mo ago
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

UniLM is Microsoft's large-scale self-supervised pre-training framework for unified modeling across tasks (NLU & NLG), languages (100+), and modalities (text, vision, audio, multimodal). It provides foundation architectures like DeepNet (1000-layer Transformers), Magneto (general-purpose Transformers), and specialized models like BitNet (1-bit LLMs), RetNet, and LongNet, enabling a single model to handle predictive, generative, and cross-modal tasks. Monorepo structure with independent research projects in top-level directories: Diff-Transformer/ (efficient attention kernels), LatentLM/ (diffusion models with vision tokenizers), deepnet/ (scaled Transformers), xmoe/ (sparse MoE), kosmos-2.5/ & kosmos-2/ (multimodal LLMs), unilm/ (unified language models), infoxlm/ (multilingual pretraining). Each subproject has its own models/, tokenizers/, training scripts, and evaluation code. TorchScale library sits as foundational architecture layer consumed by subprojects.

Researchers and ML engineers building or fine-tuning large foundation models, multimodal LLMs (Kosmos series), and multilingual systems (InfoXLM, XLM-E); teams implementing custom architectures on top of TorchScale; practitioners deploying vision-language models or long-context transformers in production.

Actively developed and mature: Microsoft research project with multiple published papers (BitNet, RetNet, LongNet, Kosmos-2.5), organized submodules for distinct architectures (Diff-Transformer/, LatentLM/, kosmos-2/, deepnet/), and clear integration with HuggingFace Transformers. Production-ready for research and enterprise use, though some experimental components (RetNet, BitNet) are newer. Last significant work visible across multiple subdirectories suggests ongoing maintenance.

High dependency footprint (torch, transformers, vllm, datasets, flash_attn, sympy, antlr4) with some version constraints (antlr4-python3-runtime==4.11.1 pinned for sympy compatibility). Monorepo structure spans wildly different model architectures (diffusion-based LatentLM, sparse MoE in xmoe, 1-bit BitNet), creating high cognitive load and potential for breaking changes across subprojects. No clear CI/test results visible in provided data; maintenance burden distributed across multiple architectural experiments.

Active development across: Kosmos-2.5 (multimodal literate model with improved vision grounding), Diff-Transformer-V2 (optimized differential attention with FlashDiff kernels), LatentLM (generative modeling via latent diffusion with vision VAE tokenizers), and foundational work on length-extrapolatable Transformers and sparse MoE systems. Emphasis on efficiency (flash_attn, custom CUDA kernels in Diff-Transformer/kernel/) and multimodal capabilities.

Clone the repo and install core dependencies: git clone https://github.com/microsoft/unilm.git && cd unilm && pip install torch transformers datasets vllm tqdm flash_attn. For specific subprojects, enter their directories (e.g., cd kosmos-2.5 or cd Diff-Transformer) and check local README.md for model-specific setup (Kosmos requires vision model downloads, LatentLM requires VAE checkpoints).

Daily commands: No single entry point; run subproject-specific training: cd Diff-Transformer && python example.py for differential attention demo. For LatentLM: python train_hf.py or python sample_hf.py (requires pretrained checkpoints). For Kosmos: follow kosmos-2.5/README.md for inference. For general model evaluation, use LatentLM/evaluate_fid.py or LatentLM/metrics/. VLLMs integrated for inference optimization via vllm dependency.

• README.md — Foundational overview of UniLM's multi-task, multi-modal pre-training vision and TorchScale architecture library integration.
• Diff-Transformer/multihead_diffattn.py — Core differential attention mechanism implementation—critical architectural innovation for efficient transformer variants.
• LatentLM/models/Transformer.py — Foundation transformer model definition used across vision and language latent space generation tasks.
• LatentLM/models/DiT.py — Diffusion Transformer implementation for latent generation—key model for image/vision token generation pipeline.
• LatentLM/train_hf.py — Primary training entry point integrating HuggingFace transformers, diffusion scheduling, and model optimization.
• PFPO/conf/api/vllm — Production config hierarchy for API-based inference using vLLM, defining model variants and deployment parameters.
• LatentLM/tokenizer_models/vae.py — VAE tokenizer models for vision encoding/decoding—critical for latent space representation in multi-modal tasks.

Add a new Transformer variant

1. Create a new model class in LatentLM/models/ directory inheriting from torch.nn.Module (LatentLM/models/Transformer.py)
2. Implement forward pass with optional diffusion timestep embedding for DiT compatibility (LatentLM/models/DiT.py)
3. Register model in init.py with model factory pattern for instantiation (LatentLM/models/__init__.py)
4. Add configuration YAML file in PFPO/conf/api/vllm/apps/ for production deployment (PFPO/conf/api/vllm/apps/deepseek_coder/train_v2_0.yaml)
5. Update LatentLM/train_hf.py to parse and instantiate your model variant (LatentLM/train_hf.py)

Add a new attention mechanism

1. Implement attention class in Diff-Transformer/ directory (e.g., multihead_*.py) (Diff-Transformer/multihead_diffattn.py)
2. Create optimized kernel wrapper if CUDA optimization needed (Diff-Transformer/kernel/rotary.py)
3. Benchmark against baseline in Diff-Transformer/example.py (Diff-Transformer/example.py)
4. Integrate into LatentLM/models/Transformer.py as pluggable attention option (LatentLM/models/Transformer.py)

Add support for a new vision tokenizer

1. Create tokenizer class in LatentLM/tokenizer_models/ inheriting from vae.py base (LatentLM/tokenizer_models/vae.py)
2. Implement encode/decode and perplexity computation methods (LatentLM/tokenizer_models/modeling_common.py)
3. Add evaluation metric in LatentLM/metrics/ (e.g., fidelity or inception scores) (LatentLM/metrics/IS.py)
4. Update LatentLM/train_hf.py to instantiate and integrate your tokenizer (LatentLM/train_hf.py)
5. Add FID evaluation script referencing new tokenizer output format (LatentLM/evaluate_fid.py)

Add a new training configuration for PFPO

1. Create YAML config in PFPO/conf/api/vllm/apps/{model_family}/{variant}.yaml (PFPO/conf/api/vllm/apps/deepseek_coder/dev_v1_0.yaml)
2. Define vLLM API parameters, model ID, tokenizer, and sampling hyperparameters (PFPO/conf/api/vllm/apps/general_eval/dev_v2_2.yaml)
3. Register validation/test IDs in apps_train_sub_val_ids.json if using dataset splits (PFPO/apps_train_sub_val_ids.json)
4. Execute via PFPO training framework with config reference for reproducible runs (PFPO/README.md)

VersionPinning: antlr4-python3-runtime==4.11.1 must match sympy==1.12 for math evaluation—upgrading either breaks the other. CUDA Kernels: Diff-Transformer code requires CUDA-capable GPU and compiled kernels; CPU-only setups will fail silently on flash_attn and custom attention ops. Missing Checkpoints: LatentLM, Kosmos models require downloading pretrained VAE, vision encoder, and language model weights—these are not in the repo. Model-Specific Dependencies: kosmos-2.5 needs additional vision model setup (BEiT3, CLIP), while diffusion models need DPM-Solver. No Central Config: each subproject has different hyperparameter and data format conventions; no unified setup guide covers all modules.

• Differential Attention (Diff-Attention) — Core mechanism in Diff-Transformer reducing quadratic attention complexity via differential computation; directly enables longer context windows and efficiency gains claimed in the repo
• Rotary Position Embeddings (RoPE) — Used in Diff-Transformer/kernel/rotary.py and LatentLM/models/kernel/rotary.py for length extrapolation; fundamental to modern Transformers' ability to generalize beyond training sequence length
• Diffusion Models & Denoising — LatentLM builds on diffusion principles (DDPM, DPM-Solver in schedule/); understanding forward/reverse diffusion processes is required to modify generative components
• Vision Tokenization (VAE Latent Codes) — LatentLM/tokenizer_models/ compress images to discrete latent tokens enabling alignment with text; core to multimodal pretraining strategy in Kosmos models
• Sparse Mixture of Experts (MoE) — xmoe/ subproject implements scalable sparse MoE patterns; enables parameter efficiency and specialization critical for 100+ language models (InfoXLM/XLM-E)
• Unified Pre-training Objective (Multi-task Learning) — UniLM's core innovation: single model trained on language understanding, generation, and cross-lingual tasks simultaneously; requires understanding masked LM, causal LM, and seq2seq loss weighting
• Flash Attention (Memory-Efficient CUDA Kernels) — flash_attn dependency and custom implementations (multihead_flashdiff_*.py) provide O(N) memory and reduced compute via kernel fusion; critical for scaling to long contexts or large batches

• microsoft/torchscale — Official separate repo containing TorchScale foundation architectures (DeepNet, Magneto) that unilm builds upon; reference implementation for stable Transformer scaling
• huggingface/transformers — Core dependency and ecosystem partner—unilm models integrate via HuggingFace's model registry and tokenizer APIs; many unilm models export to transformers format
• huggingface/diffusers — Companion library for diffusion-based generative models like LatentLM; shares scheduling and sampling patterns with unilm's schedule/ modules
• microsoft/Megatron-DeepSpeed — Distributed training framework often used to scale unilm pretraining; integrates with TorchScale architectures for multi-GPU/multi-node setups
• facebookresearch/detectron2 — Vision backbone and detection utilities referenced in multimodal architectures; LatentLM vision tokenizers build on similar vision encoder patterns

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for Diff-Transformer attention implementations

The Diff-Transformer module contains multiple attention implementations (multihead_attention.py, multihead_diffattn.py, multihead_flashdiff_1.py, multihead_flashdiff_2.py) but there are no visible test files. These are performance-critical components with kernel-level optimizations (rotary.py). Adding tests would ensure correctness across CUDA/non-CUDA backends and prevent regressions when updating the attention mechanisms.

• [ ] Create Diff-Transformer/tests/ directory with init.py
• [ ] Add test_multihead_attention.py with tests for standard vs. diff attention correctness (output shapes, numerical stability)
• [ ] Add test_attention_kernels.py testing rotary.py embeddings and flash attention variants across different sequence lengths and batch sizes
• [ ] Add test_example.py to validate Diff-Transformer/example.py runs without errors
• [ ] Include GPU/CPU compatibility checks in tests using pytest fixtures

Add integration tests and CI workflow for PFPO math evaluation pipeline

The PFPO module includes specialized math evaluation dependencies (sympy==1.12, antlr4-python3-runtime==4.11.1, word2number, Pebble, timeout-decorator) but no visible tests or CI workflow to validate the math evaluation functionality. Given the strict version pinning requirements, this is critical to prevent dependency conflicts and ensure math evaluation reliability.

• [ ] Create PFPO/tests/ directory with test_math_eval.py
• [ ] Add tests validating sympy compatibility with antlr4-python3-runtime==4.11.1 specifically
• [ ] Create .github/workflows/pfpo_math_eval_tests.yml GitHub Action that installs PFPO dependencies and runs math evaluation tests
• [ ] Add tests for timeout-decorator and Pebble timeout handling in PFPO evaluation pipeline
• [ ] Document expected Python version and CUDA requirements in PFPO/README.md

Add model loading and inference tests for LatentLM tokenizer models

LatentLM contains multiple specialized vision tokenizer models (modeling_beit3_vision.py, modeling_sigma_vae.py) and a Diffusion Transformer (DiT.py) that orchestrate complex pipelines. There are no visible tests validating that these models load correctly, have correct input/output shapes, or work with the VAE and schedule modules (ddpm.py, dpm_solver.py).

• [ ] Create LatentLM/tests/ directory with init.py
• [ ] Add test_tokenizer_models.py testing that BEiT3Vision and SigmaVAE models instantiate and perform forward passes with expected tensor shapes
• [ ] Add test_dit_inference.py validating DiT.py forward pass with different noise schedules (ddpm.py, dpm_solver.py)
• [ ] Add test_vae_encoding.py for VAE tokenizer round-trip (encode/decode) consistency
• [ ] Add fixture in conftest.py for downloading/caching small pretrained weights for testing

• Add unit tests for LatentLM/metrics/fid.py and LatentLM/metrics/IS.py—these critical evaluation metrics lack test coverage and examples of expected tensor shapes/ranges
• Document the data format requirements and preprocessing steps for each subproject (Kosmos vision-text pairing, LatentLM image-caption alignment, multilingual corpus structure for InfoXLM)—currently only implied in training scripts
• Create a Makefile or setup.py that consolidates dependency installation and provides single commands to download base model checkpoints (vision encoder, VAE, language model) for LatentLM and Kosmos, reducing first-run friction

• High · Outdated and Vulnerable Dependencies — dependencies/requirements (inferred from package list). The dependency file specifies sympy==1.12 and antlr4-python3-runtime==4.11.1. These are pinned to older versions that may contain known security vulnerabilities. Additionally, no version constraints are specified for critical packages like torch, transformers, vllm, and datasets, which could lead to installation of vulnerable versions. Fix: 1) Update sympy to the latest stable version and verify antlr4-python3-runtime compatibility. 2) Add version constraints and upper bounds for all dependencies (e.g., torch>=2.0.0,<3.0.0). 3) Regularly run security audits using tools like safety, pip-audit, or dependabot to identify and patch known vulnerabilities.
• Medium · Unverified External Dependencies — Package dependencies: vllm, flash_attn, transformers. The codebase relies on external packages from PyPI (vllm, flash_attn, transformers) without explicit verification mechanisms. These packages could potentially be compromised or contain malicious code. The flash_attn package in particular is a high-performance CUDA extension with lower community scrutiny than mainstream packages. Fix: 1) Implement dependency verification using hash checking or lock files (poetry.lock, requirements.lock). 2) Audit critical dependencies (especially compiled extensions like flash_attn) for source code review. 3) Use private package mirrors if available in enterprise settings. 4) Implement Software Composition Analysis (SCA) tools to monitor for vulnerabilities.
• Medium · Absence of Security Configuration Files — Repository root and configuration directories. No security-related configuration files are visible (.env files for secrets management, security policy files, SECURITY.md, or vulnerability disclosure guidelines). The codebase contains no apparent input validation, authentication, or authorization mechanisms visible in the file structure. Fix: 1) Create a SECURITY.md file with vulnerability disclosure guidelines. 2) Implement .env.example templates showing required environment variables without actual secrets. 3) Use tools like python-dotenv for secure secret management. 4) Add pre-commit hooks to prevent accidental secrets commits (e.g., detect-secrets, git-secrets).
• Medium · Machine Learning Model Supply Chain Risk — LatentLM/sample_hf.py, LatentLM/train_hf.py, and related model loading code. The codebase involves downloading and loading pre-trained models (evident from references to HuggingFace model loading in files like sample_hf.py and train_hf.py). No apparent model verification or checksum validation is visible, creating risks for model poisoning or adversarial manipulation attacks. Fix: 1) Implement cryptographic verification (SHA-256 checksums) for downloaded models. 2) Document all model sources and versions used. 3) Use trusted model registries with signatures (HuggingFace Hub's signed releases). 4) Implement model scanning for backdoors or anomalies before deployment. 5) Maintain an approved models list.
• Low · No Apparent Input Validation in Data Processing — LatentLM/evaluate_fid.py, LatentLM/metrics/. Files like evaluate_fid.py and metrics processing scripts lack visible input sanitization. If these scripts accept user-supplied data files or paths, they could be vulnerable to path traversal or file inclusion attacks. Fix: 1) Implement strict input validation for file paths using pathlib.Path and os.path.abspath() verification. 2) Use whitelisting for allowed file extensions and locations. 3) Validate dataset formats and dimensions before processing. 4) Implement proper exception handling to avoid information disclosure.
• Low · Debug Code and Configuration Exposure Risk — PFPO/conf/api/vllm/apps/deepseek_coder/ and subdirectories. Multiple YAML configuration files in PFPO/conf suggest extensive hyperparameter and model configuration exposure. If these include sensitive information or are deployed with debug settings, they could leak system internals. Fix: 1) Audit all YAML files for sensitive information (API keys, internal paths, debug flags). 2) Implement environment variable substitution for configuration values. 3) Use separate configuration files for dev/staging/production. 4) Implement configuration validation schema to prevent invalid/dangerous settings.
• Low · Missing — undefined. undefined Fix: undefined

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/microsoft/unilm shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live microsoft/unilm repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/microsoft/unilm.

What it runs against: a local clone of microsoft/unilm — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in microsoft/unilm | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 136 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of microsoft/unilm. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/microsoft/unilm.git
#   cd unilm
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of microsoft/unilm and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "microsoft/unilm(\\.git)?\\b" \\
  && ok "origin remote is microsoft/unilm" \\
  || miss "origin remote is not microsoft/unilm (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "Diff-Transformer/multihead_diffattn.py" \\
  && ok "Diff-Transformer/multihead_diffattn.py" \\
  || miss "missing critical file: Diff-Transformer/multihead_diffattn.py"
test -f "LatentLM/models/Transformer.py" \\
  && ok "LatentLM/models/Transformer.py" \\
  || miss "missing critical file: LatentLM/models/Transformer.py"
test -f "LatentLM/models/DiT.py" \\
  && ok "LatentLM/models/DiT.py" \\
  || miss "missing critical file: LatentLM/models/DiT.py"
test -f "LatentLM/train_hf.py" \\
  && ok "LatentLM/train_hf.py" \\
  || miss "missing critical file: LatentLM/train_hf.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 136 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~106d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/microsoft/unilm"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.