WAIT — Stale — last commit 2y ago

• 3 active contributors
• Apache-2.0 licensed
• Tests present
• ⚠ Stale — last commit 2y ago
• ⚠ Small team — 3 contributors active in recent commits
• ⚠ Single-maintainer risk — top contributor 98% of recent commits
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

ControlNet is a neural network architecture that adds conditional control to diffusion models (like Stable Diffusion) by injecting spatial or semantic guidance through multiple parallel neural pathways. It uses a clever 'locked' (original) and 'trainable' (learnable) copy pattern with zero-initialized convolutions to learn new control conditions without degrading the underlying diffusion model. The repo implements 14 different control types (canny edges, depth maps, pose, semantic segmentation, etc.) via annotator modules that preprocess input images into control signals. Modular monorepo: annotator/ directory contains task-specific preprocessors (canny, hed, midas, mlsd, openpose, uniformer) each with their own models and preprocessing logic. Each annotator is a standalone module that converts input images to control maps (depth, edges, poses, segmentation masks). The design allows plugging different control conditions into a shared ControlNet injection mechanism without modifying core diffusion code.

ML researchers and practitioners building custom image generation systems who need to steer Stable Diffusion outputs with spatial constraints (pose guides, depth maps, edge maps) without retraining the base model. Also targets researchers exploring fine-tuning techniques on limited hardware by preserving pre-trained weights.

Production-ready and actively maintained. The repo has significant community adoption (referenced in the README as a standard control mechanism for diffusion), released v1.0 in early 2023, and is currently evolving with v1.1 nightly builds. Code quality is high (2.2MB Python, well-structured annotators), though it's a research repository rather than a dependency library.

Low risk for inference, moderate risk for fine-tuning workflows. Heavy CUDA/PyTorch dependency with no package.json or requirements.txt visible in the file list (dependencies inferred from code only). The architecture depends on specific Stable Diffusion model versions, so base model API changes could break compatibility. Single author (lllyasviel) creates concentration risk, though the approach is academically validated and widely adopted.

Active development on ControlNet 1.1 (nightly branch referenced in README, not yet merged to main). Recent discussions include precomputed ControlNet for 45% speedup and ablation studies on encoder depth. The project is actively integrating new condition types and optimizing inference performance.

Clone the repository with git clone https://github.com/lllyasviel/ControlNet.git && cd ControlNet. Install PyTorch and dependencies (exact command unavailable—check README for conda/pip setup). Download annotator checkpoints from annotator/ckpts/ directory (see ckpts.txt for URLs). Run inference using provided scripts (specific entry points not visible in file list—check examples/ or main README).

Daily commands: No single entry point visible. Usage pattern (from README): (1) Load a pre-trained annotator (e.g., from annotator.midas import MidasDepthEstimator), (2) Process input image to generate control map, (3) Pass control map + text prompt to ControlNet-conditioned Stable Diffusion pipeline. Exact CLI/script names require consulting the main README or example notebooks.

• annotator/openpose/__init__.py — Entry point for OpenPose annotator—a critical pose detection preprocessor that conditions diffusion models on human skeleton data
• annotator/midas/api.py — Depth estimation API wrapper—core preprocessor for depth-conditioned image generation
• annotator/canny/__init__.py — Canny edge detection implementation—fundamental edge-based control signal for diffusion models
• annotator/hed/__init__.py — Holistically-Nested Edge Detection—alternative edge representation for conditional control
• annotator/mlsd/__init__.py — M-LSD line segment detector—provides structural line features for architectural/geometric control
• annotator/uniformer/__init__.py — Semantic segmentation backbone—enables semantic mask-based conditioning of diffusion generation
• README.md — Architecture overview explaining zero-convolution training strategy and safety guarantees for fine-tuning

Add a New Annotator (Control Condition Extractor)

1. Create new annotator package directory under annotator/ (annotator/my_annotator/__init__.py)
2. Implement condition extraction model with init() constructor and call() inference method following OpenPose or MiDaS pattern (annotator/my_annotator/__init__.py)
3. Add checkpoint download URL to annotator manifest (annotator/ckpts/ckpts.txt)
4. Register annotator in main codebase entry point (typically in a control script or web UI not shown in this file list)

Add a New Semantic Segmentation Dataset

1. Create dataset config in annotator/uniformer/configs/base/datasets/ (annotator/uniformer/configs/_base_/datasets/my_dataset.py)
2. Define dataset_type, data_root, and train/val splits following cityscapes.py or pascal_voc12.py convention (annotator/uniformer/configs/_base_/datasets/my_dataset.py)
3. Create experiment config in annotator/uniformer/exp/ that imports base dataset and model configs (annotator/uniformer/exp/my_exp/config.py)

Optimize a Detection Model for Inference

1. Use lightweight variant: choose mbv2_mlsd_tiny over mbv2_mlsd_large or small UniFormer variants (annotator/mlsd/models/mbv2_mlsd_tiny.py)
2. Apply quantization utilities for int8/fp16 conversion (annotator/uniformer/mmcv/arraymisc/quantization.py)
3. Implement batching and GPU memory management in the annotator's call() method

Extend MiDaS Depth Model with Custom Backbone

1. Create new depth model class inheriting from BaseModel in annotator/midas/ (annotator/midas/midas/base_model.py)
2. Implement custom encoder/decoder following DPT or MiDasNet architecture (annotator/midas/midas/midas_net_custom.py)
3. Register new model type in annotator/midas/api.py load_model() function

• PyTorch + TorchVision — Standard deep learning framework for model training and inference; essential for diffusion model integration
• Vision Transformers (ViT) + Convolutional Encoders — Modern architecture for robust feature extraction across diverse visual conditions (depth, edges, pose, segmentation)
• Zero-Convolution (1×1 conv initialized to zero) — Enables safe, non-destructive fine-tuning—original diffusion weights locked and preserved during ControlNet training
• MiDaS + OpenPose + M-LSD modular design — Pluggable, independent annotators allow mixing/swapping control conditions without modifying core diffusion logic
• mmcv (OpenMMLab utilities) — Reusable CNN primitives (conv_module, activations, normalization) reduce code duplication across models

• Locked + Trainable copies instead of full fine-tuning

  • Why: Prevents catastrophic forgetting and model degradation on small datasets
  • Consequence: Slightly higher memory overhead (duplicate model weights) but guarantees production safety and enables personal-device training

• Multiple independent annotator modules vs single unified extractor

  • Why: Modularity and swappability—users can choose/combine conditions (pose + depth, edge + segmentation, etc.)
  • Consequence: Requires separate model checkpoints and loading per condition type; more memory at inference if using multiple conditions

• MiDaS monocular depth instead of stereo/structured-light

  • Why: Single-image depth works with any photo without paired cameras; matches text-to-image generative paradigm
  • Consequence: Depth predictions less precise than stereo but sufficient for plausible image synthesis

No explicit environment configuration or requirements.txt in visible files—dependencies are implicit in each annotator's imports. Model checkpoints must be manually downloaded to annotator/ckpts/ (see ckpts.txt for links). The 'locked' vs 'trainable' copy mechanism requires careful gradient flow management (GitHub issues mention confusion around zero-conv backprop). Different annotators have different input size requirements (e.g., Uniformer expects 769×769 in one config, 512×512 in another). CUDA version and PyTorch version matter significantly for model loading—no version pinning visible in file list.

• Zero Convolution / Zero-Initialized Bottleneck — The key innovation allowing ControlNet to be trained on small datasets without degrading pre-trained models—both weights and bias start at zero, so gradients flow through preserved locked blocks unchanged until training progresses.
• Locked vs Trainable Copy Pattern — ControlNet duplicates SD encoder blocks into 'locked' (no gradient updates, preserves original) and 'trainable' (learns condition) copies—critical for safe transfer learning on small datasets without catastrophic forgetting.
• Dense Prediction Transformers (DPT) — Backbone architecture used in MiDaS and Uniformer annotators for dense per-pixel prediction (depth, segmentation); Vision Transformer-based alternative to CNNs that ControlNet leverages.
• Cross-Attention Injection — ControlNet injects spatial control signals through cross-attention mechanisms in diffusion layers; understanding attention-based conditioning is essential for customizing control pathways.
• Monocular Depth Estimation — MiDaS annotator infers 3D depth from single RGB images; a key control modality enabling 3D-aware image generation. Common computer vision task with unique challenges for diffusion guidance.
• Structured Edge Detection (HED) — Holistically-Nested Edge Detection produces boundary maps as control signals; simpler alternative to Canny that ControlNet uses for edge-guided generation.
• Semantic Segmentation via Vision Transformers (Uniformer) — Uniformer annotator performs class-wise semantic segmentation for class-conditional control; demonstrates how ControlNet can be guided by high-level semantic labels beyond low-level geometry.

• CompVis/stable-diffusion — The underlying base diffusion model that ControlNet wraps and augments with control conditions.
• openai/CLIP — Text encoder used for text-to-image guidance in Stable Diffusion; understanding CLIP embeddings helps explain how prompts and controls interact.
• lllyasviel/ControlNet-v1-1-nightly — Nightly development branch with new condition types and performance improvements; future merge target for this stable repo.
• rwightman/pytorch-image-models — Source of pre-trained Vision Transformer and DPT encoder backbones used in Uniformer and MiDaS annotators.
• CMU-Perceptual-Computing-Lab/openpose — Original OpenPose pose detection framework; the ControlNet annotator wraps this for skeleton-based control.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add annotator initialization tests and validation suite

The annotator/ directory contains multiple specialized annotators (canny, hed, midas, mlsd, openpose, uniformer) but there's no evidence of unit tests validating their initialization, model loading, or output shapes. This is critical for a control-flow library where annotators are the entry point. A contributor could create tests that verify each annotator loads correctly, handles edge cases (invalid inputs, missing model files), and produces expected output dimensions.

• [ ] Create tests/annotator/test_canny.py validating edge detection output format
• [ ] Create tests/annotator/test_midas.py verifying depth estimation model loading and inference
• [ ] Create tests/annotator/test_openpose.py testing pose detection with sample images
• [ ] Create tests/annotator/test_model_loading.py for the annotator/ckpts/ checkpoint system
• [ ] Add pytest configuration and fixtures in tests/ for reusable test utilities

Implement missing LICENSE files and compliance documentation for annotator submodules

The repo structure shows LICENSE files exist in midas/, mlsd/, openpose/, and uniformer/ subdirectories, but annotator/canny/ and annotator/hed/ are missing LICENSE files. These are third-party implementations (canny is canonical edge detection, hed is likely from xDoG). A contributor should identify the correct licenses for these annotators and add them, plus create a CONTRIBUTORS.md or LICENSES.md explaining the provenance of each annotator module to improve attribution.

• [ ] Research and add annotator/canny/LICENSE (likely BSD or MIT based on canonical implementations)
• [ ] Research and add annotator/hed/LICENSE with proper attribution
• [ ] Create ANNOTATORS.md documenting each annotator's source, original paper, and license
• [ ] Update main README.md with a 'Credits' section linking to annotator submodule origins

Create integration tests for ControlNet pipeline with multiple annotator types

While individual annotators exist, there's no visible test suite demonstrating how annotators integrate with the diffusion model control pipeline. A contributor could create end-to-end integration tests that load a simple diffusion model, run different annotators (canny, midas, openpose), and verify the ControlNet conditioning mechanism receives properly formatted inputs. This validates the core value proposition: 'control diffusion models.'

• [ ] Create tests/integration/test_controlnet_pipeline.py with fixture for mock diffusion model
• [ ] Add test validating canny edge map → ControlNet conditioning flow
• [ ] Add test validating midas depth map → ControlNet conditioning flow
• [ ] Add test validating openpose keypoints → ControlNet conditioning flow
• [ ] Create tests/fixtures/ with small sample images for reproducible integration tests

• Add unit tests for annotator preprocessing pipelines—each module under annotator/*/ currently has no visible test coverage (test directory absent from file list). Start by writing tests for annotator/midas/api.py output validation and batch processing correctness.
• Document annotator input/output specifications and performance characteristics—create a table in README comparing resolution constraints, inference speed, VRAM usage, and output value ranges for all 6 annotators (midas, canny, hed, mlsd, openpose, uniformer).
• Add missing error handling and logging to annotator init.py files—several annotators check for model files but have no graceful fallback or verbose error messages when checkpoints are missing or corrupted. Implement try/catch with helpful messages across annotator/*/ modules.

ControlNet's static codebase analysis reveals a moderate security posture. The primary concerns are: (1) unavailable dependency file for vulnerability scanning, (2) model deserialization risks inherent to ML frameworks, (3) potential model checkpoint access control issues, and (4) multiple third-party licenses requiring compliance verification. The codebase itself appears well-structured with clear separation of concerns, but critical security details about model loading, input validation, and dependency management cannot be fully assessed without additional files. Recommend implementing automated vulnerability scanning, secure model loading practices, and comprehensive documentation of security controls.

• Medium · Dependency Management Not Provided — requirements.txt / setup.py / pyproject.toml (not provided). The dependency/package file content was not provided for analysis. This prevents verification of known vulnerabilities in third-party libraries used by ControlNet (PyTorch, Diffusers, OpenCV, etc.). Machine learning projects typically have complex dependency trees that may contain outdated or vulnerable packages. Fix: Provide and analyze dependency files using tools like: pip-audit, safety, or Snyk to identify vulnerable dependencies. Implement regular dependency scanning in CI/CD pipeline.
• Medium · Model Checkpoint Directory with Unclear Access Controls — annotator/ckpts/. The file 'annotator/ckpts/ckpts.txt' suggests storage of model checkpoints. If this directory is web-accessible or improperly configured, it could allow unauthorized download of large model files, leading to resource exhaustion or unauthorized model redistribution. Fix: Implement proper access controls for checkpoint directories. Use authentication/authorization mechanisms. Consider using cloud storage with appropriate IAM policies. Document which checkpoints are meant for public distribution.
• Medium · Potential Model Deserialization Vulnerability — annotator/midas/, annotator/mlsd/, annotator/openpose/, annotator/uniformer/ (model loading code not fully visible). ControlNet is a neural network codebase that loads model weights and checkpoints. PyTorch's pickle-based weight loading (torch.load) can execute arbitrary code if loading untrusted model files. This is a known vulnerability in ML frameworks. Fix: Validate model checksums before loading. Use torch.load with weights_only=True (PyTorch 1.13+). Only load models from trusted sources. Implement model signature verification. Document safe model loading practices.
• Low · Multiple LICENSE Files Without Clear Compliance — annotator/midas/LICENSE, annotator/mlsd/LICENSE, annotator/openpose/LICENSE, annotator/uniformer/LICENSE. The codebase contains multiple LICENSE files in subdirectories (annotator/midas/LICENSE, annotator/mlsd/LICENSE, annotator/openpose/LICENSE, annotator/uniformer/LICENSE) indicating integration of multiple third-party components. Potential licensing conflicts or compliance issues if licenses are incompatible. Fix: Conduct a complete license audit using tools like FOSSA or Black Duck. Document all third-party components and their licenses. Ensure license compatibility with ControlNet's license. Add license information to documentation.
• Low · Configuration Files May Contain Sensitive Information — annotator/uniformer/configs/, annotator/uniformer/exp/. Multiple configuration files exist throughout the codebase (annotator/uniformer/configs/, test configs, etc.). These may contain hardcoded paths, API keys, or other sensitive information if not properly reviewed. Fix: Review all configuration files for hardcoded secrets. Use environment variables for sensitive configuration. Add *.local, *.key, and sensitive config files to .gitignore. Implement configuration validation.
• Low · Lack of Input Validation Documentation — annotator/canny/, annotator/hed/, annotator/midas/, annotator/mlsd/, annotator/openpose/, annotator/uniformer/. The file structure suggests multiple model types (Canny, HED, MIDAS, MLSD, OpenPose, UniFormer) that process image inputs. Without visible input validation code, there's potential risk of processing malicious or malformed images that could cause DoS or unexpected behavior. Fix: Implement robust input validation for all image processing modules. Validate image dimensions, formats, and file sizes. Add error handling for corrupted or malicious inputs. Document expected input specifications.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/lllyasviel/ControlNet shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live lllyasviel/ControlNet repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/lllyasviel/ControlNet.

What it runs against: a local clone of lllyasviel/ControlNet — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in lllyasviel/ControlNet | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 834 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of lllyasviel/ControlNet. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/lllyasviel/ControlNet.git
#   cd ControlNet
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of lllyasviel/ControlNet and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "lllyasviel/ControlNet(\\.git)?\\b" \\
  && ok "origin remote is lllyasviel/ControlNet" \\
  || miss "origin remote is not lllyasviel/ControlNet (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "annotator/openpose/__init__.py" \\
  && ok "annotator/openpose/__init__.py" \\
  || miss "missing critical file: annotator/openpose/__init__.py"
test -f "annotator/midas/api.py" \\
  && ok "annotator/midas/api.py" \\
  || miss "missing critical file: annotator/midas/api.py"
test -f "annotator/canny/__init__.py" \\
  && ok "annotator/canny/__init__.py" \\
  || miss "missing critical file: annotator/canny/__init__.py"
test -f "annotator/hed/__init__.py" \\
  && ok "annotator/hed/__init__.py" \\
  || miss "missing critical file: annotator/hed/__init__.py"
test -f "annotator/mlsd/__init__.py" \\
  && ok "annotator/mlsd/__init__.py" \\
  || miss "missing critical file: annotator/mlsd/__init__.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 834 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~804d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/lllyasviel/ControlNet"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.