GO — Healthy across all four use cases

• Last commit 5mo ago
• 24+ active contributors
• Distributed ownership (top contributor 15% of recent commits)
• MIT licensed
• CI configured
• Tests present
• ⚠ Slowing — last commit 5mo ago
• ⚠ Scorecard: default branch unprotected (0/10)

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests + OpenSSF Scorecard}

Stability AI's open-source generative models repository implementing diffusion-based text-to-image (Stable Diffusion XL), image-to-3D (SV3D), video synthesis (SVD), and video-to-4D (SV4D 2.0) models. The core capability is sampling latent diffusion models with configurable schedulers, conditioning mechanisms, and multi-view generation pipelines, enabling researchers to generate images, videos, and 3D/4D assets from text or video inputs. Monorepo structured around scripts/sampling/ (inference entry points like simple_video_sample_4d2.py), configs/inference/ (YAML configs for each model variant: sd_xl_base.yaml, sv3d_p.yaml, svd.yaml, sv4d2.0), configs/example_training/ (training recipes for autoencoders and diffusion models), and data/ (minimal assets like fonts). Core diffusion logic lives in Python modules (exact paths unclear from file list but implied to be in root-level packages).

ML researchers and engineers building or fine-tuning generative models; computer vision engineers integrating Stable Diffusion variants into applications; 3D asset creators using SV3D/SV4D for novel-view synthesis; anyone training autoencoders or diffusion models from scratch using the example configs in configs/example_training/.

Actively maintained (May 2025 release of SV4D 2.0); repository shows strong production maturity with CI workflows (.github/workflows/ includes black.yml, test-build.yaml, test-inference.yml), extensive example training configs, and multiple stable inference scripts. However, it's primarily a research release platform rather than a packaged library, so integration requires familiarity with PyTorch and diffusion concepts.

No obvious dependency lock file visible in file list (no requirements.txt or pyproject.toml shown); model checkpoints are large and fetched from HuggingFace (external dependency); training configs suggest significant VRAM requirements with no clear fallback for resource-constrained setups. The codebase is mature but research-oriented, so API stability for internal modules is not guaranteed across releases.

SV4D 2.0 (video-to-4D) was just released (May 20, 2025) with improvements over SV4D: higher fidelity, better temporal consistency, and removal of SV3D reference dependency. The quickstart emphasizes background removal via rembg or SAM2, autoregressive frame generation, and low-VRAM modes (--encoding_t=1). CI workflows are active (black formatting, build tests, inference tests).

Clone: git clone https://github.com/Stability-AI/generative-models.git && cd generative-models. Download the model: huggingface-cli download stabilityai/sv4d2.0 sv4d2.safetensors --local-dir checkpoints. Run inference: python scripts/sampling/simple_video_sample_4d2.py --input_path assets/sv4d_videos/camel.gif --output_folder outputs. For training, edit a config in configs/example_training/ and invoke the training script (path not shown in file list; check root or scripts/).

Daily commands: Inference: python scripts/sampling/simple_video_sample_4d2.py --input_path <video.gif> --output_folder outputs --num_steps 50 --elevations_deg 0.0 --remove_bg True. Training: refer to example configs (mnist_cond.yaml, imagenet-f8_cond.yaml) and invoke the main training script (not shown; likely root-level train.py or scripts/training/). For low-VRAM, add --encoding_t=1.

• sgm/inference/api.py — Core inference API that orchestrates diffusion model sampling; entry point for all generative model operations.
• main.py — Primary entry point for the application; demonstrates how to load configs and invoke inference pipelines.
• sgm/data/dataset.py — Base dataset abstraction used across all training workflows; defines data loading contract.
• scripts/demo/sampling.py — High-level sampling orchestration for image generation; shows config-driven model instantiation pattern.
• scripts/sampling/simple_video_sample.py — Reference implementation for video diffusion sampling; demonstrates SVD/SV3D/SV4D inference workflows.
• configs/inference/sd_xl_base.yaml — YAML config schema for SDXL inference; defines model architecture and sampling parameters convention.
• pyproject.toml — Project dependencies and build configuration; critical for reproducible environment setup.

Add a New Diffusion Sampling Script

1. Create a new YAML config in scripts/sampling/configs/ with model architecture, sampler type (DDM/Euler/etc.), and conditioning parameters (scripts/sampling/configs/my_model.yaml)
2. Copy and adapt scripts/sampling/simple_video_sample.py, importing sgm.inference.api.SamplingPipeline and loading your config via load_config() from sgm.util.config (scripts/sampling/simple_video_sample_custom.py)
3. Call SamplingPipeline.get_unique_embedder() for your conditioning type (text CLIP-L, pose, video frames) and sampler.sample() with noise and conditions (scripts/sampling/simple_video_sample_custom.py)
4. Register your script in .github/workflows/test-inference.yml to validate inference on PR (.github/workflows/test-inference.yml)

Add a New Dataset for Training

1. Create a new dataset class inheriting from sgm.data.dataset.Dataset in sgm/data/my_dataset.py; implement __len__(), __getitem__() returning {"jpg": torch.Tensor, "txt": str} or similar (sgm/data/my_dataset.py)
2. Create a training config YAML in configs/example_training/ specifying dataset name, batch size, conditioning type (e.g., text, image), and diffusion parameters (configs/example_training/my_training.yaml)
3. In main.py, add a branch in the config loading logic to instantiate your dataset class and pass to the training loop (main.py)

Integrate a New Web UI Feature

1. Add your feature function to scripts/demo/gradio_app.py or scripts/demo/gradio_app_sv4d.py, importing sampling logic from scripts/demo/sampling.py or video_sampling.py (scripts/demo/gradio_app.py)
2. Create a Gradio Interface block with inputs (text prompt, image, etc.) and output (image/video) using gr.Blocks() context manager (scripts/demo/gradio_app.py)
3. Wire your function to the Gradio button/textbox events, optionally adding safety checks via scripts/util/detection/nsfw_and_watermark_dectection.py (scripts/demo/gradio_app.py)

Add Safety/Filtering for Generated Content

1. Extend scripts/util/detection/nsfw_and_watermark_dectection.py with your detector (e.g., new OpenCLIP model or classifier) (scripts/util/detection/nsfw_and_watermark_dectection.py)
2. Import and call your detector in scripts/demo/sampling.py or gradio_app.py post-generation (scripts/demo/sampling.py)
3. Return a confidence score and flag unsafe outputs before returning to user (scripts/demo/sampling.py)

• PyTorch + Lightning (implicit via sgm modules) — GPU-accelerated diffusion model training and inference; autograd enables gradient-based sampling and optimization
• YAML configuration files — Decouples model architecture from code; enables reproducible hyperparameter experiments without recompilation
• Gradio web UI — Rapid prototyping of generative model demos without frontend engineering; auto-generates REST API
• OpenCLIP embeddings — Vision-language conditioning for text-to-image and image understanding; pre-trained on large multimodal datasets
• VAE (Variational Autoencoder) latent space — Compresses high-resolution images to lower-dimensional latent representations; reduces diffusion sampling

SV4D 2.0 input videos should ideally have white backgrounds and moving objects (per README); real-world videos may require pre-segmentation via Clipdrop or SAM2. Model checkpoints (.safetensors) are fetched from HuggingFace and must be manually downloaded into checkpoints/ directory; no automated download in obvious scripts. Low-VRAM mode (--encoding_t=1) is mentioned but not well-documented—test this early. Background removal (--remove_bg=True) requires rembg library (not listed in visible dependencies); ensure it's installed. Elevation/azimuth control is specific to SV3D/SV4D; text-to-image models (SDXL) don't use these parameters. No obvious requirements.txt or setup.py shown; dependency management may be implicit or in an unshown file.

• Latent Diffusion Models — Core technique underlying all models in this repo (SDXL, SV3D, SVD, SV4D); understanding the VAE encoder/decoder and diffusion in latent space is essential to interpreting the autoencoder configs and sampling logic.
• Classifier-Free Guidance — Used throughout this repo to control generation quality and adherence to conditioning inputs (text, image, video); crucial for understanding why models have separate conditional and unconditional branches.
• Noise Scheduling (DDPM, Karras) — Diffusion models require carefully tuned noise schedules; configs specify schedulers and timestep counts, and modifying these affects quality/speed tradeoffs visible in --num_steps parameter.
• Multi-View Synthesis — Core capability of SV3D and SV4D models; generating consistent views from different camera angles requires understanding novel-view conditioning, elevation/azimuth parameterization (--elevations_deg), and temporal consistency in video variants.
• Autoregressive Frame Generation — SV4D 2.0 generates 48 frames autoregressively (12 frames at a time, reusing previous output as conditioning); understanding this pattern is critical for extending to longer sequences or higher frame counts.
• Variational Autoencoders (VAE) / Autoencoders with KL Regularization — Training configs in configs/example_training/autoencoder/ show KL-regularized VAEs (kl-f4, kl-f8); these are the encoder/decoder backbone for latent diffusion, so understanding KL divergence loss and compression ratios (f4=4x, f8=8x) is essential.
• CLIP Text Encoding for Conditioning — Text-to-image models (SDXL, txt2img configs) use CLIP-L embeddings; understanding how text is encoded into the diffusion cross-attention mechanism is key to modifying text conditioning behavior.

• replicate/cog-comfyui — Alternative Stable Diffusion inference framework (ComfyUI) with node-based UI; users choosing between this repo's CLI and ComfyUI's graphical approach for the same models.
• huggingface/diffusers — Canonical diffusers library that abstracts Stability's models into a pip-installable interface; this repo is the 'source of truth' implementation while diffusers is the high-level wrapper most users install.
• Stability-AI/stablediffusion — Original Stable Diffusion repository (v1.x); predecessor to this generative-models monorepo (v2.x+, SDXL, video variants).
• facebookresearch/segment-anything-2 — Companion tool mentioned in README for background segmentation on real-world video inputs before running SV4D 2.0; essential preprocessing for high-quality results.
• danielgatis/rembg — Background removal library integrated into SV4D 2.0 pipeline (--remove_bg=True flag); handles preprocessing for white-background video inputs.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add integration tests for inference pipelines (SVD, SV3D, SV4D 2.0)

The repo has multiple inference configs (configs/inference/svd.yaml, sv3d_p.yaml, sv3d_u.yaml, sv4d*.yaml) and demo scripts (scripts/demo/sampling.py, gradio_app.py, gradio_app_sv4d.py) but pytest.ini exists with no corresponding test files visible. Adding integration tests would validate that each model's inference pipeline works end-to-end, preventing regressions as the codebase evolves.

• [ ] Create tests/test_inference_svd.py to test SVD sampling with configs/inference/svd.yaml
• [ ] Create tests/test_inference_sv3d.py to test SV3D variants (P and U) with their respective configs
• [ ] Create tests/test_inference_sv4d.py to test SV4D 2.0 inference pipeline referenced in README
• [ ] Add fixtures to load model configs from configs/inference/ directory
• [ ] Add pytest markers to skip heavy GPU tests in CI (already has .github/workflows/test-inference.yml)
• [ ] Document test expectations in tests/README.md

Add model card generation script for HuggingFace Hub uploads

The repo references multiple HuggingFace model releases (sv4d2.0, svd, sv3d) in the README and has model_licenses/ with LICENSE files for each model variant. Currently there's no tooling to auto-generate or validate model cards (README.md, model.safetensors metadata) for these uploads. This would ensure consistent documentation across all released models.

• [ ] Create scripts/upload/generate_model_card.py to template model cards from model_licenses/ metadata
• [ ] Add model card templates in configs/model_cards/ for each variant (sdxl-turbo, sv3d, svd, sv4d2.0)
• [ ] Reference existing licenses in model_licenses/LICENSE-* files to auto-populate license sections
• [ ] Add GitHub Actions workflow .github/workflows/validate-model-cards.yml to lint cards before release
• [ ] Document the workflow in README.md's release process section

Add configuration validation and schema tests for all training/inference configs

The configs/ directory has 10+ YAML files (example_training/ and inference/) but no validation layer. Invalid configs can fail silently mid-training or during inference. Adding a schema validator would catch config errors early and support new contributors in writing correct configs.

• [ ] Create scripts/config_validation.py with schema definitions for training (configs/example_training/.yaml) and inference (configs/inference/.yaml) configs
• [ ] Add tests/test_config_schema.py to validate all YAML files match their schema (e.g., required keys, type checking)
• [ ] Create a config schema reference document in docs/CONFIG_SCHEMA.md listing all valid keys per config type
• [ ] Integrate into .github/workflows/test-build.yaml to run config validation on every PR
• [ ] Add example: add a validation error catch for missing 'model' or 'batch_size' keys in training configs

• Add automated checkpoint download to simple_video_sample_4d2.py using huggingface_hub.hf_hub_download() to avoid manual setup; currently requires users to run huggingface-cli separately.
• Create a unified inference wrapper script (scripts/sampling/unified_sampler.py) that auto-detects input type (video, image, text) and routes to the correct sampling script (SV4D, SV3D, SDXL) with consistent argument parsing.
• Add unit tests for each config YAML in configs/inference/ and configs/example_training/ to validate schema and required fields; currently no visible test coverage for configuration validation despite test-build.yaml and test-inference.yml CI workflows.

The Stability-AI generative-models codebase demonstrates a reasonable security posture with a modular structure and clear separation of concerns. However, the analysis is limited by missing dependency file contents, which prevents comprehensive vulnerability assessment. Key recommendations: (1) Provide and scan dependency files for known vulnerabilities; (2) Implement safe model loading practices with checksum verification; (3) Add input validation to web-facing scripts (Gradio/Streamlit apps); (4) Create a SECURITY.md for vulnerability disclosure; (5) Implement automated security testing in CI/CD pipeline (.github/workflows files suggest some testing infrastructure exists). The codebase lacks evidence of common injection vulnerabilities (SQLi, XSS) given its machine learning focus, but web interface security should be hardened. No hardcoded secrets are apparent in the provided file listing.

• Medium · Missing dependency file for vulnerability analysis — pyproject.toml, requirements/pt2.txt. The pyproject.toml and requirements files are referenced in the file structure but their contents were not provided for analysis. This prevents comprehensive assessment of dependency vulnerabilities, version pinning practices, and supply chain security. Fix: Provide dependency files for analysis. Implement automated dependency scanning using tools like pip-audit, safety, or dependabot. Ensure all dependencies are pinned to specific versions and regularly updated.
• Low · Potential unsafe model loading patterns — scripts/sampling/simple_video_sample.py, scripts/sampling/simple_video_sample_4d.py, scripts/demo/gradio_app.py. The codebase contains multiple sampling and inference scripts that load pretrained models. Without seeing the actual implementation, there is a risk of unsafe deserialization if pickle or other unsafe formats are used to load model weights. Fix: Ensure only safe serialization formats (safetensors, ONNX) are used for model loading. Implement checksum verification for downloaded models. Use torch.load() with appropriate restrictions (weights_only=True in PyTorch 2.6+).
• Low · No visible input validation in script files — scripts/demo/gradio_app.py, scripts/demo/gradio_app_sv4d.py, scripts/demo/streamlit_helpers.py. Demo scripts (gradio_app.py, streamlit_helpers.py) accept user inputs but the actual validation logic cannot be assessed from the file structure alone. Potential for inadequate input sanitization in web-facing applications. Fix: Implement strict input validation for all user-facing APIs. Validate image/video dimensions, formats, and file sizes. Implement rate limiting and request throttling on web interfaces.
• Low · Missing security policy documentation — Repository root. No SECURITY.md or security policy file is visible in the repository structure, making it unclear how security vulnerabilities should be reported. Fix: Create a SECURITY.md file documenting responsible disclosure practices and security contact information. Follow CNCF or GitHub's recommended format.
• Low · Third-party font file included — data/DejaVuSans.ttf. The repository includes a TTF font file (DejaVuSans.ttf) which may have potential security implications if not properly validated. While DejaVuSans is a standard open font, including binary assets should be minimized. Fix: Consider referencing system fonts or loading from trusted CDNs rather than bundling. Verify the integrity of any bundled binary files and document their source.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/Stability-AI/generative-models shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live Stability-AI/generative-models repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/Stability-AI/generative-models.

What it runs against: a local clone of Stability-AI/generative-models — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in Stability-AI/generative-models | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 180 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of Stability-AI/generative-models. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/Stability-AI/generative-models.git
#   cd generative-models
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of Stability-AI/generative-models and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "Stability-AI/generative-models(\\.git)?\\b" \\
  && ok "origin remote is Stability-AI/generative-models" \\
  || miss "origin remote is not Stability-AI/generative-models (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "sgm/inference/api.py" \\
  && ok "sgm/inference/api.py" \\
  || miss "missing critical file: sgm/inference/api.py"
test -f "main.py" \\
  && ok "main.py" \\
  || miss "missing critical file: main.py"
test -f "sgm/data/dataset.py" \\
  && ok "sgm/data/dataset.py" \\
  || miss "missing critical file: sgm/data/dataset.py"
test -f "scripts/demo/sampling.py" \\
  && ok "scripts/demo/sampling.py" \\
  || miss "missing critical file: scripts/demo/sampling.py"
test -f "scripts/sampling/simple_video_sample.py" \\
  && ok "scripts/sampling/simple_video_sample.py" \\
  || miss "missing critical file: scripts/sampling/simple_video_sample.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 180 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~150d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/Stability-AI/generative-models"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.