If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/cumulo-autumn/StreamDiffusion shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

WAIT — Stale — last commit 1y ago

• 18 active contributors
• Distributed ownership (top contributor 30% of recent commits)
• Apache-2.0 licensed
• CI configured
• ⚠ Stale — last commit 1y ago
• ⚠ No test directory detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live cumulo-autumn/StreamDiffusion repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/cumulo-autumn/StreamDiffusion.

What it runs against: a local clone of cumulo-autumn/StreamDiffusion — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in cumulo-autumn/StreamDiffusion | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 549 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of cumulo-autumn/StreamDiffusion. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/cumulo-autumn/StreamDiffusion.git
#   cd StreamDiffusion
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of cumulo-autumn/StreamDiffusion and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "cumulo-autumn/StreamDiffusion(\\.git)?\\b" \\
  && ok "origin remote is cumulo-autumn/StreamDiffusion" \\
  || miss "origin remote is not cumulo-autumn/StreamDiffusion (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "demo/realtime-txt2img/main.py" \\
  && ok "demo/realtime-txt2img/main.py" \\
  || miss "missing critical file: demo/realtime-txt2img/main.py"
test -f "demo/realtime-img2img/main.py" \\
  && ok "demo/realtime-img2img/main.py" \\
  || miss "missing critical file: demo/realtime-img2img/main.py"
test -f "demo/realtime-img2img/frontend/src/lib/lcmLive.ts" \\
  && ok "demo/realtime-img2img/frontend/src/lib/lcmLive.ts" \\
  || miss "missing critical file: demo/realtime-img2img/frontend/src/lib/lcmLive.ts"
test -f "demo/realtime-img2img/connection_manager.py" \\
  && ok "demo/realtime-img2img/connection_manager.py" \\
  || miss "missing critical file: demo/realtime-img2img/connection_manager.py"
test -f "demo/realtime-txt2img/frontend/src/app.tsx" \\
  && ok "demo/realtime-txt2img/frontend/src/app.tsx" \\
  || miss "missing critical file: demo/realtime-txt2img/frontend/src/app.tsx"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 549 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~519d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/cumulo-autumn/StreamDiffusion"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

StreamDiffusion is a pipeline-level optimization framework that enables real-time interactive image generation using diffusion models, achieving 106 fps on SD-Turbo and 38 fps on LCM-LoRA through techniques like Residual Classifier-Free Guidance, Stochastic Similarity Filtering, and KV-Cache pre-computation. It transforms latency-heavy diffusion pipelines into interactive tools by batching inference, reducing computational redundancy, and optimizing GPU utilization. Dual-layer architecture: core Python library (99,698 LOC) implementing StreamDiffusion pipeline optimizations in src/, paired with full-stack demo in demo/realtime-img2img/ (SvelteKit frontend, FastAPI backend via connection_manager.py, config.py). Frontend uses Svelte components for UI controls (PipelineOptions.svelte, ImagePlayer.svelte) communicating via WebSocket to Python inference backend.

ML engineers and researchers building real-time generative AI applications (text-to-image, image-to-image) who need production-grade inference optimization without sacrificing model quality. Also targets demo developers using the demo/realtime-img2img web interface with SvelteKit frontend.

Actively maintained but research-focused: published arXiv paper (2312.12491) with benchmarks on RTX 4090, CI/CD via GitHub Actions (release.yml), but limited visible test suite in file listing. Code is well-documented with multiple language READMEs (English, Japanese, Korean) suggesting academic rigor, yet commit recency and issue backlog not visible in provided data.

Research project with limited test coverage visible in file structure—no tests/ directory shown. Depends on cutting-edge diffusion model infrastructure (Hugging Face integration implied) that may have breaking API changes. Single-authored feel with key paper authors listed; community contribution patterns unclear. GPU-heavy optimization (RTX 4090 benchmarks) may not translate predictably to consumer hardware.

Primary focus on real-time image generation optimization. The demo/realtime-img2img/ application is fully featured with config system, WebSocket connection management, and SvelteKit scaffolding (vite dev setup). GitHub Actions release workflow suggests ongoing publication; no PR/issue data provided limits visibility into active development.

Clone: git clone https://github.com/cumulo-autumn/StreamDiffusion.git && cd StreamDiffusion. Install: pip install -r requirements.txt (inferred, verify exact path). For frontend demo: cd demo/realtime-img2img/frontend && npm install && npm run dev to start SvelteKit dev server on port 5173 (Vite default). Backend requires Python 3.8+ and CUDA-capable GPU.

Daily commands: Frontend: npm run dev (starts Vite dev server at http://localhost:5173). Backend: Inferred as python -m demo.realtime_img2img.main or similar entry point (exact command not visible; check demo/realtime-img2img/ for app.py or main.py). Build frontend: npm run build outputs to build/.

• demo/realtime-txt2img/main.py — Entry point for the real-time text-to-image demo; initializes the StreamDiffusion pipeline and WebSocket server for live generation
• demo/realtime-img2img/main.py — Entry point for the real-time image-to-image demo; orchestrates pipeline initialization and connection management for streaming inference
• demo/realtime-img2img/frontend/src/lib/lcmLive.ts — Core WebSocket client library that handles real-time bidirectional communication with the backend diffusion pipeline
• demo/realtime-img2img/connection_manager.py — Manages concurrent WebSocket connections and coordinates inference requests across multiple clients in real-time
• demo/realtime-txt2img/frontend/src/app.tsx — Main React application component that binds UI controls to streaming pipeline parameters and renders real-time outputs
• demo/realtime-img2img/img2img.py — Implements the image-to-image inference logic using StreamDiffusion for latency-optimized conditional generation
• demo/realtime-img2img/frontend/src/lib/store.ts — Svelte store managing application state (pipeline parameters, generation settings, UI state) for reactive updates across components

Add a new UI control parameter to the img2img frontend

1. Define the parameter type in the TypeScript types file (demo/realtime-img2img/frontend/src/lib/types.ts)
2. Add the reactive store state for the new parameter (demo/realtime-img2img/frontend/src/lib/store.ts)
3. Create or update a component to bind the parameter (e.g., InputRange, Selectlist) (demo/realtime-img2img/frontend/src/lib/components/PipelineOptions.svelte)
4. Update the WebSocket message in lcmLive.ts to send the parameter to the server (demo/realtime-img2img/frontend/src/lib/lcmLive.ts)
5. Handle the parameter in the backend connection_manager.py to update the pipeline state (demo/realtime-img2img/connection_manager.py)

Add a new inference mode (e.g., depth-to-image)

1. Create a new Python script analogous to img2img.py with your inference logic (demo/realtime-img2img/img2img.py)
2. Update main.py to import and conditionally initialize your new pipeline based on configuration (demo/realtime-img2img/main.py)
3. Add configuration flags in config.py to select the inference mode (demo/realtime-img2img/config.py)
4. Update the frontend types to include new input/output formats (demo/realtime-img2img/frontend/src/lib/types.ts)
5. Add a UI input component (e.g., depth map uploader) in the components directory (demo/realtime-img2img/frontend/src/lib/components)

Add a new reactive parameter to the txt2img pipeline

1. Add the parameter to the config.py to define defaults and ranges (demo/realtime-txt2img/config.py)
2. Update main.py to read and apply the parameter when creating/updating the pipeline (demo/realtime-txt2img/main.py)
3. Add a UI input element (text input, slider) in the React component (demo/realtime-txt2img/frontend/src/app.tsx)
4. Update the WebSocket message handler to serialize and send the parameter to the backend (demo/realtime-txt2img/frontend/src/app.tsx)

• StreamDiffusion (custom pipeline) — Purpose-built for real-time inference; optimizes latency through token-merging, progressive denoising, and GPU memory efficiency vs. standard Diffusers
• WebSocket (async communication) — Enables low-latency bidirectional streaming of frames and parameters without HTTP request overhead; essential for 15-30 FPS interactive generation
• SvelteKit + Svelte stores — Reactive state management and SSR-capable frontend; Svelte's fine-grained reactivity minimizes re-renders for high-frequency parameter updates
• React + Vite (txt2img demo) — Lightweight fast-refresh dev experience; Vite provides near-instant HMR for rapid UI iteration
• FastAPI / async Python (connection_manager) — Non-blocking async handling of concurrent WebSocket connections; allows N clients to queue inference without thread contention
• Docker multi-stage builds — Isolates backend and frontend build stages; reduces final image size and deployment complexity

• Client-side media capture (browser) → server inference → client display

  • Why: Offloads computation to GPU server while keeping UI responsive; reduces end-device requirements
  • Consequence: Introduces network latency (WebSocket) and frame serialization overhead; best for local/LAN deployment

• Single pipeline per server instance (connection_manager queues requests)

  • Why: Minimizes GPU memory fragmentation and kernel launch overhead; simpler synchronization logic
  • Consequence: Throughput limited by single GPU; scales via horizontal clustering (multiple server instances) rather than vertical scaling

• Frame-by-frame img2img (vs. temporal coherence models)

  • Why: Simpler, stateless inference loop; compatible with arbitrary input sources
  • Consequence: May see flicker/temporal jitter without explicit coherence loss; user must supply smooth input stream

1. Frontend dev server (Vite) runs on port 5173 by default; backend must serve on different port with CORS headers configured. 2) GPU VRAM requirements not specified—benchmarks use RTX 4090 (24GB); smaller GPUs may require model quantization or batch-size reduction. 3) Hugging Face model downloads in demo/realtime-img2img/config.py likely require HF_TOKEN env var for gated models. 4) npm lockfile (frontend/package-lock.json) suggests pinned dependencies—npm ci preferred over npm install for reproduction. 5) TypeScript/Svelte toolchain requires Node.js 16+; no .nvmrc or engine field in package.json.

• Residual Classifier-Free Guidance (RCFG) — StreamDiffusion's key optimization reducing guidance computation overhead—understanding CFG residuals is essential to modifying the guidance mechanism in demo/realtime-img2img/config.py
• Stochastic Similarity Filter — Core GPU utilization optimization in StreamDiffusion; enables skipping redundant denoising steps by filtering similar latent states—understanding probability thresholds is needed to tune performance on different hardware
• KV-Cache Pre-computation — Memory optimization technique pre-computing key-value caches for transformer attention layers; critical to understanding why StreamDiffusion achieves 106 fps—directly impacts latency bottleneck in real-time image generation
• Stream Batching — Efficient batch operation pattern that processes multiple inference requests without blocking—core architectural pattern enabling real-time interactivity vs. traditional single-request pipelines
• WebSocket Real-Time Communication — Enables live image frame streaming from Python backend to SvelteKit frontend (connection_manager.py)—non-HTTP persistent connection required for <50ms latency feedback loops in interactive generation
• Latency-Throughput Trade-off — StreamDiffusion explicitly optimizes for latency over maximum throughput by reducing denoising steps (1-4 steps vs. 20+ standard); understanding this constraint is essential when tuning performance for your use case
• Quantization & Model Acceleration — StreamDiffusion mentions 'Model Acceleration Tools' for optimization—likely includes int8/fp16 precision casting or pruning; understanding model precision trade-offs crucial when deploying to consumer GPUs below RTX 4090

• huggingface/diffusers — Official diffusion model library that StreamDiffusion builds optimization techniques on top of; core dependency
• compvis/stable-diffusion — Original Stable Diffusion implementation; StreamDiffusion applies pipeline-level optimizations to compatible model architectures
• openai/improved-diffusion — Alternative diffusion improvements framework; represents competing approach to inference acceleration
• NVIDIA/TensorRT — GPU inference optimization toolkit that StreamDiffusion could integrate with for model acceleration tools mentioned in features
• ggerganov/llama.cpp — Demonstrates feasibility of CPU-compatible optimized inference for generative models; architectural pattern reference

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add frontend E2E tests for WebSocket connection and real-time streaming

The frontend code (demo/realtime-img2img/frontend/src/lib/lcmLive.ts and connection_manager.py backend) handles critical real-time WebSocket communication for image streaming, but there are no visible test files. E2E tests using Playwright/Vitest would validate the connection lifecycle, message handling, and image streaming pipeline—essential for a real-time generation project.

• [ ] Create demo/realtime-img2img/frontend/tests/ directory with Vitest/Playwright setup
• [ ] Add tests for lcmLive.ts WebSocket connection, reconnection logic, and message parsing
• [ ] Add tests for mediaStream.ts video capture and frame transmission
• [ ] Add tests for connection_manager.py backend socket handling and frame broadcasting
• [ ] Integrate E2E tests into GitHub Actions workflow

Add frontend component unit tests for interactive controls (PipelineOptions, InputRange, SeedInput)

The frontend has several interactive Svelte components (demo/realtime-img2img/frontend/src/lib/components/) that control pipeline parameters like CFG scale, seed, and options, but no unit tests exist. These components directly affect generation quality and reproducibility—testing them ensures UI state changes correctly propagate to the backend.

• [ ] Create demo/realtime-img2img/frontend/src/lib/components/tests/ directory
• [ ] Add Vitest/svelte-testing-library tests for InputRange.svelte (value binding, min/max constraints)
• [ ] Add tests for SeedInput.svelte (number validation, random seed generation)
• [ ] Add tests for PipelineOptions.svelte (option selection, state updates to store)
• [ ] Add tests for Checkbox.svelte and Selectlist.svelte toggle/selection logic

Add GitHub Actions workflow to test and build frontend on every PR

The repo has a release.yml workflow but no CI for frontend builds/linting on every commit. The frontend has strict linting rules (ESLint, Prettier, Tailwind), and builds could fail silently. A dedicated workflow would catch TypeScript errors, style issues, and build failures before merge.

• [ ] Create .github/workflows/frontend-ci.yml
• [ ] Add job to run 'npm ci && npm run check' for type checking in demo/realtime-img2img/frontend
• [ ] Add job to run 'npm run lint' (prettier + eslint) and fail on violations
• [ ] Add job to run 'npm run build' and verify production build succeeds
• [ ] Add job to run frontend unit tests (once tests are added from PR #2)
• [ ] Set workflow to trigger on pull_request and push to main branches

• Add unit tests for demo/realtime-img2img/connection_manager.py WebSocket message handling (mocking socket.io/FastAPI WebSocket)—currently no test suite visible
• Document exact backend entry point and environment variables (HF_TOKEN, CUDA_VISIBLE_DEVICES, PORT) in demo/realtime-img2img/README.md with runnable example
• Create missing error boundary in demo/realtime-img2img/frontend/src/lib/components/ImagePlayer.svelte to gracefully handle WebSocket disconnections instead of silent failures

Failed to generate security analysis.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.