WAIT — Slowing — last commit 6mo ago

• Last commit 6mo ago
• ⚠ Slowing — last commit 6mo ago
• ⚠ Solo or near-solo (1 contributor active in recent commits)
• ⚠ No license — legally unclear to depend on
• ⚠ No CI workflows detected
• ⚠ No test directory detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

LLM Council is a web application that orchestrates multiple LLMs (via OpenRouter) to collaboratively answer questions through a three-stage deliberation process: Stage 1 collects independent responses from all council members, Stage 2 has each LLM anonymously review and rank the others' outputs, and Stage 3 has a Chairman LLM synthesize the reviews into a final answer. It's a ChatGPT-like UI that exposes the full reasoning pipeline rather than hiding it behind a single model. Monorepo structure with backend/ (FastAPI server: config.py defines council members, council.py orchestrates the three-stage flow, openrouter.py handles API calls, storage.py manages conversation history) and frontend/ (React + Vite with three-stage UI components: Stage1.jsx shows individual responses in tabs, Stage2.jsx displays review rankings, Stage3.jsx shows the final answer). The main.py at root appears to be a legacy entry point.

Researchers and power users evaluating multiple LLM providers (GPT, Gemini, Claude, Grok) who want to see side-by-side responses and understand how different models critique each other's reasoning. Also useful for anyone wanting to understand LLM model bias and relative strengths on complex questions.

This is experimental/prototype-grade software. The README explicitly states it was '99% vibe coded as a fun Saturday hack' with no intention of ongoing support or maintenance. No test suite present, no CI/CD setup visible, and the code is intentionally ephemeral. Users are expected to modify it themselves rather than expect stability.

High risk for production use: no automated tests, single-maintainer (Andrej Karpathy), OpenRouter API dependency with no fallback, requires valid API credentials and paid credits to function at all, and the codebase is explicitly unsupported. The project also depends on frontier LLM availability (GPT-5.1, Gemini 3.0 Pro) which may not exist or be accessible long-term.

This is a static snapshot project with no ongoing active development. The repository exists in its current state as a reference implementation for the LLM Council concept. No active PRs, issues, or commits are mentioned in the provided data, suggesting this is archived for reference rather than actively maintained.

git clone https://github.com/karpathy/llm-council.git
cd llm-council
uv sync
cd frontend && npm install && cd ..
echo 'OPENROUTER_API_KEY=sk-or-v1-...' > .env
./start.sh

Then open http://localhost:5173 in your browser.

Daily commands: Option A: ./start.sh (runs both backend and frontend). Option B: Terminal 1: uv run python -m backend.main (starts FastAPI on port 8000 implied). Terminal 2: cd frontend && npm run dev (starts Vite dev server on 5173). Then navigate to http://localhost:5173.

• backend/council.py — Core orchestration logic that manages the three-stage LLM council workflow (first opinions, review, final response); every backend change flows through here.
• backend/main.py — FastAPI application entry point that defines all HTTP endpoints; critical for understanding request routing and WebSocket integration.
• backend/openrouter.py — Abstraction layer for OpenRouter API calls; all LLM interactions depend on this module's structure and error handling.
• frontend/src/App.jsx — Root React component orchestrating the multi-stage UI and state management; prerequisite for understanding frontend architecture.
• frontend/src/components/ChatInterface.jsx — Primary chat component handling user input, message rendering, and API communication; where most user interactions originate.
• backend/config.py — Configuration and environment management for backend; defines API keys, model selections, and system prompts that drive council behavior.
• frontend/src/api.js — Frontend HTTP client wrapping backend API calls; essential for understanding client-server contract and error boundaries.

Add a new LLM model to the council

1. Add model identifier and endpoint to the model roster in backend/config.py (backend/config.py)
2. Ensure backend/openrouter.py can dispatch to the new model (may require no changes if OpenRouter handles it) (backend/openrouter.py)
3. Update system prompts in backend/config.py if model requires special instructions (backend/config.py)
4. Test Stage 1, 2, and 3 flows via ChatInterface to verify council orchestration includes new model (backend/council.py)

Add a new UI component for council visualization

1. Create new JSX file in frontend/src/components/ (e.g., StageX.jsx) following existing Stage1/2/3 patterns (frontend/src/components/ChatInterface.jsx)
2. Import and integrate the component into App.jsx state logic for conditional rendering (frontend/src/App.jsx)
3. Create corresponding CSS file in frontend/src/components/ with naming convention (e.g., StageX.css) (frontend/src/components/Stage1.css)
4. Update frontend/src/api.js if new component requires additional API endpoints (frontend/src/api.js)

Add a new backend API endpoint for council customization

1. Define the route in backend/main.py using FastAPI @app.post() or @app.get() decorators (backend/main.py)
2. Implement business logic in backend/council.py if orchestration is needed, or backend/config.py for settings (backend/council.py)
3. Add corresponding client function in frontend/src/api.js to call the new endpoint (frontend/src/api.js)
4. Call the new API function from frontend/src/App.jsx or specific components as needed (frontend/src/App.jsx)

Modify the three-stage workflow logic

1. Review and edit the Stage 1, 2, 3 orchestration methods in backend/council.py (backend/council.py)
2. Update system prompts or ranking instructions in backend/config.py if changing evaluation criteria (backend/config.py)
3. Adjust UI rendering in Stage1.jsx, Stage2.jsx, Stage3.jsx to reflect workflow changes (frontend/src/components/Stage1.jsx)
4. Test end-to-end via ChatInterface.jsx with sample queries (frontend/src/components/ChatInterface.jsx)

• FastAPI (Backend) — Provides async-first HTTP server suitable for handling multiple concurrent LLM queries and WebSocket support for real-time UI updates
• React with Vite (Frontend) — Enables dynamic multi-stage UI with tab switching and reactive re-renders as council stages complete; Vite provides fast HMR during development
• OpenRouter API — Abstracts multiple LLM providers (OpenAI, Google, Anthropic, xAI) behind a single unified API, allowing council to aggregate diverse model responses
• react-markdown — Renders LLM responses (which are often markdown) with proper formatting for readability in the chat interface

• Anonymize model identities in Stage 2 reviews

  • Why: Prevents LLM favoritism or bias when evaluating peer responses
  • Consequence: Users cannot directly see which model ranked whom, adding interpretive overhead but improving objectivity

• Three-stage sequential workflow instead of single LLM call

  • Why: Enables ensemble reasoning where models critique each other, improving response quality through iterative refinement
  • Consequence: Latency increases significantly (~20–40s total vs 3–5s for single LLM); requires careful prompt engineering for each stage

• Tab-based UI in Stage 1 to inspect individual responses

  • Why: Gives users visibility into council deliberation and confidence in final answer
  • Consequence: UX complexity increases; requires state management for tab switching and scrolling context

• Does not persist chat history across sessions (no database like PostgreSQL)
• Does not authenticate users; runs as a local web app without access control
• Does not support streaming responses from individual LLMs (likely waits for full completion)
• Does not handle rate limiting or quota management for OpenRouter API internally

Required: OPENROUTER_API_KEY environment variable must be set in .env or the backend will fail silently or with cryptic errors. Blocking: You need active OpenRouter credits; free tier may not have access to the latest models (GPT-5.1, Gemini 3.0 Pro) listed in default config.py. Model availability: Default COUNCIL_MODELS in config.py may reference models that don't exist or are paywalled; you must verify all model IDs at openrouter.ai/models before deploying. No persistence by default: backend/storage.py likely isn't fully implemented; conversations may not persist across restarts. Async quirks: Backend uses async/await heavily with httpx; blocking calls anywhere will stall all concurrent requests.

• LLM Model Anonymization — Stage 2 of this repo deliberately hides LLM identities from each other so models can't be biased by brand names or reputations; understanding how to implement this in prompts is key to fair cross-evaluation
• Asynchronous HTTP Batching — backend/openrouter.py needs to call multiple LLMs concurrently in Stage 1 to avoid sequential delay; httpx async context is how you avoid blocking waits
• Prompt Engineering for Peer Review — Stage 2 requires carefully crafted prompts that ask LLMs to rank anonymized outputs objectively; the phrasing directly impacts the quality of cross-critique
• OpenRouter API Routing — OpenRouter abstracts away model-specific API differences (OpenAI vs Anthropic vs Google vs xAI) behind a unified interface; this is why the code doesn't have separate client instantiation for each provider
• Server-Sent Events (SSE) for Streaming — The three-stage UI likely streams LLM responses incrementally rather than waiting for full completion; SSE or WebSocket patterns would enable showing model responses as they arrive
• Component-Based State Management in React — The frontend uses local component state (no global Redux); tracking which stage is active, which model's tab is visible, and streaming responses requires careful prop threading
• FastAPI Dependency Injection — backend/main.py uses FastAPI's dependency system to handle request validation and potentially auth; this pattern keeps endpoint handlers clean and testable

• lm-sys/FastChat — Battle-tested multi-model evaluation UI with similar tabbed comparison view; good reference for handling many LLM responses simultaneously
• LASR-LSE/Multi-LLM-Judge — Research repo that uses multiple LLMs to judge each other's outputs, similar core concept but with focus on evaluation metrics rather than user-facing interface
• open-telemetry/opentelemetry-python — If you extend this repo for production, you'll likely want structured tracing of the three-stage pipeline; OpenTelemetry provides the instrumentation
• langchain-ai/langchain — LangChain provides higher-level abstractions for LLM orchestration and could replace the manual httpx calls in backend/openrouter.py
• anthropics/anthropic-sdk-python — Direct Anthropic SDK; if you want to use Claude without OpenRouter routing, this is the native client (though OpenRouter is the current approach)

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add error handling and retry logic for OpenRouter API calls in backend/openrouter.py

The backend/openrouter.py module likely makes multiple concurrent API calls to different LLM providers during the 3-stage council process. Without explicit error handling, a single provider timeout or rate limit could fail the entire request. This PR would add retry logic with exponential backoff, fallback handling, and graceful degradation (e.g., if one LLM fails, continue with others). This is critical for reliability in a production council system.

• [ ] Add try-catch blocks around API calls in backend/openrouter.py with specific exception types (timeouts, rate limits, invalid responses)
• [ ] Implement exponential backoff retry logic with configurable max retries in backend/config.py
• [ ] Add logging to backend/main.py to surface which LLMs succeeded/failed and why
• [ ] Create a basic integration test that mocks OpenRouter failures and verifies council still produces output

Implement storage persistence for chat history in backend/storage.py

The repo has a backend/storage.py file but no indication of whether multi-turn conversations or chat history is persisted. Currently, users likely lose their council discussions on page refresh. Implementing proper storage (JSON file, SQLite, or similar) with session management would allow users to resume conversations and review past council decisions.

• [ ] Design and implement persistence layer in backend/storage.py (e.g., SQLite schema with tables for sessions, queries, and council responses)
• [ ] Add endpoints to backend/main.py for fetching historical queries and their multi-stage responses
• [ ] Update frontend/src/api.js to call new history endpoints and display past council sessions in frontend/src/components/Sidebar.jsx
• [ ] Add configuration options in backend/config.py for storage backend selection and retention policies

Add unit and integration tests for the 3-stage council workflow in backend/council.py

The core logic of the 3-stage process (initial responses, review/ranking, chairman synthesis) lives in backend/council.py, but there are no tests listed in the repo structure. Without tests, refactoring the council logic is risky and regressions are hard to catch. New contributors can add comprehensive tests that verify stage transitions, response anonymization, and ranking aggregation.

• [ ] Create tests/test_council.py with fixtures that mock OpenRouter API responses
• [ ] Add tests for Stage 1 (verify all LLMs receive query and responses are collected correctly)
• [ ] Add tests for Stage 2 (verify anonymization works, each LLM sees others' responses, ranking is captured)
• [ ] Add tests for Stage 3 (verify chairman LLM receives aggregated rankings and produces final output)
• [ ] Add an end-to-end test that runs a full 3-stage council workflow with mock data

• Add a conversation history feature: backend/storage.py is stubbed out but not integrated; implement SQLite-based persistence and wire it through backend/main.py endpoints so users can see past council discussions
• Create a configuration UI: currently COUNCIL_MODELS and CHAIRMAN_MODEL are hardcoded in backend/config.py; build a frontend settings panel that lets users swap models without restarting, validating against OpenRouter's /models endpoint
• Add LLM response time tracking: modify backend/council.py to measure and return latency for each stage and each model, then display it in the frontend components (Stage1.jsx, Stage2.jsx, Stage3.jsx) so users understand which models are slowest

Failed to generate security analysis.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/karpathy/llm-council shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live karpathy/llm-council repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/karpathy/llm-council.

What it runs against: a local clone of karpathy/llm-council — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in karpathy/llm-council | Confirms the artifact applies here, not a fork | | 2 | Default branch master exists | Catches branch renames | | 3 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 4 | Last commit ≤ 197 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of karpathy/llm-council. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/karpathy/llm-council.git
#   cd llm-council
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of karpathy/llm-council and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "karpathy/llm-council(\\.git)?\\b" \\
  && ok "origin remote is karpathy/llm-council" \\
  || miss "origin remote is not karpathy/llm-council (artifact may be from a fork)"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "backend/council.py" \\
  && ok "backend/council.py" \\
  || miss "missing critical file: backend/council.py"
test -f "backend/main.py" \\
  && ok "backend/main.py" \\
  || miss "missing critical file: backend/main.py"
test -f "backend/openrouter.py" \\
  && ok "backend/openrouter.py" \\
  || miss "missing critical file: backend/openrouter.py"
test -f "frontend/src/App.jsx" \\
  && ok "frontend/src/App.jsx" \\
  || miss "missing critical file: frontend/src/App.jsx"
test -f "frontend/src/components/ChatInterface.jsx" \\
  && ok "frontend/src/components/ChatInterface.jsx" \\
  || miss "missing critical file: frontend/src/components/ChatInterface.jsx"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 197 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~167d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/karpathy/llm-council"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.