Healthy — Healthy across all four use cases

• Last commit 2w ago
• MIT licensed
• CI configured
• Tests present
• ⚠ Solo or near-solo (1 contributor active in recent commits)
• ⚠ Scorecard: default branch unprotected (0/10)

_{Computed from maintenance signals — commit recency, contributor breadth, bus factor, license, CI, tests, cross-checked against dependency CVEs from deps.dev and OpenSSF Scorecard}

BitBudget is a reproducible benchmark for embedding compression that measures retrieval quality (nDCG@10, recall@10) against bytes-per-vector storage cost. It evaluates compression methods like binarization, product quantization, RaBitQ, and Matryoshka truncation across BEIR corpora, answering the core question: when you compress embeddings, what retrieval quality do you actually lose per byte stored? Monolithic CLI-driven package under src/bitbudget/: the core logic splits into embedders.py (model wrappers), indexes.py (HNSW/IVF-PQ/bittrie), methods.py (compression algorithms), metrics.py (nDCG/recall), and eval.py (orchestration). Board cards in board_cards/ are JSON result artifacts. The site/ directory holds a static leaderboard (data.json + index.html). A small C extension (_bittrie.c) compiled via _bittrie_build.py accelerates the bittrie index's query hot-path.

LLM-derived; treat as a starting point, not verified fact.

ML/RAG engineers and researchers building vector databases and retrieval systems who need to decide between compression methods and quantization strategies. They use BitBudget to benchmark their embedder+compression pipeline and make storage vs. recall trade-off decisions backed by reproducible empirical data rather than vendor claims.

LLM-derived; treat as a starting point, not verified fact.

Actively developed but research-stage. The repo has CI/CD workflows (.github/workflows/), a published leaderboard (LEADERBOARD.md), and test coverage (tests/ directory), but the commit frequency and issue volume are not visible from file metadata. The CITATION.cff and companion survey paper suggest academic-grade rigor, but lack of GitHub stars/issues data means it's likely still building adoption beyond the research community.

Low-to-moderate risk. The project is heavily Python-based with a small C extension (_bittrie.c for query performance), meaning C compilation could fail on some platforms. Dependencies on sentence-transformers and faiss are optional but recommended, creating a multi-version testing surface. Single-author repo (sjmoran) increases bus factor; no visible maintainer team or recent commit date metadata provided.

LLM-derived; treat as a starting point, not verified fact.

The repo is actively maintaining the compression leaderboard and expanding benchmark coverage. The card files (card_bge-base.json through card_openai-3-large.json) suggest ongoing evaluation of new embedder models. The build_board.sh and cards_to_data.py pipeline indicates active leaderboard publishing. The companion survey paper on projection and quantization (arxiv 2510.04127) suggests the research narrative is maturing.

LLM-derived; treat as a starting point, not verified fact.

git clone https://github.com/sjmoran/bitbudget.git
cd bitbudget
pip install -e '.[all]'  # installs with sentence-transformers + faiss
bitbudget methods           # list available compression methods
bitbudget run --embedder mxbai --corpus scifact  # embed + evaluate

Daily commands:

bitbudget run --embedder mxbai --corpus scifact     # full pipeline: embed + compress + eval
bitbudget bench-index --synthetic 100000 128         # benchmark index recall/QPS/bytes trade-offs
bitbudget leaderboard results/card_*.json            # render markdown leaderboard from result cards
bitbudget methods                                     # list all compression strategies
bitbudget indexes                                     # list all index types

• src/bitbudget/__init__.py — Package entry point exposing the main API for compression benchmarking.
• src/bitbudget/cli.py — Command-line interface orchestrating the benchmark workflow; start here for understanding how experiments run.
• src/bitbudget/methods.py — Core compression methods (binarization, quantization, product quantization) that form the heart of the benchmark.
• src/bitbudget/eval.py — Evaluation logic computing nDCG@10 and recall@10 metrics against retrieved neighbors.
• src/bitbudget/indexes.py — Index abstractions (in-memory and faiss-based) for storing and searching compressed embeddings.
• src/bitbudget/embedders.py — Embedder wrappers for different models (OpenAI, BGE, E5) that produce baseline vectors.
• README.md — Explains the benchmark philosophy, headline findings, and how to use BitBudget.

• Embedders (Transformers (HuggingFace), OpenAI API, sentence-transformers) — Fetch pretrained models and produce fixed-size float32 vectors from text.
  • Failure mode: API rate limit, OOM, model weight download failure → benchmark hangs or crashes.
• Methods (Compression) (NumPy, sklearn.decomposition, custom quantization logic) — Transform full vectors into lower-bit representations while preserving retrieval structure.
  • Failure mode: Numerical instability, incorrect bit-packing → silent correctness errors in downstream ranking.
• Indexes (FAISS, brute-force L2, bit-trie (optional)) — Store compressed vectors and perform efficient nearest-neighbor retrieval.
  • Failure mode: FAISS build failure, OOM during large-scale search → benchmark cannot complete.
• Evaluation (NumPy, NDCG calculation, ranking metrics) — Compute nDCG@10 and recall@10 by comparing retrieved neighbors against ground-truth rankings.
  • Failure mode: Ground-truth mismatch, metric computation bugs → incorrect leaderboard scores.
• CLI Orchestrator (argparse, Python logging) — Parse arguments, sequence component calls, and manage experiment state.
  • Failure mode: Incorrect argument validation, state corruption across runs → experiment reproducibility broken.
• Datasets (Hugging Face datasets library, JSON parsing) — Download and load BEIR corpora, queries, and ground-truth relevance judgments.
  • Failure mode: Network failure, corrupted cache, missing splits → benchmark cannot initialize.

• Embedder → Method — Full-precision float32 vectors (shape: [n, d]) passed to compression method.
• Method → Index — Compressed bit/int8 vectors stored in index for retrieval.
• Index → Eval — Top-k neighbor indices and distances returned for each query.
• Eval → CLI — Aggregated nDCG@10 and recall@10 metrics written to results JSON.
• CLI → Leaderboard — Results cards serialized to board_cards/ and ingested by cards_to_data.py for website rendering.

Add a new compression method

1. Define a class inheriting from Method in src/bitbudget/methods.py with compress() and decompress() implementations. (src/bitbudget/methods.py)
2. Register it in the method registry (likely a dict in methods.py or init.py). (src/bitbudget/methods.py)
3. Add a test case in tests/test_protocol.py to verify it satisfies the Method protocol. (tests/test_protocol.py)

Add a new embedder model

1. Create an Embedder subclass in src/bitbudget/embedders.py with an embed() method returning vectors. (src/bitbudget/embedders.py)
2. Register it in the embedder registry so CLI can discover it. (src/bitbudget/embedders.py)
3. Run benchmark and save results to board_cards/card_<model-name>.json. (board_cards)

Run a new benchmark experiment

1. Invoke src/bitbudget/cli.py with --embedder, --dataset, and --methods flags. (src/bitbudget/cli.py)
2. CLI orchestrates embedding → compression → indexing → evaluation via eval.py. (src/bitbudget/eval.py)
3. Results are saved and can be added to leaderboard via cards_to_data.py. (cards_to_data.py)

• Python + NumPy — Rapid prototyping and data manipulation for embedding vectors and experiments.
• FAISS — Efficient approximate nearest-neighbor search at scale for retrieval evaluation.
• C extension (_bittrie.c) — Optional low-level optimization for bit-trie indexing when binarized retrieval is critical.
• BEIR benchmark — Standardized, reproducible evaluation corpora covering diverse retrieval scenarios.

• In-memory indexing (no persistent storage layer)

  • Why: Simpler implementation and faster iteration for research; benchmark is reproducible from code alone.
  • Consequence: Benchmark runs must re-embed and re-index on each invocation; not suitable for long-lived deployments.

• Support multiple embedder APIs (OpenAI, BGE, E5) as separate classes

  • Why: Each model has different rate limits, authentication, and tensor formats.
  • Consequence: Added code complexity; no unified interface, but full control over each model's quirks.

• Evaluate on BEIR corpora only

  • Why: Single standardized benchmark ensures reproducibility and fair comparison.
  • Consequence: Results may not generalize to specialized domains (medical, legal, code retrieval).

• Report bytes-per-vector, not compression ratio

  • Why: Aligns with real deployment constraints (memory budgets, network bandwidth).
  • Consequence: Requires explicit byte-size accounting for each method; less intuitive than 'compression ratio %'.

• Does not perform online indexing or real-time vector updates.
• Does not provide distributed/multi-GPU benchmark execution.
• Does not handle authentication or multi-tenant isolation.
• Does not support custom datasets outside BEIR.
• Does not optimize for latency; focuses on retrieval quality vs. storage trade-off.

• Avg cyclomatic complexity: ~6 — Moderate cyclomatic complexity: orchestration logic in cli.py and eval.py, but individual methods are straightforward quantization routines. Multiple branching paths for embedder APIs and dataset loading.
• Largest file: src/bitbudget/methods.py (450 lines)
• Estimated quality issues: ~3 — Missing input validation on vector dimensions, sparse docstrings in methods.py, and inconsistent error handling between embedder subclasses. No type hints in several key functions.

• Tight coupling between Method and Index (Medium) — src/bitbudget/methods.py, src/bitbudget/indexes.py: Methods return compressed vectors without metadata; Index must infer storage format and bit-width, risking silent mismatches.
• No validation of embedder output shape (Medium) — src/bitbudget/embedders.py, src/bitbudget/eval.py: If embedder returns wrong dimensions, error only surfaces deep in FAISS index construction, making debugging hard.
• Global metric computation without per-query logging (Low) — src/bitbudget/eval.py: Aggregated metrics hide outlier queries that perform poorly under specific compression methods.

• src/bitbudget/embedders.py (embedder initialization) (I/O-bound, initialization latency) — Model weight download and GPU allocation can take 30–60 seconds per embedder before any benchmark runs.
• src/bitbudget/eval.py (query-by-query evaluation loop) (CPU-bound, algorithmic) — Nested loop over queries × methods × FAISS search calls; no vectorization or parallelization across methods.
• src/bitbudget/indexes.py (FAISS index construction) (Memory and CPU-bound) — Building FAISS indices for large BEIR corpora (>100k docs) can dominate total runtime, especially for exact search.

1. Optional dependencies matter: bitbudget run works with just NumPy, but --embedder mxbai silently requires sentence-transformers (torch+transformers) to be installed; bench-index with HNSW/IVF-PQ requires faiss. Errors only surface at runtime. 2. C extension compilation: _bittrie.c must compile on your platform; if pip install fails with C compiler errors, you lose the fast bittrie query path. 3. BEIR corpus auto-download: datasets.py downloads multi-GB corpora on first run to ~/.cache/ or $XDG_CACHE_HOME; no disk check or resume logic. 4. Float precision in metrics: numpy broadcasting in metrics.py can silently lose precision on large matrices; nDCG@10 is hardcoded, no CLI override. 5. Reproducibility risk: embeddings vary by sentence-transformers/transformers version; LEADERBOARD.md results may not match if dependencies drift.

• Product Quantization (PQ) — One of the core compression methods in BitBudget (methods.py); decomposes high-d vectors into subspaces and quantizes each independently, balancing compression ratio and recall loss
• Binary Embeddings / Hamming Distance — BitBudget's headline finding is that 1-bit codes (binarization) with re-ranking beat full-precision embeddings per byte; requires understanding bit-level distance metrics and re-ranking trade-offs
• nDCG@k (Normalized Discounted Cumulative Gain) — The primary ranking metric BitBudget uses to measure retrieval quality; essential for interpreting the leaderboard and understanding recall@byte trade-offs
• Vector Index Structures (HNSW, IVF-PQ, KD-Trees) — BitBudget's bench-index command evaluates compression methods on different index types; understanding index recall, QPS, and bytes-per-vector overhead is core to the 'organisation axis' benchmark
• Approximate Nearest Neighbor (ANN) Search — BitBudget evaluates compression in the context of ANN retrieval, not exact search; recall@k and QPS metrics depend on index implementation and are different from offline compression-only benchmarks
• Matryoshka Embeddings — A learned projection method that truncates embeddings to lower dimensions; BitBudget includes this as a baseline compression strategy and shows it underperforms quantization at fixed byte budgets
• Re-ranking (Two-Stage Retrieval) — BitBudget's binary+rerank result uses a two-stage pipeline (cheap retrieval on 1-bit codes, then re-rank with full embeddings); critical for understanding how quantized indexes can be lossless in practice

• facebookresearch/faiss — Provides the approximate nearest neighbor indexes (HNSW, IVF-PQ) that BitBudget uses for the 'organisation axis' benchmarking (bench-index command)
• UKPLab/sentence-transformers — Embedding model library used by BitBudget's embedders.py to generate the vectors being compressed; essential dependency for reproducible benchmarks
• beir-cellar/beir — The Information Retrieval benchmark suite; BitBudget wraps BEIR corpora (scifact, nfcorpus, etc.) for evaluation and computes nDCG@10 using BEIR's metrics
• microsoft/unilm — Includes Matryoshka Embedding research; BitBudget includes Matryoshka truncation as one compression baseline and compares it against other methods
• ingestion-edge/rabitq — The RaBitQ quantization method is one of the key compression baselines evaluated in BitBudget's methods.py and featured in LEADERBOARD.md results

• Add benchmarking for smaller embedders (e.g., MiniLM, ONNX-quantized variants) by extending src/bitbudget/embedders.py with lazy-load options and profiling memory/latency overhead per embedder, then add a new bench-embedder command to cli.py.
• Write integration tests for the evaluation pipeline (src/bitbudget/eval.py) that mock BEIR corpus downloads and verify nDCG/recall output stability across Python versions; currently tests/test_protocol.py only validates compression interface, not end-to-end retrieval correctness.
• Implement a new compression method (e.g., scalar quantization with learned thresholds, or learned rotations like ITQ) in src/bitbudget/methods.py following the existing protocol, add CLI registration in cli.py, benchmark it on 2-3 corpora, and submit a board_card/ JSON to the leaderboard.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Computed from the actual import graph (no LLM). Read in this order to learn the codebase from the foundation up — each step builds on the previous ones.

1. src/bitbudget/methods.py — Foundation: doesn't import anything internally and is imported by 3 other files. Read first to learn the vocabulary.
2. src/bitbudget/embedders.py — Foundation: imported by 2, no internal dependencies of its own.
3. src/bitbudget/eval.py — Built on the foundation; imported by 2 downstream files.
4. src/bitbudget/bittrie.py — Built on the foundation; imported by 2 downstream files.
5. src/bitbudget/indexes.py — Layer 2 — application-level code that wires the lower layers together.
6. src/bitbudget/__init__.py — Layer 3 — application-level code that wires the lower layers together.
7. src/bitbudget/cli.py — Layer 4 — application-level code that wires the lower layers together.

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.