GO — Healthy across the board

• Last commit 2mo ago
• 20 active contributors
• Distributed ownership (top contributor 20% of recent commits)
• MIT licensed
• Tests present
• ⚠ No CI workflows detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

bitnet.cpp is the official inference engine for 1-bit quantized LLMs (BitNet b1.58 models), delivering optimized CPU and GPU kernels that achieve 1.37–6.17x speedups and 55–82% energy reduction. It enables running a 100B-parameter 1.58-bit model on a single CPU at 5–7 tokens/second without quantization loss, making edge inference of billion-parameter models practical. Dual-path architecture: src/ contains CPU inference (C++ core with optimized GEMM kernels), gpu/ contains CUDA/GPU path with custom kernels (gpu/bitnet_kernels/bitnet_kernels.cu), and preset_kernels/ holds pre-compiled lookup-table (LUT) kernels for specific model/hardware combinations. Entry points are run_inference.py (CLI) and run_inference_server.py (Flask server); Python bindings wrap the C++ engine. Build via CMake; model weights are loaded in .safetensors format.

ML engineers and systems developers deploying large language models on resource-constrained edge devices (ARM/x86 CPUs, GPUs); researchers studying 1-bit model inference; teams wanting to run billion-parameter models locally without cloud infrastructure or high power consumption.

Production-ready and actively developed: released October 2024 (v1.0), with major optimizations added January 2025 (parallel kernels, embedding quantization) and GPU support (May 2025). The codebase is substantial (427K Python, 391K C++), includes preset kernel configurations for Llama3-8B and BitNet b1.58 models, and ships with a web demo. However, this is specialized inference software with evolving optimization techniques, so expect incremental refinements rather than breaking changes.

Low dependency risk: core runtime uses only fire, sentencepiece, torch (with pinned xformers≥0.0.22), tiktoken, and minimal Python stdlib. Main risk is hardware specificity—preset kernels are model/architecture-locked (see /preset_kernels/ structure by model variant), so adding support for new hardware or models requires careful kernel tuning. GPU path is newer (2025) than CPU (2024), so GPU code may have fewer field-tested configurations. Single-maintainer (Microsoft Research) reduces breakage risk but increases future support uncertainty.

Most recent work (Jan 2025): parallel kernel implementations with tiling configurations (see src/README.md optimization guide) adding 1.15–2.1x additional speedup. Prior work included GPU kernel release (May 2025) and BitNet b1.58 2B model release (April 2025). Optimization focus is expanding preset kernel coverage and reducing activation quantization overhead via embedding quantization.

git clone https://github.com/microsoft/BitNet.git
cd BitNet
pip install -r requirements.txt
# For GPU support: pip install -r gpu/requirements.txt
python run_inference.py --model_path <path_to_bitnet_model.safetensors> --prompt "Hello"

For GPU inference, see gpu/README.md. Preset kernels are auto-selected based on detected hardware; custom kernels require gpu/bitnet_kernels/compile.sh.

Daily commands: CLI: python run_inference.py --model_path model.safetensors --prompt "text" (uses CPU by default). Server: python run_inference_server.py --port 8000, then POST to /generate. GPU: cd gpu && python generate.py --model_path model.safetensors (auto-detects CUDA). Tests: cd gpu && python test.py. See gpu/sample_utils.py for tokenizer setup.

• run_inference.py — Primary entry point for CPU inference that orchestrates model loading and generation on the BitNet framework.
• src/ggml-bitnet-lut.cpp — Core CPU kernel implementation for 1-bit LLM inference using lookup tables, the performance-critical path for all BitNet operations.
• gpu/model.py — GPU model abstraction that bridges the gap between PyTorch/HuggingFace models and BitNet GPU inference kernels.
• gpu/bitnet_kernels/bitnet_kernels.cu — CUDA kernel implementations for GPU-accelerated 1-bit matrix operations, essential for GPU inference performance.
• include/ggml-bitnet.h — Core header defining the ggml BitNet API surface and data structures that both CPU and GPU implementations depend on.
• utils/convert-hf-to-gguf-bitnet.py — Model conversion utility that transforms HuggingFace BitNet models into GGUF format required for inference.
• CMakeLists.txt — Build configuration that ties together CPU/GPU kernels and defines compilation flags critical for performance.

Add Support for a New 1-Bit Model Variant

1. Convert the model from HuggingFace/checkpoint format to GGUF using the conversion pipeline (utils/convert-hf-to-gguf-bitnet.py)
2. Quantize embeddings if needed for the new model dimensions (utils/quantize_embeddings.py)
3. Run GEMM kernel tuning on target hardware to generate optimized config (utils/tune_gemm_config.py)
4. Generate model-specific kernel code for tensor layouts TL1 and TL2 (utils/codegen_tl1.py)
5. Add preset kernel headers under new model directory matching BitNet 3B structure (preset_kernels/bitnet_b1_58-3B/bitnet-lut-kernels-tl1.h)
6. Test end-to-end inference with benchmark suite (utils/e2e_benchmark.py)

Add GPU Support for a New CUDA Architecture

1. Extend CUDA kernel implementations to target new architecture (gpu/bitnet_kernels/bitnet_kernels.cu)
2. Update kernel header signatures and config structures if needed (gpu/bitnet_kernels/bitnet_kernels.h)
3. Modify GPU model loading to detect and configure for new hardware (gpu/model.py)
4. Update CUDA compilation script with new architecture flags (gpu/bitnet_kernels/compile.sh)
5. Add GPU tests for the new architecture (gpu/test.py)

Optimize Inference for a Specific CPU (ARM/x86)

1. Profile current kernel performance to identify bottlenecks (utils/e2e_benchmark.py)
2. Run automatic kernel configuration tuning for target CPU (utils/tune_gemm_config.py)
3. Generate CPU-specific LUT kernel code using codegen (utils/codegen_tl1.py)
4. Update GEMM configuration headers to reflect tuned parameters (include/gemm-config.h)
5. Re-compile kernels with CMake and validate performance improvement (CMakeLists.txt)

Expose Inference via HTTP API

1. Start with the existing inference server template (run_inference_server.py)
2. Add Flask routes for generation endpoints matching desired API contract (run_inference_server.py)
3. Ensure tokenizer and model loading follow the inference pipeline (gpu/tokenizer.py)
4. Test full server functionality with sample requests (gpu/generate.py)

• C++ with GGML backend — Enables efficient CPU inference with minimal dependencies and strong numerical performance on quantized 1-bit operations
• CUDA kernels — Provides GPU acceleration for 1-bit matrix operations, critical for achieving 5x+ speedups on GPU hardware
• Python bindings + PyTorch — Bridges HuggingFace/PyTorch model ecosystem to compiled inference kernels, easing model development and conversion
• Lookup Table (LUT) GEMM kernels — Avoids expensive floating-point arithmetic in 1-bit inference by pre-computing and indexing quantized dot products
• Preset kernel headers — Trades storage for startup latency; pre-compiled kernels eliminate per-run code generation overhead

• undefined
  • Why: undefined
  • Consequence: undefined

1. Preset kernel binding: LUT kernels in preset_kernels/ are model+tiling-specific; running inference on unsupported hardware falls back to slower scalar operations. Check kernel_config_tl1.ini / tl2.ini matches your model and tile strategy. 2. Tokenizer mismatch: gpu/tokenizer.model is SentencePiece-encoded; BitNet models expect this exact tokenizer. Custom models need matching tokenizer registration. 3. CUDA version lock: GPU kernels compiled against specific CUDA versions; mismatch causes silent kernel fallback to CPU. 4. Weight format: .safetensors required; .pt checkpoints need conversion via convert_checkpoint.py which expects specific layer naming (e.g., layers.*.wqkv). 5. Environment: C++ build requires CMake ≥3.15 and compiler supporting C++17; GPU path requires CUDA toolkit + compatible GPU (tested on A100, RTX 40 series).

• 1-bit quantization / Ternary quantization — BitNet b1.58 weights are ternary (−1, 0, +1); understanding this is fundamental to why lookup-table kernels work and why energy savings are so high
• Lookup-table (LUT) based GEMM — Core optimization: instead of multiply-accumulate on ternary values, BitNet uses pre-computed tables mapping bit patterns to sums; enables massive speedup on both CPUs and GPUs
• Activation quantization / a4.8 — BitNet a4.8 uses 4-bit activations with 8-bit scales; this is orthogonal to weight quantization and enables further optimization in preset kernels
• Memory-mapped inference / safetensors — Models are loaded via safetensors format with lazy loading; critical for fitting 100B models on constrained devices without loading entire checkpoint into RAM
• SIMD / vectorization (ARM NEON, AVX2/AVX512) — CPU kernels use architecture-specific vectorized operations to exploit parallelism; understanding SIMD bottlenecks is essential for tuning new hardware
• Kernel tiling / loop optimization (TL1, TL2) — Preset kernels include tiling variants (TL1, TL2 in config files) to optimize cache locality; different tile sizes suit different CPUs, which is why multiple preset kernels exist per model
• Inference quantization (post-training vs. per-layer) — BitNet b1.58 is post-training quantized without retraining loss; understanding losslessness is key to why this achieves standard model accuracy despite extreme quantization

• ggerganov/llama.cpp — Predecessor CPU inference framework for quantized LLMs; BitNet.cpp borrows the ggml-bitnet quantization abstraction but optimizes specifically for 1-bit ternary arithmetic
• huggingface/transformers — Upstream model definitions and tokenizers; BitNet models are on Hugging Face Hub and use transformers for model conversion
• microsoft/Olive — Microsoft's model optimization toolkit; complementary framework for quantizing and optimizing models before BitNet.cpp inference
• vLLM-project/vLLM — High-throughput LLM inference serving; alternative approach for GPU inference that BitNet users might compare against for batched/server workloads

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive GPU kernel testing suite (gpu/test.py enhancement)

The gpu/test.py file exists but lacks visible test coverage for the critical bitnet_kernels.cu CUDA kernels. With GPU inference being a major feature, a structured test suite validating kernel correctness across different batch sizes, quantization levels, and hardware would prevent regressions and improve contributor confidence.

• [ ] Expand gpu/test.py with parametrized tests for bitnet_kernels.cu (LUT-based and MAD-based operations)
• [ ] Add numerical validation tests comparing GPU kernel outputs against CPU reference implementations (src/ggml-bitnet-lut.cpp and src/ggml-bitnet-mad.cpp)
• [ ] Include benchmark assertions to catch performance regressions in gpu/bitnet_kernels/bitnet_kernels.cu
• [ ] Document test execution in gpu/README.md

Implement preset kernel code generation workflow (utils/codegen_tl1.py & utils/codegen_tl2.py automation)

The preset_kernels/ directory contains pre-generated optimized kernels for specific models (bitnet_b1_58-3B, bitnet_b1_58-large, Llama3-8B-1.58), but there's no documented or automated process for generating new kernels. Creating a CI workflow or utility script would enable the community to optimize for new models and hardware targets.

• [ ] Create utils/generate_preset_kernels.py that orchestrates utils/codegen_tl1.py and utils/codegen_tl2.py with model config inputs
• [ ] Add validation that generated preset_kernels//bitnet-lut-kernels-tl.h files compile correctly via gpu/bitnet_kernels/compile.sh
• [ ] Document the kernel generation workflow in docs/codegen.md with end-to-end examples
• [ ] Add GitHub Actions workflow (.github/workflows/kernel-gen.yml) triggered on model config changes

Add E2E integration tests across inference paths (run_inference.py, gpu/generate.py, run_inference_server.py)

Three separate inference entry points exist (CPU via run_inference.py, GPU via gpu/generate.py, and server via run_inference_server.py) but there's no test validating they produce consistent outputs or covering the conversion pipeline (utils/convert*.py). This is critical for a production inference framework.

• [ ] Create tests/test_e2e_inference.py validating that utils/convert-hf-to-gguf-bitnet.py and utils/convert-ms-to-gguf-bitnet.py produce valid GGUF models
• [ ] Add parametrized tests running the same prompt through run_inference.py (CPU) and gpu/generate.py (GPU) and asserting token sequence consistency
• [ ] Test run_inference_server.py endpoint responses against known model outputs using a small preset model
• [ ] Add this test suite to a GitHub Actions workflow triggered on src/, gpu/, and utils/convert*.py changes

• Add CPU benchmark suite for src/ kernels comparable to gpu/test.py. Currently only GPU has systematic benchmarking; CPU kernels lack instrumentation for tiling strategy comparison.
• Document preset kernel generation pipeline: preset_kernels/ contains only final .h headers. Add script to regenerate kernels from a model + hardware profile (requires reversing LUT kernel codegen logic from gpu/pack_weight.py).
• Extend model support documentation: gpu/convert_safetensors.py assumes BitNet layer naming; add layer mapping guide for importing Llama or other quantized models, with test cases for Llama3-8B conversion (mentioned in preset_kernels/ but undocumented).

The BitNet inference framework has a moderate security posture. The primary concerns are: (1) unsafe model deserialization patterns common in ML frameworks, (2) unpinned dependencies increasing supply chain risk, (3) potential code injection in dynamic kernel generation, and (4) inadequate input validation in the inference server. The codebase demonstrates good security awareness through the presence of SECURITY.md and responsible disclosure policy. Recommendations include implementing strict input validation, pinning dependencies with version scanning, using safe deserialization methods, and adding runtime security controls to the Flask server. The CUDA kernel code warrants focused security review but is less critical than Python-level vulnerabilities.

• Medium · Potential Unsafe Deserialization in Model Loading — gpu/convert_checkpoint.py, gpu/convert_safetensors.py, gpu/model.py, run_inference.py, utils/convert-hf-to-gguf-bitnet.py. The codebase includes model conversion and loading utilities (convert_checkpoint.py, convert_safetensors.py, convert-hf-to-gguf-bitnet.py) that use PyTorch and potentially pickle-based formats. PyTorch's torch.load() can execute arbitrary code if untrusted model files are loaded. The run_inference.py and gpu/model.py files appear to load models from external sources without explicit validation. Fix: Use torch.load() with weights_only=True parameter (PyTorch 2.0+), validate model file checksums/signatures before loading, or restrict model sources to trusted registries only.
• Medium · Unvalidated External Dependencies with Known Vulnerabilities — requirements.txt, run_inference_server.py. The requirements.txt includes multiple dependencies (torch, transformers, xformers, flask) without pinned versions, making the project vulnerable to supply chain attacks and known CVEs. Flask is used in run_inference_server.py without visible security headers or input validation mechanisms. Fix: Pin all dependencies to specific secure versions, implement regular dependency scanning using tools like pip-audit or Safety, add security headers to Flask app (using flask-talisman), and enable CORS restrictions.
• Medium · Potential Code Injection in Dynamic Code Generation — utils/codegen_tl1.py, utils/codegen_tl2.py, preset_kernels/*/kernel_config_*.ini. The codebase includes code generation utilities (utils/codegen_tl1.py, utils/codegen_tl2.py) that generate kernel code dynamically. If these tools process user-supplied configuration files (kernel_config_*.ini) without proper validation, they could be vulnerable to code injection attacks that generate malicious kernels. Fix: Implement strict input validation for all configuration files, use allowlist-based parameter validation, avoid eval() or exec() patterns, and sandboxe code generation processes.
• Low · Missing Input Validation in Flask Server — run_inference_server.py. run_inference_server.py likely exposes inference endpoints via Flask without visible request validation. This could allow various attacks including resource exhaustion, path traversal, or prompt injection attacks against the model. Fix: Add comprehensive input validation for all endpoints, implement rate limiting, add request size limits, validate all user inputs against expected formats, and implement authentication/authorization.
• Low · Tokenizer File Security — gpu/tokenizer.model. The tokenizer.model file in gpu/ directory is a binary model artifact. If this file is not verified or is sourced from untrusted locations, it could be manipulated to cause unexpected behavior or information disclosure. Fix: Implement hash verification (SHA-256) for tokenizer and model files, use signed/authenticated artifact sources, document the expected checksums in version control, and verify during runtime.
• Low · CUDA Kernel Security — gpu/bitnet_kernels/bitnet_kernels.cu. The CUDA kernels in gpu/bitnet_kernels/bitnet_kernels.cu are low-level code that could contain memory safety issues (buffer overflows, out-of-bounds access). While compiled code is generally safer than interpreted code, GPU kernel bugs can cause system crashes or information disclosure. Fix: Perform security-focused code review of CUDA kernels, use bounds checking in memory operations, test with memory sanitizers, and keep CUDA toolkit updated to latest security patches.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/microsoft/BitNet shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live microsoft/BitNet repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/microsoft/BitNet.

What it runs against: a local clone of microsoft/BitNet — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in microsoft/BitNet | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 90 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of microsoft/BitNet. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/microsoft/BitNet.git
#   cd BitNet
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of microsoft/BitNet and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "microsoft/BitNet(\\.git)?\\b" \\
  && ok "origin remote is microsoft/BitNet" \\
  || miss "origin remote is not microsoft/BitNet (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "run_inference.py" \\
  && ok "run_inference.py" \\
  || miss "missing critical file: run_inference.py"
test -f "src/ggml-bitnet-lut.cpp" \\
  && ok "src/ggml-bitnet-lut.cpp" \\
  || miss "missing critical file: src/ggml-bitnet-lut.cpp"
test -f "gpu/model.py" \\
  && ok "gpu/model.py" \\
  || miss "missing critical file: gpu/model.py"
test -f "gpu/bitnet_kernels/bitnet_kernels.cu" \\
  && ok "gpu/bitnet_kernels/bitnet_kernels.cu" \\
  || miss "missing critical file: gpu/bitnet_kernels/bitnet_kernels.cu"
test -f "include/ggml-bitnet.h" \\
  && ok "include/ggml-bitnet.h" \\
  || miss "missing critical file: include/ggml-bitnet.h"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 90 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~60d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/microsoft/BitNet"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.