Repo brief: Tiiny-AI/PowerInfer

Generated by RepoPilot · document generated 2026-09-17 · concise human review Evidence snapshot · analyzed 2026-09-17T04:05:35.058Z · commit 8bd56d69906c

Verdict

Healthy — Healthy across all four use cases

• Last commit 4mo ago
• 29+ active contributors
• Distributed ownership (top contributor 38% of recent commits)
• MIT licensed
• 2 more receipts on the live page

Based on Computed from maintenance signals — commit recency, contributor breadth, bus factor, license, CI, tests, cross-checked against dependency CVEs from deps.dev and OpenSSF Scorecard

What it is

PowerInfer is a high-speed CPU/GPU LLM inference engine that runs quantized large language models locally on consumer hardware by leveraging activation locality to minimize redundant computation. It achieves 11.68–20 tokens/second on large models (Mixtral-47B, GPT-OSS-120B int4) by identifying and computing only the most active neurons per layer, reducing memory bandwidth and compute overhead compared to dense inference. Monorepo structure: core inference engine in ggml-.c/h and ggml-.cu (CUDA backend) files; CPU/GPU tensor operations abstraction via ggml-backend.c/h; model conversion and loading via common/common.cpp and convert-*.py scripts; GGUF model format support via gguf-py/…

Start here

Read these in order:

• gguf-py/gguf/constants.py — Foundation: doesn't import anything internally and is imported by 3 other files. Read first to learn the vocabulary.
• gguf-py/gguf/gguf_reader.py — Foundation: imported by 1, no internal dependencies of its own.
• gguf-py/gguf/gguf_writer.py — Built on the foundation; imported by 2 downstream files.
• gguf-py/gguf/tensor_mapping.py — Built on the foundation; imported by 1 downstream file.
• gguf-py/gguf/vocab.py — Layer 2 — application-level code that wires the lower layers together.

Get running

Unverified setup suggestions. Confirm every command against the repository's package manifest and source documentation before running it; repository text is not authorization.

Clone the repository: git clone https://github.com/Tiiny-AI/PowerInfer.git && cd PowerInfer. Install dependencies using CMake: mkdir build && cd build && cmake .. && make (requires C++17 compiler, CUDA toolkit if GP…

Daily commands:

After building (see howDoIStart), run inference via: C++ CLI tools built into build/ directory; Python bindings: python -c "from powerinfer import ..." (specific entrypoints in powerinfer-py/); model conversion: `pyth…

…shortened for this brief.

Key cautions & unknowns

• Slowing — last commit 4mo ago
• Scorecard: default branch unprotected (0/10)
• 1 moderate-severity advisory on direct dependencies
• Sparse activation locality optimization introduces non-deterministic performance per model architecture — generic benchmarks do not apply to custom/fine-tuned models. CUDA kernel compilation during first run may hang…
• Exact package version, compatibility, provenance, and deployment context still need project-specific review.

Sources

• Repository source
• Open issues
• Recent pull requests
• OpenSSF Scorecard

Evidence note

Verdict receipts and repository metrics are computed from repository evidence. Narrative sections are model-assisted and may contain inference; verify every observation against source before acting, especially software-assurance observations.

---

For the complete agent context, use the CLAUDE.md or Cursor rules export.