WAIT — Stale — last commit 1y ago

• 33+ active contributors
• Distributed ownership (top contributor 38% of recent commits)
• Other licensed
• ⚠ Stale — last commit 1y ago
• ⚠ Non-standard license (Other) — review terms
• ⚠ No CI workflows detected
• ⚠ No test directory detected

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This is a minimal inference engine for running Meta's Llama 2 large language models (7B to 70B parameters) locally. It provides Python utilities to load pre-trained model weights, tokenize text via SentencePiece, and generate completions using efficient inference—both for raw text completion and chat-based interactions. The core capability is executing Llama 2 forward passes with configurable sampling strategies (temperature, top-p) on consumer or server hardware. Single-package structure: llama/ contains three core modules—model.py (model loading and forward pass), generation.py (sampling and text generation logic), and tokenizer.py (SentencePiece wrapper). Entry points are example_*.py scripts (text_completion and chat_completion). The download.sh script manages model weight distribution via signed URLs.

Researchers, ML engineers, and developers who have received access to Llama 2 model weights and want to prototype inference without heavy frameworks. Users building standalone chat applications or text generation services who need a lightweight, dependency-minimal entry point before moving to production setups (like vLLM or ONNX pipelines).

This repository is explicitly deprecated as of Llama 3.1 (per README). It is no longer the canonical inference reference—Meta has consolidated into meta-llama/llama-models, llama-toolchain, and llama-cookbook. The codebase is stable but receiving no new features; treat it as legacy for Llama 2 only.

Risk is low for Llama 2 inference reproducibility, but high for forward compatibility: this repo will not receive Llama 3+ model support. Dependencies (torch, fairscale, sentencepiece) are mature but versions are unpinned in requirements.txt. The repository is transitioning to 'archived/read-only' status, so filing issues or PRs should target the newer consolidated repos instead.

Development is frozen. No active PRs, commits, or issues are being addressed. The README explicitly directs users to newer repositories (llama-models for model cards, llama-toolchain for canonical inference, llama-recipes for community scripts). This is intentional consolidation, not abandonment.

1. Clone the repo: git clone https://github.com/meta-llama/llama.git && cd llama. 2. Install dependencies: pip install -r requirements.txt (torch, fairscale, fire, sentencepiece). 3. Download model weights: visit https://ai.meta.com/resources/models-and-libraries/llama-downloads/, get a signed URL, then run ./download.sh and paste the URL when prompted.

Daily commands: After model download: python example_text_completion.py --model_dir ./model_weights --prompt 'Once upon a time' or python example_chat_completion.py for interactive chat. Both scripts accept temperature, top_p, max_seq_len flags via fire's auto-CLI.

• llama/model.py — Core model loading and inference engine; defines the Llama class that wraps transformer architecture and generation logic.
• llama/generation.py — Text generation pipeline including sampling strategies, token decoding, and sequence completion; critical for inference quality.
• llama/tokenizer.py — Sentence-piece tokenizer integration for encoding/decoding text; essential for prompt preparation and output parsing.
• example_text_completion.py — Canonical example of basic inference workflow; demonstrates the minimal API contract every contributor should understand.
• example_chat_completion.py — Reference implementation for conversation-based inference; shows multi-turn prompt formatting and response handling.
• README.md — Deprecation notice and migration guide; required reading to understand this repo is archived in favor of llama-models and llama-toolchain.
• requirements.txt — PyTorch, fairscale, and fire dependencies; critical for understanding runtime constraints and distributed inference support.

• llama.model.Llama (PyTorch, fairscale) — Transformer model orchestrator; loads checkpoint, manages device placement, runs forward inference, maintains KV cache.
  • Failure mode: Out-of-memory on GPU, dtype mismatch (FP32 vs BF16), corrupt checkpoint; unrecoverable without user intervention (reduce batch size, use smaller model).
• llama.generation (PyTorch) — Sampling and decoding loop; applies temperature, top-p, top-k filtering, stops at EOS or max_tokens.
  • Failure mode: Infinite loop if EOS token never sampled (rare); exceeds max_tokens silently if limit is too small.
• llama.tokenizer (sentencepiece) — Text ↔ token ID conversion using SentencePiece BPE; handles special tokens and chat markers.
  • Failure mode: Tokenizer model file missing or corrupted; raises exception at Llama.build() time.
• PyTorch backend (PyTorch, CUDA/ROCm, fairscale) — Tensor operations, GPU kernels, distributed collective ops (fairscale); underlying compute substrate.
  • Failure mode: GPU memory exhausted, CUDA runtime error, out-of-sync training/inference state; typically fatal process exit.

• User (text prompt) → llama.tokenizer.encode() — Prompt string → token ID sequence
• Token IDs → llama.model.Llama (forward pass) — Input tokens → logits tensor [batch, seq_len, vocab]
• Logits → llama.generation (sampling) — Logits → temperature scaling, top-p filter, argmax sample → next_token_id
• next_token_id → llama.model.Llama (KV cache update & next forward) — Append to sequence, update cache; loop until EOS or max_tokens
• Completion token IDs → ll — undefined

Add a custom text generation strategy

1. Define a new sampling function in llama/generation.py following the existing temperature/top_p/top_k pattern. (llama/generation.py)
2. Export the function signature in llama/init.py so users can import it. (llama/__init__.py)
3. Create an example script (e.g., example_beam_search.py) demonstrating the new strategy with before/after prompts. (example_beam_search.py)

Support a new tokenizer backend

1. Add tokenizer initialization logic to llama/tokenizer.py; conditionally load based on model type or config parameter. (llama/tokenizer.py)
2. Update llama/model.py to accept tokenizer_type argument in init and instantiate the correct tokenizer. (llama/model.py)
3. Test with example_text_completion.py to verify encode/decode round-trip. (example_text_completion.py)

Optimize model loading for a new hardware accelerator

1. Modify llama/model.py load() method to detect accelerator and override device placement logic. (llama/model.py)
2. Add optional accelerator dependency to requirements.txt (e.g., triton, torch_xla). (requirements.txt)
3. Document the new accelerator option in MODEL_CARD.md and add a reference example. (MODEL_CARD.md)

• PyTorch — Industry-standard tensor computation framework; enables GPU inference and distributed multi-GPU loading via fairscale.
• SentencePiece — Language-agnostic tokenizer; used by Llama models for encoding prompts and decoding completions with subword units.
• fairscale — Enables distributed inference across multiple GPUs; used for model parallelism and checkpoint sharding on large models.
• fire — CLI flag parsing library; provides simple command-line argument binding for example scripts without boilerplate.

• Deprecated in favor of llama-models/llama-toolchain

  • Why: Meta consolidated Llama repositories to centralize foundation model updates, safety mitigations, and tooling.
  • Consequence: This repo receives no new features; users should migrate to llama-toolchain for ongoing support, fine-tuning, and agentic systems.

• No built-in distributed serving (no vLLM, TGI, Ray integration)

  • Why: Repo is minimal reference implementation; production serving is out of scope.
  • Consequence: Users must wrap model.py with their own batching, request queue, and multi-user concurrency logic.

• Synchronous inference API (no async/streaming built-in)

  • Why: Simplifies the reference implementation and keeps PyTorch logic linear.
  • Consequence: High-throughput services must implement their own async wrapper or use external serving frameworks.

• Real-time latency optimization; this is a reference implementation, not production-optimized inference engine.
• Support for LoRA, QLoRA, or parameter-efficient fine-tuning; use llama-toolchain for training workflows.
• Multi-modal input (images, audio); Llama models in this repo are text-only.
• Distributed inference beyond single-machine multi-GPU via fairscale; no cluster/network parallelism.
• Safety guardrails, content filtering, or harm mitigation; see PurpleLlama for safety shields.
• Chat APIs or role-based prompt formatting beyond reference examples; see llama-agentic-system for conversational agents.

1. Signed URL expiry: download.sh links expire after 24h; if 403 errors occur, revisit the Meta website to request a fresh URL. 2. No batch inference: generate() runs one prompt at a time; using fairscale's distributed utilities requires manual setup and model replication across GPUs. 3. SentencePiece model file required: tokenizer.py expects a .model file in the checkpoint dir—missing it causes silent-ish failures. 4. Torch device not auto-detected: no automatic CUDA fallback; set device manually if needed. 5. Fire CLI argument parsing quirk: complex arguments (dicts, lists) require special syntax; see fire docs for edge cases.

• Rotary Position Embeddings (RoPE) — Llama 2 uses RoPE instead of ALiBi for position encoding; understanding this affects how you modify context length or fine-tune the model
• Grouped Query Attention (GQA) — Llama 2 70B uses GQA to reduce KV cache size and improve inference speed; this architectural choice affects memory usage and sampling strategy design
• Top-p (Nucleus) Sampling — The default decoding strategy in generation.py; knowing when to use top-p vs. temperature affects output quality and diversity for text generation
• SentencePiece Byte-Pair Encoding — Llama 2 tokenizes via SentencePiece, not tiktoken; this determines vocab size (32k tokens) and how special tokens (BOS, EOS, padding) are handled
• Causal Language Modeling (CLM) — Llama 2 is a causal (autoregressive) LM, not masked; this constraint affects how attention masks work and why you can only generate left-to-right
• Signed URL Distribution & Checksum Validation — Meta uses signed URLs with MD5 validation (in download.sh) to gate model access; understanding this mechanism is essential for reproducible setups and troubleshooting download failures

• meta-llama/llama-models — The official successor: central repo for Llama foundation models, model cards, and licensing—recommended for Llama 2 and 3+ model access going forward
• meta-llama/llama-toolchain — Canonical inference, fine-tuning, and safety interface: the production-grade replacement for this repository's inference examples
• meta-llama/llama-recipes — Community-driven cookbook: shows integration with Hugging Face transformers, vLLM, ONNX, and deployment patterns beyond this repo's minimal scope
• vllm-project/vllm — High-performance inference engine: if you need batch processing, tensor parallelism, and optimized KV caching beyond llama/ utilities
• huggingface/transformers — Alternative model loading path: many prefer Hugging Face's Llama2ForCausalLM for broader ecosystem integration (LoRA, quantization, fine-tuning)

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive unit tests for llama/tokenizer.py

The repo has example scripts (example_chat_completion.py, example_text_completion.py) but no test suite visible in the file structure. The tokenizer is a critical component for model inference. Adding unit tests would catch regressions, validate edge cases (special tokens, encoding/decoding round-trips), and serve as documentation for expected tokenizer behavior. This is especially important given the repo's deprecation status—tests ensure stability during the transition period.

• [ ] Create tests/test_tokenizer.py with fixtures for sample inputs
• [ ] Test basic encoding/decoding operations using sentencepiece tokenizer
• [ ] Add edge case tests: empty strings, special tokens, max length handling
• [ ] Test round-trip consistency (encode → decode → encode)
• [ ] Add tests for different model variants if tokenizer behavior varies
• [ ] Document test execution in README or CONTRIBUTING.md

Add GitHub Actions workflow to validate example scripts and dependencies

The repo provides example_chat_completion.py and example_text_completion.py but has no visible CI workflow (.github/workflows/ directory is absent). New contributors can't easily validate that examples run correctly or that dependency versions in requirements.txt are compatible. A basic workflow would catch breakage early and reduce friction for contributors testing changes.

• [ ] Create .github/workflows/validate-examples.yml
• [ ] Add steps to install dependencies from requirements.txt
• [ ] Add static analysis: flake8 or pylint on llama/ and example scripts
• [ ] Add import validation step (python -c 'import llama')
• [ ] Document expected test coverage in CONTRIBUTING.md
• [ ] Consider adding a 'dry-run' step that imports models without full inference (to avoid GPU requirements)

Create llama/utils.py and refactor common utilities from generation.py and model.py

As the repo consolidates functionality (per the deprecation notice), llama/generation.py and llama/model.py likely share helper functions for device management, dtype handling, or tensor operations. Extracting these into a dedicated llama/utils.py would reduce code duplication, improve maintainability, and make it easier for contributors to extend functionality. This is valuable groundwork before users migrate to the new llama-toolchain repo.

• [ ] Audit llama/generation.py and llama/model.py for shared utilities (e.g., device setup, tensor operations, parameter loading)
• [ ] Create llama/utils.py with extracted helper functions
• [ ] Add docstrings explaining each utility's purpose and usage
• [ ] Update imports in generation.py and model.py to use llama.utils
• [ ] Add basic tests for utilities in tests/test_utils.py
• [ ] Update llama/init.py to expose public utilities if appropriate

1. Add unit tests for tokenizer edge cases: llama/tokenizer.py has no test coverage; write tests in tests/test_tokenizer.py for special tokens, BOS/EOS handling, and round-trip encode-decode. 2. Document model loading failures: add a troubleshooting section to README.md explaining common errors (wrong checkpoint path, missing .model file, OOM on small GPUs) with debugging steps. 3. Add dtype control example: create example_quantized.py showing how to load Llama 2 in float16 or int8 using torch.load(map_location=...) to reduce VRAM—currently only float32 is documented.

The meta-llama/llama repository shows moderate security posture. The primary concern is that this is a deprecated repository with active development discontinued, exposing users to potential unpatched vulnerabilities. Secondary issues include unspecified dependency versions lacking security pinning and potential risks in model loading/execution code. The codebase lacks visible Docker/infrastructure issues and hardcoded secrets. Immediate action: users should migrate to actively maintained repositories (llama-models, PurpleLlama, llama-toolchain). For any continued use: implement dependency version pinning and validate input handling in model execution paths.

• Medium · Deprecated Repository - Active Development Discontinued — README.md. The repository is marked as deprecated as of Llama 3.1 release. Users are directed to use newer repositories (llama-models, PurpleLlama, llama-toolchain, llama-agentic-system) instead. Using deprecated code may expose users to unpatched security vulnerabilities and lack of ongoing maintenance. Fix: Migrate to the recommended repositories: llama-models for foundation models, PurpleLlama for safety and inference mitigations, and llama-toolchain for model development interfaces.
• Medium · Unspecified Dependency Versions — requirements.txt. The requirements.txt file does not specify pinned versions for dependencies (torch, fairscale, fire, sentencepiece). This allows installation of any version including those with known vulnerabilities, increasing the risk of supply chain attacks and compatibility issues. Fix: Pin all dependency versions explicitly (e.g., 'torch==2.0.1' instead of 'torch'). Regularly audit dependencies using tools like safety or pip-audit and update to patched versions.
• Low · Potential Model Execution Security Concerns — llama/model.py, llama/generation.py. The codebase includes model loading and execution in llama/model.py and llama/generation.py. Without visible code review, there may be risks related to arbitrary code execution through model loading, unsafe deserialization, or improper input validation. Fix: Implement strict input validation before model execution. Use safe model loading practices (verify checksums, use sandboxing). Review code for unsafe pickle operations or arbitrary code execution vectors.
• Low · Download Script Security — download.sh. The download.sh script's security practices are not visible. Shell scripts downloading models or artifacts can be vulnerable to MITM attacks, command injection, or unsafe execution if not properly implemented. Fix: Verify that download.sh implements secure practices: HTTPS-only downloads, checksum verification, proper quoting to prevent injection, and no execution of untrusted content.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/meta-llama/llama shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live meta-llama/llama repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/meta-llama/llama.

What it runs against: a local clone of meta-llama/llama — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in meta-llama/llama | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 497 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of meta-llama/llama. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/meta-llama/llama.git
#   cd llama
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of meta-llama/llama and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "meta-llama/llama(\\.git)?\\b" \\
  && ok "origin remote is meta-llama/llama" \\
  || miss "origin remote is not meta-llama/llama (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "llama/model.py" \\
  && ok "llama/model.py" \\
  || miss "missing critical file: llama/model.py"
test -f "llama/generation.py" \\
  && ok "llama/generation.py" \\
  || miss "missing critical file: llama/generation.py"
test -f "llama/tokenizer.py" \\
  && ok "llama/tokenizer.py" \\
  || miss "missing critical file: llama/tokenizer.py"
test -f "example_text_completion.py" \\
  && ok "example_text_completion.py" \\
  || miss "missing critical file: example_text_completion.py"
test -f "example_chat_completion.py" \\
  && ok "example_chat_completion.py" \\
  || miss "missing critical file: example_chat_completion.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 497 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~467d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/meta-llama/llama"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.