GO — Healthy across all four use cases

• Last commit today
• 5 active contributors
• BSD-3-Clause licensed
• CI configured
• Tests present
• ⚠ Single-maintainer risk — top contributor 89% of recent commits

_{Computed from maintenance signals — commit recency, contributor breadth, bus factor, license, CI, tests}

Pebble is a high-performance, LevelDB/RocksDB-inspired key-value store written in Go, designed specifically for CockroachDB's needs. It implements a Log-Structured Merge (LSM) tree with optimizations like L0 sublevels, range deletion tombstones, table-level bloom filters, and backwards-linked skiplist memtables for fast reverse iteration—trading off some RocksDB feature completeness for speed and internal CockroachDB compatibility. Flat monorepo structure: core LSM logic lives at the root (batch.go, cache.go, checkpoint.go, blob_rewrite.go), with specialized subpackages like batchrepr/ (batch binary format reader/writer), and cmd/pebble/ (CLI tools). No traditional layered architecture—instead, direct file organization by concern (manifest.go for file metadata, sstable.go for table format, internal/ for private APIs). Test files colocated with implementation (*_test.go pattern).

CockroachDB maintainers and developers building distributed SQL databases, storage engineers integrating alternative LSM engines into Go applications, and contributors optimizing key-value store performance at the layer between application and disk.

Production-ready and stable; it became the default storage engine in CockroachDB v20.2 (Nov 2020) and is running at scale in production. The repo shows active development with comprehensive CI workflows (tests.yaml, stress.yaml, crossversion.yaml, nightly-25.4.yaml, etc.), extensive test coverage instrumentation, and regular commits. This is a mature, actively maintained project with high stability expectations.

Low risk for stability but high coupling to CockroachDB's versioning strategy—the nightlies-26.1.yaml and nightlies-26.2.yaml workflows suggest tight version-locking. Dependency count is moderate (~35 direct, ~45 total) with critical compression/hash libraries (zstd, xorfilter, xxhash) managed tightly. No apparent single-maintainer risk given CockroachDB Labs backing, but the explicit WARNING in README about silent corruption if used with unsupported RocksDB features means misuse is possible.

Active development on compression/performance (minlz, zstd updates in dependencies), LSM refinements (L0 sublevel logic, range deletion optimization), and cross-version compatibility testing (multiple nightlies for CockroachDB versions 25.2–26.2). The nightlies workflows and code-cover instrumentation suggest focus on correctness under stress and coverage metrics. PR analyzer workflow hints at structured code review practices.

git clone https://github.com/cockroachdb/pebble.git
cd pebble
make test          # or 'make bench' for benchmarks
go test ./...      # run all tests
go run cmd/pebble/db.go  # CLI tool for manual testing

Daily commands: Development: make test (unit tests), make bench (benchmarks). CLI interactive mode: go run cmd/pebble/db.go. Stress testing: see stress.yaml workflow. Cross-version: run nightly-*.yaml workflows or go test -run TestCrossVersion. No server mode—Pebble is an embedded database, used as a Go package import.

• db.go — Core database struct and public API entry point; defines DB interface and lifecycle methods used by all operations
• compaction.go — Level-based compaction logic that drives performance; essential for understanding write amplification and file organization
• batch.go — Batch operations abstraction for atomic writes; critical for understanding transactional semantics
• sstable — SSTable (Sorted String Table) format implementation; foundational to Pebble's storage format and RocksDB compatibility
• internal/manifest.go — MANIFEST file management tracking all SSTable metadata across levels; essential for recovery and consistency
• cache.go — Block-level caching system; critical performance component affecting read latency
• iterator.go — Iterator abstraction for key-value traversal with range support; fundamental for scan and prefix operations

• DB (db.go, db_internals.go) (Go sync.R) — Lifecycle, routing reads/writes to memtable/levels, triggering flushes and compactions

Add a new comparison function for custom key ordering

1. Define a new Comparer struct implementing Compare, AbbreviatedKey, FormatKey, and Separator methods (comparer.go)
2. Register comparer in Options.Comparer and pass to DB.Open() (db.go)
3. Add tests covering edge cases in comparison logic (comparer_test.go)

Implement a new compaction strategy or optimization

1. Study existing compaction picker in compaction_picker.go and scoring logic (compaction_picker.go)
2. Modify or extend PickCompaction() or add new scoring function (compaction_picker.go)
3. Update compaction_scheduler.go if scheduling behavior changes (compaction_scheduler.go)
4. Add benchmarks and metamorphic tests in compaction_picker_test.go (compaction_picker_test.go)

Add support for a new SSTable block encoding format

1. Define encoder/decoder in sstable/block.go or new file (sstable/block.go)
2. Update Writer to support new format version during table creation (sstable/writer.go)
3. Update Reader to recognize and decompress new format (sstable/reader.go)
4. Add version constant to manifest and test format compatibility (internal/manifest.go)

Extend batch operations with a new operation type

1. Add new opcode constant in batch.go (batch.go)
2. Implement new method in Batch struct (e.g., BatchOpX) to add operation to batch (batch.go)
3. Update batchrepr/writer.go to serialize new operation (batchrepr/writer.go)
4. Update batchrepr/reader.go to deserialize and validate new operation (batchrepr/reader.go)
5. Add unit tests in batch_test.go (batch_test.go)

• Go — Memory-safe, concurrent, excellent performance for systems code, matches CockroachDB codebase
• Block-based SSTable format — RocksDB compatibility, efficient range queries, supports bloom filters and compression per block
• LSM tree (Log-Structured Merge) — Optimizes write-heavy workloads common in distributed databases, sequential I/O
• Zstandard compression — High compression ratio with reasonable speed; configurable per level
• Range deletion tombstones — Efficient bulk deletion without rewriting all keys; critical for CockroachDB schema evolution

• LSM tree vs. B+tree

  • Why: LSM trades random read latency for better write throughput and sequential I/O patterns
  • Consequence: Reads may require checking multiple levels; writes amortize cost into background compactions

• Block-level cache vs. page cache

  • Why: Application-level cache provides better control and metrics; avoids OS page cache unpredictability
  • Consequence: Must manage memory explicitly; requires cache size tuning by operator

• Level-based compaction vs. leveled

  • Why: Tunable via Options; balances write amplification against read cost
  • Consequence: Must choose levels/size ratios; different workloads have different optima

• Single-threaded compaction scheduler

  • Why: Simplifies concurrency; reduces metadata lock contention
  • Consequence: Compactions cannot parallelize; sustained write rate limited by single compactor thread

• Does not support distributed replication or consensus; by design a local key-value store
• Does not include SQL query planning or execution; low-level storage engine only
• Does not provide built-in authentication, encryption at rest, or network RPC layer
• Not a document database; no JSON indexing, full-text search, or schema enforcement
• Does not aim for feature parity with RocksDB; intentionally omits rarely-used features

No persistent server mode: Pebble is Go-import-only; you can't pebble-server :6379 like Redis. Must embed it as import "github.com/cockroachdb/pebble" and manage DB lifecycle in your own process. Endianness assumptions: Block-based format uses specific byte ordering; cross-architecture MANIFEST compatibility is tested but not guaranteed for exotic platforms. Batch encoding stability: batchrepr binary format is stable across versions, but metamorphic tests must pass to ensure backward compatibility; modifying batch.proto-like encoding risks silent data loss. Compression mandatory on disk: All SSTable writes use compression (Snappy default, Zstd configurable); uncompressed reads will fail. No transactions: Unlike RocksDB, no cross-key ACID guarantees; CockroachDB handles that layer. CockroachDB version coupling: Nightlies tie to specific CockroachDB versions; using Pebble master in older CockroachDB builds may fail.

• Log-Structured Merge (LSM) Tree — Core data structure of Pebble; understanding levels, compaction, and write amplification is essential to grasp why Pebble makes the design choices it does.
• Bloom Filter (and Cuckoo Filter) — Pebble uses table-level and prefix bloom filters (via xorfilter/cuckoo) to skip unnecessary disk reads; critical to query optimization.
• Range Deletion Tombstones — Pebble's extension allowing efficient deletion of key ranges without touching every key; enables delete-only compactions and is a major performance win.
• Skiplist with Backward Links — Pebble's memtable uses a skiplist with reverse pointers, enabling O(1) backward iteration versus RocksDB's O(n) reverse scan.
• Copy-on-Write B-tree — Used for MANIFEST file metadata storage and version management; enables efficient handling of thousands of SSTables without per-operation allocations.
• L0 Sublevels — Pebble's innovation for ordering L0 SSTables without full compaction, reducing read amplification and enabling concurrent L0→L1 compactions.
• Metamorphic Testing — Pebble uses cockroachdb/metamorphic to generate randomized operation sequences and verify multiple implementations produce identical results; critical for correctness at this abstraction level.

• cockroachdb/cockroachdb — Primary consumer and sponsor of Pebble; storage engine integration point and motivation for feature set.
• facebook/rocksdb — Predecessor design and file format compatibility reference; Pebble inherits RocksDB SSTable layout and manifest structure.
• google/leveldb — Original LSM tree inspiration and baseline API design; Pebble modernizes LevelDB's core concepts.
• etcd-io/bbolt — Alternative Go key-value store using B+tree instead of LSM; useful for comparing trade-offs in embedded Go databases.
• dgraph-io/badger — Another Go LSM implementation with value log separation; reference for alternative design choices in Go KV stores.

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive integration tests for blob rewriting operations

The repo contains blob_rewrite.go and blob_rewrite_test.go, but given the complexity of blob rewriting in a LSM context and its critical nature for storage efficiency, there are likely edge cases and failure scenarios not covered. This is valuable because blob rewriting affects data durability and storage optimization - two critical concerns for a KV store used by CockroachDB.

• [ ] Review blob_rewrite.go and blob_rewrite_test.go to identify untested code paths (focus on error conditions, concurrent operations, and recovery scenarios)
• [ ] Add tests for blob rewriting during compaction with concurrent reads/writes
• [ ] Add tests for blob rewrite failure and recovery scenarios (e.g., partial writes, disk full conditions)
• [ ] Add stress test cases in the existing test suite targeting blob_rewrite operations
• [ ] Ensure tests cover interactions with checkpoint.go since checkpoints may interact with blob metadata

Add cross-version compatibility tests for batch representation format changes

The batchrepr package handles serialization/deserialization of batch data with testdata files (batchrepr/testdata/reader and writer). However, there's likely no automated testing that ensures older batch formats can be read by newer versions. This is critical for forward/backward compatibility in distributed systems like CockroachDB where different nodes may be running different versions temporarily.

• [ ] Review batchrepr/reader.go and batchrepr/writer.go to understand current format versioning
• [ ] Create a .github/workflows entry (similar to crossversion.yaml that already exists) specifically for batch format compatibility
• [ ] Add test cases in batchrepr/reader_test.go that verify reading of batches written by previous Pebble versions
• [ ] Generate and commit historical batch format test fixtures to batchrepr/testdata/ for version compatibility validation
• [ ] Add documentation in CLAUDE.md or a new batchrepr/README.md explaining the format versioning strategy

Add comprehensive unit tests for cache.go eviction and concurrency scenarios

Cache behavior is foundational to KV store performance, and cache.go is a core module. Given the concurrent nature of LSM operations and the dependencies on github.com/puzpuzpuz/xsync/v3 for concurrent data structures, edge cases in eviction under contention are likely under-tested. This affects both performance characteristics and correctness of cached data.

• [ ] Review cache.go to identify missing test coverage (especially around concurrent eviction, capacity limits, and hit/miss ratios)
• [ ] Add stress tests with concurrent read/write operations on the cache in cache_test.go (use similar patterns from existing *_test.go files)
• [ ] Add tests verifying correct eviction behavior under memory pressure and with different eviction policies
• [ ] Add tests for cache behavior during compaction-induced metadata changes
• [ ] Profile and document cache performance characteristics by adding benchmarks to cache_test.go if they're missing

• Add comprehensive documentation to vfs.go interfaces (FileSystem, File abstractions) with concrete examples of in-memory and mmap backends; currently sparse comments make it hard for new contributors to extend storage layers.
• Expand blob_rewrite_test.go to cover range deletion tombstone interactions during blob rewriting—currently test coverage doesn't explicitly verify correctness when delete ranges overlap with blob boundaries.
• Create a runnable example in cmd/pebble/ showing ETL-style batch load + iteration + snapshot isolation patterns, documented with inline comments; useful for onboarding and demo purposes.

The Pebble repository demonstrates reasonable security practices as a mature CockroachDB component. No critical vulnerabilities were identified in the visible structure. Primary concerns are: (1) dependency management for compression and custom fork components should undergo continuous security monitoring, (2) GitHub Actions workflows require audit for secure credential handling, and (3) formal security policies and SBOM documentation should be established. The codebase appears well-maintained with no obvious hardcoded secrets, injection points, or configuration mismanagement in the visible file structure. Recommended actions include implementing govulncheck scanning in CI/CD pipeline, documenting security policies, and maintaining comprehensive dependency audits.

• Medium · Outdated or Unverified Compression Dependencies — go.mod - compression dependencies (github.com/DataDog/zstd, github.com/golang/snappy, github.com/klauspost/compress, github.com/minio/minlz). The codebase uses multiple compression libraries (zstd, snappy, minlz, compress) with specific versions that may contain known vulnerabilities. The DataDog/zstd (v1.5.7) and klauspost/compress (v1.17.11) should be verified against CVE databases for known security issues in compression algorithms. Fix: Run go list -u -m all to check for available updates and nancy or govulncheck to identify known vulnerabilities in these dependencies. Keep compression libraries updated to the latest stable versions.
• Medium · Custom Fork Dependencies with Limited Transparency — go.mod - custom dependencies (cockroachdb/crlib, RaduBerinde/axisds, RaduBerinde/tdigest, FastFilter/xorfilter). The project uses several custom CockroachDB forks and internal dependencies (github.com/cockroachdb/crlib, github.com/RaduBerinde/axisds) with recent timestamps but limited public visibility. These may bypass standard security review processes if not properly vetted. Fix: Ensure all custom forks and internal dependencies have documented security review processes. Use dependency scanning tools and maintain an internal vulnerability tracking system for these components.
• Low · No Evidence of SBOM or Dependency Lock Strategy — Repository root - missing SBOM documentation. While go.mod exists, there is no visible Software Bill of Materials (SBOM) generation or explicit dependency pinning strategy documented in the repository structure. Fix: Generate and maintain SBOM files using tools like 'syft' or 'cyclonedx'. Document dependency management policies in SECURITY.md or similar documentation.
• Low · GitHub Actions Workflows Not Fully Analyzed — .github/workflows/ directory (ci.yaml, tests.yaml, stress.yaml, etc.). Multiple GitHub Actions workflows exist (.github/workflows/) that could potentially contain insecure patterns such as unsafe environment variable handling, unvetted third-party actions, or secrets exposure. The CI/CD pipeline visibility is limited without full workflow content analysis. Fix: Audit all GitHub Actions workflows for: 1) Use of official vs third-party actions, 2) Secrets handling (never log secrets), 3) Permission scoping (principle of least privilege), 4) Dependency version pinning in actions.
• Low · No Explicit Security Policy Visible — Repository root. No SECURITY.md or SECURITY.txt file is evident in the root directory, which is a best practice for responsible disclosure and security coordination. Fix: Create a SECURITY.md file that includes: 1) Instructions for reporting vulnerabilities, 2) Security contact information, 3) Responsible disclosure timeline, 4) Known security limitations.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/cockroachdb/pebble shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live cockroachdb/pebble repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/cockroachdb/pebble.

What it runs against: a local clone of cockroachdb/pebble — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in cockroachdb/pebble | Confirms the artifact applies here, not a fork | | 2 | License is still BSD-3-Clause | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 30 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of cockroachdb/pebble. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/cockroachdb/pebble.git
#   cd pebble
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of cockroachdb/pebble and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "cockroachdb/pebble(\\.git)?\\b" \\
  && ok "origin remote is cockroachdb/pebble" \\
  || miss "origin remote is not cockroachdb/pebble (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(BSD-3-Clause)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"BSD-3-Clause\"" package.json 2>/dev/null) \\
  && ok "license is BSD-3-Clause" \\
  || miss "license drift — was BSD-3-Clause at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "db.go" \\
  && ok "db.go" \\
  || miss "missing critical file: db.go"
test -f "compaction.go" \\
  && ok "compaction.go" \\
  || miss "missing critical file: compaction.go"
test -f "batch.go" \\
  && ok "batch.go" \\
  || miss "missing critical file: batch.go"
test -f "sstable" \\
  && ok "sstable" \\
  || miss "missing critical file: sstable"
test -f "internal/manifest.go" \\
  && ok "internal/manifest.go" \\
  || miss "missing critical file: internal/manifest.go"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 30 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~0d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/cockroachdb/pebble"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.