If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/bottlerocket-os/bottlerocket shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across the board

• Last commit 2d ago
• 13 active contributors
• Distributed ownership (top contributor 21% of recent commits)
• Other licensed
• CI configured
• Tests present
• ⚠ Non-standard license (Other) — review terms

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live bottlerocket-os/bottlerocket repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/bottlerocket-os/bottlerocket.

What it runs against: a local clone of bottlerocket-os/bottlerocket — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in bottlerocket-os/bottlerocket | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch develop exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 32 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of bottlerocket-os/bottlerocket. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/bottlerocket-os/bottlerocket.git
#   cd bottlerocket
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of bottlerocket-os/bottlerocket and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "bottlerocket-os/bottlerocket(\\.git)?\\b" \\
  && ok "origin remote is bottlerocket-os/bottlerocket" \\
  || miss "origin remote is not bottlerocket-os/bottlerocket (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify develop >/dev/null 2>&1 \\
  && ok "default branch develop exists" \\
  || miss "default branch develop no longer exists"

# 4. Critical files exist
test -f "Cargo.toml" \\
  && ok "Cargo.toml" \\
  || miss "missing critical file: Cargo.toml"
test -f "sources/api/datastore/src/lib.rs" \\
  && ok "sources/api/datastore/src/lib.rs" \\
  || miss "missing critical file: sources/api/datastore/src/lib.rs"
test -f "sources/api/migration/migration-helpers/src/lib.rs" \\
  && ok "sources/api/migration/migration-helpers/src/lib.rs" \\
  || miss "missing critical file: sources/api/migration/migration-helpers/src/lib.rs"
test -f "packages/settings-defaults/Cargo.toml" \\
  && ok "packages/settings-defaults/Cargo.toml" \\
  || miss "missing critical file: packages/settings-defaults/Cargo.toml"
test -f "BUILDING.md" \\
  && ok "BUILDING.md" \\
  || miss "missing critical file: BUILDING.md"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 32 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~2d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/bottlerocket-os/bottlerocket"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Bottlerocket is a minimal, hardened Linux operating system written primarily in Rust designed exclusively for running containerized workloads on AWS (ECS, EKS), VMware, and bare metal. It replaces traditional general-purpose Linux distributions with an OS that has only the components needed to reliably host containers, delivering fast partition-flip-based updates and configuration management via a secure API rather than manual edits. Monorepo with workspace root at Cargo.toml. Structure: packages/ contains core tooling (settings-defaults, settings-migrations, settings-plugins), variants/ contains 45+ OS variants for different platforms/versions (aws-k8s-1.30, aws-ecs-3, vmware-k8s-1.34, metal-dev, etc.). Build system uses Makefile.toml (Cargo-make), Twoliter.toml for composition. GitHub Actions orchestrate CI/build/cache/weekly validation workflows.

DevOps engineers and infrastructure teams deploying containers at scale who want a security-hardened, minimal OS that automatically migrates configuration through updates without manual intervention. Also relevant to organizations running Kubernetes or ECS at scale who need reliable, consistent host OS updates.

Production-ready and actively maintained. The codebase shows 542K lines of Rust with comprehensive CI/CD workflows (.github/workflows/build.yml, cache.yml, weekly.yml), extensive documentation (QUICKSTART-EKS.md, QUICKSTART-ECS.md, PROVISIONING-METAL.md), and a structured workspace with 45+ variant builds across AWS ECS/EKS versions and VMware. Active community with Meetup groups and regular issue triage suggests steady ongoing development.

Low risk for mature users but ecosystem-dependent. The project maintains 45+ variant configurations (aws-k8s-1.30 through 1.35, ECS versions 2-3, NVIDIA variants, FIPS variants) which creates testing and maintenance surface area. Dependency on external Kubernetes versions and container runtimes means breaking changes upstream could require rapid response. Single language focus (Rust) means all contributors must be Rust-capable.

Active variant maintenance across Kubernetes 1.30-1.35 (with FIPS and NVIDIA GPU support), ECS 2-3 support, and VMware variants. Weekly CI runs (workflows/weekly.yml) suggest continuous testing. Dependabot integration (dependabot.yaml) indicates automated dependency updates. Recent additions of 1.35 variants (aws-k8s-1.35, vmware-k8s-1.35) with both standard and FIPS versions show ongoing version tracking.

git clone https://github.com/bottlerocket-os/bottlerocket.git
cd bottlerocket
# Install Rust (if not present)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Build (uses Makefile.toml with cargo-make)
cargo install cargo-make
cargo make build
# See BUILDING.md for detailed platform-specific build instructions

Daily commands: This is an OS, not an application. To build for a specific platform: cargo make build -e VARIANT=aws-k8s-1.35 (builds that variant). To run: deploy the resulting AMI/image to EKS, ECS, or VMware using QUICKSTART-EKS.md, QUICKSTART-ECS.md, or QUICKSTART-VMWARE.md. Local development uses Makefile.toml targets defined in the workspace.

• Cargo.toml — Root workspace definition containing all member crates including API, datastore, and variant packages—understanding the monorepo structure is essential.
• sources/api/datastore/src/lib.rs — Core datastore abstraction for Bottlerocket's settings API; the fundamental data persistence layer all configuration flows through.
• sources/api/migration/migration-helpers/src/lib.rs — Migration framework helpers enabling safe schema and settings evolution across OS updates—critical for OS reliability.
• packages/settings-defaults/Cargo.toml — Defines default settings schema and packages; every contributor modifying system behavior must understand defaults.
• BUILDING.md — Build instructions and toolchain setup; required reading before contributing or understanding the container-focused OS build process.
• CHARTER.md — Articulates Bottlerocket's design philosophy around security, maintainability, and container-centric principles guiding all architectural decisions.
• sources/constants/src/lib.rs — Central constants and build-time configuration for the OS; informs version, feature flags, and cross-crate configuration.

Add a new OS setting to the schema

1. Define the new setting type in the datastore constraints or settings-defaults schema (packages/settings-defaults/Cargo.toml)
2. Add validation logic if the setting has constraints (sources/api/datastore/src/constraints_check.rs)
3. If adding a new setting to an existing structure, create a migration in migration-helpers to handle upgrade (sources/api/migration/migration-helpers/src/common_migrations.rs)
4. Update the datastore serialization layer if the new type needs custom serialization (sources/api/datastore/src/serialization/mod.rs)

Add support for a new container runtime variant

1. Create a new variant directory under variants/ (e.g., variants/aws-ecs-4/Cargo.toml) (Cargo.toml)
2. Add the variant to the workspace members list in root Cargo.toml (Cargo.toml)
3. Define runtime-specific settings in the variant's settings-defaults (packages/settings-defaults/Cargo.toml)
4. Build and test using Makefile.toml targets for the new variant (Makefile.toml)

Handle a settings migration during a breaking change

1. Implement the migration logic using common patterns from migration-helpers (sources/api/migration/migration-helpers/src/common_migrations.rs)
2. Use datastore_helper utilities to read/write settings during migration (sources/api/migration/migration-helpers/src/datastore_helper.rs)
3. Test the migration against both in-memory and filesystem datastores (sources/api/datastore/src/memory.rs)
4. Document the migration reason and version in CHANGELOG.md (CHANGELOG.md)

Extend datastore with a new backend

1. Implement the Datastore trait from the abstract definition (sources/api/datastore/src/lib.rs)
2. Implement serialization/deserialization for the new backend format (sources/api/datastore/src/serialization/mod.rs)
3. Define custom error types if needed for the backend (sources/api/datastore/src/error.rs)
4. Add build-time feature flag in root Cargo.toml to conditionally compile the new backend (Cargo.toml)

• Rust — Memory safety and zero-cost abstractions critical for a bare-metal container OS with uptime requirements; no GC pauses.
• Datastore abstraction (trait-based) — Allows testing against in-memory backend while production uses persistent filesystem, enabling both fast unit tests and reliable persistence.
• Settings-as-API model — Decouples manual SSH config edits from declarative, versioned configuration management—foundation of reliable updates and reproducibility.
• Cargo workspace + variants — Monorepo structure allows shared core crates while maintaining separate, curated builds for different container orchestrators (ECS, EKS, bare metal).
• Migration framework — Enables backwards-compatible schema evolution; critical for OS updates that must preserve and transform user settings without data loss.

• Filesystem-based settings storage instead of database

  • Why: Minimizes OS footprint, avoids daemon dependencies, and keeps settings human-readable/inspectable on-disk.
  • Consequence: Eventual consistency; concurrent writes require external coordination; limits query sophistication (no complex filters).

• Settings-as-API instead of manual config files

  • Why: Enforces schema validation, version tracking, and atomic updates; enables reproducible deployments.
  • Consequence: Adds API layer complexity; operators must learn API instead of direct file editing; requires migration tooling for breaking changes.

• Multiple variants (ECS, EKS, bare metal) in single repo

  • Why: Shares core subsystems, reduces duplication, ensures consistency across platforms.
  • Consequence: Larger repo; build matrix complexity; variant-specific bugs harder to isolate; CI/CD must test all variants.

• Constraint validation at datastore layer, not API layer

  • Why: Ensures constraints are enforced regardless of how settings are modified (API, migration, direct write).
  • Consequence: Validation errors bubbled up from datastore; constraints must be serializable/storable; adds latency to writes.

• Package management (Bottlerocket is immutable; use container registries for application updates)
• Interactive shell or SSH daemon (container-centric automation, not interactive admin access by default)
• General-purpose Linux distribution (strictly scoped to container workloads; minimal userland)
• Real-time kernel features (designed for batch container scheduling, not hard real-time systems)
• Multi-tenant isolation (single-OS-instance-per-account model assumes infrastructure-level isolation)

Build complexity: Requires Twoliter tooling (Twoliter.toml, Twoliter.lock) for image composition—building requires cargo-make and platform-specific toolchain (not just cargo build). Variant explosion: 45+ variants means testing/docs must stay in sync across aws-k8s-1.30 through 1.35, both FIPS and non-FIPS, plus ECS and NVIDIA variants—easy to miss updates. Settings migration burden: Any configuration schema change in settings-defaults requires corresponding migration in settings-migrations or old instances fail on update. Platform lock-in: Most docs/quickstarts assume AWS (ECS/EKS); metal and VMware paths are less documented. No traditional package manager: Bottlerocket intentionally lacks apt/yum—all changes must go through the API or require rebuilding the image.

• Partition-flip updates — Core to Bottlerocket's reliability claim; the OS boots from alternating disk partitions, making rollback atomic and instant—understanding this explains why Bottlerocket can guarantee zero-downtime updates
• Modeled configuration with automatic migration — Bottlerocket versioning schema migrates settings across OS updates automatically; you must understand this to extend the configuration system without breaking existing deployments
• Out-of-band API access — Bottlerocket exposes a secure, privileged API for configuration even when container workloads are compromised; critical to the security model and what makes it 'immutable OS' compatible
• Variant-based build system — The 45+ variants (aws-k8s-1.35, aws-ecs-3, vmware-k8s-1.34, etc.) are built from shared packages with variant-specific overrides; essential to contributing safely to the right variant
• FIPS compliance variants — aws-k8s-1.35-fips and vmware-k8s-1.35-fips enforce FIPS 140-2 cryptography; regulatory compliance feature that affects build tooling and acceptable dependencies
• Twoliter composition — Build orchestration layer (Twoliter.toml) that composes Bottlerocket OS images from packages and variants; understanding this is necessary to modify the image build pipeline
• Settings plugins architecture — packages/settings-plugins allows extending the configuration API without modifying core; key pattern for custom deployments and vendor integrations

• flatcar/flatcar-linux — Alternative minimal Linux OS for containers (CoreOS successor); same problem domain but different approach to updates and configuration
• fedora-iot/ignition — Configuration management tool used in some container OS distributions; Bottlerocket uses similar declarative configuration principles
• aws/amazon-ecs-agent — ECS agent that runs on Bottlerocket instances; directly compatible and key component of the aws-ecs-2 and aws-ecs-3 variants
• kubernetes/kubernetes — Core dependency for all Bottlerocket aws-k8s-* and vmware-k8s-* variants; Bottlerocket tracks stable K8s versions 1.30-1.35
• moby/moby — Container runtime underlying Bottlerocket's container hosting; essential to understanding the lower-level runtime behavior

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add integration tests for settings-migrations package

The packages/settings-migrations directory exists with a Cargo.toml and spec file, but there's no visible test suite. Settings migrations are critical for OS upgrades and version compatibility. Adding comprehensive tests would ensure migrations work correctly across Bottlerocket variants (ECS, Kubernetes 1.30-1.35, FIPS, NVIDIA). This is high-value because a broken migration could affect all users during updates.

• [ ] Create tests/ directory in packages/settings-migrations/
• [ ] Add integration tests for common migration patterns (e.g., schema additions, field renames, deprecations)
• [ ] Test migrations across at least 3 variant configurations to ensure they work universally
• [ ] Add test execution to .github/workflows/build.yml or create a new workflow for migration-specific testing
• [ ] Document test setup in packages/settings-migrations/README.md

Add variant-specific build validation workflow

The Cargo.toml workspace defines 40+ variants (AWS ECS, Kubernetes 1.30-1.35, NVIDIA, FIPS, VMware, metal). Currently there's no visible CI workflow that validates all variants can build successfully. A new GitHub Action would catch variant-specific regressions early, which is critical given the combinatorial explosion of variant combinations.

• [ ] Create .github/workflows/variant-matrix.yml that builds all workspace members in parallel
• [ ] Use matrix strategy to test representative variants: aws-ecs-3, aws-k8s-1.35, aws-k8s-1.35-nvidia-fips, vmware-k8s-1.35, metal-dev
• [ ] Add quick sanity checks: cargo build for each variant, verify .spec files are valid
• [ ] Cache Cargo artifacts to keep run time under 30 minutes
• [ ] Document in CONTRIBUTING.md which variants are tested in CI vs. optional local testing

Create ARCHITECTURE.md documenting the settings system and package structure

While BUILDING.md, CONTRIBUTING.md, and multiple QUICKSTART guides exist, there's no high-level architectural documentation explaining how settings-defaults, settings-migrations, and settings-plugins interact, or why the workspace has 40+ variant crates. New contributors struggle to understand the OS design. This doc would reference existing files like packages/settings-migrations/README.md and sources/api/datastore/README.md to tie them together.

• [ ] Create ARCHITECTURE.md at repo root explaining: (1) How Bottlerocket's settings system works, (2) Why variant approach exists, (3) Relationship between packages/, sources/, and variants/
• [ ] Add diagrams or flow charts showing settings flow: defaults → migrations → plugins
• [ ] Document the purpose of each major package: settings-defaults (immutable base config), settings-migrations (upgrade path), settings-plugins (extensibility)
• [ ] Explain variant strategy: base variants (aws-dev, metal-dev, vmware-dev) vs. derived (aws-k8s-*, with FIPS/NVIDIA combinations)
• [ ] Link to sources/api/datastore/README.md and explain how datastore integrates with settings
• [ ] Add examples: 'To add a new Kubernetes variant, copy an existing aws-k8s-X and update these 3 files...'

• Add integration tests for settings migration on at least 2 variant types (e.g., test that aws-k8s-1.35 and vmware-k8s-1.35 apply schema migrations identically); currently only code exists with no visible test coverage in packages/settings-migrations
• Expand QUICKSTART-VMWARE.md with concrete vSphere provisioning examples (IP configuration, network setup steps); currently it's minimal compared to QUICKSTART-EKS.md and QUICKSTART-ECS.md
• Create a decision matrix document comparing when to use aws-ecs vs aws-k8s vs vmware variants based on workload requirements; would help new users navigate the 45+ variant options without trial-and-error

Bottlerocket demonstrates a strong security foundation with responsible vulnerability disclosure practices, security-focused documentation, and a clear security charter. The codebase is organized and follows Rust security best practices. However, the large workspace with 51+ variants and multiple Kubernetes versions creates significant maintenance complexity that could introduce security gaps. The main concerns are: (1) ensuring consistent security practices across all variants, (2) comprehensive dependency vulnerability scanning in CI/CD, and (3) secure management of signing key infrastructure. The medium-sized attack surface from variant proliferation and lack of visible automated security scanning in the provided workflow files are the primary areas for improvement. Consider implementing workspace-wide security policies, automated vulnerability scanning, and stricter version support policies.

• Low · Multiple Kubernetes Versions in Maintenance — Cargo.toml - workspace members (variants/aws-k8s-*, variants/vmware-k8s-*). The workspace includes multiple Kubernetes versions (1.30 through 1.35) with both standard and FIPS variants. While this provides flexibility, it increases the maintenance burden and potential for version-specific vulnerabilities to be missed in older versions. Ensure all supported versions receive security updates in a timely manner. Fix: Implement a clear EOL (End-of-Life) policy for older Kubernetes versions. Establish a maximum number of concurrent supported versions and automate security update testing across all variants.
• Low · Debug Symbols Disabled in Dev Profile — Cargo.toml - [profile.dev] section. The dev profile has debug = false which removes debug symbols. While this reduces binary size, it may complicate security debugging and incident response. This appears intentional for optimization, but consider the trade-offs. Fix: Evaluate whether debug symbols should be available in development builds for security analysis. Consider using a separate debug build profile for security testing.
• Medium · Large Workspace Attack Surface — Cargo.toml - entire workspace definition. The workspace contains 51+ members across multiple variants (AWS, VMware, Metal) with multiple Kubernetes versions and FIPS variants. This large number of build targets increases the attack surface and complexity of ensuring consistent security practices across all builds. Fix: Implement automated security scanning (SAST, dependency auditing) that applies to all workspace members. Use workspace-level linting and policy enforcement to ensure consistent security configurations.
• Low · Incomplete Visibility into Dependencies — sources/Cargo.lock (not fully provided). Only high-level workspace structure and Cargo.toml is provided. Actual dependency vulnerabilities cannot be assessed without examining Cargo.lock and the specific dependency trees for each crate. Fix: Regularly run cargo audit to identify known vulnerabilities in dependencies. Implement automated dependency scanning in CI/CD pipeline. Set up Dependabot or similar tools for proactive updates.
• Low · No Visible Security Policy Enforcement in CI/CD — .github/workflows/. While SECURITY.md exists with responsible disclosure guidelines, the visible GitHub workflows (.github/workflows/) do not show explicit security scanning steps in the build process (only build.yml, cache.yml, weekly.yml are mentioned). Fix: Add explicit security scanning steps to CI/CD: cargo audit for dependency vulnerabilities, SAST tools for code analysis, container image scanning before publishing, and security linting.
• Low · Potential Secrets in Signing Keys Directory — sbkeys/ directory. The sbkeys/ directory contains signing key generation scripts. While these appear to be generation tools rather than actual keys, the presence of signing key infrastructure needs careful access control. Fix: Ensure signing keys are never committed to the repository. Use secure key management (AWS KMS, HashiCorp Vault). Restrict access to key generation scripts and implement audit logging for all key operations.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.