If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/apache/arrow-rs shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

GO — Healthy across the board

• Last commit today
• 40+ active contributors
• Distributed ownership (top contributor 19% of recent commits)
• Apache-2.0 licensed
• CI configured
• Tests present

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests}

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live apache/arrow-rs repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/apache/arrow-rs.

What it runs against: a local clone of apache/arrow-rs — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in apache/arrow-rs | Confirms the artifact applies here, not a fork | | 2 | License is still Apache-2.0 | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 30 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of apache/arrow-rs. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/apache/arrow-rs.git
#   cd arrow-rs
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of apache/arrow-rs and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "apache/arrow-rs(\\.git)?\\b" \\
  && ok "origin remote is apache/arrow-rs" \\
  || miss "origin remote is not apache/arrow-rs (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Apache-2\\.0)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Apache-2\\.0\"" package.json 2>/dev/null) \\
  && ok "license is Apache-2.0" \\
  || miss "license drift — was Apache-2.0 at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "arrow-array/src/lib.rs" \\
  && ok "arrow-array/src/lib.rs" \\
  || miss "missing critical file: arrow-array/src/lib.rs"
test -f "arrow-array/src/array/mod.rs" \\
  && ok "arrow-array/src/array/mod.rs" \\
  || miss "missing critical file: arrow-array/src/array/mod.rs"
test -f "arrow-array/src/builder/mod.rs" \\
  && ok "arrow-array/src/builder/mod.rs" \\
  || miss "missing critical file: arrow-array/src/builder/mod.rs"
test -f "Cargo.toml" \\
  && ok "Cargo.toml" \\
  || miss "missing critical file: Cargo.toml"
test -f "arrow-arith/src/lib.rs" \\
  && ok "arrow-arith/src/lib.rs" \\
  || miss "missing critical file: arrow-arith/src/lib.rs"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 30 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~0d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/apache/arrow-rs"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Apache Arrow Rust is the official native Rust implementation of Apache Arrow columnar memory format and Apache Parquet file format, providing high-performance in-memory data structures and compute kernels for analytical workloads. It enables zero-copy data sharing between processes and efficient columnar operations like filtering, aggregation, and casting without serialization overhead. Workspace monorepo with 25+ crates under Cargo.toml: core arrow/ crate provides memory layout and base arrays; specialized crates like arrow-arith (arithmetic ops), arrow-cast (type conversions), arrow-csv (I/O), arrow-json (JSON readers); compute kernels separated into focused modules (aggregate.rs, arithmetic.rs, bitwise.rs); parquet/ is a separate top-level crate for file format. Tests and benchmarks colocated in each crate.

Data engineers, database developers, and analytics platform builders who need to process columnar data in Rust with Arrow interoperability—particularly those building data pipelines, query engines, or tools that must exchange data with Python/Java Arrow implementations.

Highly mature and production-ready. The Apache Arrow project is 7+ years old, this repository shows consistent commits across 25+ crates, extensive CI/CD in .github/workflows (arrow.yml, parquet.yml, integration.yml, miri.yaml for memory safety), and is actively maintained by ASF. Daily contributions visible in commit frequency.

Low risk for core functionality, moderate risk for cutting-edge features. Single dependency on Apache governance (not single maintainer—ASF umbrella project) reduces risk. Main concerns: large monorepo (Cargo.lock suggests many interdependencies), some crates like parquet-variant and parquet-geospatial are newer/experimental, and feature flag interactions require care (resolver = '2' explicitly enabled to prevent feature unification bugs).

Active development across multiple areas: Arrow Flight RPC (arrow-flight/), Parquet variant support (parquet-variant, parquet-variant-compute, parquet-variant-json), geospatial Parquet extensions (parquet-geospatial), memory audit tooling (audit.yml workflow), and miri safety checks (miri.yaml). Multiple parallel workflows indicate concurrent feature development.

git clone https://github.com/apache/arrow-rs.git && cd arrow-rs && cargo build && cargo test. No external services required for local development; uses standard Rust toolchain.

Daily commands: For library: cargo build --all. For tests: cargo test --all. For benchmarks: cargo bench --all. For specific crate: cargo test -p arrow-arith. CI runs: cargo clippy, cargo fmt, cargo miri to catch memory issues.

• arrow-array/src/lib.rs — Main entry point for the arrow-array crate; defines core array types and fundamental data structure abstractions that everything else depends on.
• arrow-array/src/array/mod.rs — Central module exporting all array implementations (primitive, binary, struct, union, etc.); essential to understand the type hierarchy.
• arrow-array/src/builder/mod.rs — Builder pattern implementations for constructing arrays; critical for understanding how data is loaded and validated.
• Cargo.toml — Workspace manifest defining all crates, features, and dependencies; required reading for build configuration and feature flags.
• arrow-arith/src/lib.rs — Core arithmetic operations module; shows how kernels are composed and how the library exports compute functionality.
• arrow-array/src/ffi.rs — Foreign Function Interface layer for Arrow C Data Interface; essential for understanding inter-language compatibility.
• README.md — Project overview, goals, and contribution guidelines; sets context for the entire Rust Arrow implementation.

Add a New Primitive Array Type

1. Create a new struct in arrow-array/src/array/ (e.g., my_type_array.rs) that implements core traits: Array, ArrowArray, and appropriate accessors. (arrow-array/src/array/my_type_array.rs)
2. Export the new type in arrow-array/src/array/mod.rs with pub use and add to the ArrowArray enum. (arrow-array/src/array/mod.rs)
3. Create a corresponding builder in arrow-array/src/builder/my_type_builder.rs following the builder pattern used by PrimitiveBuilder. (arrow-array/src/builder/my_type_builder.rs)
4. Export the builder in arrow-array/src/builder/mod.rs. (arrow-array/src/builder/mod.rs)
5. Add arithmetic kernels in arrow-arith/src/numeric.rs or appropriate module for operations specific to your type. (arrow-arith/src/numeric.rs)

Add a New Compute Kernel (Arithmetic Operation)

1. Determine the operation category (arithmetic, aggregate, boolean, numeric, or temporal) and open the corresponding file in arrow-arith/src/. (arrow-arith/src/arithmetic.rs)
2. Implement the kernel function following the naming convention (e.g., add, subtract, multiply) with generic type parameters for array types. (arrow-arith/src/arithmetic.rs)
3. Add comprehensive unit tests at the bottom of the same file testing edge cases, null handling, and overflow behavior. (arrow-arith/src/arithmetic.rs)
4. Export the new kernel in arrow-arith/src/lib.rs via pub use. (arrow-arith/src/lib.rs)
5. Document with examples showing common usage patterns. (arrow-arith/src/arithmetic.rs)

Add FFI Support for a New Array Type

1. Implement ArrowArrayRef trait and from_ffi for your array type in the array module, handling the Arrow C Data Interface struct layout. (arrow-array/src/array/my_type_array.rs)
2. Add the type's FFI conversion logic to the ffi module, mapping between Rust structs and C FFI pointers. (arrow-array/src/ffi.rs)
3. If the type supports chunked/streaming data, add support in arrow-array/src/ffi_stream.rs for the Arrow C Stream Interface. (arrow-array/src/ffi_stream.rs)
4. Write round-trip tests in the test module to verify FFI serialization and deserialization. (arrow-array/src/ffi.rs)

Enable or Add a Cargo Feature

1. Define the feature in Cargo.toml under [features] section, specifying required dependencies. (Cargo.toml)
2. Guard code with #[cfg(feature = "feature_name")] in the relevant modules (e.g., in arrow-array/src/lib.rs or specific operation files). (arrow-array/src/lib.rs)
3. Update CI workflow files (.github/workflows/arrow.yml) to test the new feature combination. (.github/workflows/arrow.yml)
4. Document the feature in README.md and CONTRIBUTING.md explaining when and why to use it. (README.md)

• Rust with unsafe for memory layout control — Arrow requires precise control over in-memory columnar layout and zero-copy semantics; Rust's type system and unsafe blocks enable safe abstraction while maintaining C ABI compatibility.
• Builder pattern for array construction — Provides ergonomic, type-safe construction while enforcing Arrow invariants (nullability, offset correctness, capacity management) at compile and runtime.
• Trait-based polymorphism (Array, ArrowArray, etc.) — Allows generic compute kernels to work across all array types without code duplication while maintaining strong type safety.
• C FFI modules (Arrow C Data/Stream Interface) — Enables seamless interoperability with other Arrow implementations (Python, C++, Go) without copying data.
• Modular crate structure (arrow-array, arrow-arith, arrow-parquet, etc.) — Allows users to depend on only needed functionality (lightweight for embedded use, full suite for analytics) and enables parallel development.

• Unsafe code for low-level buffer operations
  • Why: Arrow requires precise memory layout and zero-
  • Consequence: undefined

Feature flag interactions: must use resolver = '2' (already set) to avoid dev-dependencies leaking features into library builds. Some crates optional (parquet-geospatial, parquet-variant marked experimental). Miri can be slow—.github/workflows/miri.sh limits to specific targets. Arrow Flight code generation in arrow-flight/gen requires protobuf; protoc must be available. Memory safety is heavily tested (miri.yaml) so failing that test usually means real bug, not flake. Pre-commit hooks (.pre-commit-config.yaml) may be used locally—check CONTRIBUTING.md for expectations.

• Arrow Format / Columnar Memory Layout — This entire repo implements the Arrow columnar specification—understanding fixed-width columns, validity bitmaps, offsets buffers, and memory alignment is essential to reading any code here
• Zero-Copy Data Sharing — Arrow's key advantage is sharing data between languages without serialization; arrow-buffer and array designs are built around this principle using Rust's ownership to enforce safety
• Parquet Encoding (RLE, Dictionary, etc.) — parquet/ crate implements multiple compression and encoding schemes; understanding RLE, dictionary encoding, and delta encoding is critical for file format changes
• Validity Bitmaps / Null Handling — Arrow uses compact bitmaps for nullability rather than per-value null markers; this design decision appears throughout arrow-array and compute kernels and affects performance
• Unsafe Rust + Memory Safety Auditing (Miri) — Low-level operations in arrow-buffer require unsafe blocks for performance; the repo uses miri (memory safety checker) in CI to prevent UB—understanding safe abstractions over unsafe is a pattern here
• Type Coercion and Type System Dispatch — arrow-cast and compute kernels must handle dynamic type coercion; Rust's trait system (DataType enum, ArrowDataType trait) models Arrow's polymorphic types
• Flight RPC Protocol — arrow-flight/ implements Protobuf-based RPC for Arrow data exchange; used for distributed query execution and data sharing between services

• apache/arrow — Reference C++ implementation; Arrow Rust aims for spec parity and data interchange—Rust repo cross-tests with this via arrow-integration-testing crate
• apache/parquet-cpp — C++ Parquet implementation that this Rust version mirrors in functionality and spec compliance
• hyperdb/datafusion — Query execution engine built atop Arrow Rust; primary downstream consumer demonstrating real-world usage patterns
• pola-rs/polars — High-performance DataFrame library built using Arrow Rust as core—shows how arrow-rs integrates into larger analytical stacks

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive integration tests for arrow-flight serialization/deserialization

The repo has arrow-flight and arrow-integration-test packages, but the file structure shows no dedicated integration test files for flight protocol edge cases. Arrow Flight is critical for inter-process communication in Arrow ecosystems. New contributors could add tests for malformed messages, large payload handling, and protocol version compatibility across the integration test suite.

• [ ] Review existing tests in arrow-flight/tests/ and arrow-integration-test/src/
• [ ] Identify untested Flight message types and error scenarios
• [ ] Add test cases in arrow-integration-test/ for serialization round-trips with various data types
• [ ] Add tests for Flight protocol version negotiation edge cases
• [ ] Run existing CI workflows (arrow_flight.yml) to validate

Add specialized benchmarks for view array types in parquet read/write paths

The repo has arrow-array/benches/view_types.rs but no corresponding benchmarks in the parquet crate for view type I/O performance. View arrays (ByteViewArray, StringViewArray) are relatively new in Arrow and their parquet interop performance is critical. This PR would establish performance baselines for future optimization work.

• [ ] Create parquet/benches/view_array_io.rs with read/write benchmarks
• [ ] Add benchmarks for encoding/decoding StringView and ByteView columns
• [ ] Compare performance vs legacy BinaryArray to establish delta metrics
• [ ] Document findings in parquet/README.md if performance characteristics differ
• [ ] Integrate into rust.yml or create parquet-specific workflow trigger

Add missing feature-gated tests for arrow-cast type coercion with optional dependencies

The workspace uses resolver = "2" to prevent dev-dependencies from leaking features, but arrow-cast (a core casting crate) likely has incomplete test coverage for conditional feature paths. New contributors could add tests for casting operations when temporal, decimal, or other optional features are enabled/disabled.

• [ ] Review arrow-cast/Cargo.toml for optional features and dependencies
• [ ] Audit arrow-cast/src/ for #[cfg(feature = ...)] code paths without corresponding tests
• [ ] Add test module in arrow-cast/src/ with #[test] functions for each feature gate combination
• [ ] Use cargo test --features='' and cargo test --all-features to validate coverage
• [ ] Update .github/workflows/arrow.yml or arrow-cast-specific workflow to run feature-gated test matrix

• Add missing unit tests for bitwise operations in arrow-arith/src/bitwise.rs—currently sparse coverage for XOR/OR/AND edge cases with nulls
• Document the unsafe blocks in arrow-buffer/src—audit trail shows they're safe but lack inline comments explaining invariants; add SAFETY comments per Rust guidelines
• Implement missing Display trait implementations for error types across crates (arrow-cast, arrow-csv) to improve user debugging experience—low complexity, high impact

The apache/arrow-rs repository demonstrates generally good security practices as an Apache Foundation project, with appropriate use of workspace feature resolver v2 to prevent feature flag exploits. However, the primary concerns are: (1) incomplete SECURITY.md documentation which is critical for a security-conscious library, (2) inability to assess dependency security without seeing actual Cargo.lock content, and (3) standard Rust project configuration without visible advanced security hardening. The large, mature codebase across multiple interdependent packages suggests established security review processes, but they are not visible in the provided file structure. No hardcoded secrets, SQL injection risks, or obvious misconfigurations were detected in the available snippets.

• Medium · Incomplete Security Policy Documentation — SECURITY.md. The SECURITY.md file is incomplete and truncated at 'This document outlines the security model for the Rust implementation of Apache Arrow (arrow-rs) and how to report vulnerabilities. ## Security Model The arrow-rs project follows th'. This means critical security policy information is missing, including vulnerability reporting procedures, disclosure timelines, and supported versions. Fix: Complete the SECURITY.md file with comprehensive security policy information, including: clear vulnerability reporting process, responsible disclosure timeline, contact information for security issues, supported versions receiving security updates, and security model overview.
• Low · Workspace Feature Resolver Configuration — Cargo.toml (resolver = '2'). The workspace uses feature resolver version 2, which is good practice. However, the configuration comment indicates this was implemented to prevent dev-dependencies from affecting library features. While this is correct, ensure all crates properly validate their feature flag combinations. Fix: Continue enforcing strict feature flag validation across all workspace members. Consider adding CI checks to verify feature combinations don't introduce security issues (e.g., unsafe code being conditionally enabled).
• Low · Limited Visibility into Dependency Security — Cargo.lock, workspace member Cargo.toml files. While Cargo.lock is present (good for reproducibility), the actual dependency content and versions are not shown in the provided analysis. This makes it difficult to assess whether vulnerable dependencies are present. Fix: Implement continuous dependency scanning using tools like cargo-audit or cargo-deny in CI/CD pipeline. Run cargo audit regularly and keep dependencies updated. Document and justify any allowed vulnerabilities.
• Low · Incomplete Repository Security Visibility — .asf.yaml. The .asf.yaml configuration file is present but not shown. This file often contains security-related settings for GitHub integration, branch protection rules, and access controls. Fix: Verify .asf.yaml contains appropriate branch protection rules, requires status checks before merge, enforces code review requirements, and maintains strict access controls.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.