WAIT — Mixed signals — read the receipts

• Last commit 3d ago
• 7 active contributors
• Other licensed
• CI configured
• Tests present
• ⚠ Concentrated ownership — top contributor handles 58% of recent commits
• ⚠ Non-standard license (Other) — review terms

_{Computed from maintenance signals — commit recency, contributor breadth, bus factor, license, CI, tests}

Miller is a command-line data transformation tool that operates on key-value-pair data in formats like CSV, TSV, JSON, and JSON Lines—similar to how awk/sed/cut work on positional fields. The Go rewrite (main codebase) replaces the original C implementation, providing a performant, portable alternative to Unix text processing tools for named-field data manipulation. Monolithic single-binary project: cmd/mlr/main.go is the entry point; cmd/experiments/ contains prototyping code (CLI parser variants, line-parsing strategies); core logic appears in internal packages (inferred from Go pattern but exact internal/ structure not visible in provided file list). Multiple language implementations suggest modular format-handling code that could be ported.

Data engineers, sysadmins, and analysts who regularly process CSV/JSON/TSV files via CLI and want named-field operations (filtering, aggregation, stateful transforms) without writing awk scripts or Python one-liners. Contributors range from Go developers maintaining the core engine to documentation writers and test engineers.

Production-ready and actively maintained. The project has comprehensive CI/CD (GitHub Actions in .github/workflows/), a mature release process (.goreleaser.yml), extensive documentation (README-*.md files), and multi-language ecosystem (Go primary, but C/Ruby/Python/Zig variants exist). Recent activity evident in release workflow and dependabot configuration.

Standard open source risks apply.

Active development on Go port completion and feature parity with C version. Workflow files indicate ongoing testing (go.yml), release automation (.goreleaser.yml), and documentation generation (.readthedocs.yaml). Experiments directory suggests active work on CLI parsing and performance optimization (profiling README mentioned).

git clone https://github.com/johnkerl/miller.git
cd miller
make    # uses Makefile (visible in file list)
./mlr --version  # verify install

Refer to README-dev.md for development setup; requires Go 1.16+ (per go.mod).

Daily commands:

cd cmd/mlr
go run main.go --version
go run main.go --csv cat < input.csv  # example: cat CSV records

For production: go build -o mlr cmd/mlr/main.go produces the binary.

• cmd/mlr/main.go — Entry point for the Miller CLI application; all contributors must understand how command-line arguments flow into the core processing pipeline
• README.md — Defines Miller's purpose, scope, and primary use cases; essential for understanding what problems the codebase solves
• go.mod — Specifies Go version (1.16+) and minimal external dependencies (cliparse, go-shellquote, sys); critical for build and development environment setup
• Makefile — Build orchestration for compilation, testing, and release artifacts; every contributor must know how to build and test locally
• README-dev.md — Developer-focused documentation on architecture, build process, and contribution workflow; load-bearing for onboarding
• .github/workflows/go.yml — CI/CD pipeline that validates all commits; defines the test and build requirements every change must pass
• .goreleaser.yml — Release configuration for multi-platform binary distribution; critical for understanding how Miller reaches end-users

• cmd/mlr/main.go — CLI entry point; parses arguments, initial

Add a new data format parser

1. Create a new format handler in cmd/experiments/line_parser/ following the pattern in splitter.go (cmd/experiments/line_parser/splitter.go)
2. Implement record splitting logic for your format (e.g., Protocol Buffers, Avro, Parquet) (cmd/experiments/line_parser/scanner.go)
3. Register format in CLI argument parsing within cmd/mlr/main.go (cmd/mlr/main.go)
4. Add test fixtures to data/ directory and corresponding test cases (data/small.csv)

Add a new transformation verb (e.g., filter, group, aggregate)

1. Document the verb's usage and examples in docs/src/ (docs/src/10min.md.in)
2. Implement verb logic in the core processing engine (likely in Go code under cmd/) (cmd/mlr/main.go)
3. Add CLI flag/option handling in cmd/mlr/main.go or argument parser (cmd/experiments/cli_parser/cliparse.go)
4. Create test cases using sample data from data/ directory and verify via Makefile (Makefile)

Improve performance or add a new diagnostic tool

1. Create new utility binary in cmd/<tool-name>/main.go (e.g., cmd/scan, cmd/sizes pattern) (cmd/sizes/main.go)
2. Use data generators in data/generators/ to create large test datasets (data/generators/abixy.c)
3. Add build target to Makefile and profiling steps (Makefile)
4. Document findings and performance improvements in README-profiling.md (README-profiling.md)

• Go 1.16+ — Cross-platform compilation to single binary; fast startup and memory efficiency for CLI tool; strong built-in string and I/O handling for data processing
• cliparse (CLI parsing library) — Handles complex command-line argument parsing with subcommands and flags; reduces boilerplate for building intuitive CLI interfaces
• go-shellquote — Safely parses shell-quoted arguments and escape sequences; critical for correct handling of user input with spaces and special characters
• GitHub Actions CI/CD — Automated testing and release pipeline on multiple OS/architecture combinations (Linux, macOS, Windows); .goreleaser integration enables one-command multi-platform releases

• Streaming record-by-record processing over loading entire dataset into memory

  • Why: Miller targets Unix-like environments where piping and stream processing are fundamental; enables processing of files larger than available RAM
  • Consequence: Some operations (e.g., global sorting, aggregation) may require buffering; multi-pass algorithms are harder to implement

• Named fields (key-value data) as primary abstraction over positional indices

  • Why: Aligns with natural CSV/JSON/TSV semantics; eliminates need for manual column counting; self-documenting transformations
  • Consequence: Loses positional optimizations available in awk; requires field lookups in each record

• Single-binary distribution with minimal external dependencies (only 2 non-stdlib deps)

  • Why: Maximizes portability, simplifies deployment, reduces supply chain risk, and ensures predictable performance
  • Consequence: Complex features (e.g., native SQL support, advanced ML) deferred to user composition with external tools

• Experimental code in cmd/experiments/ (line_parser, cli_parser) kept separate from main

  • Why: Allows iterating on new parsing strategies without destabilizing stable CLI
  • Consequence: Main CLI may lag behind experimental improvements; requires deliberate promotion of stable features

• Real-time streaming analytics (batch/pipeline-oriented, not event-stream processing)
• SQL database replacement (in-process data transformation only, no persistence layer)
• Distributed computation (single-machine tool, designed for Unix pipes and composition)
• Authentication/authorization (data transformation tool, not a service)
• GUI interface (command-line only, intentionally minimal surface area)

No Go modules in main: cmd/experiments/cli_parser/ has its own go.mod—ensure you're building from cmd/mlr/, not root. Parallel C codebase: The C implementation still exists; bugs filed may reference C behavior as 'expected'—check README-go-port.md for known divergences. Data path assumptions: If testing with data/ generators (abixy.c, fuzz.rb), they may require compilation or Ruby runtime. Platform-specific code: cmd/experiments/cli_parser/arch/ shows Windows vs. non-Windows branching; test on target platform. Documentation lag: Docs may reference C version features not yet ported to Go.

• Insertion-Ordered Hash Maps (IOM) — Miller's core data structure: unlike arrays (awk) or maps (generic), IOMs preserve field order from input while allowing O(1) named-field access, enabling Miller to transform CSV seamlessly
• Streaming Data Processing — Miller processes one record at a time without loading entire files into memory, critical for large datasets; see README for 'on the fly' transformations
• Format-Agnostic Record Manipulation — Miller abstracts CSV/JSON/TSV differences so the same filter/transform/aggregate logic works across formats; core abstraction in the codebase
• Lexer/Parser Patterns — cmd/experiments/line_parser/ and cli_parser/ show tokenization and syntax parsing; essential for Miller's DSL (user-defined aggregations and filters)
• Cross-Platform Binary Delivery — .goreleaser.yml automates building for Linux/macOS/Windows; critical for a CLI tool targeting sysadmins on heterogeneous infrastructure
• Pluggable Codec Architecture — Miller's design allows new format readers/writers to be added without modifying core logic; inferred from multi-format support and separate codec implementations
• Stateful Record Aggregation — Miller can compute running sums, counts, min/max across records (e.g., group-by + aggregate); requires maintaining state across streaming records

• stedolan/jq — Similar domain (streaming JSON transformation via CLI), but jq is JSON-only while Miller handles multiple formats; jq is inspiration for Miller's design
• BurntSushi/xsv — Alternative CSV-focused CLI tool in Rust; solves overlapping problems but less general than Miller
• aws/amazon-athena-query-federation — Ecosystem companion; Athena users can query Miller-formatted data via federation connectors
• nushell/nushell — Modern shell with structured data pipeline support; philosophical cousin to Miller for tabular data processing
• wfxr/csview — Lightweight CSV viewer; uses Miller-like concepts but simpler scope (display vs. transformation)

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add integration tests for Go CLI parser in cmd/experiments/cli_parser

The cli_parser experiment has platform-specific implementations (args_notwindows.go and args_windows.go) but no corresponding test files. This is critical for a CLI tool like Miller where argument parsing must work correctly across Windows, Linux, and macOS. New contributor could write tests to validate the cliparse module handles edge cases like quoted arguments, escape sequences, and platform-specific path separators.

• [ ] Create cmd/experiments/cli_parser/cliparse_test.go with unit tests for the cliparse module
• [ ] Add platform-specific tests in cmd/experiments/cli_parser/arch/args_windows_test.go and args_notwindows_test.go
• [ ] Test edge cases: quoted strings, escape sequences, empty arguments, and special characters
• [ ] Integrate tests into GitHub Actions workflow (reference: .github/workflows/go.yml) to run on multiple OS

Add Go benchmarks for Miller's data format parsers

Miller handles multiple data formats (CSV, TSV, JSON, DKVP) but there are no visible benchmark files to measure parser performance across formats. This is valuable for performance-sensitive operations on large datasets. New contributor could add benchmarks in cmd/mlr/ to compare parsing speed and identify bottlenecks.

• [ ] Create benchmark files for CSV, JSON, and DKVP parsers in appropriate subdirectories under cmd/mlr
• [ ] Use Go's testing.B framework with realistic datasets from data/ directory (e.g., data/het.csv, data/small.csv)
• [ ] Add 'go test -bench' targets to Makefile for easy local benchmarking
• [ ] Document benchmark results in README-profiling.md with instructions for contributors to run benchmarks

Complete missing documentation for Go port in README-go-port.md

README-go-port.md exists but the repo README snippet shows Miller is being ported to Go. The file likely documents the migration status but may have gaps on feature parity, known limitations, or implementation architecture. New contributor could audit feature completeness by comparing original Miller functionality against Go port and document missing features or TODOs.

• [ ] Review README-go-port.md and identify sections marked as incomplete or TODO
• [ ] Cross-reference cmd/mlr/main.go with known Miller features (transformations, formats, filters)
• [ ] Create a feature matrix table showing which Miller features are implemented, in-progress, or planned in the Go port
• [ ] Add architecture overview explaining how Go port maps to original Miller's module structure
• [ ] Link to specific GitHub issues for each missing feature to guide future contributors

• Add more test cases for CSV edge cases (empty fields, quoted commas, CRLF line endings) in Go implementation; the data/generators/ directory suggests test data exists but test coverage may be incomplete.
• Write a missing format handler for a popular data format (e.g., YAML, Protocol Buffers) by following the pattern of existing CSV/JSON/TSV readers; cmd/experiments/line_parser/ shows line-splitting strategies that could be reused.
• Improve CLI help text and add examples for lesser-used subcommands by updating cmd/mlr/main.go help strings and cross-referencing docs/src/ so mlr --help matches online documentation.

The Miller codebase shows a reasonable security posture with version control, CI/CD automation, and organized project structure. However, the main concerns are outdated dependencies (Go 1.16, golang.org/x/sys v0.1.0) that no longer receive security updates. The go-shellquote dependency, while functional for parsing, is also dated. The project handles data processing (CSV, JSON, etc.) which could present injection risks if user input is not properly validated, though no direct code review was performed. Immediate attention should be given to updating Go to a supported LTS version and dependencies. The absence of a SECURITY.md file should be addressed for responsible vulnerability disclosure.

• High · Outdated golang.org/x/sys Dependency — cmd/experiments/cli_parser/go.mod. The go.mod file specifies golang.org/x/sys v0.1.0, which is significantly outdated. This package contains low-level system interactions and has likely received security patches in newer versions. Current stable versions are much newer (v0.15.0+). Fix: Update golang.org/x/sys to the latest stable version (0.15.0 or newer). Run 'go get -u golang.org/x/sys' and test thoroughly.
• Medium · Outdated go-shellquote Dependency — cmd/experiments/cli_parser/go.mod. The kballard/go-shellquote dependency is pinned to v0.0.0-20180428030007-95032a82bc51, which is from 2018 (over 6 years old). While shellquote is relatively simple, outdated dependencies increase attack surface and should be kept current. Fix: Review and update go-shellquote to the latest available version. Verify compatibility before deploying.
• Medium · Outdated Go Version Target — cmd/experiments/cli_parser/go.mod. The go.mod specifies Go 1.16, which reached end-of-life in September 2022. This version no longer receives security updates from the Go team. Fix: Update the Go version to 1.21 or later (current stable releases are 1.22+). Update the 'go' directive in go.mod and test the codebase with the new version.
• Low · Missing SECURITY.md File — Repository root. There is no visible SECURITY.md file in the repository root. This file typically provides security reporting guidelines and allows for responsible disclosure of vulnerabilities. Fix: Create a SECURITY.md file that includes instructions for reporting security vulnerabilities privately, security contact information, and any security policies.
• Low · Limited SBOM Visibility — .github/workflows/. While the repository uses Go modules (go.mod/go.sum), there is no evidence of SBOM (Software Bill of Materials) generation or dependency scanning configuration in the provided GitHub workflows or repository structure. Fix: Consider adding automated dependency scanning (e.g., Dependabot, Snyk, or similar) and SBOM generation to CI/CD pipelines for better supply chain security.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/johnkerl/miller shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live johnkerl/miller repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/johnkerl/miller.

What it runs against: a local clone of johnkerl/miller — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in johnkerl/miller | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 33 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of johnkerl/miller. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/johnkerl/miller.git
#   cd miller
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of johnkerl/miller and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "johnkerl/miller(\\.git)?\\b" \\
  && ok "origin remote is johnkerl/miller" \\
  || miss "origin remote is not johnkerl/miller (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "cmd/mlr/main.go" \\
  && ok "cmd/mlr/main.go" \\
  || miss "missing critical file: cmd/mlr/main.go"
test -f "README.md" \\
  && ok "README.md" \\
  || miss "missing critical file: README.md"
test -f "go.mod" \\
  && ok "go.mod" \\
  || miss "missing critical file: go.mod"
test -f "Makefile" \\
  && ok "Makefile" \\
  || miss "missing critical file: Makefile"
test -f "README-dev.md" \\
  && ok "README-dev.md" \\
  || miss "missing critical file: README-dev.md"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 33 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~3d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/johnkerl/miller"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.