GO — Healthy across the board

• Last commit 1d ago
• 12 active contributors
• Distributed ownership (top contributor 29% of recent commits)
• MIT licensed
• CI configured
• ⚠ No test directory detected

_{Computed from maintenance signals — commit recency, contributor breadth, bus factor, license, CI, tests}

Sarama is a pure Go client library for Apache Kafka that handles all protocol-level communication, message production/consumption, cluster metadata management, and transaction coordination. It implements the complete Kafka wire protocol (v0–v49+) allowing Go applications to produce, consume, and administer Kafka clusters without JVM dependencies. Monolithic single-package design: root directory contains ~200 .go files organized by Kafka API (acl_.go, add_offsets_to_txn_.go, etc.) with corresponding _test.go files, separate mocks/ subpackage for testing helpers, examples/ directory with runnable producer/consumer/admin samples, and tools/ for CLI utilities. No internal package structure; all public APIs exposed at github.com/IBM/sarama level.

Go backend engineers building event-driven systems, microservices, or data pipelines who need native Kafka integration without wrapping the Java client. DevOps/SRE teams managing Kafka infrastructure via Go tools.

Production-ready and actively maintained by IBM. Sarama is 2M+ lines of Go code with extensive test coverage (test files visible across every feature module), comprehensive CI/CD via 20+ GitHub workflows (ci.yml, fvt.yml, fuzz.yml, CodeQL), and a "2 releases + 2 months" compatibility guarantee for Kafka and Go versions. Regular dependency updates via Renovate and security scanning (WhiteSource, OpenSSF Scorecard).

Low risk for active Kafka use cases, but moderate dependency footprint: requires gokrb5 (Kerberos), compression libraries (lz4, klauspost/compress), and go-metrics for instrumentation. Breaking changes follow semver strictly (last major bump visible in CHANGELOG.md), but older Kafka brokers may have protocol gaps. Single-maintainer perception mitigated by IBM backing and active community contributions visible in wiki + Google Groups.

Active dependency maintenance (Renovate config present, Go 1.25.0 target visible in examples), security auditing (OpenSSF scorecard integration, CodeQL workflows), and protocol compatibility expansion. Workflows show focus on FVT (Functional Verification Testing) against multiple Kafka versions (fvt.yml, fvt-main.yml, fvt-pr.yml) and fuzz testing (fuzz.yml).

git clone https://github.com/IBM/sarama.git
cd sarama
make  # Review Makefile for targets
go mod download
go test ./...

Refer to examples/ directory for runnable demos: cd examples/consumer && go run consumer.go --brokers=localhost:9092.

Daily commands: Dev environment: make test runs Go test suite; make lint runs golangci-lint (configured in .golangci.yml); make gofmt checks formatting. Local Kafka: Use Dockerfile.kafka or Vagrantfile to spin up test cluster. Integration tests: GitHub workflows show FVT runs against real Kafka brokers (see .github/workflows/fvt.yml for matrix config).

• client.go — Core client implementation managing broker connections and cluster metadata; entry point for all Kafka interactions.
• broker.go — Broker communication layer handling request/response serialization and connection pooling to Kafka brokers.
• async_producer.go — Primary producer implementation with batching, partitioning, and reliability guarantees; critical for understanding message publishing flow.
• consumer_group.go — Consumer group coordinator managing rebalancing, offset management, and multi-partition consumption.
• admin.go — Administrative API interface for topic/partition management, ACLs, and cluster administration.
• config.go — Configuration schema and validation for all client, producer, and consumer settings; essential for understanding runtime behavior.
• compress.go — Compression/decompression abstraction supporting gzip, snappy, lz4, and zstd; handles codec negotiation.

Add a new Kafka admin API operation

1. Create request struct in new file following naming pattern *_request.go with encoder methods (alter_configs_request.go)
2. Create response struct in *_response.go with decoder methods (alter_configs_response.go)
3. Add method to Admin interface in admin.go that calls broker.FetchMetadata or other broker methods (admin.go)
4. Register API key in api_versions.go and add version matrix (api_versions.go)
5. Create *_test.go with request/response encoding/decoding tests (alter_configs_request_test.go)

Add support for a new message compression codec

1. Add CompressionCodec constant and codec name in compress.go (compress.go)
2. Add encoder function (e.g., compressGzip) in compress.go (compress.go)
3. Add decoder function in decompress.go (decompress.go)
4. Update config.go to include codec in Producer.Compression options (config.go)
5. Add test cases in compress.go and decompress.go test files (compress.go)

Implement a custom partition assignment strategy

1. Create new struct implementing BalanceStrategy interface (Plan, Name methods) (balance_strategy.go)
2. Implement Plan method that distributes partitions across members (balance_strategy.go)
3. Register strategy in consumer_group.go's defaultBalanceStrategyName map (consumer_group.go)
4. Add tests validating partition distribution and fairness (balance_strategy_test.go)

Add a new consumer group event or callback hook

1. Add event type struct (e.g., RebalanceStart) in consumer_group.go (consumer_group.go)
2. Define callback function signature in config.go Consumer.* struct (config.go)
3. Invoke callback at appropriate lifecycle point in consumer_group.go handleSessionStateChange or sync loop (consumer_group.go)
4. Test callback invocation in consumer_group_test.go (consumer_group_test.go)

• Go — Goroutines and channels enable efficient async I/O and concurrent connection pooling; lightweight concurrency for high throughput
• Apache Kafka protocol (binary wire format) — Direct protocol implementation enables full feature support, tight version control, and zero-copy optimizations
• SASL/SCRAM, TLS/SSL — Enterprise security requirements for authentication and encryption in Kafka deployments
• Multiple compression codecs (gzip, snappy, lz4, zstd) — Trade-offs between compression ratio, CPU cost, and latency; codec choice per topic/message
• Consumer group coordination protocol — Stateful rebalancing with configurable strategies (RoundRobin, Range, Sticky, Custom) matching diverse consumption patterns

• Async producer with callback channels vs sync producer with blocking send

  • Why: Async batching maximizes throughput and broker efficiency; sync mode simplifies error handling but reduces performance
  • Consequence: Developers must handle errors via callbacks; callback ordering guarantees depend on partition/broker affinity

• Consumer offset stored in Kafka (__consumer_offsets) vs external store

  • Why: Kafka-managed offsets provide durability and atomicity with consumer group semantics; external stores require custom coordination
  • Consequence: Tightly coupled to Kafka cluster; offset resets require broker coordination

• Connection multiplexing with request queuing per broker vs dedicated per-operation connections

  • Why: Multiplexing reduces file descriptor overhead and leverages persistent TCP connections
  • Consequence: Head-of-line blocking on slow requests; requires careful timeout tuning

• Partition assignment strategies pluggable via interface

  • Why: Enables custom strategies (Sticky, Custom) without core library changes; matches diverse scaling patterns
  • Consequence: Added complexity in consumer group rebalancing logic; must implement Plan() correctly

• Transactional producer with idempotent writes

  • Why: Prevents duplicates and out-of-order messages under failures; required for exactly-once semantics
  • Consequence: Increased latency from transactional bookkeeping; coordinator discovery overhead

• Does not manage topic creation/deletion workflows (admin APIs present, orchestration is caller's responsibility)
• Does not provide query/stream processing (see Kafka Streams for that; sarama is transport-only)
• Does not implement leader election or cluster management (pure client library)
• Does not handle schema management (no Avro/Protobuf codecs; messages are opaque bytes)
• Does not provide

Kafka broker compatibility: Sarama sends protocol versions in HandshakeRequest; older brokers may reject newer client versions—check CHANGELOG for breaking protocol changes. Consumer group rebalancing: Default rebalance strategy is RoundRobin; changing it requires config.Consumer.Group.Rebalance.Strategy tuning. Offset management: Offsets are auto-committed by default with 1s interval; disable via config.Consumer.Offsets.AutoCommit.Enable = false to prevent data loss on crash. TLS/Kerberos setup: Requires proper config.Net.TLS.Config and config.Net.SASL.* setup; missing cipher suites or KDC config will silently fail auth. Go module replace directive: Examples use replace github.com/IBM/sarama => ../../ to test local changes; remove before publishing.

• Kafka Wire Protocol — Sarama's entire codebase is built on marshaling/unmarshaling Kafka's binary protocol; understanding request/response structures is essential to reading any *_request.go or *_response.go file
• Consumer Group Coordination — Sarama handles partition assignment, offset management, and rebalancing transparently; grasping the JoinGroup, SyncGroup, and Heartbeat protocol flow is critical for consumer reliability
• Broker Metadata and Leader Election — Sarama maintains partition replica leadership state and redirects requests accordingly; this drives routing logic and failover behavior
• Transactional Writes (Idempotence + Atomic Multi-Partition Commits) — Sarama implements full Kafka transaction protocol (AddPartitionsToTxn, AddOffsetsToTxn, WriteTxnMarker); essential for exactly-once semantics in streaming pipelines
• SASL/Kerberos Authentication — Sarama integrates gokrb5 for enterprise auth; understanding broker credential negotiation is required for secure cluster access
• Codec Compression (Snappy, LZ4, Gzip, Zstd) — Sarama handles transparent compression in message batches via dependencies like klauspost/compress and pierrec/lz4; choosing codecs trades CPU vs network I/O
• Offset Commit Strategies (Auto vs Manual, Broker vs Zookeeper) — Sarama's offset management is a major source of data loss bugs; understanding when and how offsets are committed affects delivery guarantees

• segmentio/kafka-go — Alternative pure-Go Kafka client with simpler API surface; choose Sarama for feature completeness and transactions, kafka-go for minimal dependencies
• confluentinc/confluent-kafka-go — CGo wrapper around librdkafka; higher performance but requires C compiler; Sarama is pure Go alternative
• apache/kafka — Official Kafka repository and protocol specification source; Sarama implements this protocol; reference for version compatibility matrix
• eapache/go-resiliency — Circuit breaker and retry logic used internally by Sarama; understand this to customize broker connection resilience
• Shopify/sarama-cluster — Deprecated wrapper adding consumer group coordination helpers; Sarama now includes this functionality natively

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add comprehensive integration tests for ACL operations in fvt-pr.yml workflow

The repo has extensive ACL-related files (acl_.go, acl__test.go) but the FVT (Functional Verification Test) workflows (fvt.yml, fvt-pr.yml, fvt-main.yml) lack documented ACL-specific test coverage. Adding dedicated ACL integration tests would ensure ACL create/delete/describe/alter operations work correctly against real Kafka brokers in PR validation, catching regressions early.

• [ ] Review existing acl_*_test.go files to understand current unit test coverage
• [ ] Create a new test file or section in the FVT workflow for ACL-specific scenarios (e.g., CreateACL, DescribeACL with filters, DeleteACL)
• [ ] Add test cases to fvt-pr.yml that validate ACL operations against a Kafka test cluster
• [ ] Document expected ACL behavior in CONTRIBUTING.md or a test guide

Add unit tests for alter_user_scram_credentials_request.go and alter_user_scram_credentials_response.go

The file structure shows alter_user_scram_credentials_request.go and alter_user_scram_credentials_response.go exist, but there are no corresponding *_test.go files listed. This is inconsistent with other request/response pairs (e.g., alter_configs_request_test.go, acl_create_request_test.go) and represents a gap in test coverage for SCRAM credential management.

• [ ] Create alter_user_scram_credentials_request_test.go with tests for serialization/deserialization
• [ ] Create alter_user_scram_credentials_response_test.go with tests for response parsing and error handling
• [ ] Follow existing test patterns from alter_configs_request_test.go or acl_create_request_test.go
• [ ] Verify tests pass in the ci.yml workflow

Add i386 architecture test coverage for compression libraries (klauspost/compress, lz4)

The i386.yml workflow exists for 32-bit testing, but there's no documented validation that compression-heavy dependencies (klauspost/compress v1.18.3, pierrec/lz4/v4 v4.1.25) work correctly on 32-bit systems. Sarama's consumer/producer heavily rely on compression, making 32-bit compatibility critical for embedded or legacy systems.

• [ ] Review .github/workflows/i386.yml to understand current 32-bit test scope
• [ ] Add specific compression codec tests (gzip, snappy, lz4, zstd) to i386.yml to validate 32-bit compatibility
• [ ] Test producer/consumer message compression flows on 386 architecture
• [ ] Document any architecture-specific limitations in README.md or CONTRIBUTING.md

• Add integration tests for add_partitions_to_txn_request.go and add_partitions_to_txn_response.go—files exist but no _test.go coverage visible in top 60 file list: Medium: Expands transaction protocol test matrix and improves confidence in stateful features
• Document Kerberos/SASL configuration with a worked example in examples/ directory (e.g., examples/kerberos_consumer.go) since auth setup is listed as a hidden trap: Medium: Reduces onboarding friction for users needing enterprise authentication; visible gap in examples/
• Add benchmark comparison tests (bench_test.go) for consumer group rebalance strategies (RoundRobin vs Range) using rcrowley/go-metrics to measure latency and throughput: Low-Medium: Helps users choose optimal strategy; leverages existing go-metrics dependency and benchmarking infrastructure

The Sarama library demonstrates a strong security posture with proper security disclosure practices (SECURITY.md), OpenSSF scorecard badge, and best practices compliance. However, there are concerns with the example application's build configuration (invalid Go version specification) and dependency management practices. The infrastructure configuration (docker-compose) lacks explicit security hardening measures such as resource limits and network segmentation. No hardcoded secrets or injection vulnerabilities were detected in the provided file structure. Recommended actions: (1) Fix the Go version to a valid release, (2) Ensure regular dependency updates through automated tools, (3) Add security-focused configuration to deployment manifests.

• Medium · Outdated Go Version Specified — examples/consumer/go.mod. The go.mod file specifies 'go 1.25.0' which is not a valid Go release version. Current Go versions are 1.21.x, 1.22.x, etc. This indicates either a typo or use of an unreleased/invalid version, which could cause build inconsistencies and prevent use of security patches. Fix: Update to a stable, supported Go version (e.g., 'go 1.22' or 'go 1.23'). Verify against the main sarama module's go.mod requirement.
• Medium · Potential Dependency Version Pinning Issue — examples/consumer/go.mod - indirect dependencies. Several indirect dependencies use older versions that may contain security vulnerabilities: golang.org/x/crypto v0.48.0 (consider upgrading to latest), golang.org/x/net v0.51.0. While these are indirect dependencies managed by sarama, keeping them updated is important for security. Fix: Run 'go get -u' or use 'go get -u golang.org/x/crypto' and 'go get -u golang.org/x/net' to update to latest patch versions. Configure dependabot or renovate to automatically update indirect dependencies.
• Low · Docker Compose Services May Require Network Segmentation — docker-compose.yml. The docker-compose.yml shows Kafka and Zookeeper services with 'restart: always' policy without explicit network policies or resource limits defined in the snippet. This could lead to uncontrolled resource consumption in production-like environments. Fix: Add resource limits (memory, CPU), configure restart policies with limits, and implement network policies. Use a 'networks' section to isolate services and control inter-service communication.
• Low · Zookeeper Configuration Allows Unlimited Client Connections — docker-compose.yml - zookeeper-base environment. The Zookeeper configuration sets 'ZOO_MAX_CLIENT_CNXNS: 0' which means unlimited client connections per IP. While 0 is a common default, this could be exploited for resource exhaustion attacks in untrusted network environments. Fix: Consider setting ZOO_MAX_CLIENT_CNXNS to a reasonable limit based on expected load (e.g., '100' or '500') when deploying in production environments, and implement network-level rate limiting.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/IBM/sarama shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live IBM/sarama repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/IBM/sarama.

What it runs against: a local clone of IBM/sarama — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in IBM/sarama | Confirms the artifact applies here, not a fork | | 2 | License is still MIT | Catches relicense before you depend on it | | 3 | Default branch main exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 31 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of IBM/sarama. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/IBM/sarama.git
#   cd sarama
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of IBM/sarama and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "IBM/sarama(\\.git)?\\b" \\
  && ok "origin remote is IBM/sarama" \\
  || miss "origin remote is not IBM/sarama (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(MIT)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"MIT\"" package.json 2>/dev/null) \\
  && ok "license is MIT" \\
  || miss "license drift — was MIT at generation time"

# 3. Default branch
git rev-parse --verify main >/dev/null 2>&1 \\
  && ok "default branch main exists" \\
  || miss "default branch main no longer exists"

# 4. Critical files exist
test -f "client.go" \\
  && ok "client.go" \\
  || miss "missing critical file: client.go"
test -f "broker.go" \\
  && ok "broker.go" \\
  || miss "missing critical file: broker.go"
test -f "async_producer.go" \\
  && ok "async_producer.go" \\
  || miss "missing critical file: async_producer.go"
test -f "consumer_group.go" \\
  && ok "consumer_group.go" \\
  || miss "missing critical file: consumer_group.go"
test -f "admin.go" \\
  && ok "admin.go" \\
  || miss "missing critical file: admin.go"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 31 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~1d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/IBM/sarama"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.