Mixed — Slowing — last commit 4mo ago

• Last commit 4mo ago
• GPL-3.0 licensed
• Tests present
• ⚠ Slowing — last commit 4mo ago
• ⚠ Solo or near-solo (1 contributor active in recent commits)
• ⚠ GPL-3.0 is copyleft — check downstream compatibility
• ⚠ No CI workflows detected
• ⚠ Scorecard: default branch unprotected (0/10)

_{Computed from maintenance signals — commit recency, contributor breadth, bus factor, license, CI, tests, cross-checked against OpenSSF Scorecard}

mtx is a command-line rapid prototyping toolkit for Information Retrieval and Machine Learning that represents almost everything (documents, queries, rankings, networks) as sparse matrices and performs experiments through memory-mapped matrix operations. It trades memory efficiency for disk I/O, allowing Wikipedia-scale datasets to be processed interactively on a laptop by keeping matrices on disk and streaming them in small chunks rather than loading into RAM. Monolithic single-directory C codebase: core matrix/IR operations in *.c files (matrix.c, mtx.c, cache.c, compress.c, hash.c, kvs.c, cluster.c), utility modules for compression and hashing, GPU acceleration via cumtx.cu, and shell/awk scripts in scripts/ directory for gluing operations together. Memory-mapped I/O (mmap.c/h) and custom hash tables (hash.c, hashf.c) provide the low-level machinery; Makefile orchestrates compilation.

LLM-derived; treat as a starting point, not verified fact.

Unix-savvy IR researchers and ML practitioners who prototype novel algorithms (BM25 variants, PageRank combinations, clustering approaches) and need to iterate quickly on large datasets without writing custom C code or managing Matlab licenses. Also targets academic groups building ad-hoc data pipelines where tab-completion and shell history matter more than GUI niceties.

LLM-derived; treat as a starting point, not verified fact.

Mature and stable but dormant: the codebase is substantial (~900KB C, well-organized modules like cache.c, compress.c, matrix.c) suggesting years of refinement, but no recent activity is evident. No CI/CD files visible (no .github, no Travis/GitHub Actions), no test directory, and the single maintainer (v-lavrenko) pattern suggests low bus-factor risk is outweighed by maintenance risk. Production-ready for the specific use case it was designed for, but not actively developed.

Single-author, dormant repo with no visible dependency manifest (no package.json, no requirements.txt), meaning security vulnerabilities in upstream C libraries (curl.c, compress.c) would require manual patching. The codebase mixes C (916KB), AWK, shell, Python, and CUDA without visible integration tests or CI, making refactoring dangerous. Heavy reliance on POSIX shell scripting (scripts/ directory) and awk (email2xml.awk, evl2recall.awk) means breakage on non-Unix platforms is undetected.

LLM-derived; treat as a starting point, not verified fact.

No recent activity visible from file list alone. The repo structure suggests a completed, mature tool that is feature-complete and maintained reactively. Last commits, open issues, and PR counts cannot be inferred from the provided metadata.

LLM-derived; treat as a starting point, not verified fact.

Clone and build: git clone https://github.com/v-lavrenko/yari.git && cd yari && make. Binaries will be created by the Makefile. Interactive usage: mtx (bash tab-completion recommended). Example: zcat docs.gz | mtx load:xml DOCS to import data.

Daily commands: Interactive: ./mtx after make. One-off commands: mtx load:xml DOCS < file.xml, mtx multiply A B C, mtx transpose DOCS DOCS_T. Batch scripting via Makefile (k-means.csh example visible). See scripts/ directory for domain-specific pipelines (batch-gemini.py for Gemini API integration).

• matrix.h — Core sparse matrix abstraction that represents documents, queries, indices, and rankings—the foundational data structure for all IR/ML operations.
• matrix.c — Implementation of sparse matrix operations, storage, and serialization; handles the heaviest computational workloads.
• mtx.c — Main CLI entry point and orchestrator for command dispatch, input parsing, and matrix pipeline execution.
• cache.h — Cache abstraction that enables yari's core design principle of avoiding re-computation by persisting intermediate results.
• hash.h — Hash table implementation used throughout for vocabulary, document mapping, and fast lookups in IR workflows.
• query.h — Query parsing and execution interface that bridges CLI input to matrix operations and ranking algorithms.
• README.md — Design philosophy and quick-start guide explaining yari's Unix-hacker ethos and matrix-centric IR/ML approach.

• Matrix Engine (C, mmap, custom sparse format) — Stores, loads, iterates, and performs arithmetic on sparse matrices (transpose, multiply, normalize).
  • Failure mode: Out-of-memory on very large matrices; graceful degradation to disk-based operations via mmap.
• Cache Layer (C, disk I/O, hash-based lookup) — Saves intermediate computation results to disk; retrieves on subsequent queries to avoid re-running expensive algorithms.
  • Failure mode: Cache invalidation bugs if algorithm parameters change; user must manually clear stale cache entries.
• Query Executor (C, CLI argument parsing) — Parses user command, orchestrates algorithm selection, and pipes results through output formatters.
  • Failure mode: Malformed queries crash or produce silent errors; error messages are minimal.
• Ranking/Clustering Algorithms (C, matrix operations) — Implement IR scoring functions (BM25, vector space, PageRank) and unsupervised learning (K-means, HAC).
  • Failure mode: Numerical instability on edge cases (zero vectors, sparse data); no automatic regularization.
• Evaluation Metrics (C, matrix iteration) — Computes IR quality measures (MAP, NDCG, P@k) against relevance judgments stored as sparse matrices.
  • Failure mode: Assumes standard evaluation set format; incompatible formats silently produce garbage scores.
• Data Transformation Scripts (AWK, shell, awk arrays) — AWK/shell utilities convert between formats (TSV, RCV, JSON) and perform standard text operations (tokenization, grouping, aggregation).
  • Failure mode: Memory limits on large files; slow on very large datasets; no streaming aggregation.

• User shell → mtx CLI — Command-line query with algorithm name, parameters, and input/output file paths.
• mtx CLI → Cache layer — CLI checks if cached result exists for given command+parameters.
• Cache miss → Matrix Engine → Ranking/Clustering Algorithm — Engine loads sparse matrix from disk, algorithm computes scores/clusters.
• Algorithm output → Cache layer — Result matrix persisted to disk cache for reuse in future queries.
• Cached/computed matrix → Output formatters — Result piped through AWK scripts or formatters to TSV/JSON for downstream analysis.

Add a new ranking algorithm

1. Create new .c file in root (e.g., bm25.c) implementing your ranking function (bm25.c (new))
2. Add corresponding .h header with function signature (bm25.h (new))
3. Register command dispatch in mtx.c's main command switch (mtx.c)
4. Use matrix.h sparse matrix API to iterate docs/terms and compute scores (matrix.h)

Add a new data transformation script

1. Create .awk or .sh script in scripts/ following naming convention (e.g., scripts/normalize.awk) (scripts/normalize.awk (new))
2. Use standard input/output and Unix pipes for streaming data (scripts/normalize.awk (new))
3. Reference in Makefile if it's a critical workflow component (Makefile)

Add a new evaluation metric

1. Add metric computation function to eval.c (eval.c)
2. Export function signature in query.h if metric is query-dependent (query.h)
3. Call metric function from query execution pipeline in query.c (query.c)
4. Test with standard IR collections (RCV, TREC) using scripts like rcv_eval.awk (scripts/rcv_eval.awk)

• Sparse matrix representation — Enables compact storage of high-dimensional, low-density IR/ML data (documents, queries, features) and makes algorithms naturally scale to Wikipedia-sized datasets.
• Unix CLI + shell pipes — Allows interactive, exploratory workflows with tab-completion and command history; users can debug mid-pipeline using awk/perl without restarting long jobs.
• Result caching layer — Core design principle: never recompute intermediate results, enabling rapid iteration on expensive algorithms without losing progress.
• C for core algorithms — Provides performance necessary for Wikipedia-scale matrix operations while remaining portable across Unix systems.
• AWK/shell scripts in scripts/ — Unix philosophy: small, composable tools for common data transformations (format conversion, aggregation, filtering) without heavyweight dependencies.

• Sparse matrix as universal abstraction

  • Why: Unifies treatment of documents, queries, indices, ratings, and features under single representation.
  • Consequence: Some operations (e.g., dense neural networks) are awkward to express; not ideal for dense computations.

• No built-in parallel/GPU support in core

  • Why: Keeps CLI fast and interactive on single machine; caching provides most speedup for iterative workflows.
  • Consequence: Large-scale distributed training requires external tools; cumtx.cu is experimental CUDA extension.

• Shell-first interface, not library

  • Why: Lowers barrier to entry for Unix hackers; encourages ad-hoc combination of existing tools.
  • Consequence: Harder to embed in larger applications; no Python/Java bindings.

• Minimal dependencies (C, AWK, POSIX)

  • Why: Ensures portability and reliability; no dependency hell.
  • Consequence: Cannot leverage modern ML libraries (PyTorch, TensorFlow); must reimplement algorithms in C.

• Does not handle distributed/parallel computation; designed for interactive single-machine exploration
• Not a real-time system; trade-off favors batch processing and caching over latency
• Does not provide web UI or REST API; pure Unix CLI tool
• Not a general-purpose ML framework; focused on IR-specific tasks and matrix operations
• No built-in authentication or multi-user support; assumes single-user Unix environment

• Avg cyclomatic complexity: ~2.3 — Most algorithms are O(n) or O(n log n) for sparse operations; dense matrix ops can be O(n³); high variance across files.
• Largest file: matrix.c (2,400 lines)
• Estimated quality issues: ~18 — Minimal error handling, implicit format assumptions, missing documentation, weak test coverage; Unix-hacker ethos prioritizes agility over robustness.

• Implicit file format assumptions (High) — matrix.c, query.c: Code assumes specific sparse matrix binary format without version checking; format changes break backwards compatibility silently.
• Minimal error handling (High) — eval.c, cluster.c, mtx.c: malloc/file operations lack null checks; invalid inputs cause silent data corruption or crashes rather than user-friendly errors.
• Global cache invalidation (Medium) — cache.c: Cache does not track parameter changes; stale results returned if algorithm parameters updated without explicit cache clear.
• Linear search in hot loops (Low) — dense.c, matrix.c: Some matrix operations use linear search instead of hash table lookups, causing O(n) overhead in repeated iterations.

• matrix.c: sparse matrix multiplication (CPU-bound) — Unoptimized triple-loop dot products; no SIMD or cache-friendly blocking; degrades on wide matrices.
• cache.c: disk I/O on large matrices (I/O-bound) — Serial write of matrices to disk; no incremental writes or compression by default.
• eval.c: ranking evaluation on large judgment sets (CPU-bound) — Recomputes all metrics from scratch; no incremental or batch evaluation.
• hash.c: linear probing collision resolution (Memory/Cache) — High collision rates on skewed vocabulary distributions; no dynamic rehashing strategy.

No visible CMake or autoconf; Makefile is hand-written and may assume POSIX tools not available on Windows/MSVC. No .env or config files; behavior controlled entirely via command-line args and stdin/stdout piping—shell quoting and escaping pitfalls likely. GPU support (cumtx.cu) requires CUDA toolkit installed but no detection in Makefile visible. Memory limits: maximum matrix dimensions hardcoded as 2^32-1 rows/columns, and actual memory usage (16R + 8C bytes minimum) can surprise users with multi-GB intermediates. LSM directory (lsm) mentioned but purpose unclear—may be incomplete feature or stale code. No error handling conventions visible; segfaults likely from buffer overflows in hand-rolled C code given the age and single-author maintenance.

• Sparse matrix CSR/compressed format — mtx's entire design hinges on compressing large matrices (docs, queries) into compact on-disk representations that fit in memory when streamed. Understanding CSR (row pointers, column indices, values) is critical to predicting memory usage and optimizing operations.
• Memory-mapped I/O (mmap) — mtx uses mmap (cache.c, mmap.c/h) to transparently page matrices from disk without explicit read/write calls, enabling Wikipedia-scale experiments on modest RAM. Grasping mmap semantics (page faults, shared memory, file consistency) is essential to debugging performance and correctness.
• Hash-based inverted indexing — hash.c and hashf.c implement custom hash tables for vocabulary and document lookups without stdlib dependencies. Understanding collision resolution and hash function choice (hashf.c) helps when tuning performance on large vocabularies.
• BM25 and vector space models for IR — mtx is designed around composing standard IR operations (BM25 weighting, query expansion, PageRank) as matrix multiplications. Knowing BM25 term weights and how they decompose into matrix form is required to use mtx effectively.
• TREC evaluation format and metrics — mtx is built around TREC-format relevance judgments (qrels, rcv files) and metrics (scripts like evl2recall.awk compute recall/precision/NDCG). Fluency with TREC qrels structure and standard metrics is assumed throughout.
• Bloom filters for approximate membership — bloom.c implements Bloom filters for fast set membership testing (e.g., 'is this document in the relevant set?') with tunable false-positive rates, trading memory for accuracy. Critical for large-scale filtering without building full indices.
• K-means and hierarchical agglomerative clustering — cluster.c and hac.c provide these standard clustering algorithms operating on sparse matrices. Understanding how clustering is formulated as repeated matrix operations (e.g., centroid updates as weighted sums) shows mtx's composability.

• scipy/scipy — scipy.sparse provides Python sparse matrix operations for interactive IR/ML; mtx is the Unix command-line equivalent optimized for disk-resident matrices too large for RAM
• lemire/FastPFor — Bitpacking and compression for integer streams; mtx uses similar ideas (bitvec.h, compress.c) for sparse matrix on-disk compression to reduce I/O
• terrier-org/terrier-core — Terrier is a full-featured Java IR toolkit; mtx is a lighter-weight Unix-native alternative for the same domain (document ranking, indexing, evaluation)
• mgormley/pna — Probabilistic networks and structured inference; mtx's generic matrix representation (via cluster.c, maxent.c) overlaps with PNA's use of matrices for graphical models
• xwhan/trectools — TREC-compatible evaluation tools; mtx integrates with TREC relevance judgment format (*.rcv) and scripts like evl2recall.awk for standard IR metrics

• Add unit tests for matrix.c operations (multiply, transpose, compress) and cache.c mmap streaming. Currently no test/ directory exists despite complex low-level I/O code that is easy to regress.
• Document the LSM module (lsm/) and its role—currently a black box in the file list. Either complete its integration or deprecate it in README.
• Write a Python 3 wrapper module (e.g., pyarmed/) exposing core mtx operations to Jupyter notebooks, leveraging the existing scripts/batch-gemini.py pattern to make the toolkit usable without shell expertise.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub