WAIT — Stale — last commit 2y ago

• 31+ active contributors
• Distributed ownership (top contributor 45% of recent commits)
• Other licensed
• ⚠ Stale — last commit 2y ago
• ⚠ Non-standard license (Other) — review terms
• ⚠ No CI workflows detected
• ⚠ No test directory detected
• ⚠ Scorecard: marked unmaintained (0/10)
• ⚠ Scorecard: default branch unprotected (0/10)

_{Maintenance signals: commit recency, contributor breadth, bus factor, license, CI, tests + OpenSSF Scorecard}

Mask R-CNN is a deep learning framework for object detection and instance segmentation built on Keras and TensorFlow. It extends Faster R-CNN with a pixel-level segmentation branch to generate bounding boxes and segmentation masks for each object instance, using a ResNet101 backbone and Feature Pyramid Network (FPN) architecture. The implementation includes pre-trained MS COCO weights and is designed for production use on custom datasets. Flat structure: core logic in mrcnn/ directory (model.py, config.py, utils.py, parallel_model.py) implements the neural network and training utilities; samples/ directory contains example notebooks (demo.ipynb, train_shapes.ipynb, coco/) demonstrating usage on toy datasets and MS COCO; images/ holds test images; assets/ holds demo GIFs and visualizations.

Computer vision researchers and engineers building object detection and instance segmentation pipelines who need a battle-tested implementation. Specifically developers training on custom datasets (not just COCO), practitioners benchmarking against Mask R-CNN baselines, and teams deploying multi-GPU training infrastructure.

This is a mature, production-ready implementation from Matterport (2017) with 25k+ GitHub stars and extensive documentation via Jupyter notebooks. It's actively maintained with clear example code, though the codebase predates modern Python practices (Python 3.5 era); TensorFlow 1.x and Keras 2.x are pinned dependencies, making it less compatible with current TF 2.x workflows without adaptation.

Standard open source risks apply.

Repository appears to be in maintenance mode with no active development visible. No specific recent commits, PRs, or milestones are evident from the file list provided. The last activity was likely years ago; this is a stable reference implementation rather than an actively evolving project.

git clone https://github.com/matterport/Mask_RCNN.git && cd Mask_RCNN && pip install -r requirements.txt (if present, else pip install numpy scipy Pillow cython matplotlib scikit-image tensorflow keras opencv-python h5py imgaug). Then: jupyter notebook samples/demo.ipynb to run inference on pre-trained COCO weights.

Daily commands: For inference: python samples/demo.ipynb in Jupyter or python -c "from mrcnn import model; ..." for direct API use. For training: see samples/shapes/train_shapes.ipynb or samples/coco/ notebooks which contain full training loops with Config objects. No CLI entry point; primary interface is Jupyter notebooks and direct Python API.

• mrcnn/model.py — Core Mask R-CNN model definition with FPN backbone, RPN, and detection heads—every contributor must understand the architecture and layer composition.
• mrcnn/config.py — Configuration base class used across all training and inference—critical for understanding hyperparameters, data pipelines, and model settings.
• mrcnn/utils.py — Utility functions for anchor generation, NMS, IoU computation, and data loading—foundational for both training and inference pipelines.
• mrcnn/visualize.py — Visualization and debugging utilities for masks, detections, and activations—essential for model validation and interpretation.
• samples/coco/coco.py — Reference implementation of dataset loading and training loop for MS COCO—demonstrates the pattern for extending to custom datasets.
• mrcnn/parallel_model.py — Multi-GPU training wrapper handling gradient aggregation—required for scaling training beyond single GPU.
• requirements.txt — TensorFlow, Keras, and OpenCV versions—critical for environment reproducibility and understanding framework constraints.

• ResNet101 Backbone + FPN (Keras/TensorFlow layers) — Multi-scale feature extraction; outputs feature maps P2–P5 at different resolutions.
  • Failure mode: Poor feature quality if pre-trained weights corrupted; gradient flow issues if frozen layers not unfrozen.
• Region Proposal Network (RPN) (Keras/TensorFlow Conv2D, custom anchor generation) — Generates candidate object regions by sliding anchor boxes across feature maps.
  • Failure mode: Fails to detect objects if anchor scales/ratios mismatched to dataset; NMS threshold too strict.
• Detection Heads (Classification + Bbox) (Keras fully connected & conv layers) — Classifies proposals into object classes and refines bounding box coordinates.
  • Failure mode: Class imbalance causes poor performance on rare classes; bbox regression explodes if not normalized.
• Mask Head (FCN) (Keras Conv2DTranspose, up-sampling) — Fully convolutional decoder that generates binary masks for each detected instance.
  • Failure mode: Boundary artifacts if resolution mismatch; mode collapse if mask loss weight too high.
• Data Loading & Augmentation (mrcnn/utils.py, imgaug library) — Batch preparation, image augmentation, anchor matching, and target assignment.
  • Failure mode: Batch size mismatch across GPUs; incorrect IoU computation for anchor matching.

• User input image → ResNet101 backbone — Image resized to 1024×1024 (configurable), normalized by subtracting mean RGB values

Train Mask R-CNN on a Custom Dataset

1. Create a new dataset class by subclassing mrcnn.utils.Dataset (mrcnn/utils.py)
2. Implement load_image(), load_mask(), and image_reference() methods (samples/balloon/balloon.py)
3. Create a config class inheriting from mrcnn.config.Config with dataset-specific parameters (mrcnn/config.py)
4. Instantiate model, load pre-trained COCO weights, and call model.train() with your dataset (samples/balloon/balloon.py)

Modify Model Architecture (e.g., different backbone, head modifications)

1. Edit mrcnn/model.py to replace ResNet101 backbone or modify FPN construction in build_fpn_mask_graph() (mrcnn/model.py)
2. Adjust anchor scales/ratios in mrcnn/config.py if changing input resolution or detection targets (mrcnn/config.py)
3. Update model instantiation and weight loading logic in your training script (samples/coco/coco.py)

Run Inference on New Images

1. Load pre-trained weights using model.load_weights() in your script (mrcnn/model.py)
2. Prepare image using mrcnn.utils preprocessing utilities (resize, normalize) (mrcnn/utils.py)
3. Call model.detect() and retrieve masks, bboxes, and class scores (mrcnn/model.py)
4. Visualize results using mrcnn.visualize.display_instances() (mrcnn/visualize.py)

Enable Multi-GPU Training

1. Replace single-GPU model instantiation with ParallelModel wrapper (mrcnn/parallel_model.py)
2. Adjust batch size and learning rate in config proportionally to GPU count (mrcnn/config.py)
3. Call model.train() as normal; gradient synchronization handled internally (mrcnn/parallel_model.py)

• Keras + TensorFlow — High-level API for rapid prototyping; TensorFlow provides production-grade performance and distributed training support.
• ResNet101 backbone with Feature Pyramid Network (FPN) — Efficient multi-scale feature extraction for detecting objects across a wide range of sizes.
• Region Proposal Network (RPN) — Learns to generate candidate object regions, enabling efficient two-stage detection.
• ROI Align pooling — Preserves spatial structure when extracting features for proposals, improving mask quality over ROI Pooling.

• Two-stage detection (RPN → detection heads) vs. single-stage

  • Why: Two-stage allows precise bounding boxes and mask generation; single-stage is faster but less accurate for segmentation.
  • Consequence: Slower inference (~300ms per image) but higher quality instance masks and localization.

• Keras/TensorFlow v1 compatibility vs. eager execution

  • Why: Static graphs enable graph optimizations and distributed training; eager mode sacrifices some performance for debuggability.
  • Consequence: Requires careful graph construction; harder to debug but scales well to multi-GPU.

• Pre-trained COCO weights included; no random initialization

  • Why: Transfer learning from COCO accelerates training on custom datasets; requires external weight download.
  • Consequence: Users must download ~250MB weights file; eliminates need for hours of training on custom datasets.

• Real-time inference (<16ms per frame); focus is accuracy over speed.
• GPU-free deployment; CUDA is a soft requirement for practical training.
• Panoptic segmentation; handles only instance-level segmentation.
• Video object tracking; frame-by-frame inference only.

TensorFlow 1.x vs 2.x incompatibility: uses tf.Session, tf.placeholder, and Keras Sequential/Model from keras package (not tf.keras); requires TF <2.0 or major refactoring. imgaug version pinning may cause dependency conflicts; no version constraint specified. ResNet101 weights download from MS COCO must complete or training fails silently. GPU memory: default batch_size in Config is large; OOM errors on <12GB cards unless reduced. Anchor box generation in utils.py uses hardcoded scales/ratios; changing image input size requires recalibration. No Python type hints or docstring format consistency makes extending difficult.

• Region Proposal Network (RPN) — RPN is the first-stage detector generating candidate object regions; core Mask R-CNN component in model.py that separates this repo from simpler single-stage detectors and determines training complexity
• Feature Pyramid Network (FPN) — FPN generates multi-scale feature maps enabling detection of objects at vastly different scales; explicitly implemented in mrcnn/model.py's build_fpn_mask_graph() and critical for COCO dataset success
• Anchor boxes and refinement — Anchors (predefined box templates) and refinement (delta prediction) are core to RPN in model.py; utils.py's generate_anchors() and apply_box_refinement() functions are key to understanding bounding box regression
• Non-Maximum Suppression (NMS) — NMS in utils.py removes duplicate overlapping detections after RPN; critical post-processing step for both box and mask output in detect() that affects final accuracy metrics
• Region of Interest (RoI) Align — RoI Align (in model.py's PyramidROIAlign layer) crops and resizes features from FPN to fixed size for each proposal; replaces RoI Pool to preserve spatial precision for mask branch
• Instance segmentation via mask branch — Mask R-CNN adds an FCN decoder (build_fpn_mask_graph in model.py) on top of classification/detection heads to predict per-pixel masks; this is the key innovation distinguishing it from Faster R-CNN
• Focal Loss and class imbalance — mrcnn/model.py implements balanced loss weighting to handle extreme foreground/background class imbalance in anchor-based detection; understanding loss computation is critical for training stability on new datasets

• facebookresearch/detectron2 — Modern successor to Mask R-CNN built on PyTorch with better multi-GPU support, updated backbone options, and active development; the go-to for production deployments
• tensorflow/models — Official TensorFlow object detection and segmentation models including TensorFlow 2.x implementations of Mask R-CNN and related architectures
• mmcv/mmdetection — Modular framework building on PyTorch with Mask R-CNN, Cascade R-CNN, and 100+ detection models; widely used for benchmarking and custom training
• keras-team/keras — The Keras API dependency this repo was built on; understanding Keras 2.x Model, fit(), and functional API is required to modify mrcnn/model.py
• numpy/numpy — Heavily used for array operations in utils.py for anchor generation, box refinement, and mask manipulation; understanding broadcasting and indexing is essential

To work on one of these in Claude Code or Cursor, paste: Implement the "<title>" PR idea from CLAUDE.md, working through the checklist as the task list.

Add unit tests for mrcnn/utils.py core functions

The mrcnn/utils.py module contains critical utility functions for data processing, anchor generation, and mask operations, but there are no visible test files in the repository. This is a high-risk area since utils are foundational to the entire pipeline. A test suite would catch regressions, validate edge cases (empty masks, invalid anchor ratios, etc.), and make refactoring safer for future maintainers.

• [ ] Create tests/test_utils.py with unit tests for anchor generation functions
• [ ] Add tests for mask-related utilities (mask_to_bbox, minimize_mask, etc.)
• [ ] Add tests for data augmentation and batch processing functions in utils.py
• [ ] Add a GitHub Actions workflow (.github/workflows/tests.yml) to run pytest on Python 3.6+
• [ ] Document test coverage in README.md with a badge

Add GPU compatibility tests and document TensorFlow 2.x migration path

The repo specifies tensorflow>=1.3.0 and keras>=2.0.8, which are now legacy. TensorFlow 2.x has significant API changes (eager execution, keras integration, etc.). While major refactoring isn't expected, a CI workflow testing against TF 2.x would identify breaking changes early, and migration documentation would help users upgrade. Currently there's no test matrix in the codebase.

• [ ] Create .github/workflows/tensorflow-compat.yml to test against TF 1.15 and TF 2.x
• [ ] Document known compatibility issues and workarounds in samples/README.md or a MIGRATION.md file
• [ ] Test mrcnn/model.py and mrcnn/parallel_model.py against TF 2.x
• [ ] Add a requirements-tf2.txt with compatible dependency versions
• [ ] Update README.md with a compatibility matrix table

Add integration tests for the three sample datasets (balloon, nucleus, coco)

The repo includes three sample training scripts (samples/balloon/balloon.py, samples/nucleus/nucleus.py, samples/coco/coco.py) but no automated tests verify they actually work end-to-end. A single failed dependency update could break all three without anyone knowing. Integration tests would download mini datasets, train for 1 epoch, and validate model output shapes.

• [ ] Create tests/integration/test_balloon_training.py to train on balloon dataset for 1 epoch and validate output
• [ ] Create tests/integration/test_nucleus_training.py similarly for nucleus.py
• [ ] Create tests/integration/test_coco_inference.py to load pre-trained weights and run inference
• [ ] Add these to .github/workflows/integration-tests.yml (run weekly due to size)
• [ ] Document in CONTRIBUTING.md how to run integration tests locally

• Add comprehensive unit tests for mrcnn/utils.py functions (generate_anchors, apply_box_refinement, mask_to_bbox) which currently have no test coverage visible in repo
• Create a migration guide or refactored branch for TensorFlow 2.x compatibility (replace tf.Session with eager execution, update Keras imports to tf.keras, add modern training loop with tf.GradientTape)
• Build a command-line interface wrapper (e.g., argparse-based CLI in mrcnn/cli.py) so users can train/infer without writing Jupyter notebooks, taking Config file paths as arguments

• High · Outdated TensorFlow Version Specification — requirements.txt - tensorflow>=1.3.0. The requirements.txt specifies 'tensorflow>=1.3.0', which is extremely outdated (released in 2017). TensorFlow 1.3.0 contains numerous known security vulnerabilities, including remote code execution, information disclosure, and arbitrary code execution flaws. Using such old versions exposes the application to documented CVEs. Fix: Update to the latest stable TensorFlow 2.x version (e.g., tensorflow>=2.13.0). Conduct thorough testing to ensure compatibility with the Keras/Mask R-CNN codebase before upgrading.
• High · Outdated Keras Version Specification — requirements.txt - keras>=2.0.8. The requirements.txt specifies 'keras>=2.0.8', which is outdated (released in 2017). Older Keras versions may have security vulnerabilities. Additionally, Keras has been integrated into TensorFlow since 2.0, and standalone Keras maintenance has inconsistent security practices. Fix: Use tensorflow.keras instead of standalone Keras. If standalone Keras is required, update to keras>=3.0.0 and ensure it's compatible with your TensorFlow version.
• Medium · No Version Pinning for Dependencies — requirements.txt - all dependency entries. The requirements.txt uses loose version specifications (>=X.Y.Z) without upper bounds for most packages (numpy, scipy, Pillow, matplotlib, scikit-image, opencv-python, h5py, imgaug). This allows installation of potentially incompatible or vulnerable newer versions, creating supply chain risks and reproducibility issues. Fix: Pin exact versions (==X.Y.Z) or use version ranges with upper bounds (>=X.Y.Z,<X+1.0.0) for all dependencies. Example: 'tensorflow>=2.13.0,<3.0.0'. Generate and maintain a lock file (requirements.lock or Pipfile.lock).
• Medium · Deprecated IPython[all] Dependency — requirements.txt - IPython[all]. The requirements.txt includes 'IPython[all]', which pulls in all optional IPython dependencies. This includes packages that may not be necessary for the application and could introduce vulnerabilities. IPython is primarily for interactive use and shouldn't be a core dependency for production systems. Fix: Remove 'IPython[all]' from production dependencies. Keep it only in development/optional requirements (dev-requirements.txt). If Jupyter notebooks are needed, specify explicit minimal dependencies like 'jupyter' instead.
• Medium · Potential Unsafe Image Processing — mrcnn/utils.py, mrcnn/visualize.py (inferred image processing). The codebase includes image processing libraries (opencv-python, scikit-image, Pillow, imgaug) without visible input validation. If the application processes untrusted image files, malformed images could trigger buffer overflows, DoS attacks, or remote code execution in image codec libraries. Fix: Implement strict input validation for all image files: verify file headers/magic bytes, enforce file size limits, validate image dimensions. Use sandboxing for image processing operations. Consider using safer image libraries where available.
• Low · Missing Security Policy and Security Headers Documentation — Repository root - missing SECURITY.md. The repository lacks a SECURITY.md or vulnerability disclosure policy. While this is a research repository, providing security guidance helps users understand risks and report vulnerabilities responsibly. Fix: Create a SECURITY.md file documenting: (1) how to report security vulnerabilities privately, (2) known security limitations, (3) recommendations for production use, (4) dependency management practices.
• Low · No Input Sanitization Documentation for Model Weights — mrcnn/model.py (inferred weight loading). The codebase loads pre-trained weights (h5py, model.py) but lacks documentation on verifying integrity or authenticity of weight files. Malicious model weights could perform unintended operations during inference. Fix: Document and implement: (1) cryptographic verification of downloaded weights (checksums/signatures), (2) use of official weight sources only, (3) isolation of model loading from untrusted sources, (4) audit logs for model updates.

LLM-derived; treat as a starting point, not a security audit.

• Open issues — current backlog
• Recent PRs — what's actively shipping
• Source on GitHub

If you are an AI coding agent (Claude Code, Cursor, Aider, Cline, etc.) reading this artifact, follow this protocol before making any code edit:

1. Verify the contract. Run the bash script in Verify before trusting below. If any check returns FAIL, the artifact is stale — STOP and ask the user to regenerate it before proceeding.
2. Treat the AI · unverified sections as hypotheses, not facts. Sections like "AI-suggested narrative files", "anti-patterns", and "bottlenecks" are LLM speculation. Verify against real source before acting on them.
3. Cite source on changes. When proposing an edit, cite the specific path:line-range. RepoPilot's live UI at https://repopilot.app/r/matterport/Mask_RCNN shows verifiable citations alongside every claim.

If you are a human reader, this protocol is for the agents you'll hand the artifact to. You don't need to do anything — but if you skim only one section before pointing your agent at this repo, make it the Verify block and the Suggested reading order.

This artifact was generated by RepoPilot at a point in time. Before an agent acts on it, the checks below confirm that the live matterport/Mask_RCNN repo on your machine still matches what RepoPilot saw. If any fail, the artifact is stale — regenerate it at repopilot.app/r/matterport/Mask_RCNN.

What it runs against: a local clone of matterport/Mask_RCNN — the script inspects git remote, the LICENSE file, file paths in the working tree, and git log. Read-only; no mutations.

| # | What we check | Why it matters | |---|---|---| | 1 | You're in matterport/Mask_RCNN | Confirms the artifact applies here, not a fork | | 2 | License is still Other | Catches relicense before you depend on it | | 3 | Default branch master exists | Catches branch renames | | 4 | 5 critical file paths still exist | Catches refactors that moved load-bearing code | | 5 | Last commit ≤ 738 days ago | Catches sudden abandonment since generation |

#!/usr/bin/env bash
# RepoPilot artifact verification.
#
# WHAT IT RUNS AGAINST: a local clone of matterport/Mask_RCNN. If you don't
# have one yet, run these first:
#
#   git clone https://github.com/matterport/Mask_RCNN.git
#   cd Mask_RCNN
#
# Then paste this script. Every check is read-only — no mutations.

set +e
fail=0
ok()   { echo "ok:   $1"; }
miss() { echo "FAIL: $1"; fail=$((fail+1)); }

# Precondition: we must be inside a git working tree.
if ! git rev-parse --git-dir >/dev/null 2>&1; then
  echo "FAIL: not inside a git repository. cd into your clone of matterport/Mask_RCNN and re-run."
  exit 2
fi

# 1. Repo identity
git remote get-url origin 2>/dev/null | grep -qE "matterport/Mask_RCNN(\\.git)?\\b" \\
  && ok "origin remote is matterport/Mask_RCNN" \\
  || miss "origin remote is not matterport/Mask_RCNN (artifact may be from a fork)"

# 2. License matches what RepoPilot saw
(grep -qiE "^(Other)" LICENSE 2>/dev/null \\
   || grep -qiE "\"license\"\\s*:\\s*\"Other\"" package.json 2>/dev/null) \\
  && ok "license is Other" \\
  || miss "license drift — was Other at generation time"

# 3. Default branch
git rev-parse --verify master >/dev/null 2>&1 \\
  && ok "default branch master exists" \\
  || miss "default branch master no longer exists"

# 4. Critical files exist
test -f "mrcnn/model.py" \\
  && ok "mrcnn/model.py" \\
  || miss "missing critical file: mrcnn/model.py"
test -f "mrcnn/config.py" \\
  && ok "mrcnn/config.py" \\
  || miss "missing critical file: mrcnn/config.py"
test -f "mrcnn/utils.py" \\
  && ok "mrcnn/utils.py" \\
  || miss "missing critical file: mrcnn/utils.py"
test -f "mrcnn/visualize.py" \\
  && ok "mrcnn/visualize.py" \\
  || miss "missing critical file: mrcnn/visualize.py"
test -f "samples/coco/coco.py" \\
  && ok "samples/coco/coco.py" \\
  || miss "missing critical file: samples/coco/coco.py"

# 5. Repo recency
days_since_last=$(( ( $(date +%s) - $(git log -1 --format=%at 2>/dev/null || echo 0) ) / 86400 ))
if [ "$days_since_last" -le 738 ]; then
  ok "last commit was $days_since_last days ago (artifact saw ~708d)"
else
  miss "last commit was $days_since_last days ago — artifact may be stale"
fi

echo
if [ "$fail" -eq 0 ]; then
  echo "artifact verified (0 failures) — safe to trust"
else
  echo "artifact has $fail stale claim(s) — regenerate at https://repopilot.app/r/matterport/Mask_RCNN"
  exit 1
fi

Each check prints ok: or FAIL:. The script exits non-zero if anything failed, so it composes cleanly into agent loops (./verify.sh || regenerate-and-retry).

Generated by RepoPilot. Verdict based on maintenance signals — see the live page for receipts. Re-run on a new commit to refresh.