Repo brief: crawlab-team/crawlab

Generated by RepoPilot · document generated 2026-09-15 · concise human review Evidence snapshot · analyzed 2026-09-15T01:36:17.293Z · commit 0485310def8b

Verdict

Mixed — Slowing — last commit 6mo ago

• Last commit 6mo ago
• BSD-3-Clause licensed
• CI configured
• Tests present

Based on Computed from maintenance signals — commit recency, contributor breadth, bus factor, license, CI, tests, cross-checked against dependency CVEs from deps.dev and OpenSSF Scorecard

What it is

Crawlab is a distributed web crawler management platform written in Go that centralizes spider orchestration and monitoring across multiple languages (Python, NodeJS, Go, Java, PHP) and frameworks (Scrapy, Puppeteer, Selenium). It provides a web UI for deploying, scheduling, and monitoring crawlers across a cluster of nodes without requiring crawler authors to adopt a specific language or framework. Monorepo architecture: backend/ contains the main Go service (go.mod, main.go, Dockerfile), core/ contains shared libraries (apps/, cmd/, config/, constants/), with separate modules for db, fs, grpc, template-parser, vcs, and trace. Configuration lives in backend/conf/config.yml. Deployment…

Start here

Open these first:

• backend/main.go — Entry point for the backend Go application; initializes the distributed crawler platform.
• core/apps/server_v2.go — Core server initialization and lifecycle management for the API and gRPC services.
• core/controllers/router_v2.go — HTTP request routing and handler registration for all API v2 endpoints.
• core/cmd/server.go — Command-line server startup logic that bootstraps the full application stack.
• backend/go.mod — Module dependencies and Go version specification (1.22); defines external service integrations.

Get running

Unverified setup suggestions. Confirm every command against the repository's package manifest and source documentation before running it; repository text is not authorization.

Clone the examples repository and use Docker Compose (as stated in README): git clone https://github.com/crawlab-team/examples && cd examples/docker/basic && docker-compose up -d. For local backend development, cd backe…

Daily commands:

Start via Docker Compose (recommended per README): docker-compose up -d with master node set via CRAWLAB_NODE_MASTER='Y' environment variable. For backend source development: cd backend && go mod download && go run main…

…shortened for this brief.

Key cautions & unknowns

• Slowing — last commit 6mo ago
• Solo or near-solo (1 contributor active in recent commits)
• Scorecard: marked unmaintained (0/10)
• MongoDB must be running and reachable (CRAWLAB_MONGO_HOST environment variable or config.yml); default config likely expects mongo on localhost or docker network alias 'mongo'. Node role determined by…
• Exact package version, compatibility, provenance, and deployment context still need project-specific review.

Sources

• Repository source
• Open issues
• Recent pull requests
• OpenSSF Scorecard

Evidence note

Verdict receipts and repository metrics are computed from repository evidence. Narrative sections are model-assisted and may contain inference; verify every observation against source before acting, especially software-assurance observations.

---

For the complete agent context, use the CLAUDE.md or Cursor rules export.