how does it work

NiubiGEO is a GEO (Generative Engine Optimization) research tool — it runs structured "audits" that ask AI providers brand-related prompts, classifies the intent behind each question, checks whether the AI's answer satisfies that intent, and compiles the results into human-readable reports. Here's the flow based on what's in the retrieved code:

1. Planning an audit A request comes in (e.g. POST /projects or POST /audit-jobs in src/server.ts:L649-662) carrying a "confirmed plan" — target domain, competitors, provider targets, language, prompt counts, keywords, etc. If no plan is given, the server builds one via AuditPlanner().plan(...), passing along things like target, competitors, providerTargets, promptCount, keywords, and autoDiscover src/server.ts:L927-941.

2. Running the audit The plan is executed by AuditRunner().run(...), which actually queries the AI providers with the prompts and produces an output containing audit, metrics, gaps, and paths src/server.ts:L941-945. For recurring checks, a MonitoringService re-runs a baseline's existing prompts against providers on a schedule ("run-due" jobs) src/server.ts:L863-870.

3. Classifying each prompt's intent Before/while interpreting answers, prompts are run through a brand-question classifier. INTENT_DEFINITIONS in src/prompts/brand-question-classifier.ts:L17-38 define categories like product_understanding, brand_evaluation, recommendation, comparison, alternative, pricing, product_fit, purchase_decision, risk_evaluation, adoption, and source_analysis, each with a definition and a "boundary" to disambiguate overlapping intents.

A separate intentAnalyzerInstructions() in src/intent/intent-analyzer.ts:L7-27 defines the schema an LLM must fill in per prompt: primaryIntent, secondaryIntents, requestedOutputs, targetBrandRole, booleans like requiresSources/requiresComparison/requiresRecommendation, and an uncertainty level. This is how the system decides what the user was really asking before judging whether the AI's answer satisfied it.

4. Tracking whether web search was used searchSummary() in src/report/human-report.ts:L152-160 inspects run.search (or run.result?.search) to report whether the provider used live web search, requested-but-unconfirmed search, or no search at all — localized into Chinese, English, and Portuguese.

5. Persisting results as projects/baselines/runs Results are saved through a ProjectService/ProjectStore abstraction: a project is derived from the confirmed plan, a baseline captures the initial prompt set, and subsequent audits become runs tied to that baseline via RunOrchestrator src/server.ts:L663-674, L915-L919. The server exposes REST-style read endpoints for baselines, tasks, runs, observations, and events per project, plus a combined workbench view built by WorkbenchReadModelBuilder src/server.ts:L684-713.

6. Building the human report renderIntentAnswerDetails() in src/report/report-builder.ts:L153-192 turns the intent-analysis output into a narrative report section per answer: what the user asked for (requestedOutputs), what the AI answered, what it missed, what was uncertain, per-task completion status with evidence quotes, and mentioned entity relationships — all localized (zh / en / pt-BR).

In short: plan → run prompts against AI providers → classify each prompt's intent → judge whether the response satisfied that intent (including search usage) → persist as project/baseline/run data → render into a structured, localized visibility/competitor report.

Per RepoPilot's architecture signals, the most central files in the whole repo are actually the examples/lib/*.mjs helpers (io.mjs with 12 importers, study.mjs with 6), suggesting a lot of the day-to-day usage/examples logic funnels through those utility modules — worth reading first if you want the practical entry points, alongside `src

You're right to push back. Let me be more precise about what's actually non-trivial here.

The hard part isn't "run prompts and log answers." It's:

1. Intent classification at scale — The system needs to reliably infer what a question is really asking before judging whether an AI answer is good. That's why INTENT_DEFINITIONS in src/prompts/brand-question-classifier.ts:L17-38 painstakingly define 12 distinct intents with explicit "boundary" rules to disambiguate them. For example: brand_evaluation ("is it suitable?") vs. product_fit ("which users does it serve?") vs. purchase_decision ("should I buy it?") are conceptually adjacent but operationally different. The system has to distinguish them reliably so the grading is meaningful.

2. Multi-dimensional answer evaluation — renderIntentAnswerDetails() in src/report/report-builder.ts:L153-192 doesn't just check "did the AI answer yes/no?" It unpacks:

   • What was explicitly asked (requestedOutputs)
   • What the AI actually answered
   • What it missed
   • What remains uncertain
   • Per-task completion with evidence citations
   • Entity relationships the AI mentioned or didn't

   This is a rubric-based assessment, not a binary pass/fail.

3. Reproducibility and monitoring — The project/baseline/run abstraction lets you run the same audit again months later and diff the results. MonitoringService schedules recurring checks src/server.ts:L863-870. That's non-trivial infrastructure.

4. Multi-model, multi-competitor comparison — You're not testing one AI on one domain. The system coordinates runs across multiple providerTargets and competitor domains, orchestrates them, and produces comparative visibility metrics.

So yes, the basic scaffolding (HTTP routes, JSON storage) is standard. But the intent semantics, multi-dimensional grading, and reproducible monitoring are the actual product complexity. Does that land better?

This clarifies things: there is no trained model or empirical validation dataset for intent classification. It's not "validated or trained" in the ML sense at all — it's an LLM-prompted, structurally-enforced, schema-checked pipeline with cross-checking built in. Here's the actual mechanism:

1. Primary classification pass BrandQuestionClassifier.classify() in src/prompts/brand-question-classifier.ts:L240-278 first calls runStructured() with primaryPrompt(input) against a JSON schema (primaryResponseSchema). This asks the model to produce an initial classification for each question. If the provider doesn't support structured/schema-enforced output and the call fails, it retries unstructured [L252-L259].

2. Independent per-intent adjudication (the real "validation" mechanism) Rather than trusting the single classification pass, the code runs a second, independent adjudication round — one prompt per intent dimension (INTENT_ADJUDICATIONS, one entry per intent, [L39]). The adjudicationPrompt() [L172-L192] explicitly instructs the model:

• "Judge only this intent dimension. Do not choose a best label and do not suppress it because a related intent may also apply."
• "Do not use keyword matching and do not assume another classifier has selected any label."

This is deliberately designed to avoid the bias of a single multi-label classification call, where the model might anchor on one "best" label and under-report secondary intents.

3. Merging via OR logic mergeAdjudication() in src/prompts/brand-question-classifier.ts:L218-238 combines the primary classification's intent checks with each independent adjudication's boolean decisions using logical OR (Boolean(decisions.get(intent)) || requested). So if either the primary pass or the adjudication pass says an intent was requested, it's kept. This is a form of ensemble/ensemble-of-prompts cross-validation rather than statistical validation.

4. Structural validation, not semantic validation parseAdjudication() [L125-L137] enforces that every intent in the schema has a boolean decision — if the model omits one, it throws (Intent adjudicator omitted ${intent}). This is strict JSON-shape validation, not accuracy validation. There's also assertQuestionsContainTargetIdentity() called before classification [L249] as a precondition check, and a BrandQuestionScopeError for out-of-scope failures.

5. No offline eval/benchmark harness found I didn't find a test suite, labeled dataset, or scoring/accuracy benchmark for these classifiers in the retrieved code — the "boundary" text in each IntentDefinition src/prompts/brand-question-classifier.ts:L22-34 functions as prompt-engineering disambiguation guidance for the LLM at inference time, not a held-out validation set.

Bottom line: classification correctness is not validated against ground truth or trained on labeled data. It's engineered reliability — multiple independent LLM judgment passes per intent, strict schema enforcement, and OR-merging to reduce false negatives — rather than a traditional ML validation/training loop. If you want to confirm there's truly no test/eval harness anywhere in the repo, I can search specifically for test files covering these modules.