Eval Automation

What eval automation does

Eval automation extends the manual Eval Management workflow with three capabilities:

1. Nightly batch -- runs all golden queries against the live API every night, compares to baseline, and alerts on regressions
2. Auto-sampling -- generates eval queries automatically from new documents, negative feedback, and nightly failures
3. Agentic diagnosis -- when queries fail, an AI agent diagnoses root causes, proposes fixes, and (with approval) applies them

Nightly eval batch

What it does

The nightly eval batch (planara-v2/scripts/nightly_eval.py) runs at 2 AM UTC as a scheduled job. For each active namespace, it:

1. Fetches all golden eval queries from v2_eval_queries
2. Runs each against the deployed API
3. Stores results in v2_eval_runs and v2_eval_run_results
4. Compares against the stored baseline in v2_tenant_config.eval_baseline
5. Creates alerts when regressions are detected
6. Fires analytics platform events for monitoring dashboards

Running manually

# Run for all active namespaces against production
.venv/bin/python planara-v2/scripts/nightly_eval.py

# Run for a single namespace
.venv/bin/python planara-v2/scripts/nightly_eval.py --namespace your-namespace

# Run against a specific API endpoint
.venv/bin/python planara-v2/scripts/nightly_eval.py --url https://your-api-domain.planara.com

# Lock current results as the new baseline
.venv/bin/python planara-v2/scripts/nightly_eval.py --lock-baseline

Baseline comparison

Each namespace stores a baseline in v2_tenant_config.eval_baseline:

{
  "pass_rate": 0.92,
  "updated_at": "2026-03-24T02:00:00Z",
  "total_queries": 25,
  "results": {
    "query-uuid-1": { "passed": true, "confidence": 0.78 },
    "query-uuid-2": { "passed": true, "confidence": 0.65 }
  }
}

The nightly run compares each query's result against its baseline entry and classifies changes as:

Alert system

Alerts are stored in v2_eval_alerts and visible in the backoffice eval dashboard. Three alert types:

Alerts also fire as analytics platform events (eval_regression_detected) for integration with external alerting (Slack, PagerDuty, email via analytics platform webhooks).

Report format

Each nightly run produces a console report and a stored summary:

=== Nightly Eval Report ===
Namespace: your-namespace (Your Tenant)
Date: 2026-03-24 02:15
Queries: 25 golden
Passed: 23 (92%)
Failed: 2 (8%)
Confidence: avg=0.712, min=0.340, max=0.890
Latency: avg=18200ms, p50=16400ms, p95=31200ms

Baseline: 92% -> Current: 92% (+0%)
No regressions, improvements, or new failures vs baseline.

Locking a new baseline

After a major pipeline change (re-ingestion, model upgrade, prompt rewrite), lock a new baseline:

.venv/bin/python planara-v2/scripts/nightly_eval.py --lock-baseline

This stores the current run's results as the new comparison point. All future nightly runs compare against this baseline until it is locked again.

Auto-sampling

Eval queries are automatically generated from three sources:

1. New document ingestion -- after a document is ingested, the system generates 5-10 eval queries covering the new content. These are created with pending status and require human review before promotion.

2. Negative feedback -- when a technician rates a response negatively (thumbs down), the original query is added as a candidate eval query with type troubleshooting. This surfaces real-world failure patterns.

3. Nightly failures -- when a previously-passing query regresses, it is flagged for investigation and the failing query is queued for agentic diagnosis.

Recommended flow for auto-generated queries

All auto-generated queries start as Pending regardless of source. The recommended promotion path is:

Pending → Trial → Approved → Golden

• Pending → Trial: Move queries that look reasonable into trial. This is a quick triage -- you're saying "worth testing" not "definitely correct."
• Trial → Approved: After a trial query passes 2-3 runs with correct expected facts, promote it to Approved so it counts toward your official score.
• Approved → Golden: Promote critical queries (safety, key procedures, common fault codes) to Golden. These become your regression safety net.

Don't skip Trial for auto-generated queries. AI-generated expected facts are sometimes wrong -- a thermostat procedure query might expect "remove 4 bolts" when the manual says "remove 3 bolts." Trial catches these before they become false failures in your official score.

How the nightly batch handles trial vs golden

The nightly eval batch (planara-v2/scripts/nightly_eval.py) handles trial and golden queries differently:

When the nightly batch runs trial queries, it produces a separate summary:

=== Trial Summary ===
Trial queries: 12
Passed: 8 (67%)
Failed: 4 (33%)
Candidates for promotion: 6 (passed 3+ consecutive runs)

This "candidates for promotion" count surfaces trial queries that have been consistently passing and are ready for an admin to promote to Approved or Golden.

Agentic eval system

The agentic eval system transforms eval from a pass/fail test runner into a quality improvement loop. It is implemented as an orchestration pipeline with the following flow:

Failed queries
    │
    ▼
  Diagnose (per-query root cause analysis)
    │
    ▼
  Propose fixes (actionable, specific changes)
    │
    ▼
  Human review (approve/reject each fix)
    │
    ▼
  Apply fixes (with backup)
    │
    ▼
  Re-run (verify improvement)

Root cause categories

The diagnosis agent classifies each failure into one of:

Fix proposals

Each diagnosis produces a concrete fix proposal:

{
  "fix_id": "fix-abc123",
  "query_id": "query-uuid-1",
  "root_cause": "ontology",
  "description": "Query 'winterize outboard' uses term not in manufacturer manual. Manual uses 'storing' and 'transporting'.",
  "proposed_action": "add_ontology_association",
  "details": {
    "term": "winterize",
    "expansion": "storing transporting vapor separator drain flush"
  },
  "confidence": 0.85,
  "status": "pending_review"
}

Fixes require human approval before application. This is intentional -- the agent proposes, humans decide. Applied fixes create a backup of the modified file so they can be rolled back.

Running the eval improvement script

# Diagnose all failures from the latest eval run
.venv/bin/python planara-v2/scripts/eval_improve.py --namespace your-namespace

# Diagnose + propose fixes (no application)
.venv/bin/python planara-v2/scripts/eval_improve.py --namespace your-namespace --propose

# Apply approved fixes and re-run
.venv/bin/python planara-v2/scripts/eval_improve.py --namespace your-namespace --apply --rerun

Diagnostic template quality tracking

Diagnostic templates — the structured decision trees extracted from your documentation — are tracked via usage metrics in addition to eval queries. Each template records:

• Usage count — how many times a technician has been guided through this template
• Success rate — the percentage of completions that led to a confirmed correct diagnosis
• Average resolution time — how long the diagnostic flow takes from start to confirmed root cause

Templates with high success rates and usage counts are automatically prioritized in future diagnostic sessions. Templates with declining success rates are flagged for admin review. This feedback loop means diagnostic quality improves continuously from real-world usage, not just from eval query pass rates.

Diagnostic quality evaluation

The eval system extends beyond retrieval accuracy and response quality to cover diagnostic decision quality. Diagnostic eval queries test whether the platform:

• Selects the correct diagnostic path given a set of symptoms and telemetry data
• Presents checks in an efficient order (high-value eliminations first)
• Correctly narrows the diagnosis as findings are reported
• Transitions to repair at the right confidence threshold — not too early (misdiagnosis risk) and not too late (unnecessary checks)

Diagnostic quality queries are structured differently from standard eval queries: they include a sequence of simulated findings and evaluate the platform's response at each step, not just the final answer. This tests the full diagnostic reasoning chain, not just retrieval.

Hypothetical question indexing

A separate preprocessing step that improves retrieval recall by generating technician-language questions for each document chunk:

# Generate for a specific document
python scripts/generate_questions.py --namespace your-namespace --doc-id your_doc_id

# Generate for all ingested documents
python scripts/generate_questions.py --namespace your-namespace

# Dry run (generate questions but don't embed or store)
python scripts/generate_questions.py --namespace your-namespace --dry-run

# Force regeneration for chunks that already have questions
python scripts/generate_questions.py --namespace your-namespace --force

The script reads existing chunks, sends each to a fast AI model with the prompt "what questions would a technician ask that this content answers?", generates 3-5 questions per chunk, embeds them with the same embedding model, and stores them in the vector database alongside the original chunk vectors.

At query time, the technician's natural-language question matches the synthetic question, which retrieves the correct chunk. This bridges the vocabulary gap between how techs ask questions ("why is my engine overheating?") and how manuals are written ("Overheat alert indicator troubleshooting table").

Hypothetical question vectors use the ID format {chunk_id}_q0, {chunk_id}_q1, etc., with metadata linking back to the parent chunk. They are non-destructive -- the original chunk vectors are never modified or deleted.

Related

• Eval Management -- the manual eval workflow and UI
• Eval Dashboard -- aggregate metrics and trend analysis
• Tuning & Customization -- ontology and fault code configuration that eval fixes may modify
• Analytics -- analytics platform integration for eval event tracking