Evaluation System
How Planara's eval system works -- query lifecycle, golden sets, failure types, and quality metrics.
Why evals matter
Planara generates answers from your documentation using AI. AI systems can produce incorrect, incomplete, or hallucinated answers. The evaluation system is how you verify that Planara answers correctly and detect when changes (new documents, response-tuning updates, model changes) cause regressions.
Query lifecycle
Every eval query goes through this lifecycle:
Generated → Pending → Approved → Run → Pass/Fail → (if Pass) → Golden
↓
RejectedStates
| State | Description |
|---|---|
| Pending | AI-generated, awaiting human review |
| Approved | Human-confirmed as a valid test question |
| Rejected | Discarded as invalid or untestable |
| Golden | Permanently tracked. Runs on every quality check. |
What makes a good eval query
A good eval query has:
- A clear question that a technician would actually ask
- Expected facts that are specific and verifiable (e.g., "10W-40", "34 Nm", "page 5-12")
- A known source document so you can trace where the answer should come from
- An appropriate difficulty level (easy = single fact lookup, hard = multi-step reasoning)
A bad eval query has:
- Vague questions ("Tell me about the engine")
- Expected facts that are subjective ("good explanation of maintenance")
- No connection to actual document content
Failure types
When a query fails, it is classified:
| Type | Description | Common cause |
|---|---|---|
| Retrieval | The correct source content was not found in the index | Indexing gaps, scope issues |
| Generation | The AI produced a wrong answer despite having the correct source content | Response tuning, model limitations |
| Grounding | The answer contains claims not supported by retrieved content | Hallucination |
| Safety | Required safety information was missing | Safety content not retrieved or not surfaced |
| Completeness | Answer was partial -- some expected facts found, others missing | Content gaps, retrieval cutoff |
The golden set
The golden set is your regression test suite. These are queries that:
- Were generated (or manually created)
- Were reviewed and approved by a human
- Were run and passed
- Were promoted to golden status
Golden queries should run:
- After every document re-ingestion
- After response-tuning changes
- After model changes
- Before every production deployment
- At least weekly during active development
Quality metrics
Pass rate
passing_golden_queries / total_golden_queries * 100
Target: 90%+ for production readiness.
Per-document breakdown
Shows which documents have the highest/lowest pass rates. Documents with low rates need content review or re-ingestion.
Failure distribution
Shows whether failures are concentrated in retrieval, generation, or other categories. This tells you where to focus improvement efforts.
Trend line
Pass rate over the last 10 runs. Should be stable or improving. A declining trend indicates a regression that needs investigation.
Recommended targets
| Metric | Demo-ready | Pilot-ready | Production |
|---|---|---|---|
| Golden queries | 30+ | 50+ | 100+ |
| Pass rate | 80%+ | 90%+ | 95%+ |
| Coverage | Major topics | All topics | Edge cases |