Eval Dashboard

What this page does

The Eval Dashboard provides a high-level view of your evaluation quality over time. While Eval Management is where you work with individual queries, the Dashboard shows aggregate trends and identifies systemic issues.

Top-level metrics

KPI cards showing dual-track metrics:

Reading the dual-track dashboard

The dashboard separates official and trial metrics intentionally. Here's how to interpret them:

• Official score is your production quality signal. This is what you share with stakeholders and use for go/no-go decisions. It only includes Golden and Approved queries -- vetted questions with verified expected facts.
• Trial score is your exploration signal. Low trial scores are expected and healthy. They mean you're testing new boundaries. A trial score of 40% doesn't mean the system is bad -- it often means the trial queries are testing harder scenarios or have expected facts that need refinement.
• "If promoted" score shows what your official score would be if all currently-passing trial queries were promoted to Approved. This helps you decide when to promote: if "if promoted" is close to your official score, those trials are safe to promote. If it would drop significantly, investigate the failing trials first.

When to promote trial queries

• A trial query has passed 3+ consecutive runs -- it's consistently correct
• The expected facts are verified by a domain expert -- not just machine-checked
• The query covers a production-critical topic (safety, common procedures, key specs) that should be in your baseline

When NOT to promote

• The query only passed once -- could be flaky
• The expected facts haven't been reviewed by someone who knows the equipment
• The query tests an edge case you're still iterating on

Pass Rate Trend

A sparkline chart showing the pass rate from the last 10 completed eval runs. Each bar is color-coded:

• Green: 90%+ pass rate
• Yellow: 70-89%
• Red: below 70%

Hover over a bar to see the exact pass rate and date.

What to look for: The trend should be stable or improving. A sudden drop indicates a regression -- likely caused by a document re-ingestion, prompt change, or model update.

Failure Distribution

A horizontal bar chart showing how failures in the latest run break down by type:

What to look for: Retrieval failures indicate indexing or chunking problems. Generation failures indicate prompt issues. Grounding failures may indicate hallucination. Safety failures are critical and should be prioritized.

Per-Document Breakdown

A table showing eval performance grouped by source document:

Documents are sorted by pass rate (lowest first) so problem documents are at the top.

What to look for: Documents with low pass rates need attention. Common causes: poor chunking, missing content for the questions being asked, or outdated information.

Documentation Gaps

Lists all golden queries that are currently failing, showing:

• FAIL badge
• Query type
• Source document
• The query text
• Expected facts (as small chips)
• Last run date

These are prioritized by how long they have been failing -- queries failing for the longest time appear first.

Action: For each gap, determine whether the issue is:

1. Missing content -- upload a new document or supplement existing ones
2. Retrieval problem -- use the document detail Chunks tab to simulate retrieval
3. Generation problem -- review the tracing dashboard trace to see what the LLM generated

Recent Runs

A table showing the last 20 eval runs: