Performance
How Planara optimizes response time — model routing, response caching, progressive generation, streaming, and latency targets.
Why performance matters
Field technicians work with their hands. When a technician pauses a repair to ask a question, every second of wait time is a second of lost productivity and mounting frustration.
The numbers below are measured against the production deployment, not modelled. They are reported honestly, including where they are worse than we want them to be. Full-response latency is currently the platform's weakest operational dimension and is an active work item.
Current latency profile
Measured on production with a 60-query sweep, cache bypassed. Figures are p50 for full (non-streamed) responses.
| Query type | p50 (full response) | p95 |
|---|---|---|
| Cache hit | under 500ms | — |
| Acknowledgment ("thanks") | under 10ms | — |
| Context gate hit (follow-up answered from history) | ~150ms | — |
| Spec lookup | ~26s | ~50s |
| Diagram lookup | ~33s | ~55s |
| Procedural query | ~41s | ~70s |
| Diagnostic query | ~50s | ~72s |
Streaming materially changes the experience. Usable content reaches the technician at 16-19s while the full structured response completes behind it. The technician frontend streams by default; the figures above are the full-completion times, which is what an API consumer polling POST /v1/query will see.
Where time is spent
The dominant cost is orchestration, not generation and not vector scale:
| Stage | Time |
|---|---|
| Sequential routing calls (router, orchestrator, query rewrite) | ~4-5s |
| Retrieval node (embed + rerank + compliance retrieval + supplemental passes) | ~4.2s |
| — of which the actual hybrid search | 0.78s |
| Pre-first-token floor | ~7.5-11s (mean ~9.5s) |
| Generation | ~6-11s |
Two things follow from this, and they run against the usual assumptions:
- Generation is not the bottleneck. It is roughly half the total. The pre-first-token floor is the larger half.
- The floor is not a scale problem. The vector search itself is 0.78s. Narrowing corpus scope will not move the number. The floor is sequential LLM routing calls plus unconditional retrieval passes — all of which are collapsible or conditional. A sub-5s floor is reachable without changing model or infrastructure.
This is a known, diagnosed, and addressable gap rather than an architectural limit.
Optimization techniques
Model routing
Not every query needs the same AI model. The pipeline routes queries to different models based on complexity:
| Query type | Model | Why |
|---|---|---|
| Spec lookups, simple Q&A | Fast model | 3-4x faster generation, adequate quality for structured lookups |
| Procedures, diagnostics | Capable model | Complex reasoning, multi-step instructions, safety-critical content |
| Intent classification | Fast model | ~150ms, ~100 tokens |
| Query rewriting | Fast model | ~200ms, only on follow-ups |
| Context gate check | Fast model | ~150ms, simple yes/no evaluation |
| Contextual enrichment (ingestion) | Fast model | High volume, cost-sensitive |
Model evaluation results
Multiple models were evaluated across procedural, diagnostic, and spec queries. Each model was tested on the same query set with scoring for structured-output compliance, citation accuracy, safety warning inclusion, and composite quality.
| Model tier | Latency (procedural) | JSON valid | Citations | Safety warnings | Composite score |
|---|---|---|---|---|---|
| Capable model (cloud) | ~10s | Yes | Yes | Yes | 91.2 |
| Fast model (cloud) | ~4s | Yes | Yes | Yes | 81.3 |
| Fast generation model (cloud, cost-optimized) | ~5s | Yes | Yes | Inconsistent | 81.1 |
| Open-source 70B+ (capable class) | ~16-17s | Yes | Partial | No-Inconsistent | 52-85 |
| Open-source 70B (fast class) | Rate limited | Unreliable (33%) | -- | -- | N/A |
Result: The capable cloud model delivers best quality (91.2 composite). The fast cloud model offers the best speed-to-quality ratio (81.3 composite). A cost-optimized fast model is a viable alternative (81.1 composite). Open-source models do not meet the quality bar for production cloud use -- they fail on JSON compliance or lack safety warning generation.
For on-premises deployments where cloud APIs are not available, local model recommendations are available on request — contact us for the on-premises deployment guide.
System instruction caching
The system instructions are cached across requests using the model provider's native cache mechanism. On cache hits, the provider skips re-processing the system instructions entirely.
Impact: significant cost reduction on cached input, 2-4 second improvement in time-to-first-token.
The system instructions change infrequently (only when configuration updates deploy), so cache hit rates are consistently high.
Two-layer response cache
The pipeline maintains two cache layers:
-
Response cache -- full response hash match. If the exact query (same text, same role, same equipment model, no conversation history) was answered before, the cached response is returned in under 500ms. Total pipeline bypass.
-
Retrieval cache -- query+role hash match. If the same source passages were retrieved before, retrieval is skipped and the pipeline jumps directly to generation with the cached passages.
30-50% of technician queries are near-duplicates ("what oil does the engine use?" is asked constantly). Caching these serves answers instantly instead of running the full pipeline.
Cache is automatically bypassed when conversation history is present. Follow-ups need fresh context even if the query text matches a cached standalone query.
Priority routing
Requests opt into priority routing when provider capacity is available. This is free -- the model provider routes to faster infrastructure when possible and reports which tier was used.
Streaming
The pipeline supports Server-Sent Events (SSE) streaming with progressive phases:
| Phase | What the frontend shows |
|---|---|
thinking | Pulsing indicator -- query is being processed |
retrieved | Source documents identified -- retrieval complete |
generating | Response text streaming in real-time |
complete | Full validated response delivered |
Streaming is the difference between a usable and an unusable experience at current latencies. The technician sees activity within 1-2 seconds and readable content at 16-19s, while the full structured response completes behind it. Any consumer building on the API should use POST /v1/query/stream rather than POST /v1/query for interactive surfaces.
Streaming assessment
The initial assessment (diagnosis summary, confidence, safety warnings) is streamed as structured JSON. The frontend does not display raw token output -- it waits for complete structural events and renders them as typed cards. This avoids the jarring experience of watching JSON tokens appear on screen.
Progressive procedure generation
For long procedures (5+ steps), the pipeline splits generation into two calls:
- Call 1 (3-5s): Summary, step titles, safety warnings, parts list, citations. The technician sees the procedure outline and safety warnings immediately.
- Call 2 (concurrent): Full step details generated in parallel batches. Steps fill in progressively as they complete.
The technician can start reading safety warnings and reviewing the procedure outline while step details are still generating. The "Start Repair" action becomes available as soon as Call 1 completes.
Pipeline short-circuits
Several optimizations skip pipeline stages entirely when they are not needed:
| Condition | Stages skipped | Time saved |
|---|---|---|
| Acknowledgment intent ("thanks") | Entire pipeline | 10-20s |
| Context gate hit (follow-up from history) | Retrieval + generation | 10-15s |
| Response cache hit | Entire pipeline | 10-20s |
| Retrieval cache hit | Retrieval | 3-5s |
| Greeting intent | Retrieval + generation | 10-15s |
| Deterministic fault code match | Retrieval + generation | 10-15s |
Cost per query
| Component | Cost | Notes |
|---|---|---|
| Embedding | ~$0.0001 | High-dimensional, one call |
| Relevance re-ranking | ~$0.002 | 30 docs re-ranked to 12 |
| Intent classification | ~$0.0005 | Fast model |
| Query rewrite | ~$0.001 | Only on follow-ups |
| Generation (capable model) | ~$0.01-0.03 | Varies by response length |
| System instruction caching discount | significant on cached input | |
| Total per query | ~$0.015-0.035 |
What you can monitor
The Observability dashboard shows per-query latency broken down by pipeline stage. Every stage is traced with timing data, so you can identify bottlenecks specific to your content.
Key metrics to watch:
- P50/P95 latency by query type, and separately time-to-first-content on the streaming path — the second number is what the technician actually experiences
- Cache hit rate -- should be 30-50% for mature deployments
- Query-expansion trigger rate -- high rates may indicate content gaps (documentation not covering common questions)
- Context gate hit rate -- should increase as technicians have longer multi-turn sessions
Related
- Observability -- per-stage latency tracing
- Platform Overview -- full pipeline architecture
- Tuning Guide -- adjusting pipeline behavior per tenant