Ingestion Pipeline
How Planara transforms raw equipment PDFs into searchable, enriched knowledge — parsing, chunking, linking, and indexing.
What happens after upload
When you upload a PDF, Planara runs a multi-stage ingestion pipeline that transforms it from a flat document into structured, searchable knowledge. This page explains what each stage does and why it matters for answer quality.
The pipeline runs automatically after upload. You can monitor progress on the Upload page.
Pipeline stages
1. Page rendering
Every PDF page is rendered as a 300 DPI image. This gives the AI parsers a clean visual representation of the page, including diagrams, tables, and formatted text that raw PDF text extraction would miss.
2. Multi-parser extraction
Three AI parsers process each page independently, each contributing what it does best:
| Parser | Strength | Role |
|---|---|---|
| Multimodal AI parser | Page structure, content classification, diagram descriptions, numeric values | Primary parser -- establishes the page-level structure |
| AI validation model | Safety warning hierarchy, heading structure | Cross-validates the primary parser's output, especially for safety content |
| Document structure parser | Precise table cell values | Runs on raw PDF (not images) to capture exact numbers that vision models may approximate |
Results are merged using trust rules: the primary parser is trusted for structure, the document structure parser for precise numeric values, and the validation model for safety warning hierarchy. Safety warnings are unioned across all parsers -- if any parser detects a warning, it is included.
A unit-conversion self-check runs on torque values to catch parser disagreements (N-m to ft-lb, 5% tolerance).
3. Structure-aware chunking
Parsed pages are split into semantic chunks -- not by token count, but by content type. Each chunk preserves a complete semantic unit:
| Content type | Chunking rule |
|---|---|
| Spec table | Entire table as one chunk (never split rows) |
| Procedure | Procedure title + all steps + associated warnings as one chunk |
| Safety warning | Attached to parent procedure, never orphaned |
| Parts list | Linked to associated procedures |
| Diagram | One chunk per diagram with searchable description |
| Narrative | Split at heading boundaries |
Maximum chunk size is 2,500 tokens. Beyond this threshold, retrieval quality degrades (the "context cliff").
Each chunk carries metadata: document ID, page number, section hierarchy, content type, equipment model, and system tag.
4. Auto-linking
Procedures that span multiple pages get split into separate chunks during stage 3. The auto-linker reconnects them by detecting chunks that share the same section hierarchy on adjacent pages. It creates bidirectional links so that when retrieval finds any part of a multi-page procedure, the full procedure is returned.
The linker also creates cross-type links within the same section:
- Procedure to spec table (torque values referenced by the procedure)
- Procedure to safety warning (hazards associated with the procedure)
- Procedure to diagram (visual reference, max 3 diagrams per procedure)
- Procedure to parts list (parts required for the procedure)
These links power the "linked chunk expansion" at retrieval time -- finding one chunk pulls in all related content.
4a. Entity extraction
During chunking, the pipeline extracts structured entities from every chunk:
| Entity type | Examples |
|---|---|
| Part numbers | 6AW-12411-00, 90890-01653 |
| Components | thermostat, impeller, anodes |
| Specifications | torque values, fluid capacities, clearances |
| Tools | torque wrench, feeler gauge, multimeter |
Extracted entities are stored in the database and used for three purposes: improving keyword search (part number queries match even when semantic similarity does not), building the domain ontology automatically, and powering the parts identification feature in the technician frontend.
5. Contextual enrichment
Each chunk receives a 50-100 token contextual prefix generated by an AI model. The prefix situates the chunk within the document: document title, section hierarchy, content type, and applicable equipment models.
This prefix is prepended to the chunk before it is indexed. The result is that the search index captures not just what a chunk says, but where it sits in the document and what type of content it is.
Impact: 35-67% retrieval improvement (measured by published research on this technique).
6. Hypothetical question generation
For each chunk, the pipeline generates 3-5 natural language questions that a technician might ask that the chunk answers. High-value content types (spec tables, procedures, safety warnings, parts lists) get 5 questions; others get 3.
These questions are indexed as additional entries in the search index, all pointing back to the parent chunk. When a technician asks "Why is my engine overheating?", the hypothetical question matches even though the chunk text says "Overheat alert indicator troubleshooting table."
This bridges the vocabulary gap between how technicians ask questions and how manuals are written.
7. Section summary generation
The pipeline generates paragraph-level summaries for each document section (roughly 20 pages each) and one overall document summary. These summaries are indexed alongside the chunks in the search index.
Section summaries help with broad queries like "Tell me about the cooling system" that are too general to match any single chunk but should still return useful results. The summary routes the query to the right section, which then pulls in detailed child chunks.
8. Triple-representation table indexing
Specification tables receive special treatment. Each table is indexed three ways:
- Structured -- the original table with rows and columns preserved, for queries that reference specific cells ("what is the oil capacity for the F300A?")
- Natural language -- a prose description of the table contents generated by an AI model ("The F300A uses 7.4 liters of 10W-30 oil..."), for conversational queries
- Key-value -- individual spec pairs extracted as standalone entries ("F300A oil capacity: 7.4L"), for exact-match lookups
This triple representation ensures spec queries succeed regardless of how the technician phrases the question -- by table cell reference, by natural question, or by spec name lookup.
9. Indexing
Every chunk (with its contextual prefix), every hypothetical question, every section summary, and every table representation is converted into a high-dimensional semantic representation. Simultaneously, a keyword index is built for exact-term matching (part numbers, torque values, model IDs). Keyword index parameters are persisted to the database per namespace so they survive re-deployment.
Both representations are stored together in the search index with hybrid keyword + semantic search capability. The semantic representation captures meaning; the keyword index captures exact terms. The combination achieves higher recall than either alone.
10. Image extraction and classification
Images embedded in the PDF are extracted and classified by type: diagram, photo, parts view, spec label, or decorative. Diagrams receive searchable descriptions generated by vision AI. Original images are stored in cloud storage and served to technicians -- Planara never replaces original diagrams with AI interpretations.
11. QA gates
Before chunks are committed to the search index, the pipeline runs 7 automated quality checks:
| Gate | What it checks |
|---|---|
| Chunk completeness | Every chunk has required metadata (doc ID, page, type, equipment model) |
| Safety coverage | Safety warnings detected in source are present in chunks |
| Spec validation | Numeric values in chunks match source values (unit conversion tolerance) |
| Link integrity | Cross-references resolve to valid target chunks |
| Duplicate detection | No duplicate chunk IDs in the batch |
| Index quality | Indexed representations are valid and within expected ranges |
| Content coverage | Page count of chunks matches page count of source document |
Failed gates block indexing and surface the failure reason on the Upload page. You can review the failure, fix the source document, and re-upload.
12. Stale vector cleanup
When a document is re-ingested, the pipeline automatically removes old vectors that no longer correspond to current chunks. Chunk IDs include content hashes, so any content change produces new IDs. After the new vectors are indexed, the pipeline deletes orphaned vectors from the previous version. This prevents stale answers from persisting after document updates.
Equipment model tagging
During upload, documents are tagged with applicable equipment models. This metadata is carried through to every chunk and used at retrieval time to scope search results to the equipment the technician is working on.
Documents that apply across all models (general safety procedures, common maintenance) should be tagged as global. Global documents are always included in retrieval regardless of the active equipment model.
What you can configure
| Setting | Where | Effect |
|---|---|---|
| Document type | Upload form | Affects how the document is processed and classified |
| Equipment models | Upload form | Scopes retrieval to the right equipment |
| CKL designation | Upload form (document type = CKL) | Routes to the compliance index instead of the main search index |
Everything else -- chunking rules, enrichment, entity extraction, triple-representation indexing, QA gates, linking, indexing -- runs automatically using the pipeline configuration for your namespace. The Tuning Guide covers how to adjust pipeline behavior for your content.
Quality impact
Each enrichment stage contributes measurably to answer quality:
| Stage | Contribution |
|---|---|
| Structure-aware chunking | Keeps complete procedures intact -- retrieval returns full context, not fragments |
| Auto-linking | Multi-page procedures and cross-references resolved automatically |
| Entity extraction | Part number queries match even without semantic similarity |
| Contextual enrichment | Situates each chunk in its document — published research on this technique reports 35-67% fewer retrieval failures |
| Triple-representation tables | Spec queries succeed regardless of phrasing style |
| Hypothetical questions | Bridges vocabulary gap -- technician language matches manual language |
| Section summaries | Broad queries ("tell me about X system") return relevant results |
| Hybrid indexing | Part numbers and spec values found by exact match; conceptual queries found by meaning |
| QA gates | Catches quality issues before they reach technicians |
| Stale vector cleanup | Re-ingested documents never serve outdated answers |
Re-ingestion
When you re-upload a document, the entire pipeline runs again. Old chunks are replaced with new ones once processing completes. Run an evaluation after re-ingestion to verify answer quality -- chunk IDs change when content changes, so cached responses are invalidated.
Related
- Upload and Ingestion -- the upload interface and job monitoring
- Document Types -- how document type affects processing
- CKL Explained -- the compliance knowledge layer
- Tuning Guide -- adjusting pipeline behavior per tenant