Diagram Highlighting

What diagram highlighting does

When a procedure step references a component (e.g., "remove the thermostat housing"), diagram highlighting draws a semi-transparent bounding box around that component in the actual manual diagram. Instead of the technician squinting at a parts diagram to find the right part, the relevant component is visually called out.

This is a two-phase system: preprocessing (runs once per document at ingestion time) and rendering (runs on-demand at query time).

Phase 1: Diagram analysis (preprocessing)

The preprocessing script analyzes every page image in a document using multimodal AI analysis to identify:

• Whether the page contains a technical diagram (vs. text-only)
• The diagram type (exploded parts view, wiring diagram, system schematic, cross-section, etc.)
• Individual components with labels and approximate bounding box coordinates
• Component names normalized to match terminology used in procedure steps

Running the analyzer

The analyzer runs per document and supports analyzing all pages, a specific page range, a dry run that lists pages without analyzing, and a force option to re-analyze previously processed pages.

The analysis is incremental -- it skips pages that have already been analyzed unless re-analysis is forced. Checkpoints are saved periodically so interrupted runs can resume.

Analysis output format

Each page produces an analysis record:

{
  "page_number": 47,
  "is_diagram": true,
  "diagram_type": "exploded_parts",
  "components": [
    {
      "name": "Thermostat Housing",
      "label": "6AW-12411-00-00",
      "bbox": [0.12, 0.25, 0.35, 0.48],
      "confidence": 0.92
    },
    {
      "name": "Water Pump Impeller",
      "label": "6CB-W0078-00-00",
      "bbox": [0.45, 0.30, 0.68, 0.55],
      "confidence": 0.87
    }
  ]
}

Bounding boxes use normalized coordinates (0.0-1.0) relative to the image dimensions, so they work regardless of image resolution.

Phase 2: Highlight rendering (query time)

At query time, when a procedure step references a component, the frontend requests the highlighted diagram image:

GET /v1/diagrams/{namespace}/{doc_id}/{page}/highlight?components=thermostat+housing,water+pump

The rendering pipeline:

1. Loads the pre-computed diagram analysis for the document
2. Finds the analysis for the requested page number
3. Matches requested component names against analyzed components (fuzzy matching)
4. Opens the original page image
5. Draws semi-transparent overlays:
   • Primary highlight (sky blue, 30% opacity) for directly matching components
   • Label (dark background, light text) with the component name
6. Returns the composited PNG

Rendering uses lightweight CPU-only image processing -- no GPU required. Typical render time is under 500ms.

Fallback behavior

If any step fails gracefully:

• No analysis data: returns the original page image unmodified
• No matching components: returns null (caller shows original image)
• Image load failure: returns null with a warning log

The technician frontend always falls back to the full page image when highlighting is unavailable.

How it appears in the frontend

In the procedure widget, each step that references a component on an analyzed diagram page shows the diagram inline with the component highlighted. Tapping the diagram opens it fullscreen in the lightbox overlay.

Steps that reference pages without diagram analysis show the full page image with a caption but no highlighting.

Prerequisites

Diagram highlighting requires:

1. Page images for the document (generated during ingestion)
2. Diagram analysis -- a one-time analysis pass that locates components on each diagram page

Diagram analysis is not part of the standard ingestion pipeline -- it is run separately as a preprocessing step. This is intentional: the analysis uses AI vision processing that adds cost and time, so it should be run deliberately on documents where diagram highlighting adds value (service manuals with exploded parts views, wiring diagrams, system schematics).

Related

• Widget API -- the highlight endpoint specification
• Context Streams -- stream 8 (diagram context) feeds page/image metadata into generation
• Dashboard Configuration -- widget configuration for procedure steps with diagrams