Ingestion Worker
The background service that automatically processes queued document ingestion jobs.
What the worker does
The ingestion worker is a background service that monitors the ingestion queue and automatically processes new jobs. When you upload a document through the backoffice or API, it enters a queue. The worker picks up queued jobs and runs the full ingestion pipeline — parsing, chunking, entity extraction, linking, embedding, QA gates, and indexing.
This decouples the upload action from the processing work. Uploading returns immediately, and the worker handles the compute-intensive pipeline stages asynchronously.
How it works
The worker polls the ingestion queue on a configurable interval (default: 10 seconds). When it finds a queued job:
- Claims the job (prevents other workers from picking it up)
- Updates the job status to "processing"
- Runs the unified ingestion pipeline — the same code path used by both the worker and the CLI
- Updates progress as each stage completes (visible on the Upload page)
- Runs QA gates on the output
- If all gates pass, commits vectors to the vector index and marks the job "complete"
- If any gate fails, marks the job with the failure reason
Pipeline unification
The worker uses the same ingestion pipeline that the CLI ingestion script uses. There is one canonical pipeline implementation, not separate code paths for CLI and worker. This means improvements to the pipeline (new stages, better chunking, additional QA gates) automatically apply to both CLI and backoffice ingestion.
The CLI script is a thin wrapper that calls the pipeline directly. The worker calls the same function with the same parameters, just triggered by a queue entry instead of a command line.
Monitoring
Worker activity is visible in two places:
- Backoffice Upload page — job cards show real-time progress, stage completion, and QA gate results
- Observability traces — every pipeline run is traced with per-stage timing, so you can identify slow stages or failures
Configuration
The worker is deployed as its own service, separate from the API. This keeps long-running document processing off the request path — a multi-minute ingest cannot degrade query latency, and the two can be scaled independently.
| Setting | Default | Description |
|---|---|---|
| Poll interval | 10s | How often the worker checks for new jobs |
| Concurrent jobs | 1 | Number of jobs processed simultaneously (bounded by model-provider rate limits, not local compute) |
| Timeout | 30 min | Maximum time per job before marking as failed |
Defaults are per-deployment and may be tuned for your environment.
Related
- Upload and Ingestion — the upload interface and job monitoring
- Ingestion Pipeline — what happens inside each processing stage
- Tenant Provisioning — creating a new tenant, which may trigger initial ingestion