Retrieval for content that text-only pipelines miss.
- Status
- Planned
- Priority
- 5 of 6
- Best use
- Scanned forms, slide decks, charts and photos, cited to the page region
- Deployment
- Standalone agent; can call Hybrid and Structured components
- Relative effort
- M-L
Pick it when the answer lives in a scan, a chart or a slide
Best for
- Scanned forms, signed documents, photos of whiteboards
- Slide decks and reports where the answer is in a chart or a table image
- Answers that must point to the exact page and region for a reviewer
Ask it things like
- How many new lines were requested on service order SO-88213?
- What was network availability in August 2026? (only in a chart)
- When does the 5G standalone pilot go live? (only in a photo on a slide)
Not the right choice for
- Plain text documents (Hybrid is simpler and faster)
- Totals across many rows (Structured)
Choose it over Hybrid when most of the content is visual, or the citation must point to a region of a page. Since Nov 11 Hybrid also reads chart data and scans, so for mostly-text sets with a few visuals, Hybrid is enough.
- Data it reads
- PDF, PNG and JPG scans, PowerPoint, Word.
- How it gets the data
- Extracts each modality at load time, stores it with its page and region, then searches like Hybrid and checks the answer against the cited region.
- How fresh the answer is
- As of the last time the files were read.
- Relative cost
- Moderate (OCR and optional vision model at load time)
How Multimodal RAG works
- 1
Register & Classify Sources
Identify supported modalities, privacy restrictions, source identity, access policy and original artifact locations.
- 2
Extract by Modality
Parse text and tables; apply OCR, vision processing, speech transcription or video segmentation where needed; retain extraction uncertainty.
- 3
Preserve Structure & Provenance
Link extracted values to source page/region/cell or media time/frame; retain original evidence and any OCR/model version.
- 4
Index & Enforce Policy
Create appropriate textual, visual and other indexes with source-level and derived-evidence authorization controls.
- 5
Retrieve & Reconcile
Retrieve authorized multimodal candidates and verify critical values/relationships against the original media, not only generated captions.
- 6
Draft with Source Locations
Produce an evidence-grounded response with page, region, table, cell, frame or timestamp citations where supported.
- 7
Validate & Return
Check especially numbers, units, tables and visual claims; flag uncertain extraction or ask for review.
Use it when the answer depends on content that text-only pipelines miss: scanned files, engineering diagrams, multi-page tables, visual evidence, audio and video.
One router picks the right method for each question
Context66 combines these methods as Ontology-guided Adaptive RAG. Every answer is checked against its evidence, and the system abstains when the evidence is incomplete.
- User question
- Intent and context detection
- Ontology and semantic interpretation
- Query router
All branches, orchestrated by the agent
- Evidence fusion and ranking
- Answer written only from retrieved evidence
- Claim-level validation and citation checking

