> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reducto.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Multimodal RAG with Image Results
> Build a RAG system that retrieves and reasons over both text and images from documents
Standard RAG extracts text and misses charts, figures, and tables. Multimodal RAG indexes both text and images, then passes relevant visuals to a vision LLM at query time.
```
PDF → Reducto Parse → S3 (images) → Pinecone (embeddings) → Vision LLM
```
This cookbook builds a pipeline that answers questions using both text context and document images.
***
## Create Reducto API Key
Go to [studio.reducto.ai](https://studio.reducto.ai) and sign in. From the home page, click **API Keys** in the left sidebar.
The API Keys page shows your existing keys. Click **+ Create new API key** in the top right corner.
In the modal, enter a name for your key and set an expiration policy (or select "Never" for no expiration). Click **Create**.
Copy your new API key and store it securely. You won't be able to see it again after closing this dialog.
Set the key as an environment variable:
```bash theme={null}
export REDUCTO_API_KEY="your-api-key-here"
```
***
## Prerequisites
You'll also need accounts and API keys from these services:
| Service | Purpose | Sign up |
| -------- | ----------------------------------- | ---------------------------------------- |
| AWS S3 | Permanent image storage | [aws.amazon.com](https://aws.amazon.com) |
| Pinecone | Vector database for semantic search | [pinecone.io](https://www.pinecone.io) |
| VoyageAI | Text embeddings | [voyageai.com](https://www.voyageai.com) |
VoyageAI's free tier has a rate limit of 3 requests per minute. For production use, add delays between embedding calls or upgrade to a paid plan.
You'll also need a vision-capable LLM (Claude, GPT-4V, Gemini, etc.) for the final generation step.
Install the required packages:
```bash Python theme={null}
pip install reductoai pinecone voyageai requests boto3
```
```bash JavaScript theme={null}
npm install reductoai @aws-sdk/client-s3 @aws-sdk/s3-request-presigner @pinecone-database/pinecone voyageai
```
***
## Setup: AWS S3 bucket
Reducto image URLs expire after 1 hour, so upload extracted images to your own S3 bucket for durable storage. Keep the bucket private. When an image needs to be read, your application mints a short-lived presigned URL.
Never use your AWS root account for applications. Instead, create a dedicated IAM user with limited permissions.
Go to [IAM Console](https://console.aws.amazon.com/iam) → **Users** → **Create user**.
Select **Attach policies directly** → **Create policy**, switch to the JSON editor, and paste a policy that grants access only to your bucket's `multimodal-rag/` prefix:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:PutObject", "s3:GetObject"],
"Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/multimodal-rag/*"
}]
}
```
Replace `YOUR-BUCKET-NAME` with the bucket name you plan to use in the next step. Click **Next**, name the policy (e.g., `multimodal-rag-s3`), and click **Create policy**. Back in the user wizard, refresh the policy list, select the policy you just created, then click **Next** → **Create user**.
When the pipeline runs on AWS, such as EC2, ECS, or Lambda, use an IAM role or other temporary credentials instead of long-lived access keys. Use access keys for local development only. See [AWS IAM best practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html).
Click on your new user → **Security credentials** → **Create access key**.
Select "Command Line Interface (CLI)", confirm, and create.
**Save both keys now** - you won't see the secret again. You'll need these for the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables.
Go to [S3 Console](https://s3.console.aws.amazon.com) → **Create bucket**.
Choose a unique name (e.g., `my-multimodal-rag-images`) and select a region close to you for lower latency.
Leave S3 Block Public Access enabled, which is the default, and do not add a bucket policy. Your application reads objects through presigned URLs created with its IAM permissions. See [S3 Block Public Access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-control-block-public-access.html).
***
## Setup: Pinecone index
We need a vector index to store text embeddings. Each vector will also include the S3 bucket and object key, so we can mint a presigned URL when we need the corresponding image.
Go to [Pinecone Console](https://app.pinecone.io) → **Create Index**.
* **Name**: `multimodal-rag`
* **Dimensions**: `1024` (the default dimension for VoyageAI's `voyage-4` model)
* **Metric**: `cosine`
The dimension setting must match the embedding model. VoyageAI's `voyage-4` model defaults to 1024 dimensions. If you use a different embedding model, check its output dimensions.
***
## Set environment variables
Before running any code, set these environment variables in your terminal:
```bash theme={null}
export REDUCTO_API_KEY="your-reducto-key"
export PINECONE_API_KEY="your-pinecone-key"
export VOYAGEAI_API_KEY="your-voyageai-key"
export AWS_ACCESS_KEY_ID="your-aws-access-key"
export AWS_SECRET_ACCESS_KEY="your-aws-secret-key"
export S3_BUCKET_NAME="your-bucket-name"
export AWS_REGION="your-bucket-region"
```
***
## Sample document
For this cookbook, we'll use Deng et al.'s 2024 PLOS ONE paper, "Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language." The paper has 24 pages, 8 figures, and 6 tables, making it suitable for demonstrating multimodal RAG.
**Download the sample PDF:**
```
https://cdn.reducto.ai/samples/multimodal-rag-research-paper.pdf
```
Save it as `research-paper.pdf` in your working directory.
**Attribution:** Deng, Xizi, Elise McClay, Erin Jastrzebski, Yue Wang, and H. Henny Yeung. PLOS ONE, 2024. Licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
You can use any PDF with charts or figures. Annual reports, scientific papers, and technical documentation work well for multimodal RAG.
***
## Step 1: Parse the document with image extraction
### Initialize the client
First, we create a Reducto client using our API key:
```python Python theme={null}
import os
from reducto import Reducto
client = Reducto(api_key=os.environ["REDUCTO_API_KEY"])
```
```javascript JavaScript theme={null}
import Reducto from "reductoai";
const client = new Reducto({ apiKey: process.env.REDUCTO_API_KEY });
```
### Upload the document
Before parsing, we need to upload the file to Reducto's servers. The `upload()` method returns a file ID that we'll use in the parse call:
```python Python theme={null}
from pathlib import Path
upload = client.upload(file=Path("research-paper.pdf"))
print(f"File ID: {upload.file_id}")
```
```javascript JavaScript theme={null}
import fs from "fs";
const upload = await client.upload({
file: fs.createReadStream("research-paper.pdf"),
});
console.log(`File ID: ${upload.file_id}`);
```
```
File ID: reducto://c0584170-17a0-44f5-baf4-727467b71b84.pdf
```
### Parse with image extraction
Now we parse the document. The key settings here are:
* **`return_images`**: Tells Reducto to crop and return images for figures and tables
* **`chunk_mode`**: Controls how text is grouped into chunks
```python Python theme={null}
result = client.parse.run(
input=upload.file_id,
settings={
"return_images": ["figure", "table"]
},
retrieval={
"chunking": {
"chunk_mode": "section"
}
}
)
print(f"Parsed {result.usage.num_pages} pages")
print(f"Got {len(result.result.chunks)} chunks")
```
```javascript JavaScript theme={null}
const result = await client.parse.run({
input: upload.file_id,
settings: {
return_images: ["figure", "table"]
},
retrieval: {
chunking: {
chunk_mode: "section"
}
}
});
console.log(`Parsed ${result.usage.num_pages} pages`);
console.log(`Got ${result.result.chunks.length} chunks`);
```
```
Parsed 24 pages
Got 33 chunks
```
**Why `chunk_mode: "section"`?**
We use section-based chunking because it keeps figures together with their surrounding explanatory text. If a figure appears in the "Results" section, the chunk will include both the figure and the text that explains it. This improves retrieval quality because the embedding captures the full context.
Other options:
* `page`: One chunk per page. Simpler but may split related content.
* `variable`: Adaptive chunking based on content density.
***
## Step 2: Understand the response structure
Before we start uploading images, let's look at what Reducto actually returns. This helps us understand what data we're working with.
### Exploring chunks and blocks
Each chunk contains multiple blocks. A block can be text, a figure, a table, or other content types:
```python Python theme={null}
# Skip the front matter and look at a content chunk
chunks = result.result.chunks
chunk = chunks[2] if len(chunks) > 2 else chunks[0]
print(f"Chunk has {len(chunk.blocks)} blocks")
print(f"Embed text preview: {chunk.embed[:200]}...")
```
```javascript JavaScript theme={null}
// Skip the front matter and look at a content chunk
const chunks = result.result.chunks;
const chunk = chunks[2] ?? chunks[0];
console.log(`Chunk has ${chunk.blocks.length} blocks`);
console.log(`Embed text preview: ${chunk.embed.slice(0, 200)}...`);
```
```
Chunk has 5 blocks
Embed text preview: # Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language
Xizi DengⓇ*, Elise McClay, Erin Jastrzebski, Yue Wang, H. Henny Yeung
Department...
```
### Finding figures and tables
Let's find all figures and tables in the document:
```python Python theme={null}
figure_count = 0
table_count = 0
for chunk in result.result.chunks:
for block in chunk.blocks:
if block.type == "Figure":
figure_count += 1
elif block.type == "Table":
table_count += 1
print(f"Found {figure_count} figures and {table_count} tables")
```
```javascript JavaScript theme={null}
let figureCount = 0;
let tableCount = 0;
for (const chunk of result.result.chunks) {
for (const block of chunk.blocks) {
if (block.type === "Figure") {
figureCount++;
} else if (block.type === "Table") {
tableCount++;
}
}
}
console.log(`Found ${figureCount} figures and ${tableCount} tables`);
```
```
Found 8 figures and 6 tables
```
### Examining a figure block
Each figure block has several important fields:
```python Python theme={null}
# The first figures in this paper are the journal masthead and a badge,
# so inspect a later one
figure_blocks = [
block
for chunk in result.result.chunks
for block in chunk.blocks
if block.type == "Figure"
]
figure = figure_blocks[2] if len(figure_blocks) > 2 else figure_blocks[0]
print(f"Type: {figure.type}")
print(f"Page: {figure.bbox.page}")
print(f"Image URL: {figure.image_url[:80]}...")
print(f"Content: {figure.content[:150]}...")
```
```javascript JavaScript theme={null}
// The first figures in this paper are the journal masthead and a badge,
// so inspect a later one
const figureBlocks = result.result.chunks
.flatMap(chunk => chunk.blocks)
.filter(block => block.type === "Figure");
const figure = figureBlocks[2] ?? figureBlocks[0];
console.log(`Type: ${figure.type}`);
console.log(`Page: ${figure.bbox.page}`);
console.log(`Image URL: ${figure.image_url.slice(0, 80)}...`);
console.log(`Content: ${figure.content.slice(0, 150)}...`);
```
```
Type: Figure
Page: 6
Image URL: https://prod-storage20241010144745140900000001.s3.amazonaws.com/org/25e89aa6...
Content: Trial-structure diagram for a sentence-matching task: Sentence A (~3-6 s) is followed by a 300 ms pause, Sentence B (~3-6 s), a 500 ms pause, a silent...
```
**Key fields explained:**
| Field | What it contains |
| ----------------- | --------------------------------------------------------------- |
| `block.type` | "Figure", "Table", "Text", etc. |
| `block.bbox.page` | Page number (1-indexed) |
| `block.image_url` | Temporary URL to the cropped image. **Expires in 1 hour.** |
| `block.content` | AI-generated description of the figure |
| `chunk.embed` | Full text optimized for embedding, includes figure descriptions |
The `content` field contains Reducto's AI-generated description of the figure. This is incredibly useful, as it means your vector search can find figures based on what they show, not just the surrounding text.
***
## Step 3: Upload images to S3
Now we need to save these images permanently. Reducto's URLs expire in 1 hour, so we'll upload each image to S3 immediately.
### Initialize the S3 client
```python Python theme={null}
import boto3
s3 = boto3.client("s3", region_name=os.environ["AWS_REGION"])
bucket_name = os.environ["S3_BUCKET_NAME"]
```
```javascript JavaScript theme={null}
import { S3Client, PutObjectCommand, HeadObjectCommand } from "@aws-sdk/client-s3";
const s3 = new S3Client({ region: process.env.AWS_REGION });
const bucketName = process.env.S3_BUCKET_NAME;
```
### Create the upload function
This function downloads an image from Reducto and uploads it to S3. It returns the private object key, not a public URL.
```python Python theme={null}
import requests
def upload_image_to_s3(image_url, s3_key):
"""Download image from Reducto and upload to S3."""
# Download from Reducto
response = requests.get(image_url)
response.raise_for_status()
# Upload to S3
s3.put_object(
Bucket=bucket_name,
Key=s3_key,
Body=response.content,
ContentType="image/png",
ServerSideEncryption="AES256"
)
return s3_key
```
```javascript JavaScript theme={null}
async function uploadImageToS3(imageUrl, s3Key) {
// Download from Reducto
const response = await fetch(imageUrl);
if (!response.ok) throw new Error(`Failed to fetch: ${response.status}`);
const imageBuffer = Buffer.from(await response.arrayBuffer());
// Upload to S3
await s3.send(new PutObjectCommand({
Bucket: bucketName,
Key: s3Key,
Body: imageBuffer,
ContentType: "image/png",
ServerSideEncryption: "AES256"
}));
return s3Key;
}
```
### Upload all images
Now we loop through all chunks. For each chunk, we check if it contains any figures or tables. If it does, we upload those images to S3 and store the object key. If it doesn't, we still index the chunk without image metadata.
This is the key difference from image-only indexing: **we index everything**, both text-only chunks and chunks with figures.
```python Python theme={null}
all_items = []
for chunk_idx, chunk in enumerate(result.result.chunks):
# Find any figures/tables in this chunk
images_in_chunk = []
for block in chunk.blocks:
if block.type in ["Figure", "Table"] and block.image_url:
image_id = f"chunk-{chunk_idx}-{block.type.lower()}-page{block.bbox.page}"
s3_key = f"multimodal-rag/{image_id}.png"
object_key = upload_image_to_s3(block.image_url, s3_key)
images_in_chunk.append({
"s3_key": object_key,
"block_type": block.type,
"page": block.bbox.page
})
# Get the page number from the first block
page = chunk.blocks[0].bbox.page if chunk.blocks else 1
# Index this chunk (with or without images)
item = {
"id": f"chunk-{chunk_idx}",
"text": chunk.embed,
"page": page,
"has_images": len(images_in_chunk) > 0
}
# If this chunk has images, include the first one
# (for chunks with multiple figures, you could store all keys)
if images_in_chunk:
item["s3_key"] = images_in_chunk[0]["s3_key"]
item["block_type"] = images_in_chunk[0]["block_type"]
all_items.append(item)
# Count what we have
chunks_with_images = sum(1 for item in all_items if item.get("has_images"))
chunks_text_only = len(all_items) - chunks_with_images
print(f"Total chunks: {len(all_items)}")
print(f" - With images: {chunks_with_images}")
print(f" - Text only: {chunks_text_only}")
```
```javascript JavaScript theme={null}
const allItems = [];
for (let chunkIdx = 0; chunkIdx < result.result.chunks.length; chunkIdx++) {
const chunk = result.result.chunks[chunkIdx];
// Find any figures/tables in this chunk
const imagesInChunk = [];
for (const block of chunk.blocks) {
if (["Figure", "Table"].includes(block.type) && block.image_url) {
const imageId = `chunk-${chunkIdx}-${block.type.toLowerCase()}-page${block.bbox.page}`;
const s3Key = `multimodal-rag/${imageId}.png`;
const objectKey = await uploadImageToS3(block.image_url, s3Key);
imagesInChunk.push({
s3_key: objectKey,
block_type: block.type,
page: block.bbox.page
});
}
}
// Get the page number from the first block
const page = chunk.blocks.length > 0 ? chunk.blocks[0].bbox.page : 1;
// Index this chunk (with or without images)
const item = {
id: `chunk-${chunkIdx}`,
text: chunk.embed,
page: page,
has_images: imagesInChunk.length > 0
};
// If this chunk has images, include the first one
if (imagesInChunk.length > 0) {
item.s3_key = imagesInChunk[0].s3_key;
item.block_type = imagesInChunk[0].block_type;
}
allItems.push(item);
}
// Count what we have
const chunksWithImages = allItems.filter(item => item.has_images).length;
const chunksTextOnly = allItems.length - chunksWithImages;
console.log(`Total chunks: ${allItems.length}`);
console.log(` - With images: ${chunksWithImages}`);
console.log(` - Text only: ${chunksTextOnly}`);
```
```
Total chunks: 33
- With images: 11
- Text only: 22
```
Now our index will contain the entire document. Text-only queries will find relevant text chunks, while queries about figures will find chunks that have associated images.
### Verify an image is accessible
Use the S3 API to verify that the object exists in your private bucket:
```python Python theme={null}
# Find a chunk that has an image
test_item = next(item for item in all_items if item.get("s3_key"))
test_key = test_item["s3_key"]
response = s3.head_object(Bucket=bucket_name, Key=test_key)
print(f"Testing: s3://{bucket_name}/{test_key}")
print(f"Content length: {response['ContentLength']}")
```
```javascript JavaScript theme={null}
// Find a chunk that has an image
const testItem = allItems.find(item => item.s3_key);
const testKey = testItem.s3_key;
const response = await s3.send(
new HeadObjectCommand({ Bucket: bucketName, Key: testKey })
);
console.log(`Testing: s3://${bucketName}/${testKey}`);
console.log(`Content length: ${response.ContentLength}`);
```
```
Testing: s3://reducto-multimodal-rag-demo/multimodal-rag/chunk-0-figure-page1.png
Content length: 48231
```
If you get `AccessDenied`, check that the IAM policy covers the exact bucket and the `multimodal-rag/*` prefix. Keep S3 Block Public Access enabled and do not add a public bucket policy.
***
## Step 4: Index into Pinecone
With images safely stored in S3, we can now create vector embeddings and store them in Pinecone.
### Initialize the clients
```python Python theme={null}
from pinecone import Pinecone
import voyageai
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("multimodal-rag")
vo = voyageai.Client(api_key=os.environ["VOYAGEAI_API_KEY"])
```
```javascript JavaScript theme={null}
import { Pinecone } from "@pinecone-database/pinecone";
import { VoyageAIClient } from "voyageai";
const pc = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });
const index = pc.index("multimodal-rag");
const vo = new VoyageAIClient({ apiKey: process.env.VOYAGEAI_API_KEY });
```
### Understanding what we're indexing
For each chunk, we store:
* **Vector**: Embedding of the text (from `chunk.embed`)
* **Metadata**: Text preview, page number, and optionally the S3 bucket and object key
The vector enables semantic search across the entire document. When a chunk has an associated image, the metadata includes its S3 bucket and object key. Never persist presigned URLs in Pinecone because they expire.
### Create embeddings and upsert
```python Python theme={null}
for item in all_items:
# Create embedding from the chunk's embed text
embedding_response = vo.embed(
[item["text"][:8000]], # VoyageAI has input limits
model="voyage-4"
)
embedding = embedding_response.embeddings[0]
# Build metadata (S3 location only included if chunk has images)
metadata = {
"text": item["text"][:1000], # Preview for display
"page": item["page"],
"has_images": item["has_images"]
}
if item.get("s3_key"):
metadata["s3_bucket"] = bucket_name
metadata["s3_key"] = item["s3_key"]
metadata["block_type"] = item.get("block_type", "Figure")
# Upsert to Pinecone
index.upsert(vectors=[{
"id": item["id"],
"values": embedding,
"metadata": metadata
}])
print(f"Indexed {len(all_items)} items")
```
```javascript JavaScript theme={null}
for (const item of allItems) {
// Create embedding from the chunk's embed text
const embeddingResponse = await vo.embed({
input: [item.text.slice(0, 8000)], // VoyageAI has input limits
model: "voyage-4"
});
const embedding = embeddingResponse.data[0].embedding;
// Build metadata (S3 location only included if chunk has images)
const metadata = {
text: item.text.slice(0, 1000), // Preview for display
page: item.page,
has_images: item.has_images
};
if (item.s3_key) {
metadata.s3_bucket = bucketName;
metadata.s3_key = item.s3_key;
metadata.block_type = item.block_type || "Figure";
}
// Upsert to Pinecone
await index.upsert([{
id: item.id,
values: embedding,
metadata: metadata
}]);
}
console.log(`Indexed ${allItems.length} items`);
```
```
Indexed 33 items
```
**Why do we truncate the text?**
* For embeddings (`[:8000]`): VoyageAI has input token limits
* For metadata (`[:1000]`): Pinecone has metadata size limits (\~40KB per vector)
We store enough text in metadata to display a preview, but the full context is captured in the embedding.
VoyageAI's free tier has rate limits (3 requests per minute). For production use with many documents, add rate limiting or upgrade your plan.
***
## Step 5: Query and retrieve
Now we can search our index. When someone asks a question, we:
1. Embed their query using the same model (`voyage-4`)
2. Find the most similar vectors in Pinecone
3. Return the matches with their S3 object keys
### Create the search function
```python Python theme={null}
def search(query: str, top_k: int = 3):
"""Search for relevant chunks with images."""
# Embed the query
query_embedding = vo.embed(
[query],
model="voyage-4"
).embeddings[0]
# Search Pinecone
results = index.query(
vector=query_embedding,
top_k=top_k,
include_metadata=True
)
return results.matches
def create_presigned_image_url(bucket: str, key: str) -> str:
return s3.generate_presigned_url(
"get_object",
Params={"Bucket": bucket, "Key": key},
ExpiresIn=900,
)
```
```javascript JavaScript theme={null}
import { GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
async function search(query, topK = 3) {
// Embed the query
const embeddingResponse = await vo.embed({
input: [query],
model: "voyage-4"
});
const queryEmbedding = embeddingResponse.data[0].embedding;
// Search Pinecone
const results = await index.query({
vector: queryEmbedding,
topK: topK,
includeMetadata: true
});
return results.matches;
}
async function createPresignedImageUrl(bucket, key) {
return getSignedUrl(
s3,
new GetObjectCommand({ Bucket: bucket, Key: key }),
{ expiresIn: 900 }
);
}
```
The presigned URL expires in 15 minutes. That only needs to outlive the LLM call.
### Test the search
```python Python theme={null}
matches = search("eye and mouth looking patterns by language")
for match in matches:
print(f"Score: {match.score:.3f}")
print(f"Page: {match.metadata['page']}")
print(f"Has image: {match.metadata.get('has_images', False)}")
if match.metadata.get("s3_key"):
print(f"Image: s3://{match.metadata['s3_bucket']}/{match.metadata['s3_key']}")
print(f"Text preview: {match.metadata['text'][:100]}...")
print("---")
```
```javascript JavaScript theme={null}
const matches = await search("eye and mouth looking patterns by language");
for (const match of matches) {
console.log(`Score: ${match.score.toFixed(3)}`);
console.log(`Page: ${match.metadata.page}`);
console.log(`Has image: ${match.metadata.has_images || false}`);
if (match.metadata.s3_key) {
console.log(`Image: s3://${match.metadata.s3_bucket}/${match.metadata.s3_key}`);
}
console.log(`Text preview: ${match.metadata.text.slice(0, 100)}...`);
console.log("---");
}
```
```
Score: 0.412
Page: 10
Has image: True
Image: s3://reducto-multimodal-rag-demo/multimodal-rag/chunk-14-figure-page10.png
Text preview: # Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language (cont.)...
---
Score: 0.287
Page: 6
Has image: True
Image: s3://reducto-multimodal-rag-demo/multimodal-rag/chunk-9-figure-page6.png
Text preview: # Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language (cont.)...
---
Score: 0.194
Page: 7
Has image: True
Image: s3://reducto-multimodal-rag-demo/multimodal-rag/chunk-10-table-page7.png
Text preview: # Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language (cont.)...
---
```
Every top match for this query carries an image, because the question asks about a charted result. Text-only chunks compete in the same index and rank higher for prose questions. Exact scores depend on the embedding model and index contents.
***
## Step 6: Send to your LLM
You now have everything needed for multimodal generation:
* **Text context**: `match.metadata["text"]`
* **Images**: Mint a presigned S3 GET URL from `match.metadata["s3_bucket"]` and `match.metadata["s3_key"]`, or fetch the raw image bytes.
Pass the presigned URL or raw image bytes to any vision-capable LLM (Claude, GPT-4V, Gemini). See [S3 presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html). Set the URL expiry longer than the LLM call, such as 15 minutes.
***
## Reducto features for better results
These Reducto settings can improve your multimodal RAG pipeline:
Reducto automatically generates AI descriptions of figures and includes them in the `embed` field. This is why queries like "show me the revenue chart" can find relevant figures even if "revenue" doesn't appear in surrounding text.
This is controlled by `summarize_figures` (default: `True`). Keep it enabled for multimodal RAG.
For documents with complex charts, enable agentic figure extraction for higher accuracy:
```python Python theme={null}
result = client.parse.run(
input=upload.file_id,
settings={"return_images": ["figure", "table"]},
enhance={
"agentic": [{"scope": "figure"}]
}
)
```
```javascript JavaScript theme={null}
const result = await client.parse.run({
input: upload.file_id,
settings: { return_images: ["figure", "table"] },
enhance: {
agentic: [{ scope: "figure" }]
}
});
```
This uses vision LLMs to better understand chart content. It adds latency and cost but improves extraction quality.
Enable `embedding_optimized` for output specifically tuned for vector embeddings:
```python Python theme={null}
result = client.parse.run(
input=upload.file_id,
retrieval={
"chunking": {"chunk_mode": "section"},
"embedding_optimized": True
}
)
```
```javascript JavaScript theme={null}
const result = await client.parse.run({
input: upload.file_id,
retrieval: {
chunking: { chunk_mode: "section" },
embedding_optimized: true
}
});
```
If you only want certain content types, use `filter_blocks` to exclude others from the embed field:
```python Python theme={null}
result = client.parse.run(
input=upload.file_id,
retrieval={
"filter_blocks": ["Header", "Footer", "Page Number"]
}
)
```
```javascript JavaScript theme={null}
const result = await client.parse.run({
input: upload.file_id,
retrieval: {
filter_blocks: ["Header", "Footer", "Page Number"]
}
});
```
***
## Best practices
The `chunk_mode` setting affects how figures relate to surrounding text:
* **`section`**: Groups content by document sections. Best for structured documents like research papers.
* **`page`**: One chunk per page. Simple and predictable.
* **`variable`**: Adaptive chunking based on content density.
For multimodal RAG, `section` usually works best because it keeps figures with their explanatory text.
Always use `chunk.embed` (not `chunk.content`) for your vector embeddings. The embed field includes AI-generated figure descriptions that make visual content searchable.
Sending images to your LLM adds latency and cost. Consider routing:
* Simple factual questions → Text-only RAG
* Questions about trends, comparisons, or visuals → Multimodal RAG
You can implement this by checking `has_images` in retrieved chunks before deciding which path to take.
VoyageAI's free tier has a 3 requests per minute limit. For production:
* Batch multiple texts in a single embedding call
* Add delays between calls
* Upgrade to a paid plan for higher limits
More images means higher LLM costs and slower responses. For most questions, 2-3 images is sufficient. Use `top_k=3` in your search function.
***
## Complete example
Here's the full pipeline in a single script:
```python Python theme={null}
import os
from pathlib import Path
import requests
import boto3
from pinecone import Pinecone
import voyageai
from reducto import Reducto
# Initialize clients
reducto = Reducto()
s3 = boto3.client("s3", region_name=os.environ["AWS_REGION"])
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
voyage = voyageai.Client()
bucket_name = os.environ["S3_BUCKET_NAME"]
index_name = "multimodal-rag"
def upload_image_to_s3(image_url: str, s3_key: str) -> str:
"""Download image from Reducto and upload to S3."""
response = requests.get(image_url)
response.raise_for_status()
s3.put_object(
Bucket=bucket_name,
Key=s3_key,
Body=response.content,
ContentType="image/png",
ServerSideEncryption="AES256",
)
return s3_key
def index_document(file_path):
"""Parse document, upload images to S3, and index in Pinecone."""
# Parse with image extraction
upload = reducto.upload(file=Path(file_path))
result = reducto.parse.run(
input=upload.file_id,
settings={"return_images": ["figure", "table"]},
retrieval={"chunking": {"chunk_mode": "section"}}
)
# Process chunks and upload images
records = []
for i, chunk in enumerate(result.result.chunks):
s3_key = None
for block in chunk.blocks:
if block.type in ["Figure", "Table"] and block.image_url:
image_id = f"{result.job_id}-chunk-{i}"
s3_key = f"multimodal-rag/{image_id}.png"
upload_image_to_s3(block.image_url, s3_key)
break
# Embed and prepare record
embedding = voyage.embed([chunk.embed], model="voyage-4").embeddings[0]
records.append({
"id": f"{result.job_id}_{i}",
"values": embedding,
"metadata": {
"text": chunk.embed,
"page": chunk.blocks[0].bbox.page if chunk.blocks else 0,
"s3_bucket": bucket_name if s3_key else "",
"s3_key": s3_key or "",
"has_image": s3_key is not None
}
})
# Upsert to Pinecone
index = pc.Index(index_name)
index.upsert(vectors=records)
return len(records)
def search(query, top_k=3):
"""Search for relevant chunks."""
embedding = voyage.embed([query], model="voyage-4").embeddings[0]
index = pc.Index(index_name)
results = index.query(vector=embedding, top_k=top_k, include_metadata=True)
return results.matches
def query_with_images(query):
"""Search and prepare context for LLM."""
matches = search(query, top_k=3)
context = "\n\n".join([m.metadata["text"] for m in matches])
images = [
s3.generate_presigned_url(
"get_object",
Params={
"Bucket": m.metadata["s3_bucket"],
"Key": m.metadata["s3_key"],
},
ExpiresIn=900,
)
for m in matches
if m.metadata.get("s3_key")
]
return {"context": context, "images": images, "query": query}
# Usage
indexed = index_document("research-paper.pdf")
print(f"Indexed {indexed} chunks")
result = query_with_images("What do the experimental results show?")
print(f"Context: {len(result['context'])} chars, Images: {len(result['images'])}")
# Pass result to your vision LLM
```
```javascript JavaScript theme={null}
import fs from "fs";
import Reducto from "reductoai";
import { S3Client, PutObjectCommand, GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { Pinecone } from "@pinecone-database/pinecone";
import { VoyageAIClient } from "voyageai";
// Initialize clients
const reducto = new Reducto();
const s3 = new S3Client({ region: process.env.AWS_REGION });
const pc = new Pinecone({ apiKey: process.env.PINECONE_API_KEY });
const voyage = new VoyageAIClient();
const bucketName = process.env.S3_BUCKET_NAME;
const indexName = "multimodal-rag";
async function uploadImageToS3(imageUrl, s3Key) {
const response = await fetch(imageUrl);
if (!response.ok) throw new Error(`Failed to fetch: ${response.status}`);
const imageBuffer = Buffer.from(await response.arrayBuffer());
await s3.send(new PutObjectCommand({
Bucket: bucketName,
Key: s3Key,
Body: imageBuffer,
ContentType: "image/png",
ServerSideEncryption: "AES256"
}));
return s3Key;
}
async function indexDocument(filePath) {
// Parse with image extraction
const upload = await reducto.upload({ file: fs.createReadStream(filePath) });
const result = await reducto.parse.run({
input: upload.file_id,
settings: { return_images: ["figure", "table"] },
retrieval: { chunking: { chunk_mode: "section" } }
});
// Process chunks and upload images
const records = [];
for (let i = 0; i < result.result.chunks.length; i++) {
const chunk = result.result.chunks[i];
let s3Key = null;
for (const block of chunk.blocks) {
if (["Figure", "Table"].includes(block.type) && block.image_url) {
const imageId = `${result.job_id}-chunk-${i}`;
s3Key = `multimodal-rag/${imageId}.png`;
await uploadImageToS3(block.image_url, s3Key);
break;
}
}
// Embed and prepare record
const embeddingResponse = await voyage.embed({ input: [chunk.embed], model: "voyage-4" });
records.push({
id: `${result.job_id}_${i}`,
values: embeddingResponse.data[0].embedding,
metadata: {
text: chunk.embed,
page: chunk.blocks.length > 0 ? chunk.blocks[0].bbox.page : 0,
s3_bucket: s3Key ? bucketName : "",
s3_key: s3Key || "",
has_image: s3Key !== null
}
});
}
// Upsert to Pinecone
const index = pc.index(indexName);
await index.upsert(records);
return records.length;
}
async function search(query, topK = 3) {
const embeddingResponse = await voyage.embed({ input: [query], model: "voyage-4" });
const index = pc.index(indexName);
const results = await index.query({
vector: embeddingResponse.data[0].embedding, topK, includeMetadata: true
});
return results.matches;
}
async function queryWithImages(query) {
const matches = await search(query, 3);
const context = matches.map(m => m.metadata.text).join("\n\n");
const images = await Promise.all(
matches
.filter(m => m.metadata.s3_key)
.map(m => getSignedUrl(
s3,
new GetObjectCommand({
Bucket: m.metadata.s3_bucket,
Key: m.metadata.s3_key
}),
{ expiresIn: 900 }
))
);
return { context, images, query };
}
// Usage
const indexed = await indexDocument("research-paper.pdf");
console.log(`Indexed ${indexed} chunks`);
const result = await queryWithImages("What do the experimental results show?");
console.log(`Context: ${result.context.length} chars, Images: ${result.images.length}`);
// Pass result to your vision LLM
```
***
## Next steps
Process multiple documents in parallel.
Learn about different chunking strategies.
Configure image extraction and other parse options.
Understand the full parse response structure.