Multimodal RAG with Image Results

Build a RAG system that retrieves and reasons over both text and images from documents

Standard RAG extracts text and misses charts, figures, and tables. Multimodal RAG indexes both text and images, then passes relevant visuals to a vision LLM at query time.

PDF → Reducto Parse → S3 (images) → Pinecone (embeddings) → Vision LLM

This cookbook builds a pipeline that answers questions using both text context and document images.

---

Create Reducto API Key

1. Open Studio

   Go to studio.reducto.ai and sign in. From the home page, click API Keys in the left sidebar.

   
2. View API Keys

   The API Keys page shows your existing keys. Click + Create new API key in the top right corner.

   
3. Configure Key

   In the modal, enter a name for your key and set an expiration policy (or select “Never” for no expiration). Click Create.

   
4. Copy Your Key

   Copy your new API key and store it securely. You won’t be able to see it again after closing this dialog.

   

   Set the key as an environment variable:

   export REDUCTO_API_KEY="your-api-key-here"
   

---

Prerequisites

You’ll also need accounts and API keys from these services:

Service	Purpose	Sign up
AWS S3	Permanent image storage	aws.amazon.com
Pinecone	Vector database for semantic search	pinecone.io
VoyageAI	Text embeddings	voyageai.com

VoyageAI’s free tier has a rate limit of 3 requests per minute. For production use, add delays between embedding calls or upgrade to a paid plan.

You’ll also need a vision-capable LLM (Claude, GPT-4V, Gemini, etc.) for the final generation step.

Install the required packages:

Python

pip install reductoai pinecone voyageai requests boto3

---

Setup: AWS S3 bucket

Reducto image URLs expire after 1 hour, so upload extracted images to your own S3 bucket for durable storage. Keep the bucket private. When an image needs to be read, your application mints a short-lived presigned URL.

1. Create an IAM user

   Never use your AWS root account for applications. Instead, create a dedicated IAM user with limited permissions.

   Go to IAM Console → Users → Create user.

   
2. Attach least-privilege S3 permissions

   Select Attach policies directly → Create policy, switch to the JSON editor, and paste a policy that grants access only to your bucket’s multimodal-rag/ prefix:

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": ["s3:PutObject", "s3:GetObject"],
       "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/multimodal-rag/*"
     }]
   }
   

   Replace YOUR-BUCKET-NAME with the bucket name you plan to use in the next step. Click Next, name the policy (e.g., multimodal-rag-s3), and click Create policy. Back in the user wizard, refresh the policy list, select the policy you just created, then click Next → Create user.

   When the pipeline runs on AWS, such as EC2, ECS, or Lambda, use an IAM role or other temporary credentials instead of long-lived access keys. Use access keys for local development only. See AWS IAM best practices.

3. Create access keys

   Click on your new user → Security credentials → Create access key.

   Select “Command Line Interface (CLI)”, confirm, and create.

   Save both keys now - you won’t see the secret again. You’ll need these for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

4. Create an S3 bucket

   Go to S3 Console → Create bucket.

   

   Choose a unique name (e.g., my-multimodal-rag-images) and select a region close to you for lower latency.

5. Keep public access blocked

   Leave S3 Block Public Access enabled, which is the default, and do not add a bucket policy. Your application reads objects through presigned URLs created with its IAM permissions. See S3 Block Public Access.

---

Setup: Pinecone index

We need a vector index to store text embeddings. Each vector will also include the S3 bucket and object key, so we can mint a presigned URL when we need the corresponding image.

1. Create index

   

   Go to Pinecone Console → Create Index.

   • Name: multimodal-rag
   • Dimensions: 1024 (the default dimension for VoyageAI’s voyage-4 model)
   • Metric: cosine

   The dimension setting must match the embedding model. VoyageAI’s voyage-4 model defaults to 1024 dimensions. If you use a different embedding model, check its output dimensions.

---

Set environment variables

Before running any code, set these environment variables in your terminal:

export REDUCTO_API_KEY="your-reducto-key"
export PINECONE_API_KEY="your-pinecone-key"
export VOYAGEAI_API_KEY="your-voyageai-key"
export AWS_ACCESS_KEY_ID="your-aws-access-key"
export AWS_SECRET_ACCESS_KEY="your-aws-secret-key"
export S3_BUCKET_NAME="your-bucket-name"
export AWS_REGION="your-bucket-region"

---

Sample document

For this cookbook, we’ll use Deng et al.’s 2024 PLOS ONE paper, “Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language.” The paper has 24 pages, 8 figures, and 6 tables, making it suitable for demonstrating multimodal RAG.

Download the sample PDF:

https://cdn.reducto.ai/samples/multimodal-rag-research-paper.pdf

Save it as research-paper.pdf in your working directory.

Attribution: Deng, Xizi, Elise McClay, Erin Jastrzebski, Yue Wang, and H. Henny Yeung. PLOS ONE, 2024. Licensed under CC BY 4.0.

You can use any PDF with charts or figures. Annual reports, scientific papers, and technical documentation work well for multimodal RAG.

---

Step 1: Parse the document with image extraction

Initialize the client

First, we create a Reducto client using our API key:

Python

import os
from reducto import Reducto

client = Reducto(api_key=os.environ["REDUCTO_API_KEY"])

Upload the document

Before parsing, we need to upload the file to Reducto’s servers. The upload() method returns a file ID that we’ll use in the parse call:

Python

from pathlib import Path

upload = client.upload(file=Path("research-paper.pdf"))

print(f"File ID: {upload.file_id}")

File ID: reducto://c0584170-17a0-44f5-baf4-727467b71b84.pdf

Parse with image extraction

Now we parse the document. The key settings here are:

• return_images: Tells Reducto to crop and return images for figures and tables
• chunk_mode: Controls how text is grouped into chunks

Python

result = client.parse.run(
    input=upload.file_id,
    settings={
        "return_images": ["figure", "table"]
    },
    retrieval={
        "chunking": {
            "chunk_mode": "section"
        }
    }
)

print(f"Parsed {result.usage.num_pages} pages")
print(f"Got {len(result.result.chunks)} chunks")

Parsed 24 pages
Got 33 chunks

Why chunk_mode: "section"?

We use section-based chunking because it keeps figures together with their surrounding explanatory text. If a figure appears in the “Results” section, the chunk will include both the figure and the text that explains it. This improves retrieval quality because the embedding captures the full context.

Other options:

• page: One chunk per page. Simpler but may split related content.
• variable: Adaptive chunking based on content density.

---

Step 2: Understand the response structure

Before we start uploading images, let’s look at what Reducto actually returns. This helps us understand what data we’re working with.

Exploring chunks and blocks

Each chunk contains multiple blocks. A block can be text, a figure, a table, or other content types:

Python

# Skip the front matter and look at a content chunk
chunks = result.result.chunks
chunk = chunks[2] if len(chunks) > 2 else chunks[0]
print(f"Chunk has {len(chunk.blocks)} blocks")
print(f"Embed text preview: {chunk.embed[:200]}...")

Chunk has 5 blocks
Embed text preview: # Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language

Xizi DengⓇ*, Elise McClay, Erin Jastrzebski, Yue Wang, H. Henny Yeung

Department...

Finding figures and tables

Let’s find all figures and tables in the document:

Python

figure_count = 0
table_count = 0

for chunk in result.result.chunks:
    for block in chunk.blocks:
        if block.type == "Figure":
            figure_count += 1
        elif block.type == "Table":
            table_count += 1

print(f"Found {figure_count} figures and {table_count} tables")

Found 8 figures and 6 tables

Examining a figure block

Each figure block has several important fields:

Python

# The first figures in this paper are the journal masthead and a badge,
# so inspect a later one
figure_blocks = [
    block
    for chunk in result.result.chunks
    for block in chunk.blocks
    if block.type == "Figure"
]
figure = figure_blocks[2] if len(figure_blocks) > 2 else figure_blocks[0]
print(f"Type: {figure.type}")
print(f"Page: {figure.bbox.page}")
print(f"Image URL: {figure.image_url[:80]}...")
print(f"Content: {figure.content[:150]}...")

Type: Figure
Page: 6
Image URL: https://prod-storage20241010144745140900000001.s3.amazonaws.com/org/25e89aa6...
Content: Trial-structure diagram for a sentence-matching task: Sentence A (~3-6 s) is followed by a 300 ms pause, Sentence B (~3-6 s), a 500 ms pause, a silent...

Key fields explained:

Field	What it contains
block.type	”Figure”, “Table”, “Text”, etc.
block.bbox.page	Page number (1-indexed)
block.image_url	Temporary URL to the cropped image. Expires in 1 hour.
block.content	AI-generated description of the figure
chunk.embed	Full text optimized for embedding, includes figure descriptions

The content field contains Reducto’s AI-generated description of the figure. This is incredibly useful, as it means your vector search can find figures based on what they show, not just the surrounding text.

---

Step 3: Upload images to S3

Now we need to save these images permanently. Reducto’s URLs expire in 1 hour, so we’ll upload each image to S3 immediately.

Initialize the S3 client

Python

import boto3

s3 = boto3.client("s3", region_name=os.environ["AWS_REGION"])
bucket_name = os.environ["S3_BUCKET_NAME"]

Create the upload function

This function downloads an image from Reducto and uploads it to S3. It returns the private object key, not a public URL.

Python

import requests

def upload_image_to_s3(image_url, s3_key):
    """Download image from Reducto and upload to S3."""
    # Download from Reducto
    response = requests.get(image_url)
    response.raise_for_status()

    # Upload to S3
    s3.put_object(
        Bucket=bucket_name,
        Key=s3_key,
        Body=response.content,
        ContentType="image/png",
        ServerSideEncryption="AES256"
    )

    return s3_key

Upload all images

Now we loop through all chunks. For each chunk, we check if it contains any figures or tables. If it does, we upload those images to S3 and store the object key. If it doesn’t, we still index the chunk without image metadata.

This is the key difference from image-only indexing: we index everything, both text-only chunks and chunks with figures.

Python

all_items = []

for chunk_idx, chunk in enumerate(result.result.chunks):
    # Find any figures/tables in this chunk
    images_in_chunk = []
    for block in chunk.blocks:
        if block.type in ["Figure", "Table"] and block.image_url:
            image_id = f"chunk-{chunk_idx}-{block.type.lower()}-page{block.bbox.page}"
            s3_key = f"multimodal-rag/{image_id}.png"
            object_key = upload_image_to_s3(block.image_url, s3_key)
            images_in_chunk.append({
                "s3_key": object_key,
                "block_type": block.type,
                "page": block.bbox.page
            })

    # Get the page number from the first block
    page = chunk.blocks[0].bbox.page if chunk.blocks else 1

    # Index this chunk (with or without images)
    item = {
        "id": f"chunk-{chunk_idx}",
        "text": chunk.embed,
        "page": page,
        "has_images": len(images_in_chunk) > 0
    }

    # If this chunk has images, include the first one
    # (for chunks with multiple figures, you could store all keys)
    if images_in_chunk:
        item["s3_key"] = images_in_chunk[0]["s3_key"]
        item["block_type"] = images_in_chunk[0]["block_type"]

    all_items.append(item)

# Count what we have
chunks_with_images = sum(1 for item in all_items if item.get("has_images"))
chunks_text_only = len(all_items) - chunks_with_images

print(f"Total chunks: {len(all_items)}")
print(f"  - With images: {chunks_with_images}")
print(f"  - Text only: {chunks_text_only}")

Total chunks: 33
  - With images: 11
  - Text only: 22

Now our index will contain the entire document. Text-only queries will find relevant text chunks, while queries about figures will find chunks that have associated images.

Verify an image is accessible

Use the S3 API to verify that the object exists in your private bucket:

Python

# Find a chunk that has an image
test_item = next(item for item in all_items if item.get("s3_key"))
test_key = test_item["s3_key"]
response = s3.head_object(Bucket=bucket_name, Key=test_key)
print(f"Testing: s3://{bucket_name}/{test_key}")
print(f"Content length: {response['ContentLength']}")

Testing: s3://reducto-multimodal-rag-demo/multimodal-rag/chunk-0-figure-page1.png
Content length: 48231

If you get AccessDenied, check that the IAM policy covers the exact bucket and the multimodal-rag/* prefix. Keep S3 Block Public Access enabled and do not add a public bucket policy.

---

Step 4: Index into Pinecone

With images safely stored in S3, we can now create vector embeddings and store them in Pinecone.

Initialize the clients

Python

from pinecone import Pinecone
import voyageai

pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
index = pc.Index("multimodal-rag")

vo = voyageai.Client(api_key=os.environ["VOYAGEAI_API_KEY"])

Understanding what we’re indexing

For each chunk, we store:

• Vector: Embedding of the text (from chunk.embed)
• Metadata: Text preview, page number, and optionally the S3 bucket and object key

The vector enables semantic search across the entire document. When a chunk has an associated image, the metadata includes its S3 bucket and object key. Never persist presigned URLs in Pinecone because they expire.

Create embeddings and upsert

Python

for item in all_items:
    # Create embedding from the chunk's embed text
    embedding_response = vo.embed(
        [item["text"][:8000]],  # VoyageAI has input limits
        model="voyage-4"
    )
    embedding = embedding_response.embeddings[0]

    # Build metadata (S3 location only included if chunk has images)
    metadata = {
        "text": item["text"][:1000],  # Preview for display
        "page": item["page"],
        "has_images": item["has_images"]
    }

    if item.get("s3_key"):
        metadata["s3_bucket"] = bucket_name
        metadata["s3_key"] = item["s3_key"]
        metadata["block_type"] = item.get("block_type", "Figure")

    # Upsert to Pinecone
    index.upsert(vectors=[{
        "id": item["id"],
        "values": embedding,
        "metadata": metadata
    }])

print(f"Indexed {len(all_items)} items")

Indexed 33 items

Why do we truncate the text?

• For embeddings ([:8000]): VoyageAI has input token limits
• For metadata ([:1000]): Pinecone has metadata size limits (~40KB per vector)

We store enough text in metadata to display a preview, but the full context is captured in the embedding.

VoyageAI’s free tier has rate limits (3 requests per minute). For production use with many documents, add rate limiting or upgrade your plan.

---

Step 5: Query and retrieve

Now we can search our index. When someone asks a question, we:

1. Embed their query using the same model (voyage-4)
2. Find the most similar vectors in Pinecone
3. Return the matches with their S3 object keys

Create the search function

Python

def search(query: str, top_k: int = 3):
    """Search for relevant chunks with images."""
    # Embed the query
    query_embedding = vo.embed(
        [query],
        model="voyage-4"
    ).embeddings[0]

    # Search Pinecone
    results = index.query(
        vector=query_embedding,
        top_k=top_k,
        include_metadata=True
    )

    return results.matches

def create_presigned_image_url(bucket: str, key: str) -> str:
    return s3.generate_presigned_url(
        "get_object",
        Params={"Bucket": bucket, "Key": key},
        ExpiresIn=900,
    )

The presigned URL expires in 15 minutes. That only needs to outlive the LLM call.

Test the search

Python

matches = search("eye and mouth looking patterns by language")

for match in matches:
    print(f"Score: {match.score:.3f}")
    print(f"Page: {match.metadata['page']}")
    print(f"Has image: {match.metadata.get('has_images', False)}")
    if match.metadata.get("s3_key"):
        print(f"Image: s3://{match.metadata['s3_bucket']}/{match.metadata['s3_key']}")
    print(f"Text preview: {match.metadata['text'][:100]}...")
    print("---")

Score: 0.412
Page: 10
Has image: True
Image: s3://reducto-multimodal-rag-demo/multimodal-rag/chunk-14-figure-page10.png
Text preview: # Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language (cont.)...
---
Score: 0.287
Page: 6
Has image: True
Image: s3://reducto-multimodal-rag-demo/multimodal-rag/chunk-9-figure-page6.png
Text preview: # Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language (cont.)...
---
Score: 0.194
Page: 7
Has image: True
Image: s3://reducto-multimodal-rag-demo/multimodal-rag/chunk-10-table-page7.png
Text preview: # Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language (cont.)...
---

Every top match for this query carries an image, because the question asks about a charted result. Text-only chunks compete in the same index and rank higher for prose questions. Exact scores depend on the embedding model and index contents.

---

Step 6: Send to your LLM

You now have everything needed for multimodal generation:

• Text context: match.metadata["text"]
• Images: Mint a presigned S3 GET URL from match.metadata["s3_bucket"] and match.metadata["s3_key"], or fetch the raw image bytes.

Pass the presigned URL or raw image bytes to any vision-capable LLM (Claude, GPT-4V, Gemini). See S3 presigned URLs. Set the URL expiry longer than the LLM call, such as 15 minutes.

---

Reducto features for better results

These Reducto settings can improve your multimodal RAG pipeline:

Figure summaries (enabled by default)

Reducto automatically generates AI descriptions of figures and includes them in the embed field. This is why queries like “show me the revenue chart” can find relevant figures even if “revenue” doesn’t appear in surrounding text.

This is controlled by summarize_figures (default: True). Keep it enabled for multimodal RAG.

Agentic figure extraction

For documents with complex charts, enable agentic figure extraction for higher accuracy:

Python

result = client.parse.run(
    input=upload.file_id,
    settings={"return_images": ["figure", "table"]},
    enhance={
        "agentic": [{"scope": "figure"}]
    }
)

This uses vision LLMs to better understand chart content. It adds latency and cost but improves extraction quality.

Embedding-optimized output

Enable embedding_optimized for output specifically tuned for vector embeddings:

Python

result = client.parse.run(
    input=upload.file_id,
    retrieval={
        "chunking": {"chunk_mode": "section"},
        "embedding_optimized": True
    }
)

Filter block types

If you only want certain content types, use filter_blocks to exclude others from the embed field:

Python

result = client.parse.run(
    input=upload.file_id,
    retrieval={
        "filter_blocks": ["Header", "Footer", "Page Number"]
    }
)

---

Best practices

Choose the right chunking strategy

The chunk_mode setting affects how figures relate to surrounding text:

• section: Groups content by document sections. Best for structured documents like research papers.
• page: One chunk per page. Simple and predictable.
• variable: Adaptive chunking based on content density.

For multimodal RAG, section usually works best because it keeps figures with their explanatory text.

Use the embed field for vector search

Always use chunk.embed (not chunk.content) for your vector embeddings. The embed field includes AI-generated figure descriptions that make visual content searchable.

Not every query needs images

Sending images to your LLM adds latency and cost. Consider routing:

• Simple factual questions → Text-only RAG
• Questions about trends, comparisons, or visuals → Multimodal RAG

You can implement this by checking has_images in retrieved chunks before deciding which path to take.

Handle rate limits gracefully

VoyageAI’s free tier has a 3 requests per minute limit. For production:

• Batch multiple texts in a single embedding call
• Add delays between calls
• Upgrade to a paid plan for higher limits

Limit retrieved images

More images means higher LLM costs and slower responses. For most questions, 2-3 images is sufficient. Use top_k=3 in your search function.

---

Complete example

Here’s the full pipeline in a single script:

Python

import os
from pathlib import Path
import requests
import boto3
from pinecone import Pinecone
import voyageai
from reducto import Reducto

# Initialize clients
reducto = Reducto()
s3 = boto3.client("s3", region_name=os.environ["AWS_REGION"])
pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])
voyage = voyageai.Client()

bucket_name = os.environ["S3_BUCKET_NAME"]
index_name = "multimodal-rag"

def upload_image_to_s3(image_url: str, s3_key: str) -> str:
    """Download image from Reducto and upload to S3."""
    response = requests.get(image_url)
    response.raise_for_status()
    s3.put_object(
        Bucket=bucket_name,
        Key=s3_key,
        Body=response.content,
        ContentType="image/png",
        ServerSideEncryption="AES256",
    )
    return s3_key

def index_document(file_path):
    """Parse document, upload images to S3, and index in Pinecone."""
    # Parse with image extraction
    upload = reducto.upload(file=Path(file_path))

    result = reducto.parse.run(
        input=upload.file_id,
        settings={"return_images": ["figure", "table"]},
        retrieval={"chunking": {"chunk_mode": "section"}}
    )

    # Process chunks and upload images
    records = []
    for i, chunk in enumerate(result.result.chunks):
        s3_key = None
        for block in chunk.blocks:
            if block.type in ["Figure", "Table"] and block.image_url:
                image_id = f"{result.job_id}-chunk-{i}"
                s3_key = f"multimodal-rag/{image_id}.png"
                upload_image_to_s3(block.image_url, s3_key)
                break

        # Embed and prepare record
        embedding = voyage.embed([chunk.embed], model="voyage-4").embeddings[0]
        records.append({
            "id": f"{result.job_id}_{i}",
            "values": embedding,
            "metadata": {
                "text": chunk.embed,
                "page": chunk.blocks[0].bbox.page if chunk.blocks else 0,
                "s3_bucket": bucket_name if s3_key else "",
                "s3_key": s3_key or "",
                "has_image": s3_key is not None
            }
        })

    # Upsert to Pinecone
    index = pc.Index(index_name)
    index.upsert(vectors=records)
    return len(records)

def search(query, top_k=3):
    """Search for relevant chunks."""
    embedding = voyage.embed([query], model="voyage-4").embeddings[0]
    index = pc.Index(index_name)
    results = index.query(vector=embedding, top_k=top_k, include_metadata=True)
    return results.matches

def query_with_images(query):
    """Search and prepare context for LLM."""
    matches = search(query, top_k=3)
    context = "\n\n".join([m.metadata["text"] for m in matches])
    images = [
        s3.generate_presigned_url(
            "get_object",
            Params={
                "Bucket": m.metadata["s3_bucket"],
                "Key": m.metadata["s3_key"],
            },
            ExpiresIn=900,
        )
        for m in matches
        if m.metadata.get("s3_key")
    ]
    return {"context": context, "images": images, "query": query}

# Usage
indexed = index_document("research-paper.pdf")
print(f"Indexed {indexed} chunks")

result = query_with_images("What do the experimental results show?")
print(f"Context: {len(result['context'])} chars, Images: {len(result['images'])}")
# Pass result to your vision LLM

---

Next steps

Batch Processing

Process multiple documents in parallel.

Chunking Methods

Learn about different chunking strategies.

Chart Extraction

Configure image extraction and other parse options.

Response Format

Understand the full parse response structure.