Bulletproof RAG: Production Curriculum

Module 01: Introduction

Module 02: Naive RAG

Module 03: Data Engineering

Module 04: Query Intelligence

Module 05: Advanced Retrieval

Module 06: Agentic Architectures

Module 07: Evaluation

RAG Logging and Observability

Advanced Data Parsing with unpdf

Optimizing PDF extraction for multi-column layouts, page continuity, and structural integrity in RAG pipelines.

The biggest bottleneck in any RAG system is the "Garbage In, Garbage Out" (GIGO) principle. Standard PDF extraction often fails to handle multi-column layouts or splits words across pages (e.g., "hyphen-ation"). unpdf provides a high-fidelity proxy to the underlying document structure, allowing us to merge pages and preserve semantic flow.

Engineering Trade-offs

Feature	Technical Impact
Pros	Handles multi-page context merging; provides low-level access to document metadata; lightweight and fast.
Cons	Requires binary buffer management; memory intensive for massive (500MB+) documents.

Architecture Overview

The Parsing data flow converts binary buffers into clean, traversable strings:

       [Binary PDF]
          │
          ▼
   ┌───────────────┐
   │ Buffer Load   │ (Optimized for non-blocking I/O)
   └───────┬───────┘
           │
   ┌───────▼───────┐
   │ unpdf Proxy   │ (Structure Identification)
   └───────┬───────┘
           │
   ┌───────▼───────┐
   │ Page Merging  │ (Context Continuity)
   └───────────────┘

Implementation Walkthrough

The following steps demonstrate how to handle multi-page technical documents while preserving sentence integrity.

Production Buffer Loading

We use fs/promises to read files into a Uint8Array. This ensures compatibility with the unpdf proxy engine.

import fs from 'fs/promises';
import { getDocumentProxy } from 'unpdf';
 
const buffer = await fs.readFile('./data/AI_Information.pdf');
const pdfProxy = await getDocumentProxy(new Uint8Array(buffer));
 
console.log(`[SYS] Document Proxy created. Version: ${pdfProxy.version}`);

Heuristic Page Merging

By setting mergePages: true, unpdf eliminates page-break artifacts that typically fragment RAG embeddings.

import { extractText } from 'unpdf';
 
const { text, totalPages } = await extractText(pdfProxy, { mergePages: true });
 
console.log(`[DATA] Extracted ${text.length} characters from ${totalPages} pages.`);

Complete Production Script

This script provides a production-grade parser with error handling and metadata extraction.

import fs from 'fs/promises';
import { getDocumentProxy, extractText } from 'unpdf';
import 'dotenv/config';
 
/**
 * Senior Developer PDF Ingestion Pipeline
 * Optimized for unpdf integration.
 */
async function parseDocument(filePath) {
  const logger = {
    info: (msg) => console.log(`[INFO]  ${new Date().toISOString()} | ${msg}`),
    warn: (msg) => console.warn(`[WARN]  ${new Date().toISOString()} | ${msg}`),
  };
 
  try {
    logger.info(`Loading document: ${filePath}`);
    const buffer = await fs.readFile(filePath);
    
    // Create optimized proxy
    const pdf = await getDocumentProxy(new Uint8Array(buffer));
    
    // Perform extraction with page merging
    const { text, totalPages } = await extractText(pdf, { mergePages: true });
    
    logger.info(`Parsing SUCCESS. Pages: ${totalPages}. Chars: ${text.length}.`);
    
    return { text, metadata: { pages: totalPages, source: filePath } };
  } catch (error) {
    logger.warn(`Failure extracting data: ${error.message}`);
    throw error;
  }
}
 
// Execution block
const pathToFile = './data/AI_Information.pdf';
parseDocument(pathToFile).then(data => {
  console.log(`\n--- DATA PARSING REPORT ---\nContent Sample: ${data.text.slice(0, 50)}...\n---------------------------\n`);
});

Summary of Impact

Metric	Standard PDF Loader	unpdf Implementation
Parsing Latency	Variable (Heavier deps)	Low (Lightweight engine)
Context Consistency	Poor (Page interruptions)	High (Unified buffer)
Dependency Weight	Significant	Minimal

When to Use This

When parsing dense research papers with complex layouts.
For high-volume ingestion where speed and memory efficiency are critical.
When your RAG system is failing due to "half-sentences" cut off by page breaks.

Next PageAdvanced Chunking Strategies

On This Page

Engineering Trade-offs Architecture Overview Implementation Walkthrough Complete Production Script Summary of Impact When to Use This

Connect with the Author

Have questions or suggestions? Feel free to reach out.

Feature

Technical Impact

Pros

Handles multi-page context merging; provides low-level access to document metadata; lightweight and fast.

Cons

Requires binary buffer management; memory intensive for massive (500MB+) documents.

Architecture Overview

The Parsing data flow converts binary buffers into clean, traversable strings:

       [Binary PDF]
          │
          ▼
   ┌───────────────┐
   │ Buffer Load   │ (Optimized for non-blocking I/O)
   └───────┬───────┘
           │
   ┌───────▼───────┐
   │ unpdf Proxy   │ (Structure Identification)
   └───────┬───────┘
           │
   ┌───────▼───────┐
   │ Page Merging  │ (Context Continuity)
   └───────────────┘

Implementation Walkthrough

The following steps demonstrate how to handle multi-page technical documents while preserving sentence integrity.

Production Buffer Loading

We use fs/promises to read files into a Uint8Array. This ensures compatibility with the unpdf proxy engine.

import fs from 'fs/promises';
import { getDocumentProxy } from 'unpdf';
 
const buffer = await fs.readFile('./data/AI_Information.pdf');
const pdfProxy = await getDocumentProxy(new Uint8Array(buffer));
 
console.log(`[SYS] Document Proxy created. Version: ${pdfProxy.version}`);

Heuristic Page Merging

By setting mergePages: true, unpdf eliminates page-break artifacts that typically fragment RAG embeddings.

import { extractText } from 'unpdf';
 
const { text, totalPages } = await extractText(pdfProxy, { mergePages: true });
 
console.log(`[DATA] Extracted ${text.length} characters from ${totalPages} pages.`);

Complete Production Script

This script provides a production-grade parser with error handling and metadata extraction.

import fs from 'fs/promises';
import { getDocumentProxy, extractText } from 'unpdf';
import 'dotenv/config';
 
/**
 * Senior Developer PDF Ingestion Pipeline
 * Optimized for unpdf integration.
 */
async function parseDocument(filePath) {
  const logger = {
    info: (msg) => console.log(`[INFO]  ${new Date().toISOString()} | ${msg}`),
    warn: (msg) => console.warn(`[WARN]  ${new Date().toISOString()} | ${msg}`),
  };
 
  try {
    logger.info(`Loading document: ${filePath}`);
    const buffer = await fs.readFile(filePath);
    
    // Create optimized proxy
    const pdf = await getDocumentProxy(new Uint8Array(buffer));
    
    // Perform extraction with page merging
    const { text, totalPages } = await extractText(pdf, { mergePages: true });
    
    logger.info(`Parsing SUCCESS. Pages: ${totalPages}. Chars: ${text.length}.`);
    
    return { text, metadata: { pages: totalPages, source: filePath } };
  } catch (error) {
    logger.warn(`Failure extracting data: ${error.message}`);
    throw error;
  }
}
 
// Execution block
const pathToFile = './data/AI_Information.pdf';
parseDocument(pathToFile).then(data => {
  console.log(`\n--- DATA PARSING REPORT ---\nContent Sample: ${data.text.slice(0, 50)}...\n---------------------------\n`);
});

Metric

Standard PDF Loader

unpdf Implementation

Parsing Latency

Variable (Heavier deps)

Low (Lightweight engine)

Context Consistency

Poor (Page interruptions)

High (Unified buffer)

Dependency Weight

Significant

Minimal