From Rules to Reasoning: A Practical Showdown Between Traditional PDF Extraction and LLM-Based Document Parsing

Introduction

In the world of B2B operations, extracting structured data from PDF documents—such as purchase orders, invoices, and shipping manifests—remains a persistent challenge. Traditional rule-based methods rely on optical character recognition (OCR) and template matching, while modern large language models (LLMs) promise more flexible, context-aware extraction. This article presents a hands-on comparison between a rule-based approach using pytesseract and an LLM-based pipeline built with Ollama and LLaMA 3, applied to a realistic B2B order scenario. We'll explore the strengths, weaknesses, and practical trade-offs of each method.

The B2B Order Scenario

To make the comparison grounded, we used a sample purchase order PDF typical in B2B transactions. The document contained fields such as Order Number, Customer Name, Order Date, Line Items (with descriptions, quantities, unit prices, and totals), Shipping Address, and Total Amount. The goal was to extract these fields accurately and reliably, mimicking a real-world document processing pipeline.

Rule-Based Extraction with pytesseract

Approach

The rule-based pipeline followed these steps:

1. Image Preprocessing: Convert PDF pages to high-resolution images, apply grayscale, thresholding, and deskewing to improve OCR accuracy.
2. OCR with pytesseract: Use Tesseract’s OCR engine to extract raw text from the images.
3. Post-Processing: Apply regular expressions and heuristic rules to locate and extract specific fields. For example, a pattern like Order No:\s*([A-Z0-9]+) to grab the order number, or finding rows that match expected line‑item patterns.

Results and Limitations

On well-formatted, clean PDFs, the rule-based system performed reasonably well. It extracted most fields with high precision when the document adhered to a known layout. However, it struggled with variations in formatting, inconsistent spacing, or unexpected table structures. Key failure points included:

• Misaligned tables: When columns shifted slightly, line‑item extraction often broke.
• Noise from OCR errors: Poor image quality or non‑standard fonts introduced typos that broke regex patterns.
• Hard‑coded templates: Every new document layout required manual rule adjustments, making maintenance expensive.

The rule-based method proved fast (processing a page in under one second) but brittle. It could not adapt to unseen formats without significant developer effort.

LLM-Based Extraction with Ollama and LLaMA 3

Approach

The LLM pipeline used Ollama to run the LLaMA 3 model locally (8B parameter variant). The steps were:

1. PDF to Text: First convert the PDF to plain text using basic layout‑preserving tools (e.g., PyMuPDF). No OCR required—LLMs handle text directly.
2. Prompt Engineering: Design a structured prompt instructing the model to extract specific fields from the document text. The prompt included a schema for the expected output (JSON format) and examples of correct extraction.
3. Inference: Feed the document text and prompt into LLaMA 3 via Ollama’s API. The model returns a JSON object with the extracted fields.

Results and Limitations

The LLM approach showed remarkable flexibility. It correctly extracted fields even from documents with variable layouts, different fonts, and occasional typos. It handled ambiguous cases—like optional fields or multiple line‑items—better than the rule-based approach. However, it had its own challenges:

• Inference latency: Processing a page took 5–15 seconds on consumer hardware (CPU), which is 10× slower than OCR.
• Occasional hallucinations: The model sometimes invented plausible‑looking data (e.g., guessing a missing order number) or mis‑interpreted ambiguous text.
• Cost of compute: Running a 8B model locally still requires significant RAM and CPU/GPU resources; cloud LLM APIs would add per‑request costs.

Nevertheless, the LLM eliminated the need for template maintenance and adapted to new document types with only prompt modifications.

Side‑by‑Side Comparison

The table below summarizes key differences:

• Accuracy on known layouts: Rule-based ~95%, LLM ~92% (due to occasional mis‑extraction of numbers).
• Accuracy on unknown layouts: Rule‑based ~40%, LLM ~88%.
• Processing speed: Rule‑based <1 sec/page, LLM 5–15 sec/page.
• Maintenance effort: Rule‑based high (manual regex rules), LLM low (prompt updates).
• Hardware requirements: Rule‑based minimal (CPU only), LLM moderate (8GB+ RAM, GPU beneficial).

Conclusion: Which Approach Wins?

Neither approach is universally superior; the choice depends on your constraints. If you have a stable set of document templates and need high‑throughput, low‑latency extraction, a well‑tuned rule‑based system with pytesseract remains effective and cost‑efficient. But if your documents vary wildly in layout, or you must quickly support new formats without re‑coding, the LLM approach with Ollama and LLaMA 3 provides unprecedented adaptability—at the cost of slower inference and potential inaccuracies.

For many B2B scenarios, a hybrid solution may be best: use rule‑based extraction as the primary pipeline, and fallback to an LLM when confidence scores drop below a threshold. This balances speed and flexibility while keeping costs manageable. The key takeaway: rules excel in repetition; LLMs excel in reasoning. Choose your tool based on the chaos you expect in your documents.