PDF Processing Pipeline Documentation ==================================== Overview ------- .. raw:: html

A document processing pipeline is a structured sequence of steps that transforms raw, unstructured documents into formats suitable for training or inference with Large Language Models (LLMs). Its importance lies in ensuring data quality, relevance, and usability, which directly impact LLM performance.

-------------------------------------------------------------------------------------------------------------------- Docling: Next‑Generation Document Processing ----------------------------- Overview _________________________________ .. raw:: html

Docling is a powerful document parsing tool designed to extract structured, layout-aware content from various file types such as PDF, DOCX, PPTX, HTML, and more. It transforms these documents into a rich unified representation that retains important elements like layout, headings, tables, and sections. This makes the documents ideal for generative AI workflows, including Retrieval-Augmented Generation (RAG).

Docling is integrated with LangChain via the DoclingLoader, which allows seamless integration of complex documents into LLM pipelines.

Architecture ____________________ .. raw:: html

The architecture consists of the following components:

.. figure:: /Documentation/images/docling_arch.png :width: 100% :alt: Alternative text for the image :name: logo In a nutshell, Docling's architecture is outlined in the diagram above. .. raw:: html

For each document format, the document converter knows which format-specific backend to employ for parsing the document and which pipeline to use for orchestrating the execution, along with any relevant options.

🔧 Key Features _____________________ ✅ Supports multiple document formats: PDF, DOCX, PPTX, HTML, and more. ✅ Preserves layout, tables, headings, and sections. ✅ Converts documents into formats optimized for LLMs. ✅ Works natively with LangChain. ✅ Supports two export modes for flexibility: .. raw:: html

ExportType.DOC_CHUNKS: Splits documents into smaller chunks, each treated as a separate LangChain Document.

ExportType.MARKDOWN: Converts the entire document into a single markdown-formatted LangChain Document.

🚀 Installation ____________________ .. raw:: html

To install Docling, use the following command:

```bash pip install -qU langchain-docling ``` .. raw:: html

This will install the latest version of Docling along with the necessary integrations for LangChain.

📦 Usage ____________________ .. raw:: html

To use Docling, you can import the DoclingLoader and specify the document path and export type. Here’s a simple example:

1. Import Required Modules

.. code-block:: python from langchain_docling import DoclingLoader from langchain_docling.schema import ExportType .. raw:: html

2. Choose Export Type

You can choose between:

* ExportType.DOC_CHUNKS – Document is split into chunks * ExportType.MARKDOWN – Whole document as a single markdown output .. code-block:: python export_type = ExportType.DOC_CHUNKS # or ExportType.MARKDOWN .. raw:: html

3. Load Document

Specify the document path and export type:

.. code-block:: python doc_path = "path/to/your/document.pdf" # Specify your document path loader = DoclingLoader(doc_path, export_type=export_type) # Load the document documents = loader.load() .. raw:: html

This will load the document and return a list of LangChain Document objects, each containing the extracted content.

4. Use with LangChain

Once loaded, the documents can be passed into any LangChain component like a retriever, summarizer, or question-answering chain.

📁 Supported File Types ____________________ .pdf .docx .pptx .html .txt .. raw:: html

Other common document formats

--------------------------------------------------------------------------------------------------------------------------------------- 📄 PDF Data Extraction docling ==================================== 🧠 Goal ------------------- **We will:** .. raw:: html

Extract all content from a PDF

Save the content in a DocLing-compatible format (e.g., HTML or Markdown with clear structure)

Optionally generate metadata for easier annotation

✅ 📄 Code: PDF Extraction for DocLing Annotation -------------------------------------------------- .. code-block:: python # Required Libraries import os import pdfplumber import fitz # PyMuPDF from pathlib import Path # Create Output Folder output_dir = Path("docling_output") output_dir.mkdir(exist_ok=True) def extract_pdf_content(pdf_path): """ Extracts text, images, and tables from a PDF file. Outputs a structured HTML-like file ready for DocLing annotation. """ output_file = output_dir / f"{Path(pdf_path).stem}_docling_ready.html" .................................................. .................................................. # The function code is in `structuration_pdf.ipynb`. Open the notebook to view and experiment. .................................................. ........................................................ print(f"✅ Extracted content saved to: {output_file}") # Example usage extract_pdf_content("example.pdf") .. raw:: html

code-link:

       
        GitHub  link - for DocLing Annotation code

🔎 Output ------------------- ✅ docling_output/example_docling_ready.html: contains all structured content ✅ Images saved as page1_img1.png, page2_img1.jpg etc. ✅ Human-readable format for manual annotation in DocLing 🧠 How to Use with DocLing -------------------- .. raw:: html

1. Open the HTML file in a browser or upload it to a DocLing-compatible editor

.. raw:: html

2. Use DocLing to:

* Add metadata (e.g., title, author) * Annotate sections (e.g., introduction, methods, conclusion) * Tag named entities (dates, places, etc.) .. raw:: html

3. Export as JSON, XML, or HTML for NLP training or archiving

✅ Summary ------------------ .. raw:: html

Feature	Included
Text Extraction	✅
Table Extraction	✅
Image Extraction	✅
Prepares for DocLing	✅

🧠 Final Notes ------------------- .. raw:: html

DocLing is not a Python API, but a manual annotation platform.

This script prepares your PDFs to be easily annotated using DocLing.

If you need a script where annotations are automated, consider:

* Using spaCy + pdfplumber * Or using Label Studio (which has a Python API) Practice ---------------------- .. raw:: html

link:

        Colab Notebook -docling extracted PDF 
        
        GitHub Repository - PDF Extraction Script "docling"