{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "b2c3c554daa1" }, "source": [ "# Comparing LlamaIndex and LlamaParse for Dense Document Questioning Answering on Vertex AI\n", "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \"Google
Open in Colab\n", " \n", " 		\n", " \n", " \"Google
Open in Colab Enterprise\n", " \n", " 		\n", " \n", " \"Vertex
Open in Vertex AI Workbench\n", " \n", " 		\n", " \n", " \"GitHub
View on GitHub\n", " \n", "
\n", "\n", "
\n", "\n", "Share to:\n", "\n", "\n", " \"LinkedIn\n", "\n", "\n", "\n", " \"Bluesky\n", "\n", "\n", "\n", " \"X\n", "\n", "\n", "\n", " \"Reddit\n", "\n", "\n", "\n", " \"Facebook\n", " " ] }, { "cell_type": "markdown", "metadata": { "id": "1bc291e87bd1" }, "source": [ "| | |\n", "|-|-|\n", "| Author(s) | [Noa Ben-Efraim](https://github.com/noabenefraim/) |" ] }, { "cell_type": "markdown", "metadata": { "id": "648701a7bf44" }, "source": [ "## Overview\n", "There are many ways to customize RAG pipelines by choosing how to ingest, parse, chunk, and retrieve your data. This notebook focuses on comparing different document parsing capabilities offered by LlamaIndex.\n", "\n", "We will compare document parsing with LlamaIndex and LlamaParse on a 10-Q financial document, which is heavily populated with complex tables.\n", "\n", "### Objectives\n", "This notebook compare using LlamaIndex and LlamaParse for ingesting and indexing a complex document. \n", "\n", "You will complete the following tasks:\n", "1. Ingest and parse document using LlamaIndex SimpleDataReader, LlamaIndex LangchainNodeParser, and LlamaParse Parser using Gemini models.\n", "2. Index your parsed document in a VectorStore.\n", "3. Create a a query agent for each parsing technique that can answer questions against the input document.\n", "4. Compare results across LlamaIndex and LlamaParse.\n", "\n", "### LlamaIndex\n", "LlamaIndex is a foundational data framework for building LLM applications. A few of its main capabilities are:\n", "\n", "+ Data Ingestion: Loads your data from various sources (documents, databases, APIs).   \n", "+ Indexing: Structures your data into efficient formats for LLM retrieval (e.g., vector stores, tree structures).   \n", "+ Querying: Enables you to ask questions or give instructions to the LLM, referencing your indexed data for answers.   \n", "+ Integration: Connects with various LLMs, vector databases, and other tools.   \n", "  \n", "\n", "### LlamaParse\n", "LlamaParse is a tool within the LlamaIndex ecosystem, focused on parsing complex documents:\n", "\n", "+ PDFs: Handles PDFs with tables, charts, and other embedded elements that can be challenging for standard parsing.  \n", "+ Semi-structured Data: Extracts structured information from documents that aren't fully formatted databases.   \n", "+ Enhanced Retrieval: Works seamlessly with LlamaIndex to improve retrieval accuracy for complex documents." ] }, { "cell_type": "markdown", "metadata": { "id": "87008e44295a" }, "source": [ "## Getting Started\n", "\n", "### Authenticate your notebook environment\n", "\n", "This notebook expects the following resources to exists:\n", "+ Initialized Google Cloud project \n", "+ Vertex AI API enabled\n", "+ GCS Bucket and Vertex AI Search Index and Endpoint\n", "+ A LlamaParse API Key [request a key here](https://docs.cloud.llamaindex.ai/llamacloud/getting_started/api_key)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "516b8d774169" }, "outputs": [], "source": [ "PROJECT_ID = \"\" # @param {type:\"string\"}\n", "REGION = \"\" # @param {type:\"string\"}\n", "GCS_BUCKET = \"\" # @param {type:\"string\"}\n", "VS_INDEX_NAME = \"\" # @param {type:\"string\"}\n", "VS_INDEX_ENDPOINT_NAME = \"\" # @param {type:\"string\"}\n", "DATA_FOLDER = \"./data\" # @param {type:\"string\"}" ] }, { "cell_type": "markdown", "metadata": { "id": "205f199d8a53" }, "source": [ "### Set Google Cloud project information and initialize Vertex AI SDK\n", "\n", "To get started using Vertex AI, you must have an existing Google Cloud project and [enable the Vertex AI API](https://console.cloud.google.com/flows/enableapi?apiid=aiplatform.googleapis.com).\n", "\n", "Learn more about [setting up a project and a development environment](https://cloud.google.com/vertex-ai/docs/start/cloud-environment)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "dd11490e3764" }, "outputs": [], "source": [ "PROJECT_ID = \"\" # @param {type:\"string\"}\n", "LOCATION = \"\" # @param {type:\"string\"}\n", "\n", "import vertexai\n", "\n", "vertexai.init(project=PROJECT_ID, location=LOCATION)" ] }, { "cell_type": "markdown", "metadata": { "id": "f818f7127a12" }, "source": [ "### Setting up the Environment\n", "Install dependencies" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "c76df03a88f3" }, "outputs": [], "source": [ "%pip install google-cloud-aiplatform \\\n", " llama-index \\\n", " langchain-community \\\n", " llama-index-embeddings-vertex \\\n", " llama-index-llms-vertex \\\n", " termcolor \\\n", " llama-index-core -q" ] }, { "cell_type": "markdown", "metadata": { "id": "44d71c554004" }, "source": [ "Set up imports" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "fb7e65420d45" }, "outputs": [], "source": [ "from langchain.text_splitter import RecursiveCharacterTextSplitter\n", "from llama_index.core import Settings, SimpleDirectoryReader, VectorStoreIndex\n", "from llama_index.core.base.response.schema import Response\n", "from llama_index.core.extractors import KeywordExtractor, QuestionsAnsweredExtractor\n", "from llama_index.core.ingestion import IngestionPipeline\n", "from llama_index.core.node_parser import LangchainNodeParser\n", "from llama_index.embeddings.vertex import VertexTextEmbedding\n", "from llama_index.llms.vertex import Vertex\n", "from llama_index.vector_stores.vertexaivectorsearch import VertexAIVectorStore\n", "from llama_parse import LlamaParse\n", "from termcolor import colored" ] }, { "cell_type": "markdown", "metadata": { "id": "8d7a288c44bb" }, "source": [ "Generate credentials" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "id": "5e5cb3d58df5" }, "outputs": [], "source": [ "import google.auth\n", "import google.auth.transport.requests\n", "\n", "# credentials will now have an api token\n", "credentials = google.auth.default(quota_project_id=\"genai-noabe\")[0]\n", "request = google.auth.transport.requests.Request()\n", "credentials.refresh(request)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0e9455150d49" }, "outputs": [], "source": [ "embedding_model = VertexTextEmbedding(\"text-embedding-005\", credentials=credentials)\n", "llm = Vertex(model=\"gemini-2.5-flash\", temperature=0.0, max_tokens=5000)\n", "\n", "Settings.embed_model = embedding_model\n", "Settings.llm = llm" ] }, { "cell_type": "markdown", "metadata": { "id": "f5822037c828" }, "source": [ "Set up LlamaIndex settings to point to Gemini models." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "a6c832d2780c" }, "outputs": [], "source": [ "import IPython\n", "\n", "app = IPython.Application.instance()\n", "app.kernel.do_shutdown(True)" ] }, { "cell_type": "markdown", "metadata": { "id": "8daf430888fc" }, "source": [ "### Authenticate your notebook environment (Colab only)\n", "\n", "If you're running this notebook on Google Colab, run the cell below to authenticate your environment." ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "885a3c84ddac" }, "outputs": [], "source": [ "import sys\n", "\n", "if \"google.colab\" in sys.modules:\n", " from google.colab import auth\n", "\n", " auth.authenticate_user()" ] }, { "cell_type": "markdown", "metadata": { "id": "e9a64d5bf6c3" }, "source": [ "### Download sample data\n", "\n", "For the remainder of the notebook we will examine Alphabet Inc. 10Q document. A 10Q is a financial document that is dense with tables with financial figures. This document is a great candidate to to investigate document parsing capabilities." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "e0c896dfda70" }, "outputs": [], "source": [ "!mkdir {DATA_FOLDER}\n", "!wget \"https://abc.xyz/assets/ae/e9/753110054014b6de4d620a2853f6/goog-10-q-q2-2024.pdf\" -P {DATA_FOLDER}" ] }, { "cell_type": "markdown", "metadata": { "id": "ece6a448d7e0" }, "source": [ "## Document Parsing with LlamaIndex\n", "\n", "This section will ingest and parse the 10Q using LlamaIndex tools, specifically focusing on SimpleDirectoryReader and LangChainNodeParser.\n", "\n", "### Option 1: `SimpleDirectoryReader`\n", "The SimpleDirectoryReader is the core data ingestion tool in LlamaIndex. It's designed to load data from a variety of sources and convert it into a format suitable for further processing and indexing by LlamaIndex." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7b343b4be5d9" }, "outputs": [], "source": [ "reader = SimpleDirectoryReader(\"./data\")\n", "documents = reader.load_data(show_progress=True)\n", "print(documents[0])" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "f5e3def62a59" }, "outputs": [], "source": [ "# Index the parsed document\n", "simpledirectory_index = VectorStoreIndex.from_documents(documents)\n", "\n", "# Generate a query engine based on the SimpleDataReader\n", "simple_query_engine = simpledirectory_index.as_query_engine(similarity_top_k=2)" ] }, { "cell_type": "markdown", "metadata": { "id": "ec35c0de8639" }, "source": [ "### Option 2: `LangChainNodeParser` with LlamaIndex\n", "The LangChainNodeParser is a part of LlamaIndex. It is a specialized parser designed to extract structured information from text documents using the power of LangChain.\n", "\n", "Key Features:\n", "+ LangChain Integration: Leverages LangChain's powerful language models and tools to parse text.\n", "+ Node-Based Output: Converts unstructured text into a structured format based on a defined schema, represented as a hierarchy of nodes. This enables more sophisticated querying and analysis of the extracted information.\n", "+ Customization: Supports defining custom parsing schemas to match the structure of your specific documents.\n", "+ Flexibility: Can be used in combination with other LlamaIndex components, such as the SimpleDataReader, to process and index the extracted structured data." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "id": "c94da573044a" }, "outputs": [], "source": [ "parser = LangchainNodeParser(RecursiveCharacterTextSplitter())\n", "langchain_nodes = parser.get_nodes_from_documents(documents)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "e8e2f0b882ce" }, "outputs": [], "source": [ "# An example node that was generated using the LangChainNodeParser and the associated metadata\n", "langchain_nodes[0]" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "id": "a88552f52124" }, "outputs": [], "source": [ "# Index the document based on the LangChain nodes generated above\n", "langchainparser_index = VectorStoreIndex(nodes=langchain_nodes)\n", "\n", "# Create a query engine based off the LangChainNodeParser\n", "lg_query_engine = langchainparser_index.as_query_engine(similarity_top_k=2)" ] }, { "cell_type": "markdown", "metadata": { "id": "adc97e363bac" }, "source": [ "## LlamaParse\n", "\n", "LlamaParse Parser is a powerful tool for extracting structured data from unstructured or semi-structured text, offering flexibility, customization, and seamless integration within the LlamaIndex framework.It can take an unstructured or semi-structured text document and, using a defined schema, extract structured information from it. This structured output is represented as a nested hierarchy of nodes, facilitating further processing and analysis.\n", "\n", "A few key features include:\n", "\n", "+ JSON Schema: Leverages the standardized JSON Schema format for more complex schemas.\n", "+ Prompt Templates: Allows you to craft custom prompts to guide the language model's parsing behavior, offering greater control and adaptability.\n", "+ LLM Selection: You have the flexibility to choose the specific LLM you want to use for parsing, enabling you to tailor the performance to your specific needs and budget.\n", "+ Node-Based Output:\n", " + Structured Representation: The parsed output is organized into a hierarchy of nodes, each representing a piece of extracted information.\n", " + Nested Structure: Nodes can contain other nodes, allowing for the representation of complex relationships and nested data structures within the document.\n", " + Metadata: Nodes can also include additional metadata, such as confidence scores or source information, enriching the extracted data.\n", "+ Integration with LlamaIndex: The structured output from parser() seamlessly integrates with other LlamaIndex components, such as indexing and querying, facilitating efficient retrieval and analysis of the extracted information.\n", "\n", "#### Define a Parser\n", "\n", "Here we will define a LlamaParse() parser with specific parsing instructions, and ingest the data." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "id": "85ec700afd80" }, "outputs": [], "source": [ "parser = LlamaParse(\n", " parsing_instruction=\"You are a financial analyst working specifically with 10Q documents. Not all pages have titles. Try to reconstruct the dialogue spoken in a cohesive way.\",\n", " api_key=\"\",\n", " result_type=\"text\", # \"markdown\" and \"text\" are available\n", " language=\"en\",\n", " invalidate_cache=True,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "id": "23fd5739fa74" }, "source": [ "### Option 1 - LlamaParse with SimpleDirectoryReader\n", "\n", "This is the apples to apples comparison with LlamaIndex. We are using the SimpleDirectoryReader with the LlamaParse file extractor, and then loading the data directly from documents to a Vector Store for retrieval." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "688898e8b334" }, "outputs": [], "source": [ "import nest_asyncio\n", "\n", "nest_asyncio.apply()\n", "file_extractor = {\".pdf\": parser}\n", "documents = SimpleDirectoryReader(\n", " input_files=[\"./data/goog-10-q-q2-2024.pdf\"], file_extractor=file_extractor # type: ignore\n", ").load_data()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "id": "d215261053fb" }, "outputs": [], "source": [ "lp_simple = VectorStoreIndex.from_documents(documents)\n", "lp_simple_engine = lp_simple.as_query_engine(similarity_top_k=2)" ] }, { "cell_type": "markdown", "metadata": { "id": "906057ddf157" }, "source": [ "### Option 2 - LlamaParse and Vertex AI Vector Search\n", "\n", "This approach is a more customized approach by defining the Vector Search mechanism through Vertex AI and extracting metadata that will be embedded and stored in the search index. \n", "\n", "Using metadata in Retrieval Augmented Generation (RAG) improves accuracy and context by focusing searches and providing additional information. This leads to efficient filtering, ranking, and personalized responses tailored to user needs and history. Metadata also facilitates handling complex multi-criteria queries, making RAG systems more versatile and effective.\n", "\n", "The following section will:\n", "+ Parse the documents using LlamaParse\n", "+ Extract metadata from documents returned from LlamaParse\n", "+ Create metadata embeddings attached to each document\n", "+ Create index in Vertex AI Vector Store\n", "+ Query against the Vector Store" ] }, { "cell_type": "markdown", "metadata": { "id": "0ab41643425c" }, "source": [ "#### Parse data using LlamaParse" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "d4d5d47ad69e" }, "outputs": [], "source": [ "documents = parser.load_data(\"./data/goog-10-q-q2-2024.pdf\")" ] }, { "cell_type": "markdown", "metadata": { "id": "e5bde318580a" }, "source": [ "#### Create Metadata from Nodes\n", "\n", "Using extractors we will generate meta-data for each node. The metadata is generated using Gemini-Pro and focuses on what questions can this text answer and what key words are meaningful in this section. Each metadata piece will be embedded with Gemini text-embedding model. \n", "\n", "Creating metadata can be useful for another lookup criteria during RAG based search." ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "id": "ea18fa35d496" }, "outputs": [], "source": [ "extractors = [\n", " QuestionsAnsweredExtractor(questions=3, llm=llm),\n", " KeywordExtractor(keywords=10, llm=llm),\n", "]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fd57de02eb9b" }, "outputs": [], "source": [ "# Run metadata transformation pipeline.\n", "pipeline = IngestionPipeline(\n", " transformations=extractors, # type: ignore\n", ")\n", "nodes = await pipeline.arun(documents=documents, in_place=False)" ] }, { "cell_type": "markdown", "metadata": { "id": "8929ab07f054" }, "source": [ "Example metadata that was generated:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "b41541f14b77" }, "outputs": [], "source": [ "print(nodes[1].metadata)" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "id": "476642ff9305" }, "outputs": [], "source": [ "# Generate embeddings for each metadata node\n", "for node in nodes:\n", " node_embedding = embedding_model.get_text_embedding(\n", " node.get_content(metadata_mode=\"all\")\n", " )\n", " node.embedding = node_embedding" ] }, { "cell_type": "markdown", "metadata": { "id": "6e52150147f8" }, "source": [ "#### Load Nodes into Predefined Vector Store\n", "\n", "This following section required a preexisting Vertex AI Vector Store. Vector stores contain embedding vectors of ingested document chunks.\n", "\n", "For information to create a vector store, refer to this link https://docs.llamaindex.ai/en/stable/examples/vector_stores/VertexAIVectorSearchDemo/" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "f8cdc3678a71" }, "outputs": [], "source": [ "vector_store = VertexAIVectorStore(\n", " project_id=PROJECT_ID,\n", " region=REGION,\n", " index_id=\"\", # Add in your Vertex AI Vector Search Index ID\n", " endpoint_id=\"\", # Add in your Vertex AI Vector Search Deployed Index ID\n", " gcs_bucket_name=GCS_BUCKET,\n", ")\n", "\n", "# Only need to run once\n", "vector_store.add(nodes)" ] }, { "cell_type": "markdown", "metadata": { "id": "d22ef6b3a7e1" }, "source": [ "#### Create a search index and search and query the Vector Store" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "id": "c135cf0ee925" }, "outputs": [], "source": [ "lp_index = VectorStoreIndex.from_vector_store(vector_store)\n", "lp_query_engine = lp_index.as_query_engine(similarity_top_k=2)" ] }, { "cell_type": "markdown", "metadata": { "id": "7064bdd2c434" }, "source": [ "## Query Comparison between LlamaIndex and LlamaParse\n", "Below are queries that responses can be found in the 10Q document within complex tables. Let's see how each approach compares." ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "id": "bef0f00ab246" }, "outputs": [], "source": [ "queries = [\n", " \"What are the total cash, cash equivalents, and marketable securities as of Dec 23 2023\",\n", " \"Total investments with fair value change reflected in other comprehensive income as of Dec 23 2023\",\n", " \"What is the corporate debt securities unrealized loss as of Dec 31 2023 for 12 months or greater?\",\n", " \"What is the coupon rate for total outstanding debt\",\n", " \"Provide the table of share repurchases\",\n", "]" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "id": "a0c410e45aea" }, "outputs": [], "source": [ "def print_output(response: Response):\n", " print(\"Response:\")\n", " print(\"-\" * 80)\n", " print(colored(response.response, color=\"red\"))\n", " print(\"-\" * 80)\n", " print(\"Source Documents:\")\n", " print(\"-\" * 80)\n", " for source in response.source_nodes:\n", " print(f\"Sample Text: {source.text[:100]}\")\n", " print(f\"Relevance score: {source.get_score():.3f}\")\n", " print(f\"File Name: {source.metadata.get('file_name')}\")\n", " print(f\"Page #: {source.metadata.get('page_label')}\")\n", " print(f\"File Path: {source.metadata.get('file_path')}\")\n", " print(\"-\" * 80)\n", "\n", "\n", "def run_query(query_idx: int):\n", " query = queries[query_idx]\n", " print(\"Query: \" + query)\n", " print(colored(\"LlamaIndex SimpleDirectoryReader response....\\n\", color=\"blue\"))\n", " print_output(simple_query_engine.query(query))\n", "\n", " print(\n", " colored(\n", " \"LlamaIndex LangChainNodeParser on LlamaIndex response....\\n\", color=\"blue\"\n", " )\n", " )\n", " print_output(lg_query_engine.query(query))\n", "\n", " print(colored(\"LlamaParse Simple response....\\n\", color=\"blue\"))\n", " print_output(lp_simple_engine.query(query))\n", "\n", " print(colored(\"LlamaParse on Vertex AI response....\\n\", color=\"blue\"))\n", " print_output(lp_query_engine.query(query))\n", " print(\"###################################################\\n\\n\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "3a90e4c451cf" }, "outputs": [], "source": [ "run_query(query_idx=0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "8095a2092eab" }, "outputs": [], "source": [ "run_query(query_idx=1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "fe4cac0c796c" }, "outputs": [], "source": [ "run_query(query_idx=2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "e4deb4eeaa0d" }, "outputs": [], "source": [ "run_query(query_idx=3)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "c837ac4f41f5" }, "outputs": [], "source": [ "run_query(query_idx=4)" ] }, { "cell_type": "markdown", "metadata": { "id": "ceb7a1bbb11e" }, "source": [ "## Observations\n", "\n", "### Answer Key\n", "| Query | Answer | Citation page |\n", "|------------------------------------------------------------------------------------------------------|------------------|---------------|\n", "| \"What are the total cash, cash equivalents, and marketable securities as of Dec 23 2023\" | $110,916 million | 5 |\n", "| \"Total investments with fair value change reflected in other comprehensive income as of Dec 23 2023\" | $78,917 million | 13 |\n", "| \"What is the corporate debt securities unrealized loss as of Dec 31 2023 for 12 months or greater? | 592 million | 15 |\n", "| \"What is the coupon rate for total outstanding debt\" | 0.45-2.25% | 22 |\n", "| \"Provide the table of share repurchases\" | Table | 27 or 49 |\n", "\n", "### Generated Answers\n", "| Document Parsing Technique | Query 1 | Query 2 | Query 3 | Query 4 | Query 5 |\n", "|------------------------------------------|---------|---------|---------|---------|---------|\n", "| LlamaIndex - SimpleDirectoryReader | (✓) | (✓) | (✓) | (✓) | (✓) |\n", "| LlamaIndex - LangChainNodeParser | (✓) | (✓) | (✓) | (✓) | (✓) |\n", "| LlamaParse - SimpleDirectoryReader | (✓) | (✓) | (✓)| (✓) | (✓) |\n", "| LlamaParse - Vertex AI Vector Search | (✓) | (✓) | (✓)| (✓) | (✓) |\n", "\n", "## Conclusion\n", "\n", "There are many ways to customize your data ingestion and retrieval pipelines for custom RAG applications. This notebook was an overview to a handful of options that work in combination with Google Gemini models. " ] } ], "metadata": { "colab": { "name": "doc_parsing_with_llamaindex_and_llamaparse.ipynb", "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" } }, "nbformat": 4, "nbformat_minor": 0 }