• ✦ Also known as recursive structure aware chunking, content based chunking which can keep the context and format of text or the specific file types, such as HTML, PDF, Markdown, JSON.

• ✦ Simply put, using the right or suitable document splitter method for the use case will help us to derive chunks that are tailored to the specific file formats that we are dealing with.

  • This choice significantly impacts the quality and relevance of derived text chunks, offering benefits such as format-specific processing, preservation of structural elements, enhanced context retention, and improved accuracy in downstream tasks.
  • For example, when dealing with HTML files, specialized loaders can retain important elements like headings (<h1>), paragraphs (<p>), and tables (<table>), enabling custom processing based on element types.
  • However, it's not a magical solution; simply applying the technique is not enough. In many cases, developers need to write custom functions to effectively process the retained structural elements and extract meaningful information. This additional step ensures that the preserved document structure is fully utilized to meet specific analytical requirements and maximize the value of the chunking process.
  • It is still a good start with a suitable splitter method(s), rather than the overly simple splitter, like CharacterTextSplitter.
• ✦ Langchain supports many of the commonly used file types. Refer to the table below:

  • The table below shows the different text splitters offered by Langchain.
    • Name: Name of the text splitter
    • Splits On: How this text splitter splits text
    • Description: Description of the splitter, including recommendation on when to use it.

• ✦ Semantic chunking is one the more sophisticated chunking methods.

  • Semantic chunking relies heavily on embeddings, powerful tools for understanding text semantics.
  • Here’s how semantic chunking works in practice:
    • Text segments with similar meanings are grouped together.
    • Leveraging embeddings, we analyze and group consecutive sentences within a specified window size.
    • Beginning with the initial sentence, we compare its embedding to the subsequent sentences, iterating through the text until a significant deviation is detected, indicating a potential break point.
    • Continuously computing embeddings within each sentence set allows for dynamic adjustments, refining the grouping process and enhancing our understanding of the text’s meaning.
    • Through this method, we identify coherent groups of sentences that form meaningful sections, aiding in analysis and comprehension.

• ✦ The easiest way to take advantage of this cutting-edge chunking approach is to use Langchain's experimental module:

!pip install --quiet langchain_experimental langchain_openai

# Load Example Data
# This is a long document we can split up.  
with open("../../state_of_the_union.txt") as f:  
state_of_the_union = f.read()

# Create Text Splitter
from langchain_experimental.text_splitter import SemanticChunker  
from langchain_openai.embeddings import OpenAIEmbeddings

# That's it. It is this simple.
text_splitter = SemanticChunker(OpenAIEmbeddings())

# Spit Text
docs = text_splitter.create_documents([state_of_the_union])  
print(docs[0].page_content)
Copy

• ✦ In the real world, a user's query may not be properly phrased or optimized to get quality retrieval. This will affect the end output.
  • To overcome this issue, we can rewrite or rephrase the query so that it can optimally retrieve relevant context.
  • To help us to better understand the intuition behind query rewriting, see the code example below. We may modify the code to suit our use case (it doesn't have to be web search engine).
  • What we have to do is to use the "improved" query, instead of the original query for the RAG.

# The main part is a rewriter to rewrite the query
prompt = """Provide a better search query for \
web search engine to answer the given question. 

Question: {user_query}
"""
Copy

• ✦ You can refer to the notebook here for the complete implementation of this technique langchain/cookbook/rewrite.ipynb at master · langchain-ai/langchain (github.com)

• ✦ If the query is complex and has multiple context then, retrieval with a single query may not be the good approach as it may fail to get the proper output you want.

  • In sub query decomposition,
    • First, the user query is decomposed into multiple sub queries using LLM,
    • Then, retrievals using this sub queries are done in parallel and after that, these retrieved contexts are combined together as a single prompt for the final answer generation.
• ✦ In LangChain, we can use MultiQueryRetriever for implementation of this technique. The MultiQueryRetriever automates the process of prompt tuning by using an LLM to generate multiple queries from different perspectives for a given user input query.

  • For each query, it retrieves a set of relevant documents and takes the unique union across all queries to get a larger set of potentially relevant documents.
  • By generating multiple perspectives on the same question, the MultiQueryRetriever might be able to overcome some of the limitations of the distance-based retrieval and get a richer set of results.
  • Below is a sample implementation using MultiQueryRetriever

from langchain_chroma import Chroma
from langchain_community.document_loaders import WebBaseLoader
from langchain_openai import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Load blog post
loader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/")
data = loader.load()

# Split
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=0)
splits = text_splitter.split_documents(data)

# VectorDB
embedding = OpenAIEmbeddings()
vectordb = Chroma.from_documents(documents=splits, embedding=embedding)

# This is the Core Part of the Code
from langchain.retrievers.multi_query import MultiQueryRetriever
from langchain_openai import ChatOpenAI

question = "What are the approaches to Task Decomposition?"
llm = ChatOpenAI(temperature=0)
retriever_from_llm = MultiQueryRetriever.from_llm(
    retriever=vectordb.as_retriever(), llm=llm
)
Copy

• ✦ To understand the intuition behind this method, we can think of it as when an original query is received, it is passed to an LLM to generate 5 different related queries. Then each of these queries is used to retrieve the relevant documents. Here is the prompt being used by LangChain:

template="""You are an AI language model assistant. Your task is to generate five  
different versions of the given user question to retrieve relevant documents from a vector  
database. By generating multiple perspectives on the user question, your goal is to help  
the user overcome some of the limitations of the distance-based similarity search.  
Provide these alternative questions separated by newlines.  
Original question: {question}"""
Copy