• ✦ LLMs offer a natural language interface between humans and data. Widely available models come pre-trained on vast amounts of publicly available data, such as Wikipedia, mailing lists, textbooks, source code, and more.

• ✦ However, while LLMs are trained on a vast amount of data, they are not trained on your data, which may be private or specific to the problem you’re trying to solve. This data could be behind APIs, in SQL databases, or trapped in PDFs and slide decks.

• ✦ You might choose to fine-tune an LLM with your data, but:

  • Training an LLM is expensive.
  • Due to the cost of training, it's difficult to update an LLM with the latest information.
  • Observability is lacking. When you ask an LLM a question, it's not clear how the LLM arrived at its answer.
• ✦ Instead of fine-tuning, you can use a context augmentation pattern called Retrieval-Augmented Generation (RAG) to obtain more accurate text generation relevant to your specific data.

  • RAG involves the following high-level steps:
    1. Retrieve information from your data sources first.
    2. Add it to your question as context.
    3. Ask the LLM to answer based on the enriched prompt.
• ✦ By doing so, RAG overcomes all three weaknesses of the fine-tuning approach:

  • There's no training involved, so it's inexpensive.
  • Data is fetched only when you request it, ensuring it's always up-to-date.
  • It's more explainable, as most RAG frameworks allow you to display the retrieved documents, making it more trustworthy.

• ✦ Use document loaders to load data from a source as Document's.

  • A Document is a piece of text and associated metadata.
  • For example, there are document loaders for loading a simple .txt file, for loading the text contents of any web page, or even for loading a transcript of a YouTube video.

• ✦ See official documentation on LangChain's Document Loaders for different kinds of loaders for different sources.

• ✦ In this particular example, we are using one of the PDF loader from LangChain to load the Prompt Engineering Playbook.

from langchain.document_loaders import PyPDFLoader

loader = PyPDFLoader("https://www.developer.tech.gov.sg/products/collections/data-science-and-artificial-intelligence/playbooks/prompt-engineering-playbook-beta-v3.pdf")
pages = loader.load()
Copy

• ✦ The loader load each page of the PDF file as a separate Document object. The code below shows the first page of the PDF, by using index 0.

Once we loaded documents, we'll often want to transform them to better suit our application.

• ✦ The simplest example is you may want to split a long document into smaller chunks that can fit into your model's context window.

• ✦ LangChain has a number of built-in document transformers that make it easy to split, combine, filter, and otherwise manipulate documents.

• ✦ At a high level, text splitters work as following:

  1. Split the text up into small, semantically meaningful chunks (often sentences).
  2. Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).
  3. Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks).

• ✦ In the example, we are using the RecursiveCharacterTextSplitter from Langchain to split the given some_text into chunks. The resulting segments will have a maximum size of 26 characters, with an overlap of 4 characters between adjacent chunks.
  

The key parameters that we often see in splitter are the following:

1. chunk_size:
   • The chunk_size parameter determines the maximum length (in characters) of each chunk or segment into which the document is split.
   • A smaller chunk_size results in more fine-grained segments, while a larger value creates larger chunks.
   • Adjusting this parameter affects the granularity of the split text.
2. chunk_overlap:
   • The chunk_overlap parameter specifies the number of characters that overlap between adjacent chunks.
   • It controls how much context is shared between neighboring segments.
   • A higher chunk_overlap value increases the overlap, allowing for smoother transitions between chunks.
   • Conversely, a lower value reduces overlap, potentially leading to more distinct segments.

• ✦ Underlying the hood, there are two operations that happen at this step.

1. Get the embeddings of the text
2. Store the embeddings into a storage (a Vector store or a Vector Database)

• ✦ However, in frameworks such as LangChain, these two operations are often completed by a single method.
  
  

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import CharacterTextSplitter
from langchain_community.vectorstores import Chroma

from langchain_openai import OpenAIEmbeddings
Copy

embeddings_model = OpenAIEmbeddings(model='text-embedding-3-small')


db = Chroma.from_documents(splitted_documents, embeddings_model, persist_directory="./chroma_db")
Copy

• ✦ The last line creates a Chroma database from a collection of splitted documents.
  • The database is built using the specified embeddings model and is stored in the directory “./chroma_db”.
  • Chroma: Chroma is a library or tool designed for efficient similarity search and indexing of such as text embeddings.
  • from_documents: This method constructs a database from a list of Documents objects (LangChain's object).
  • persist_directory: Specifies the directory where the database will be stored for future use.

For the Retrieval stage, LangChain provides a variety of retrievers, each of which is an interface that returns documents given an unstructured query.

• ✦ Retrievers are more general than vector stores.
• ✦ A retriever does not need to be able to store documents, only to return (or retrieve) them.
• ✦ Vector stores can be used as the backbone of a retriever, but there are other types of retrievers as well.
• ✦ Retrievers accept a string query as input and return a list of Document objects as output.

from langchain.chains import RetrievalQA


qa_chain = RetrievalQA.from_chain_type(
    AzureChatOpenAI(model='gpt-3.5-turbo'),
    retriever=retriever_basic
)

qa_chain.invoke("Why LLM hallucinate?")
Copy

or we can also easily write our custom Q&A prompt for generating the answer

from langchain.prompts import PromptTemplate

  
# Build prompt

template = """Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer. Use three sentences maximum. Keep the answer as concise as possible. Always say "thanks for asking!" at the end of the answer.

{context}
Question: {question}
Helpful Answer:"""


QA_CHAIN_PROMPT = PromptTemplate.from_template(template)


# Run chain
qa_chain = RetrievalQA.from_chain_type(
    AzureChatOpenAI(model='gpt-3.5-turbo'),
    retriever=retriever_basic,
    return_source_documents=True, # Make inspection of document possible
    chain_type_kwargs={"prompt": QA_CHAIN_PROMPT}
)
Copy