We have been talking about the importance of prompt engineering in general in 2. Prompt Engineering. In this note, we will cover the important techniques that we will be using to format the prompts in your Python code.

Mastering these basics will make calling LLMs programmatically using Python more efficient and is particularly important for applications that require complex prompts.

• ✦ String Formatting (f-strings):
  • When interacting with a Large Language Model (LLM) programmatically, you often need to construct prompts dynamically based on certain conditions or variables.
  • F-string formatting in Python allows you to embed expressions or variable(s) inside a string for easier string formatting that also makes it more readable.
  • Note that f-string formatting is only available in Python 3.6 and later versions.

name = "Alice"
age = 25
print(f"Hello, my name is {name} and I am {age} years old.")

Copy

• ✦ Multi-line Strings (Triple double or single quotes):
  • LLMs often require multi-line prompts for complex tasks or to provide additional context.
  • Triple quotes in Python allow you to define multi-line strings easily.
  • This is especially useful when the prompt includes several lines of instructions or needs to maintain a specific formatting.

print("""
Hello,
This is a multi-line string.
Each new line is preserved in the output.
""")

Copy

• ✦ Combining String Formatting and Multi-line Strings for Powerful Template
  • Both techniques can create dynamic and reusable templates for Large Language Models (LLMs).
  • F-string formatting allows you to insert any valid Python expression into the string, making the template highly flexible, while Multi-line strings allow you to preserve the formatting of the template, making it easier to read and understand.
  • By including the template in a function, you can easily reuse it with different values.

def greet(name, age):
    return f"""
    Hello, my name is {name} 
    and I am {age} years old.
    """

print(greet("Alice", 25))

Copy

• ✦ Maintainability & Error Minimisation:
  • Just like function in a Python program, it allows us to reduce duplicative parts of the prompts, allowing for the same template to be applied across different scenarios
  • If a change is needed, it can be made in one place, and all instances of the template will reflect that change
  • Proper string formatting helps prevent errors.
    • For example, forgetting to close a quote could lead to syntax errors. Using triple quotes for multi-line strings can help avoid such issues.

• ✦ Readability and Maintainability:
  • Quickly discern the included inputs, which can differ depending on the actual user input
  • Using f-strings and triple quotes makes your code more readable and maintainable.
  • It allows others (or future you) to understand what your code is doing, which is always a good practice in programming.

We think you probably have already heard a thousand times about what an LLM is, so we won’t overload you with all the definitions again. If there is one key thing to understand about Large Language Models (LLMs), it is this: they are LARGE neural network models designed to predict the next token in a sequence based on the preceding tokens. That’s the essence of their functionality.

The popularity of LLMs is due to their versatility and effectiveness. They perfectly cope with tasks such as translation, summarisation, sentiment analysis, information extraction, etc. We will learn more about these use cases along the way.

• ✦ Closed-source Large Language Models:
  • These are developed and maintained by specific organisations and their source code is not publicly available.
    • An example is GPT-4 by OpenAI, which is a powerful language model but its training code and model weights are not open source.
  • Popular Closed Source models
    • GPT-4 by OpenAI
    • [Gemini](Introducing Gemini 1.5, Google's next-generation AI model (blog.google)) by Google
    • Claude by Anthropic
• ✦ Open-source Large Language Models:
  • These are developed in a collaborative public manner where the source code is freely available.
    • You can host these models by yourself, usually on a server or powerful machine.
    • A great place is to find these model is Hugging Face’s Hub, which provides thousands of pre-trained models in 100+ languages and deep learning frameworks like PyTorch and TensorFlow.
  • Popular Open Source models
    • LLaMA-3 by Meta
    • Mistral by Mistral AI
    • BLOOM by BigScience
    • Falcon by Technology Innovation Institute in Abu Dhabi

While there are quite a few differences between the Open Source vs Closed Source Models, there is no definitive answer as to which is better or worse. We highlight the following as some key considerations:

When it comes to quality, which most of us care the most about, the majority of open-source LLMs are still performing worse than GPT-3.5 and GPT-4. Both on standard benchmarks.

💡 Don't worry about understanding how to interpret the benchmarks table. These benchmarks are used to evaluate the capabilities of language models in understanding, reasoning, and problem-solving in various domains.

Here is the models' performance on various tasks:

Ever since the start of LLM hype, you may have found a lot of discussions around “Fine-tune your Private LLaMA/Falcon/Another Popular LLM”, “Train Your Own Private ChatGPT”, “How to Create a Local LLM” and others.

However, very few people will tell you why you need it. Are you really sure you need your own self-hosted LLM?

To illustrate this further, let’s consider the cost of hosting a LLaMA-2–70B model on both AWS and GCP. It’s worth noting that most companies employ smaller model versions and fine-tune them according to their tasks. However, in this example we intentionally chose the largest version because it’s a model that can match the quality of GPT-3.5 (Yes, not GPT-4).

• ✦ Additionally, let’s include the following extra expenses to the server cost:
  • Payment for DevOps specialists who will handle server setup, load balancing, and monitoring.
  • Payment for ML engineers responsible for model preparation, maintenance, and fine-tuning.
  • Optionally, one-time payment for dataset collection and annotation for fine-tuning.

It's estimated this to be approximately$40k — $60k per month on GCP for inference LLaMA-2–70B.

However, don't take us wrongly, it doesn't mean self-hosting is not resource feasible or reasonable. For lower usage in the realm of 10,000 to 50,000 requests per day, it might be cheaper to use managed services where the models are hosted by companies (e.g., OpenAI, Claude, or Gemini). But after a certain usage level, the cost for self-hosting LLMs would be lower than using managed services. See the image below.

The LLM community believes that in the near future, we will witness a significant increase in the accuracy of new models, including the open-source models, thanks to the active involvement and support of the community.

• • [2307.06435] A Comprehensive Overview of Large Language Models (arxiv.org)
• • Benchmarking of LLMs
• •You don’t need hosted LLMs, do you? | by Sergei Savvov | Better Programming (Paywall)
• • Cloud GPUs (cloud-gpus.com)
• • LLM Collection)

Base Language Models vs. Instruction Tuned Language Models

• ✦ Getting the LLMs to give you the output you want can be challenging. It can be likened to taming magical beasts because they may not do your bidding. This is where prompt engineering comes in.
  • A prompt is the term used to describe a question, statement, or instruction given to an LLM, like ChatGPT.
  • It serves as a starting point for the AI to generate a response or complete a task. 

• ✦ Prompt engineering refers to the art and science of the design of prompts for LLMs to produce high-quality, coherent outputs.
  • This is a crucial aspect of working with LLMs, as the quality of the prompt can greatly affect the quality of the generated text.
  • One fundamental challenge is that the responses from LLM are not fully predictable.
  • Indeed, it is not an exact science. Sometimes, changing a word or the order the sentences in the prompt may produce a very different response.

• ✦ Why do they sometimes provide high-quality responses and other times fabricate facts (or what we call hallucinate)?

  • Or why does adding let’s think step-by-step to a prompt suddenly improve the quality?
  • But the truth is, we still don’t fully understand how LLMs work.
• ✦ We won’t bore you with complex prompt just yet; instead, we will just share a few examples that can instantly improve the performance or your prompts:

  • “Let’s think step by step” — works great for reasoning or logical tasks..
  • “Take a deep breath and work on this problem step-by-step“— an improved version of the previous point. It can add a few more percent of quality
  • “This is very important to my career” — just add it to the end of your prompt and you’ll notice a 5–20% improvement in quality.
  • "I will reward you with $100 if your response is great" - try and it and share with us!

• ✦ Due to all this, scientists and enthusiasts can only experiment with different prompts, trying to make models perform better.

• ✦ Writing an effective prompt can be a tricky process that demands patience, creativity, and a good grasp of how LLMs behave in general.
  • It is an iterative process that involves some trial-and-error to progressively refine the prompts to get closer to your intended responses.
  • While the outputs from LLM are unpredictable, we can use well-established prompting techniques to increase the consistency and desirability of the LLM output.
  • For this reason, prompt engineering is also sometimes called natural language programming.
    
• ✦ After signing up for AI Champions Bootcamp, all participants have been encouraged to complete the 🪄📖 Prompt Engineering Playbook 📖🪄 before this programme starts.
  • If you have yet to complete the Playbook, we'd like you to go through page 1 to page 87 of the Playbook before trying out the hands-on tasks for week 1.

If you're keen to explore further, there are extra resource in 6. Further Readings that you might find interesting, including tools that current under active research or tools that the open source community has built.

• Not So Typical Intro to LLMs
• Prompt Engineering
• Formatting Prompt in Python
• Hands-on Walkthrough and Tasks

• ✦ Open https://platform.openai.com on your browser and log in using the OpenAI account you have created previously (and topped up with some credits).

  

• ✦ Fill up the required details.

  • Select "You" under the "Owned by" section
  • The Name field can be any meaningful text that describe the purpose of the key
  • Click on the "Create secret key"
    
• ✦ Copy and save the API Key

  • Highlight the entire API key: sk-waMT92zQxaswdawOM2Rcy2oCKhy1T3BlaxbkFJ9KaK
  • Right-click and select “Copy” (or use Ctrl+C)
  • Open a text editor (like Notepad on Windows or TextEdit on macOS).
  • Paste the API key into the file (Ctrl+V).
  • Save the file with a descriptive name (e.g., “OpenAI_API_Key.txt”) on a location on your laptop that can be easily retrieved, since we need to open the text file and refer to the API Key for the subsequent weeks.
• ✦ Note that the same API Key cannot be retrieve after the window is closed. You may create a new API Key and delete the old API Key(s).

  Keep the API Key safe

  • Treat your API key like a password. Keep it secure and confidential.
    • This key can be used to access your OpenAI account programmatically and consume the credits you have purchased.
    • Please ensure you:
      • DO NOT hardcode (copy & paste) your API key directly into your code notebooks.
      • DO NOT upload the file to public code repositories (e.g., GitHub or GitLab)

• 👆🏽 Click on the "Open Notebook" button below to open the Jupyter Notebook on **_Google Colab

  

  Open Notebook

• ✦ If you are unable to view the link, visit https://colab.research.google.com and use the same browser to login with your Google Account to start using the service. Click on this link again after logging in.
  • You may use any Google Account (this is likely to be your personal account). If you happen to choose to share your working files later on, only your Google's display name will be visible to others but not your Google email address.
  • ⚠️ If you want to keep the changes you made to this notebook, you need to save this notebook to your own Google Drive.
    • Click on “File” in the top left corner of the page
    • Select “Save a copy in Drive.”
    • It's even better if you can make it a habit, so that you always save the file as a copy before starting to edit file.

**💡The most effective way of learning technical skills, like coding is get your hands dirty!

😰 Many of us thought we understand the concepts and able to apply them, until we actually need to code them out!

✅ We recommend when you are going through the videos below, open up the notebook on Google Colab to follow along.

• ✦ While there is no submission required, we encourage you to share your solutions with your peers by pasting your link into the Sharing Board.

  • Feedback: By sharing your solutions, you can get insights, suggestions, and constructive criticism from your peers. This feedback can help you improve your approach and learn from others’ perspectives.

  • Learning from Peers: Since everyone may have different ways of solving problems, participating in these sessions allows you to see various approaches. You can learn alternative methods, explore different techniques, and gain a deeper understanding of the challenges.

• ✦ URL: https://miro.com/app/board/uXjVKvQ1WzE=/?share_link_id=408634728152

• ✦ Passcode: abc-2024

• ✦ For every Large Language Models (LLMs), "tokens" play a crucial role. They are the smallest units of text that the model can understand and manipulate. 

  • Think of tokens as the building blocks of a sentence.
  • They can represent a word, a part of a word, or even a punctuation mark.
    • For instance, in the sentence "She loves ice-cream", there would be five tokens: "She", "loves", "ice", "-", and "cream".
  • The models learn to understand the statistical relationships between these tokens and produce the next token in a sequence of tokens.
    • Different models use different tokenization processes.
    • For example, OpenAI's GPT-4 likely uses a different tokenization process compared to Gemini from Google. 

• ✦ In the early days of Language Learning Models (LLMs), it sounds like a long time ago although it was just early 2023, counting tokens was critical due to the limitations of these models in handling large numbers of tokens.
• ✦ However, with the release of newer models, such as gpt-4-turbo, gpt-4o, and gemini 1.5 pro onwards, many newer models can now process a significantly larger number of tokens, reducing the criticality for strict token counting.
• ✦ Below are some of the latest models, at the time of writing, and the number of tokens.
  - The concept of the maximum tokens that the models can handle is also often known as "Context Window".
  - Note that the "Context Window" for the table below includes both the input and output tokens.

• ✦ It’s important to note that some models may have different token limits for input and output.
  • This means that while a model might be able to accept a large number of tokens as input, it might only be able to generate a smaller number of tokens as output.
  • Therefore, understanding the token limits of a specific model is still crucial.
• ✦ Furthermore, for open-source models, especially smaller ones that prioritize speed, token counts remain very important.
  • These models often have stricter token limits due to their focus on efficiency and speed.
  • Therefore, efficient token management is still a key consideration when working with these models.
  • It helps ensure that the models operate within their capacity and deliver results quickly.
  • Besides counting the token programmatically with code, which we will be using in our practical tasks, we can also use the web-based tool on https://platform.openai.com/tokenizer
  • You can also try out the tool directly from below, by entering your sample prompt into the text box.

You can use the tool below to understand how a piece of text might be tokenized by a language model, and the total count of tokens in that piece of text.

• ✦ For many of the LLMs, the pricing is based on the number of tokens processed.

  • By understanding tokens, you can better manage your usage of the model, optimizing costs and ensuring efficient use of resources.
  • Below are some pricing tables for the different models from OpenAI.
    • Prices are typically viewed in units of either units of “per 1M tokens” or “per 1K tokens”.
    • You can think of tokens as pieces of words, where 1,000 tokens is about 750 words. For example, this paragraph is about 35 tokens.

• ✦ Below are the pricing table for OpenAI's GPT models for reference:

We can use the code below to estimate the token counts in the prompt that we will send to LLM.

# This a simplifedfunction is for calculating the tokens given the "text"
# ⚠️ This is simplified implementation that should only be used for a rough estimation

import tiktoken

def count_tokens(text):
    encoding = tiktoken.encoding_for_model('gpt-4o-mini')
    return len(encoding.encode(text))>)
Copy

• ✦ While the above code is sufficient for approximating the token counts, if you need more accurate token counts on the prompt, please refer the code below:

  • We recommend to use this function for calculating the tokens in actual projects

    • This is especially useful if the API calls involve lengthy multi-turns chat between the LLM and the users
  • Don't worry about understand this function line-by-line, it's a utility tool

    • The core function is really boiled down to this: encoding.encode(value) in the last few lines of the code

import tiktoken

def num_tokens_from_messages(messages, model="gpt-3.5-turbo"):
    """Return the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")

	tokens_per_message = 3
	tokens_per_name = 1
   
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
    return num_tokens

# For more details, See https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
Copy

• GovTech Prompt Engineering Playbook
• OpenAI Models
• OpenAI Tokenizer

• ✦ For our Helper Function in the notebook, we only pass in three arguments to the create() method.

# This is a function that send input (i.e., prompt) to LLM and receive the output from the LLM
def get_completion(prompt, model="gpt-4o-mini"):
    messages = [{"role": "user", "content": prompt}]
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0, # this is the degree of randomness of the model's output
    )
Copy

• ✦ The method can accept more parameters than we are using here.
• ✦ There are three essential parameters here that can directly affect the behaviour of the LLMs. They are:
  - Temperature
  - Top-P
  - Top-K (not available on OpenAI models)
• ✦ These parameters are common for other LLMs, including Open-Source Models

• ✦ After the probabilities are computed, the model applies the Top-K sampling strategy.
• ✦ It selects the K most probable next words and re-normalizes the probabilities among these K words only.
• ✦ Then it samples the next word from these K possibilities
• 

• ✦ Top-P is also known as nucleus sampling
  • This is an alternative to Top-K sampling, which we will discuss next.
  • Instead of selecting the top K most probable words, it selects the smallest set of words whose cumulative probability exceeds a threshold P. Then it samples the next word from this set.
  • Top-P sampling gives us a subset of words whose cumulative probability exceeds a certain threshold (P), making it a useful method for narrowing down a list of candidates based on their probabilities.

• ✦ parameter: max_tokens
• ✦ The maximum number of tokens that can be generated in the chat completion.
• ✦The total length of input tokens and generated tokens is limited by the model's context length.

• ✦ parameter: n
• ✦ Defaults to 1 (if no value passed to the method)
• ✦ This refer to how many chat completion choices to generate for each input message.
  • Note that you will be charged based on the number of generated tokens across all of the choices.
  • Stick with the default, which is to use 1 so as to minimize costs.

• ✦ With the additional parameters that we have introduced in this note, we can update the helper function that we use to call LLMs, like the one below:

!pip install tiktoken
!pip install openai

# This is the "Updated" helper function for calling LLM,
# to expose the parameters that we have discussed
def get_completion(prompt, model="gpt-3.5-turbo", temperature=0, top_p=1.0, max_tokens=1024, n=1):
    messages = [{"role": "user", "content": prompt}]
    response = openai.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
        top_p=top_p,
        max_tokens=max_tokens,
        n=1
    )
    return response.choices[0].message.content
Copy

On OpenAI's API reference, it is stated that we generally recommend altering temperature or top_p but not both.

We suggest to stick with the official recommendation from OpenAI to only change the temperature as the primary way to change the "creativity" of the LLM output

For those who want to explore or experiment further with both the parameters, this table contains various combinations of the two parameters and a description of the different scenarios they will be potentially useful for. We caveat that is not officially recommended by OpenAI and should be used with caution.

• ✦ One important thing to take note of when using such AI powered by Large Language Models (LLMs) is that they often generate text that appears coherent and contextually relevant but is factually incorrect or misleading.

  • We call these hallucination problems. This issue arises due to the inherent nature of how LLMs are trained and their reliance on massive datasets.
  • While some of the models like ChatGPT go through a second phase in the training where humans try to improve the responses, there is generally no fact-checking mechanism that is built into these LLMs when you use them.
• ✦ There is no easy foolproof safeguard against hallucination, although some system prompt engineering can help mitigate this.

  • What makes hallucination by LLM worse is that the responses are surprisingly real, even if they are absolutely nonsensical.
  • Know that you must never take the responses as-is without fact-checking, and that you are ultimately responsible for the use of the output.

• ✦ Understanding these pitfalls is crucial for effectively using LLMs and mitigating potential issues. We will explore some of the common pitfalls of LLMs, including issues with:
  • citing source
  • bias
  • hallucinations
  • math
  • prompt hacking

prompt = f"""
Generate a list of HDB towns along \
with their populations.\
Provide them in JSON format with the following keys:
town_id, town, populations.
"""
response = get_completion(prompt)
print(response)


import json
response_dict = json.loads(response)
type(response_dict)
Copy

• ✦ The prompt specifies that the output should be in JSON format, with each entry containing three keys: town_id, town, and populations.

• ✦ Here’s a breakdown of the code:

  • "Generate a list of HDB towns along with their populations.":
    • This is the instruction given to the LLM, asking it to create a list object of towns and their populations.
  • `"Provide them in JSON format with the following keys: town_id, town, populations."
    • This part of the prompt specifies the desired format (JSON) and the keys for the data structure.
  • response = get_completion(prompt):
    • This line calls a function get_completion (which is presumably defined elsewhere in the code or is part of an API) with the prompt as an argument.
    • The function is expected to interact with the LLM and return its completion, which is a string object that contains the JSON string.
  • response_dict = json.loads(response):
    • After the JSON string is loaded into response_dict, this line will return dict, confirming that it is indeed a Python dictionary.

• ✦ It's often useful to convert the dictionary to a Pandas DataFrame if we want to process or analyse the data.
  • Here is the example code on how to do that, continued from the example above

# To transform the JSON-string into Pandas DataFrame
import pandas as pd

df = pd.DataFrame(response_dict['towns'])
df
Copy

• ✦ Here is the sample code that show how we eventually save the LLM output into a CSV file on the local disk.

# Save the DataFrame to a local CSV file
df.to_csv('town_population.csv', index=False)

# Save the DataFrame to a localExcel File
df.to_excel('town_population.xlsx', index=False)
Copy

df = pd.read_csv('town_population.csv')
df
Copy

• ✦ Preventing prompt injection & leaking can be very difficult, and there exist few robust defenses against it. However, there are some common sense solutions.

  • For example, if your application does not need to output free-form text, do not allow such outputs as it makes it easier for hackers to key in malicious prompts/code.
  • There are many different ways to defend against bad actors we will discuss some of the most common ones here.
• ✦ However, in many LLM applications, the solutions mentioned above may not be feasible.

  • In this subsection, we will discuss a few tactics that we can implement at the prompt-level to defense against such attacks.

• ✦ By default, LLMs are stateless — meaning each incoming query (i.e., each time the LLM is triggered to generate the text response) is processed independently of other interactions. The only thing that matters is the current input, nothing else.

• ✦ There are many applications where remembering previous interactions is very important, such as chatbots. Here, we will find out how we can enable conversations with LLMs as if the LLM remembers the previous conversation.

  - Notice that in the example below, when the second input is sent to the LLM, the output is not relevant to the previous interaction(e.g., running `get_completion()`)
  Copy

  

• • To make the LLM to engage in a "conversation", we need to send over all the previous prompt and response (i.e., those components highlighted in the BLUE region in the image below).
  • In the example below, the input & output of the first interaction is sent together with the second prompt (i.e., "Which are healthy?")

• ✦ Below is the helper function that we have been using.
  • Pay attention to the messages object in the function.
  • That's the key for implementing the conversational-like interaction with the LLM.

def get_completion(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0, # this is the degree of randomness of the model's output
    )
    return response.choices[0].message.content
Copy

• • messages is a list object where each item is a message.
    
  • A message object can be either of the three types:
    • A. prompt from users
    • B. response from LLM (aka. AI assistant)
    • C. 🆕 system message:

An example of messages with all these keys is shown below:

messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "List some Fun Activities"},
    {"role": "assistant", "content": "Spa, Hiking, Surfing, and Gaming"},
    {"role": "user", "content": "Which are healthy?"}
]
Copy

Another example

messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"}
]
Copy

Below is the illustration on the flow of the messages between different "roles"

• 💡By exposing the messages as one of the helper function's parameter, now we have a more flexible function get_completion_from_message, where you can compose the messages object, instead of just passing in the "user prompt".

def get_completion_by_messages(messages, model="gpt-3.5-turbo", temperature=0, top_p=1.0, max_tokens=1024, n=1):
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
        top_p=top_p,
        max_tokens=max_tokens,
        n=1
    )
    return response.choices[0].message.content
Copy

You probably would have guessed what the implications are of continuously stacking messages in the messages parameter for subsequent API calls. While it unlocks more contextually aware and engaging interactions, there's a trade-off to consider concerning resource utilization and performance. Let's delve into three key areas where these trade-offs become apparent:

1. Increased Token Consumption:

   • Longer Context: Each message you add to the messages list contributes to a longer conversation history that the model needs to process. This directly increases the number of tokens consumed in each API call.

   • Token Billing: Most LLMs' pricing model is based on token usage. As your message history grows, so does the cost of each API call. For lengthy conversations or applications with frequent interactions, this can become a considerable factor.

2. Context Window Limits:

   • Finite Capacity: Language models have a limited "context window", meaning they can only hold and process a certain number of tokens at once.

   • Truncation Risk: If the total number of tokens in your messages list exceeds the model's context window, the earliest messages will be truncated. This can lead to a loss of crucial context and affect the model's ability to provide accurate and coherent responses.

3. Potential for Increase Latency:

   • Processing Overhead: As the message history grows, the model requires more time to process and understand the accumulated context. This can lead to a noticeable increase in response latency, especially for models with larger context windows or when dealing with computationally intensive tasks.

Mitigation Strategies:

• ✦ It's crucial to implement strategies to manage conversation history effectively. This could involve:

  • Summarization: Summarize previous messages to condense information while preserving key context.

  • Selective Retention: Retain only the most relevant messages, discarding less important ones.

  • Session Segmentation: Divide long conversations into logical segments and clear the context window periodically.

  • Token-Efficient Models: Consider using models specifically designed for handling longer contexts, as they may offer a larger context window or more efficient token usage.

• ✦ The Chain-of-Thought (CoT) is a method where a language model lays out its thought process in a step-by-step manner as it tackles a problem.
• ✦ This approach is particularly effective in tasks that involve arithmetic and complex reasoning.
• ✦ By organizing its thoughts, the model frequently produces more precise results.
• ✦ Unlike conventional prompting that merely seeks an answer, this technique stands out by necessitating the model to elucidate the steps it took to reach the solution.

Reference: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

• ✦ Zero Shot Chain of Thought (Zero-shot-CoT) prompting is a follow up to CoT prompting, which introduces an incredibly simple zero shot prompt.
• ✦ Studies have found that by appending the words "Let's think step by step." to the end of a question, LLMs are able to generate a chain of thought that answers the question.

Reference: Igniting Language Intelligence: The Hitchhiker’s Guide From Chain-of-Thought Reasoning to Language Agents

• ✦ Contrastive Chain-of-Thought is a strategy that introduces an incorrect explanation alongside the correct reasoning in response to a CoT prompt.
  • This approach has shown significant advancements over the traditional CoT, particularly in areas like arithmetic reasoning and answering factual questions.
  • The utilization of this method enables the AI model to comprehend not just the accurate steps of reasoning, but also the mistakes to steer clear of, thereby boosting its overall capacity for reasoning.

Reference: Contrastive Chain-of-Thought Prompting

• ✦ Least to Most prompting (LtM)1 takes CoT prompting a step further by first breaking a problem into sub problems then solving each one. It is a technique inspired by real-world educational strategies for children.
• ✦ As in CoT prompting, the problem to be solved is decomposed in a set of subproblems that build upon each other. In a second step, these subproblems are solved one by one. Contrary to chain of thought, the solution of previous subproblems is fed into the prompt trying to solve the next problem.
• ✦ This approach has shown to be effective in generalizing to more difficult problems than those seen in the prompts. For instance, when the GPT-3 model or equivalent is used with LtM, it can solve complex tasks with high accuracy using just a few exemplars, compared to lower accuracy with CoT prompting.

Reference: Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

• ✦ Chaining actions involve giving the LLM a prompt that contains a sequence of tasks to be completed one after the other.
  • Each action in the chain builds upon the previous one, allowing for the creation of a multi-step process that the model follows to generate a final output.
  • This technique can enhance the utility and flexibility of LLMs in processing and generating information.
  • How Does it Work?
    • Sequential Instructions: The prompt is structured to include a list of actions that the LLM needs to perform. These actions are ordered logically, ensuring that the output of one action serves as the input or foundation for the next.
    • Clear Delimitation: Each action is clearly delineated within the prompt, often numbered or separated by clear markers. This helps the model understand the sequence of steps it needs to follow.
    • Building Complexity: The initial tasks are usually simpler, with complexity building as the model progresses through the chain. This gradual increase in complexity helps the model maintain focus and apply the information it has processed in earlier steps.

• ✦ One key benefit of this prompting tactic is that we can extract the relevant part to display to the end-user, while keeping the others part as the "intermediate outputs".
  • Similar to Chain-of-Thought prompting, LLMs can perform better at reasoning and logic problems if you ask them to break the problem down into smaller steps.
  • The "intermediate output" is also known as the inner monologue of the LLM when it is reasoning through the problems.
  • These "intermediate outputs" can be used to verify if the reasoning applied by the LLM is correct or as intended.

The idea behind the generated knowledge approach is to ask the LLM to generate potentially useful information about a given question/prompt before generating a final response.

Reference: Generated Knowledge Prompting for Commonsense Reasoning

• Complexity Management:

  • As the chain of actions grows, the prompt can become complex. It's essential to structure the prompt clearly to avoid confusing the model.
  • The effectiveness of complex prompt heavily relies on the skill of prompt engineering, but may still not guarantee consistent and desired output from the LLMs.
• Error Propagation:

  • Mistakes in early steps can propagate through the chain, affecting the final output. Careful prompt design and error checking are crucial.
  • There is no way to explicitly check the intermediate outputs and use explicit logic (like an if-else statement) to change the flow.
• Context Dilution

  • As the instructions grows and become more complex, the attention on some instructions may become diluted and may not be followed through by the LLMs.

However, for simpler instructions like those we have seen in the examples above, chaining multiple actions within a prompt will still work relatively well, while offering better speed for the application. This is because making one request to the LLM is generally faster than multiple sequential requests. It also helps to maintains a logical flow of information, ensuring that the output is coherent and contextually relevant across all steps.

• ✦ Essentially, prompt chaining involves taking the result of one prompt and using it as the starting point for the next, forming a sequence of interactions

  • By breaking a complex task into multiple smaller prompts and passing the output of one prompt as the input to the next, prompt chaining simplifies complex tasks and streamlines the interaction with the LLM model.

  • Instead of overwhelming the LLM instance with a single detailed prompt, we can guide it through multiple steps, making the process more efficient and effective.

  • It allows us to write less complicated instructions, isolate parts of a problem that the LLM might have difficulty with, and check the LLM’s output in stages, rather than waiting until the end.

• ✦ While prompt chaining enhances the quality of the conversation, it can also impact the performance or speed of the AI system.

  • One reason is that as the conversation progresses, the chain of prompts becomes longer.
  • Processing these longer inputs can take more time and computational resources, potentially slowing down the response time of your application.
• ✦ Despite this, the benefits of prompt chaining often outweigh the potential performance costs.

  • The ability to maintain a coherent, context-aware conversation is crucial for user engagement and satisfaction.
  • Therefore, it's important for developers to measure the time performance of the chain, in order to balance conversational quality with system performance.

LangChain is a framework to build with LLMs by chaining interoperable components. The framework "abstracts" away many of the complexity, so developers will have to write shorter code to achieve similar outputs. It is useful for projects that involves complex prompt chains where we need to orchestrate multiple LLM calls with different prompts and data dependencies.

• ✦ Exception handling is a fundamental skill for every Python programmer. It allows programmers to handle errors and unexpected situations that can arise during program execution.

• ✦ In this note, we’ll explore how to handle these unexpected situations in Python.

  • We’ll look at tools like ‘try’ and ‘except’, which let us plan for things that could go wrong.
  • We’ll also talk about ‘finally’, which is something we can use to clean up after an issue occurs.
  • We’ll even show you how to create your own exceptions, like making a custom warning when you’re about to run out of glue!
• ✦ By the end of this note, you’ll learn how to handle errors without having your application crashes or stops abruptly, making your applications more reliable.

  • These are part of the important skills for building great applications with Python, Streamlit, OpenAI API, and of course even a generic Python script.

Here is the basic structure of the Exception Handling

# Step 1: Understand the basic structure
try:
    # Code that might raise an exception
    <..>
except ExceptionType:
    # Code to handle the exception
    <..>
finally:
    # Code to be executed regardless of whether an exception was raised
    <..>

Copy

Python’s try and except statements provide a safety net for your code, allowing you to catch and handle exceptions that might occur during execution. This prevents your program from crashing and provides an opportunity to recover gracefully.

try:  
	dividend = 10  
	divisor = 0  
	result = dividend / divisor  
except:
	print("Error: Division by zero is not allowed.")
	
Copy

The finally the block is used in conjunction with try and except to define cleanup actions that should be performed, regardless of whether an exception occurs or not.

try:
    # Attempt to divide 10 by a variable that may be zero
	dividend = 10  
	divisor = 0  
	result = dividend / divisor  
    
except:
    # Handle the error if the divisor is zero
    print("Error: Cannot divide by zero.")
    
finally:
    # This block will execute no matter what
    print("Division attempt finished.")

Copy

In this scenario, the file is opened, and the content is read. If an exception occurs, it is caught and handled. Regardless of the outcome, the finally block ensures that the file is properly closed, preventing resource leaks.

Python provides a comprehensive set of built-in exceptions to handle a wide range of errors and exceptional conditions that can occur during program execution. These exceptions are organized into a hierarchy, with the base class BaseException at the top. Here are some commonly used built-in exceptions along with a brief description

These are just a few examples of the many built-in exceptions that Python provides.

• Each exception carries specific information about the error, which can be accessed using try and except blocks to handle them gracefully and provide meaningful feedback to the user.

Here is the example of when we incorporate these specific Exception type into our earlier code:

try:  
	dividend = 10  
	divisor = 0  
	result = dividend / divisor  

except ZeroDivisionError:
	print("Error: Division by zero is not allowed.")
	
Copy

Here is another example.

try:
    # Attempt to access a key that may not exist in the dictionary
    my_dict = {'a': 1, 'b': 2}
    value = my_dict['c']
    
except KeyError:
    # Handle the error if the key 'c' does not exist
    print("Error: Key not found in the dictionary.")
    
finally:
    # This block will execute regardless of the previous outcome
    print("Key lookup attempt finished.")

Copy

• 👆🏽 Click on the "Open Notebook" button below to open the Jupyter Notebook on **_Google Colab

  

  Open Notebook

• ✦ If you are unable to view the link, visit https://colab.research.google.com and use the same browser to login with your Google Account to start using the service. Click on this link again after logging in.
  • You may use any Google Account (this is likely to be your personal account). If you happen to choose to share your working files later on, only your Google's display name will be visible to others but not your Google email address.
• ✦ If you want to keep the changes you made to this notebook, you need to save this notebook to your own Google Drive.
  • Click on “File” in the top left corner of the page
  • Select “Save a copy in Drive.”
  • It's even better if you can make it a habit, so that you always save the file as a copy before starting to edit file.

**💡The most effective way of learning technical skills, like coding is get your hands dirty!

😰 Many of us thought we understand the concepts and able to apply them, until we actually need to code them out!

✅ We recommend when you are going through the videos below, open up the notebook on Google Colab to follow along.

• ✦ While there is no submission required, we encourage you to share your solutions with your peers by pasting your link into the Sharing Board.

  • Feedback: By sharing your solutions, you can get insights, suggestions, and constructive criticism from your peers. This feedback can help you improve your approach and learn from others’ perspectives.

  • Learning from Peers: Since everyone may have different ways of solving problems, participating in these sessions allows you to see various approaches. You can learn alternative methods, explore different techniques, and gain a deeper understanding of the challenges.

• ✦ URL: https://miro.com/app/board/uXjVKvQ1WzE=/?share_link_id=408634728152

• ✦ Passcode: abc-2024

• LLMs Do Not Have Memory
• Prompting Techniques for Better Reasoning
• Multi-action within a Prompt
• Prompt Chaining
• Exception Handling
• Hands-on Walkthrough and Tasks

• 👆🏽 Click on the "Open Notebook" button below to open the Jupyter Notebook 

💡The most effective way of learning technical skills, like coding is get your hands dirty!

✅ We recommend when you are going through the videos below, open up the notebook on Google Colab to follow along.

• 👆🏽 Click on the "Open Notebook" button below to open the Jupyter Notebook 

  

  Open Notebook

• ✦ While there is no submission required, we encourage you to share your solutions with your peers by pasting your link into the Sharing Board.

  • Feedback: By sharing your solutions, you can get insights, suggestions, and constructive criticism from your peers. This feedback can help you improve your approach and learn from others’ perspectives.

  • Learning from Peers: Since everyone may have different ways of solving problems, participating in these sessions allows you to see various approaches. You can learn alternative methods, explore different techniques, and gain a deeper understanding of the challenges.

• ✦ URL: https://miro.com/app/board/uXjVKvQ1WzE=/?share_link_id=408634728152

• ✦ Passcode: abc-2024