• ✦ For every Large Language Models (LLMs), "tokens" play a crucial role. They are the smallest units of text that the model can understand and manipulate. 

  • Think of tokens as the building blocks of a sentence.
  • They can represent a word, a part of a word, or even a punctuation mark.
    • For instance, in the sentence "She loves ice-cream", there would be five tokens: "She", "loves", "ice", "-", and "cream".
  • The models learn to understand the statistical relationships between these tokens and produce the next token in a sequence of tokens.
    • Different models use different tokenization processes.
    • For example, OpenAI's GPT-4 likely uses a different tokenization process compared to Gemini from Google. 

• ✦ In the early days of Language Learning Models (LLMs), it sounds like a long time ago although it was just early 2023, counting tokens was critical due to the limitations of these models in handling large numbers of tokens.
• ✦ However, with the release of newer models, such as gpt-4-turbo, gpt-4o, and gemini 1.5 pro onwards, many newer models can now process a significantly larger number of tokens, reducing the criticality for strict token counting.
• ✦ Below are some of the latest models, at the time of writing, and the number of tokens.
  - The concept of the maximum tokens that the models can handle is also often known as "Context Window".
  - Note that the "Context Window" for the table below includes both the input and output tokens.

• ✦ It’s important to note that some models may have different token limits for input and output.
  • This means that while a model might be able to accept a large number of tokens as input, it might only be able to generate a smaller number of tokens as output.
  • Therefore, understanding the token limits of a specific model is still crucial.
• ✦ Furthermore, for open-source models, especially smaller ones that prioritize speed, token counts remain very important.
  • These models often have stricter token limits due to their focus on efficiency and speed.
  • Therefore, efficient token management is still a key consideration when working with these models.
  • It helps ensure that the models operate within their capacity and deliver results quickly.
  • Besides counting the token programmatically with code, which we will be using in our practical tasks, we can also use the web-based tool on https://platform.openai.com/tokenizer
  • You can also try out the tool directly from below, by entering your sample prompt into the text box.

You can use the tool below to understand how a piece of text might be tokenized by a language model, and the total count of tokens in that piece of text.

• ✦ For many of the LLMs, the pricing is based on the number of tokens processed.

  • By understanding tokens, you can better manage your usage of the model, optimizing costs and ensuring efficient use of resources.
  • Below are some pricing tables for the different models from OpenAI.
    • Prices are typically viewed in units of either units of “per 1M tokens” or “per 1K tokens”.
    • You can think of tokens as pieces of words, where 1,000 tokens is about 750 words. For example, this paragraph is about 35 tokens.

• ✦ Below are the pricing table for OpenAI's GPT models for reference:

We can use the code below to estimate the token counts in the prompt that we will send to LLM.

# This a simplifedfunction is for calculating the tokens given the "text"
# ⚠️ This is simplified implementation that should only be used for a rough estimation

import tiktoken

def count_tokens(text):
    encoding = tiktoken.encoding_for_model('gpt-4o-mini')
    return len(encoding.encode(text))>)
Copy

• ✦ While the above code is sufficient for approximating the token counts, if you need more accurate token counts on the prompt, please refer the code below:

  • We recommend to use this function for calculating the tokens in actual projects

    • This is especially useful if the API calls involve lengthy multi-turns chat between the LLM and the users
  • Don't worry about understand this function line-by-line, it's a utility tool

    • The core function is really boiled down to this: encoding.encode(value) in the last few lines of the code

import tiktoken

def num_tokens_from_messages(messages, model="gpt-3.5-turbo"):
    """Return the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")

	tokens_per_message = 3
	tokens_per_name = 1
   
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
    return num_tokens

# For more details, See https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
Copy

• GovTech Prompt Engineering Playbook
• OpenAI Models
• OpenAI Tokenizer