• ✦ For our Helper Function in the notebook, we only pass in three arguments to the create() method.

# This is a function that send input (i.e., prompt) to LLM and receive the output from the LLM
def get_completion(prompt, model="gpt-4o-mini"):
    messages = [{"role": "user", "content": prompt}]
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0, # this is the degree of randomness of the model's output
    )
Copy

• ✦ The method can accept more parameters than we are using here.
• ✦ There are three essential parameters here that can directly affect the behaviour of the LLMs. They are:
  - Temperature
  - Top-P
  - Top-K (not available on OpenAI models)
• ✦ These parameters are common for other LLMs, including Open-Source Models

• ✦ After the probabilities are computed, the model applies the Top-K sampling strategy.
• ✦ It selects the K most probable next words and re-normalizes the probabilities among these K words only.
• ✦ Then it samples the next word from these K possibilities
• 

• ✦ Top-P is also known as nucleus sampling
  • This is an alternative to Top-K sampling, which we will discuss next.
  • Instead of selecting the top K most probable words, it selects the smallest set of words whose cumulative probability exceeds a threshold P. Then it samples the next word from this set.
  • Top-P sampling gives us a subset of words whose cumulative probability exceeds a certain threshold (P), making it a useful method for narrowing down a list of candidates based on their probabilities.

• ✦ parameter: max_tokens
• ✦ The maximum number of tokens that can be generated in the chat completion.
• ✦The total length of input tokens and generated tokens is limited by the model's context length.

• ✦ parameter: n
• ✦ Defaults to 1 (if no value passed to the method)
• ✦ This refer to how many chat completion choices to generate for each input message.
  • Note that you will be charged based on the number of generated tokens across all of the choices.
  • Stick with the default, which is to use 1 so as to minimize costs.

• ✦ With the additional parameters that we have introduced in this note, we can update the helper function that we use to call LLMs, like the one below:

!pip install tiktoken
!pip install openai

# This is the "Updated" helper function for calling LLM,
# to expose the parameters that we have discussed
def get_completion(prompt, model="gpt-3.5-turbo", temperature=0, top_p=1.0, max_tokens=1024, n=1):
    messages = [{"role": "user", "content": prompt}]
    response = openai.chat.completions.create(
        model=model,
        messages=messages,
        temperature=temperature,
        top_p=top_p,
        max_tokens=max_tokens,
        n=1
    )
    return response.choices[0].message.content
Copy

On OpenAI's API reference, it is stated that we generally recommend altering temperature or top_p but not both.

We suggest to stick with the official recommendation from OpenAI to only change the temperature as the primary way to change the "creativity" of the LLM output

For those who want to explore or experiment further with both the parameters, this table contains various combinations of the two parameters and a description of the different scenarios they will be potentially useful for. We caveat that is not officially recommended by OpenAI and should be used with caution.