OpenAI LLM¶

The openai_llm module is optimized openai llm module for AutoRAG.

Why use `openai_llm` module?¶

There are several advantages using openai module in AutoRAG.

1. Auto-truncate prompt¶

Sometimes, prompt might exceed a token limitation of the model. It will occur server-side error, and all your answer results will be gone. To prevent this, openai_llm module truncate prompt to the max length of gpt model.

2. Accurate token output¶

In llama_index_llm module, it does not return proper tokens. It just return pseudo token using GPT2 tokenizer.

When you use openai_llm module, you can get real tokens that used in gpt model. In the future, there will be a module that uses token for boosting RAG performance.

3. Accurate log prob output¶

In llama_index_llm module, it does not return proper log probs since llama index does not support it.

With openai_llm module, you can get real log probability to every token of generated answers. In the future, there will be some modules that use log probability, like answer filter.

Support chat prompt¶

From v0.3.19, you can use chat prompt with openai_llm module. For using chat prompt, you have to use chat_fstring module for prompt maker.

Support GPT-5 models¶

From v0.3.22, you can use GPT-5 models with openai_llm module. You can set reasoning efforts and verbosity level. The latest GPT-5 family models are supported, including the gpt-5.6 family (gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, and the gpt-5.6 alias), gpt-5.5, and gpt-5.4, all with a 1.05M token context window. Dated snapshots (e.g. gpt-5.6-sol-2026-07-09) resolve to their base model’s context window automatically.

modules:
  - module_type: openai_llm
    llm: [ gpt-5.6-sol ]
    max_tokens: 512
    reasoning:
      effort: high
      verbosity: low

GPT-5.6 models support reasoning efforts from none up to max (none, low, medium, high, xhigh, max).

Retired models¶

Retired OpenAI models are intentionally not supported and raise a clear error at module initialization. This follows the official OpenAI deprecations page. Notable removals: gpt-5, gpt-5.1, gpt-5-mini, gpt-5-nano, gpt-5-chat-latest, chatgpt-4o-latest, o1-preview, o1-mini, gpt-4-32k, the gpt-4 vision/turbo preview snapshots, and the gpt-3.5-turbo-0613/16k snapshots. Migrate to the gpt-5.4/5.5/5.6 family (e.g. gpt-5.6-terra as a drop-in for gpt-5/gpt-5.1, gpt-5.4-mini for gpt-5-mini).

Module Parameters¶

llm: You can type your ‘model name’ at here. For example, gpt-4-turbo-2024-04-09 or gpt-3.5-turbo-16k
batch: The batch size of openai api call. You should decrease when you got token limit error.
truncate: Whether you truncate input prompts to model’s max length. Default is True. Recommend you to keep this True.
api_key: OpenAI API key. You can also set this to env variable OPENAI_API_KEY.
And all parameters from OpenAI Chat Completion without n, logprobs, stream and top_logprobs.

Example config.yaml¶

modules:
  - module_type: openai_llm
    llm: [ gpt-3.5-turbo, gpt-4-turbo-2024-04-09 ]
    temperature: [ 0.1, 1.0 ]
    max_tokens: 512