Online Documentation for SQL Manager for PostgreSQL

Tokens

A token is the unit that language models use to measure the amount of text. Everything you send to the assistant is split into tokens: the text of your request, SQL fragments, descriptions of database objects, error messages, execution plans and the contents of attached files.

All the limits in the AI Assistant options are measured in tokens: the maximum size of an attached text file, the context size warning threshold and the maximum size of a metadata snapshot. Sizes are never given in kilobytes.

Only the model knows the exact number of tokens. It counts them while processing the request, and the assistant shows the resulting value in the status bar.

Until a request is sent, the number of tokens can only be estimated: the assistant works out the size of unsent chips and metadata snapshots based on an average conversion rate. The actual value may be higher or lower.

Context builds up over the whole conversation: each new request carries the entire conversation so far, including all the context chips. The number of tokens therefore grows as you go on, even if your requests are short. The current value is shown by the context size indicator in the status bar.

The indicator compares the accumulated number of tokens with the value set by the Warning threshold, K tokens option, changing color as the limit gets closer. This is only a warning: exceeding it does not prevent a request from being sent.

Note: Requests to a model are billed by the number of tokens sent and received, and output tokens are usually several times more expensive than input ones. Since every new request carries the whole conversation, the longer it goes on, the more tokens you pay for at each step — the cost grows faster than the number of messages.

If the context has grown too large, start a new chat: the New Chat button clears the accumulated context. To make a single request smaller, remove the context chips you do not need before sending, or trim the metadata snapshot with the options on the Context tab of the Environment Options dialog.

Hint: To keep the cost down, start a new chat whenever the subject changes, and do not attach more metadata than the task requires.