Streaming Chat

Streaming Chat API is a streaming version of LLM Chat API. In streaming chat, LLM response is divided into a series of chunks and providers send them as soon as they are available. Streaming can potentially improve latency of the final response since we don't have to wait until the whole response is generated to receive it.

Workflow