Streaming & SSE

All three inference endpoints support Server-Sent Events (SSE). Set "stream": true in the request body and the gateway responds with Content-Type: text/event-stream and relays tokens as they are generated. You are billed only for the tokens actually received.

• Send the same request you would send for non-streaming, plus "stream": true.
• The gateway streams tokens incrementally with the protocol's native event format.
• A [DONE] terminator (chat) or a terminal event (message_stop / response.completed) marks the end.
• Settlement happens after the stream ends, against the tokens actually delivered.

zrouter normalizes the chat stream to the classic OpenAI chunk protocol. Each chunk is a data: line holding a chat.completion.chunk object, and the stream always ends with data: [DONE].

curl -N -X POST "https://api.zrouter.dev/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
   -d '{"model": "<public-name>", "stream": true, "stream_options": {"include_usage": true}, "messages": [{"role": "user", "content": "Say hello!"}]}'

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"<public-name>","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"<public-name>","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"<public-name>","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":3,"total_tokens":12}}

data: [DONE]

The usage chunk is opt-in. It is sent only when the request body includes "stream_options": {"include_usage": true} — exactly one extra chunk with an empty choices array ([]) and a token-only usage object, delivered after the finish_reason chunk and immediately before data: [DONE]. Without the flag (or with false/null), the stream ends with the finish_reason chunk followed directly by data: [DONE] — no usage chunk is added.

Usage fields are token counts only: prompt_tokens, completion_tokens, and total_tokens (always prompt_tokens + completion_tokens). Cost and supplier metadata are never exposed, and if the upstream model does not report usage, no usage chunk is fabricated. Billing is independent of this flag: settlement runs server-side against the tokens actually delivered, whether or not you request the usage chunk.

The Messages stream is relayed raw in the native Anthropic format — no normalization. Each event carries an event: type line followed by a data: JSON payload, and the stream ends with message_stop.

event: message_start
data: {"type":"message_start","message":{"id":"msg_...","type":"message","role":"assistant","model":"<public-name>","content":[],"usage":{"input_tokens":9,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":3}}

event: message_stop
data: {"type":"message_stop"}

The Responses stream is also relayed raw. Incremental text arrives as response.output_text.delta events, and the terminal response.completed event carries the full assembled output — including the final usage block.

event: response.created
data: {"type":"response.created","response":{"id":"resp_...","object":"response","model":"<public-name>","status":"in_progress","output":[]}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","item_id":"msg_...","output_index":0,"content_index":0,"delta":"Hello"}

event: response.completed
data: {"type":"response.completed","response":{"id":"resp_...","object":"response","model":"<public-name>","status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"Hello!"}]}],"usage":{"input_tokens":8,"output_tokens":3,"total_tokens":11}}}

Between events, the gateway sends a keepalive comment every 15 seconds so idle proxies and load balancers do not close the connection during a slow generation:

: keepalive

If your client disconnects mid-stream, the gateway stops relaying and ends the request safely — no resource is leaked, and you are billed only for tokens delivered before the disconnect.

You do not have to parse SSE frames yourself — the official SDKs handle reconnection-free iteration, chunk decoding, and final usage for you. The same stream flag you saw in the curl examples maps to SDK-level streaming options:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.zrouter.dev/v1",
)

stream = client.chat.completions.create(
    model="<public-name>",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
    # Ask the server to include token usage on the final chunk:
    stream_options={"include_usage": True},
)

chunks = []
usage = None
for chunk in stream:
    delta = chunk.choices[0].delta.content if chunk.choices else None
    if delta:
        chunks.append(delta)
    if chunk.usage:
        usage = chunk.usage

print("".join(chunks))
print(usage)  # input_tokens / output_tokens on the last chunk