Streaming & SSE

Ketiga endpoint inferensi mendukung Server-Sent Events (SSE). Set "stream": true pada body request dan gateway membalas dengan Content-Type: text/event-stream serta mengalirkan token seiring dihasilkannya. Anda hanya ditagih atas token yang benar-benar diterima.

• Kirim request yang sama seperti non-streaming, ditambah "stream": true.
• Gateway mengalirkan token secara inkremental dalam format event native protokolnya.
• Terminator [DONE] (chat) atau event terminal (message_stop / response.completed) menandai akhir.
• Penyelesaian (settlement) terjadi setelah stream berakhir, terhadap token yang benar-benar terkirim.

zrouter menormalkan stream chat ke protokol chunk OpenAI klasik. Setiap chunk adalah baris data: berisi objek chat.completion.chunk, dan stream selalu diakhiri data: [DONE].

curl -N -X POST "https://api.zrouter.dev/v1/chat/completions" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
   -d '{"model": "<public-name>", "stream": true, "stream_options": {"include_usage": true}, "messages": [{"role": "user", "content": "Say hello!"}]}'

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"<public-name>","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"<public-name>","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"<public-name>","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":3,"total_tokens":12}}

data: [DONE]

Chunk usage bersifat opt-in. Chunk tersebut hanya dikirim bila body request menyertakan "stream_options": {"include_usage": true} — tepat satu chunk tambahan dengan array choices kosong ([]) dan objek usage yang hanya berisi token, dikirim setelah chunk finish_reason dan tepat sebelum data: [DONE]. Tanpa flag tersebut (atau dengan false/null), stream berakhir pada chunk finish_reason yang langsung diikuti data: [DONE] — tidak ada chunk usage yang ditambahkan.

Field usage hanya berupa hitungan token: prompt_tokens, completion_tokens, dan total_tokens (selalu prompt_tokens + completion_tokens). Biaya dan metadata supplier tidak pernah diekspos, dan bila model upstream tidak melaporkan usage, chunk usage tidak difabrikasi. Penagihan tidak terpengaruh flag ini: settlement berjalan di sisi server terhadap token yang benar-benar terkirim, baik Anda meminta chunk usage maupun tidak.

Stream Messages diteruskan mentah dalam format Anthropic native — tanpa normalisasi. Setiap event memuat baris tipe event: diikuti payload JSON data:, dan stream diakhiri message_stop.

event: message_start
data: {"type":"message_start","message":{"id":"msg_...","type":"message","role":"assistant","model":"<public-name>","content":[],"usage":{"input_tokens":9,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":3}}

event: message_stop
data: {"type":"message_stop"}

Stream Responses juga diteruskan mentah. Teks inkremental tiba sebagai event response.output_text.delta, dan event terminal response.completed membawa output lengkap yang sudah dirangkai — termasuk blok usage final.

event: response.created
data: {"type":"response.created","response":{"id":"resp_...","object":"response","model":"<public-name>","status":"in_progress","output":[]}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","item_id":"msg_...","output_index":0,"content_index":0,"delta":"Hello"}

event: response.completed
data: {"type":"response.completed","response":{"id":"resp_...","object":"response","model":"<public-name>","status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"Hello!"}]}],"usage":{"input_tokens":8,"output_tokens":3,"total_tokens":11}}}

Di antara event, gateway mengirim komentar keepalive setiap 15 detik agar proxy dan load balancer yang menganggur tidak menutup koneksi saat generate berjalan lambat:

: keepalive

Jika klien Anda memutus koneksi di tengah stream, gateway berhenti meneruskan dan mengakhiri request dengan aman — tidak ada resource yang bocor, dan Anda hanya ditagih atas token yang terkirim sebelum diskoneksi.

Anda tidak perlu mem-parsing frame SSE sendiri — SDK resmi menangani iterasi tanpa reconnect, decoding chunk, dan usage akhir untuk Anda. Flag stream yang sama seperti pada contoh curl dipetakan ke opsi streaming level SDK:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.zrouter.dev/v1",
)

stream = client.chat.completions.create(
    model="<public-name>",
    messages=[{"role": "user", "content": "Halo!"}],
    stream=True,
    # Minta server menyertakan token usage pada chunk terakhir:
    stream_options={"include_usage": True},
)

chunks = []
usage = None
for chunk in stream:
    delta = chunk.choices[0].delta.content if chunk.choices else None
    if delta:
        chunks.append(delta)
    if chunk.usage:
        usage = chunk.usage

print("".join(chunks))
print(usage)  # input_tokens / output_tokens di chunk terakhir