Streaming

Chat completions, Responses and Messages stream token-by-token when you pass stream: true. Streaming works on every chat model — we don't really think of it as a feature, it's the default way to consume inference, and the non-streaming shape exists for batch jobs and scripts that need the whole response atomically.

Pick the shape that matches your client. They're different on the wire but conceptually the same. Responses streams use the OpenAI Responses event types (response.output_text.delta, then response.completed or response.incomplete); see Responses.

Chat completions (OpenAI shape)

from openai import OpenAI

client = OpenAI(
    api_key="sk-mel-<YOUR_API_KEY>",
    base_url="https://api.melious.ai/v1",
)

stream = client.chat.completions.create(
    model="<MODEL_ID>",
    messages=[{"role": "user", "content": "Write a haiku about Hamburg."}],
    stream=True,
    stream_options={"include_usage": True},
)

for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print(f"\n[{chunk.usage.total_tokens} tokens]")
        impact = getattr(chunk, "environment_impact", None)
        if impact:
            print(f"[{impact['carbon_g_co2']} g CO2]")

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: "sk-mel-<YOUR_API_KEY>",
  baseURL: "https://api.melious.ai/v1",
});

const stream = await client.chat.completions.create({
  model: "<MODEL_ID>",
  messages: [{ role: "user", content: "Write a haiku about Hamburg." }],
  stream: true,
  stream_options: { include_usage: true },
});

for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) process.stdout.write(delta);
  if (chunk.usage) {
    console.log(`\n[${chunk.usage.total_tokens} tokens]`);
    if (chunk.environment_impact) {
      console.log(`[${chunk.environment_impact.carbon_g_co2} g CO2]`);
    }
  }
}

const response = await fetch("https://api.melious.ai/v1/chat/completions", {
  method: "POST",
  headers: {
    "Authorization": "Bearer sk-mel-<YOUR_API_KEY>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "<MODEL_ID>",
    messages: [{ role: "user", content: "Write a haiku about Hamburg." }],
    stream: true,
  }),
});

if (!response.ok) {
  const { error } = await response.json();
  throw new Error(`${response.status} ${error?.type ?? error?.code}: ${error?.message}`);
}

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
let failure = null;
let sawDone = false;

try {
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    // SSE messages are separated by \n\n
    const messages = buffer.split("\n\n");
    buffer = messages.pop();  // keep incomplete last piece

    for (const msg of messages) {
      if (!msg.startsWith("data: ")) continue;
      const data = msg.slice(6);
      if (data === "[DONE]") { sawDone = true; break; }
      const parsed = JSON.parse(data);
      if (parsed.error) { failure = parsed.error; break; }
      // keep reading after finish_reason — a later frame can
      // still carry usage or an error
      process.stdout.write(parsed.choices?.[0]?.delta?.content ?? "");
    }
    if (sawDone || failure) break;
  }
} finally {
  reader.cancel().catch(() => {});
}

if (failure) throw new Error(`stream failed: ${failure.code ?? failure.type}`);
if (!sawDone) throw new Error("stream ended without [DONE]");

Chunk shape

Each SSE event carries a JSON chunk:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1699999999,"model":"<MODEL_ID>","choices":[{"index":0,"delta":{"content":"The"},"finish_reason":null}]}

A chunk carries choices[].logprobs and choices[].delta.refusal only when the provider sent them; otherwise both keys are absent, not null.

By default, a successful stream ends with three separate events (subject to the omission cases and opt-out): the chunk carrying finish_reason, a usage chunk with an empty choices array, and a literal data: [DONE]\n\n:

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1699999999,"model":"<MODEL_ID>","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-...","object":"chat.completion.chunk","created":1699999999,"model":"<MODEL_ID>","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":7,"total_tokens":19,"cached_tokens":0,"reasoning_tokens":0,"prompt_tokens_details":{"cached_tokens":0},"completion_tokens_details":{"reasoning_tokens":0}},"environment_impact":{"energy_kwh":0.00015,"carbon_g_co2":0.06,"water_liters":0.0,"renewable_percent":100,"pue":1.2,"provider_id":"melious","location":"FI"},"billing_cost":{"energy":"2.62","credits":"0.000262","paid_with":"credits"}}

data: [DONE]

The final usage chunk

We send the usage chunk by default — you don't need stream_options for it. Next to usage, it carries environment_impact and billing_cost at the top level, in the same shapes as a non-streamed response. It's sent once the request is settled, so there can be a short pause between the finish_reason chunk and the usage chunk.

Usage reaches you once. Some providers attach usage to a content or finish_reason chunk; we remove it there and send it on the final usage chunk instead. If a provider reports usage more than once, only the last report is sent.

If the provider reports no usage at all, we build the chunk from the token counts we billed, with the same flat and nested token fields as a non-streamed response. Its id, created, and model come from the stream's earlier chunks.

When fields are missing

environment_impact and billing_cost only appear when the request settled cleanly. Available usage can come from a usage-only chunk or have been extracted from a content or finish_reason chunk:

A missing billing_cost isn't a zero charge — a failed or disconnected stream can still be charged for the tokens generated. See Mid-stream failures.

Opting out

Set stream_options: {"include_usage": false} to skip all of this. You get no usage chunk from us, no environment_impact or billing_cost, and nothing built from billed counts. We forward the provider's stream as it comes, including any usage the provider sends — even on a content chunk — so the opt-out doesn't guarantee a stream without usage.

Messages (Anthropic shape)

Anthropic's shape uses typed SSE events instead of a flat chunk:

event: message_start
data: {"type":"message_start","message":{...}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hamburg"}}

...

event: message_stop
data: {"type":"message_stop"}

from anthropic import Anthropic

client = Anthropic(
    api_key="sk-mel-<YOUR_API_KEY>",
    base_url="https://api.melious.ai",
)

with client.messages.stream(
    model="claude-sonnet-4",
    max_tokens=256,
    messages=[{"role": "user", "content": "Write a haiku about Hamburg."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: "sk-mel-<YOUR_API_KEY>",
  baseURL: "https://api.melious.ai",
});

const stream = client.messages.stream({
  model: "claude-sonnet-4",
  max_tokens: 256,
  messages: [{ role: "user", content: "Write a haiku about Hamburg." }],
});

for await (const text of stream.textStream) {
  process.stdout.write(text);
}

The SDKs expose a high-level text_stream / textStream that skips past the event-type dispatch. If you're building a client by hand, dispatch on the event: field — that's what the SDK does internally.

Messages streams report token counts in message_delta, but carry no environment_impact or billing_cost — and neither do non-streamed Messages responses. The same goes for Responses, streamed or not. If you need those fields per request, use Chat completions.

Browser integration

A minimal React hook for streaming chat completions:

import { useState } from "react";

export function useChat() {
  const [text, setText] = useState("");
  const [loading, setLoading] = useState(false);

  async function send(prompt: string) {
    setText("");
    setLoading(true);
    try {
      const response = await fetch("/api/chat", {  // proxy through your backend
        method: "POST",
        body: JSON.stringify({ prompt }),
      });

      if (!response.ok) throw new Error(await response.text());

      const reader = response.body!.getReader();
      const decoder = new TextDecoder();
      let buffer = "";

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        buffer += decoder.decode(value, { stream: true });
        const lines = buffer.split("\n\n");
        buffer = lines.pop()!;
        for (const line of lines) {
          if (!line.startsWith("data: ")) continue;
          const data = line.slice(6);
          if (data === "[DONE]") return;
          const chunk = JSON.parse(data);
          if (chunk.error) throw new Error(chunk.error.message);
          const delta = chunk.choices?.[0]?.delta?.content ?? "";
          setText((t) => t + delta);
        }
      }
    } finally {
      setLoading(false);
    }
  }

  return { text, loading, send };
}

Mid-stream failures

Once a stream has generated output — text, reasoning, tool calls, or on Chat Completions a refusal or token logprobs — a provider failure ends the stream with an error event. What matters for billing is what was generated, not what reached you: if your connection stalled, dropped, or buffered, you can still be charged for the tokens produced.

A request rejected before generation starts — bad input, or no provider available — isn't charged. Retryability still depends on the error type, so check the error before resending.

If output may have been generated, a resend is a new request that can incur a separate charge, and it can duplicate output you never saw. Don't resend automatically after output has started, and don't re-execute tool calls from a failed stream unless your application tolerates running the same action twice.

A stream can fail before the HTTP status is sent: an unknown model, a key without access to it, or a tools, tool_choice or response_format value the model doesn't support returns a normal 4xx with the endpoint's JSON error body, so check response.ok (or let the SDK raise) before you read the body as SSE. Once the 200 is out, every later refusal — a concurrency or daily cap, no eligible balance, a context window overflow, a provider failure — arrives inside the body. Each shape signals it differently:

• Chat completions (OpenAI shape): an error frame, then normally data: [DONE]. A stream may also end unexpectedly without [DONE], so don't rely on the sentinel alone — treat a body that ends without it as failed.
• Messages (Anthropic shape): an error event, which is the last event on the stream. Nothing from the success tail comes before it — no content_block_stop, no tool_use block, no message_delta, no message_stop — and a failure before any output skips message_start too. Treat a stream that ends with neither an error event nor message_stop as failed; it was cut off abnormally.
• Responses: a successful stream ends with response.completed or response.incomplete; a failed one with response.failed, then an error event. Treat a stream that ends with none of these as failed — it was cut off.

See Errors for when a failed request is safe to retry.

Gotchas

• Line buffering. SSE chunks aren't guaranteed to arrive one complete message at a time — you must buffer and split on \n\n, as shown above. Don't parse each value as JSON directly.
• stream_options.include_usage. On by default here, off by default at OpenAI. Setting it to true changes nothing on Melious and keeps your code portable; setting it to false drops environment_impact and billing_cost from the stream.
• Nginx buffering. If you're running your own reverse proxy, make sure it's not buffering the stream (we send X-Accel-Buffering: no to signal intent).
• Keep-alives. Chat Completions streams carry : keep-alive comment lines and Messages streams carry ping events while the model is still thinking. Skip both.

Related

Tool calling covers streaming tool-use events specifically.