Streaming

Streaming delivers AI responses incrementally as they’re generated, enabling real-time display in chat UIs and reducing perceived latency.

How It Works

sequenceDiagram
    participant Client
    participant ModelRelay
    participant Provider

    Client->>ModelRelay: POST /responses (Accept: ndjson)
    ModelRelay->>Provider: Forward request
    Provider-->>ModelRelay: start event
    ModelRelay-->>Client: {"type":"start",...}
    loop Token generation
        Provider-->>ModelRelay: token
        ModelRelay-->>Client: {"type":"update","delta":"..."}
    end
    Provider-->>ModelRelay: done
    ModelRelay-->>Client: {"type":"completion",...}

ModelRelay supports NDJSON (Newline-Delimited JSON) and Server-Sent Events (SSE) for streaming responses. Each NDJSON line (or SSE data: payload) is a complete JSON object representing a stream event:

{"type":"start","stream_mode":"text-delta","stream_version":"v2","request_id":"...","provider":"anthropic","model":"claude-sonnet-5-5"}
{"type":"update","delta":"Hello"}
{"type":"update","delta":" world"}
{"type":"completion","stop_reason":"end_turn","usage":{"input_tokens":10,"output_tokens":5,...}}

To enable streaming, set the Accept header:

Accept: application/x-ndjson; profile="responses-stream/v2"

For SSE, use:

Accept: text/event-stream

Both formats carry the same JSON events. With SSE, each event is sent as a data: payload and the SSE event name matches the type field (for example, event: update). An NDJSON Accept header without the responses-stream/v2 profile is rejected with 406.

Note: EventSource only supports GET requests, so use a POST-capable SSE client (for example, @microsoft/fetch-event-source) for /responses.

Event Types

Type	Description
`start`	Stream opened, includes model and request ID
`update`	Text delta (partial content)
`reasoning_delta`	Reasoning text delta, when the provider exposes reasoning
`completion`	Stream finished, includes usage and stop reason
`error`	Error occurred during generation
`tool_use_start`	Tool call started
`tool_use_delta`	Tool call argument delta
`tool_use_stop`	Tool call completed
`keepalive`	Heartbeat (NDJSON only; SSE sends a `: keepalive` comment instead). Ignore it.

Quick Start

Stream a response with curl. -N disables curl’s output buffering so lines print as they arrive:

curl -N https://api.modelrelay.ai/api/v1/responses \
  -H "X-ModelRelay-Api-Key: mr_sk_..." \
  -H "Content-Type: application/json" \
  -H "Accept: application/x-ndjson; profile=\"responses-stream/v2\"" \
  -d '{
    "model": "claude-sonnet-5-5",
    "input": [
      {"type": "message", "role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
      {"type": "message", "role": "user", "content": [{"type": "text", "text": "Write a haiku about programming."}]}
    ]
  }'

Reading the Stream

Any HTTP client that can read a response body incrementally can consume the stream. Split the body on newlines and parse each line as JSON. This TypeScript helper uses the standard fetch API and yields one event per line:

async function* streamResponse(body: unknown, signal?: AbortSignal) {
  const res = await fetch("https://api.modelrelay.ai/api/v1/responses", {
    method: "POST",
    headers: {
      "X-ModelRelay-Api-Key": process.env.MODELRELAY_API_KEY!,
      "Content-Type": "application/json",
      Accept: 'application/x-ndjson; profile="responses-stream/v2"',
    },
    body: JSON.stringify(body),
    signal,
  });
  if (!res.ok) {
    // Failures before the stream opens return a JSON error body.
    throw new Error(`HTTP ${res.status}: ${await res.text()}`);
  }

  const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader();
  let buffer = "";
  for (;;) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += value;
    let newline;
    while ((newline = buffer.indexOf("\n")) >= 0) {
      const line = buffer.slice(0, newline).trim();
      buffer = buffer.slice(newline + 1);
      if (!line) continue;
      const event = JSON.parse(line);
      if (event.type === "keepalive") continue;
      yield event;
    }
  }
}

Handle each event by its type:

const request = {
  model: "claude-sonnet-5-5",
  input: [
    { type: "message", role: "user", content: [{ type: "text", text: "Hello!" }] },
  ],
};

for await (const event of streamResponse(request)) {
  switch (event.type) {
    case "start":
      console.log(`Stream started: ${event.request_id} (${event.model})`);
      break;
    case "update":
      process.stdout.write(event.delta ?? "");
      break;
    case "completion":
      console.log("\n---");
      console.log("Stop reason:", event.stop_reason);
      console.log("Usage:", event.usage);
      break;
    case "error":
      throw new Error(`${event.code}: ${event.message}`);
  }
}

Collecting the Full Text

update events carry incremental deltas. To keep the complete text, concatenate the delta values as they arrive. The completion event carries usage and stop_reason; it includes content only when no update deltas were sent, so do not rely on it to repeat text you already received.

Stream Timeouts

Streams can stall when a provider is slow to start or stops sending tokens. Enforce timeouts in your client:

Timeout	Description	Use Case
TTFT	Time to first token	Detect if model is responding
Idle	Max gap between events	Detect stalled streams
Total	Overall stream duration	Prevent runaway requests

ModelRelay sends keepalives ({"type":"keepalive"} lines on NDJSON, : keepalive comments on SSE) during long gaps, so measure the idle timeout against content events rather than raw bytes if you want to detect a model that has stopped producing output. Reasoning models may send reasoning_delta events before the first update.

Error Handling

Errors surface in two places:

Before the stream opens — validation, authentication, and routing failures return a normal HTTP error status with a JSON error body. An NDJSON Accept header without the responses-stream/v2 profile returns 406.
During the stream — a failure after the stream has started arrives as an error event, and the stream ends. The event includes code, message, status_code, request_id, layer, cause, and retryable.

{"type":"error","code":"SERVICE_UNAVAILABLE","message":"...","status":503,"status_code":503,"request_id":"...","layer":"upstream","cause":"...","retryable":true}

Decide whether to retry from retryable, not from the status alone. A stream that closes without a completion or error event was interrupted in transit. See Error Handling for the error fields and retry guidance.

Cancelling Streams

To stop generation, close the HTTP connection. With fetch, pass an AbortSignal:

const controller = new AbortController();
setTimeout(() => controller.abort(), 5000);

try {
  for await (const event of streamResponse(request, controller.signal)) {
    if (event.type === "update") process.stdout.write(event.delta ?? "");
  }
} catch (error) {
  if ((error as Error).name === "AbortError") {
    console.log("\nStream cancelled");
  } else {
    throw error;
  }
}

Display Patterns

Cursor Animation

Show a typing indicator while waiting:

let content = "";
const cursor = "▋";

for await (const event of streamResponse(request)) {
  if (event.type === "update" && event.delta) {
    content += event.delta;
    // Clear line and redraw with cursor
    process.stdout.write(`\r${content}${cursor}`);
  }
}
// Remove cursor when done
process.stdout.write(`\r${content} \n`);

Buffered Updates

Batch updates for smoother rendering:

let buffer = "";
let lastRender = Date.now();
const RENDER_INTERVAL = 50; // ms

for await (const event of streamResponse(request)) {
  if (event.type === "update" && event.delta) {
    buffer += event.delta;

    const now = Date.now();
    if (now - lastRender >= RENDER_INTERVAL) {
      process.stdout.write(buffer);
      buffer = "";
      lastRender = now;
    }
  }
}
// Flush remaining buffer
if (buffer) {
  process.stdout.write(buffer);
}

Streaming with Tool Calls

When the model calls a tool, the stream carries tool events alongside text:

tool_use_start and tool_use_delta carry a tool_call_delta object with the call’s index, and, when known, its id, type, and a function fragment (name, partial arguments string). Use these to show progress.
tool_use_stop carries the finished calls in tool_calls, each with id, type, and function.name / function.arguments (a JSON string).

const toolCalls = [];

for await (const event of streamResponse(requestWithTools)) {
  switch (event.type) {
    case "update":
      process.stdout.write(event.delta ?? "");
      break;
    case "tool_use_start":
      if (event.tool_call_delta?.function?.name) {
        console.log(`\nCalling ${event.tool_call_delta.function.name}...`);
      }
      break;
    case "tool_use_stop":
      toolCalls.push(...(event.tool_calls ?? []));
      break;
  }
}

for (const call of toolCalls) {
  console.log(`Tool: ${call.function?.name}`);
  console.log(`Args: ${call.function?.arguments}`);
}

See Tool Use for sending tool results back to the model.

Best Practices

Always finish or abort the request - Read the stream to its terminal event, or abort the request so the connection is released
Set appropriate timeouts - Prevent hung connections with TTFT and idle timeouts
Accumulate deltas yourself - update deltas are incremental, and completion does not repeat text that was already streamed
Check for error events - Failures after the stream opens arrive as an event, not an HTTP status
Ignore unknown event types - Skip keepalive and any event type your client does not handle

Next Steps

First Request — Basic request examples
Responses API — Full stream event field reference
Tool Use — Streaming with function calls
Structured Output — Streaming typed JSON