Stateful Tools

Stateful tools let AI models persist data across conversation turns without client-side infrastructure. Instead of managing state yourself, the model writes and reads data through built-in tools that ModelRelay executes server-side.

Why Tools?

Making state management a tool rather than a client-side concern has several advantages:

The model can store intermediate results, track progress on complex tasks, and maintain context across a multi-turn workflow—all by calling tools.

Available Tools

ModelRelay provides two categories of stateful tools:

KV Tools

KV tools provide persistent key-value storage scoped to a run or session. Use them for storing intermediate results, caching expensive computations, or maintaining context the model needs to reference later.

Writing Data

The kv.write tool stores a string value under a namespaced key:

{
  "type": "function",
  "function": {
    "name": "kv.write",
    "arguments": "{\"key\": \"analysis/summary\", \"value\": \"The codebase uses a clean architecture...\"}"
  }
}

Key format: Keys are namespaced paths using / as a separator (e.g., user/preferences, cache/api-response). This helps organize related data.

Constraints:

• Key: Max 128 bytes, pattern [A-Za-z0-9][A-Za-z0-9_.-]*(/[A-Za-z0-9][A-Za-z0-9_.-]*)*
• Value: Max 32KB
• Total storage: Max 128KB across all keys
• Max keys: 256

Reading Data

The kv.read tool retrieves a value by key:

{
  "type": "function",
  "function": {
    "name": "kv.read",
    "arguments": "{\"key\": \"analysis/summary\"}"
  }
}

Returns:

{"found": true, "value": "The codebase uses a clean architecture..."}

Or if the key doesn’t exist:

{"found": false}

Listing Keys

The kv.list tool returns all stored keys (sorted alphabetically):

{
  "type": "function",
  "function": {
    "name": "kv.list",
    "arguments": "{}"
  }
}

Returns:

{"keys": ["analysis/summary", "cache/api-response", "user/preferences"]}

Deleting Data

The kv.delete tool removes a key:

{
  "type": "function",
  "function": {
    "name": "kv.delete",
    "arguments": "{\"key\": \"cache/api-response\"}"
  }
}

Returns:

{"ok": true, "deleted": true}

Tasks Tool

The tasks.write tool lets the model track progress on multi-step work. This is useful for complex tasks where the model needs to show what it’s working on and what remains.

Writing Tasks

{
  "type": "function",
  "function": {
    "name": "tasks.write",
    "arguments": "{\"tasks\": [{\"content\": \"Analyze authentication module\", \"status\": \"completed\"}, {\"content\": \"Review database queries\", \"status\": \"in_progress\"}, {\"content\": \"Write documentation\", \"status\": \"pending\"}]}"
  }
}

Task structure:

The entire task list is replaced on each write, giving the model full control over task state.

Enabling Stateful Tools

Stateful tools require a context to store data. There are two options:

Option 1: Session Context

Pass a session_id to enable session-scoped storage:

// Create a session first
const session = await mr.sessions.create();

// Use stateful tools with session context
const response = await mr.responses.create({
  model: "claude-sonnet-4-5",
  session_id: session.id,
  input: [
    {
      type: "message",
      role: "user",
      content: [{ type: "text", text: "Analyze this codebase and track your progress" }],
    },
  ],
  tools: [
    { type: "function", function: { name: "kv.write", description: "Store data", parameters: { type: "object", properties: { key: { type: "string" }, value: { type: "string" } }, required: ["key", "value"] } } },
    { type: "function", function: { name: "kv.read", description: "Retrieve data", parameters: { type: "object", properties: { key: { type: "string" } }, required: ["key"] } } },
    { type: "function", function: { name: "tasks.write", description: "Update task list", parameters: { type: "object", properties: { tasks: { type: "array", items: { type: "object", properties: { content: { type: "string" }, status: { type: "string", enum: ["pending", "in_progress", "completed"] } }, required: ["content", "status"] } } }, required: ["tasks"] } } },
  ],
});

// Create a session first
session, _ := client.Sessions.Create(ctx, sdk.SessionCreateRequest{})

// Use stateful tools with session context
response, _ := client.Responses.Create(ctx, sdk.ResponseRequest{
    Model:     "claude-sonnet-4-5",
    SessionID: &session.ID,
    Input: []sdk.InputItem{
        {
            Type: "message",
            Role: "user",
            Content: []sdk.ContentPart{
                {Type: "text", Text: "Analyze this codebase and track your progress"},
            },
        },
    },
    Tools: []sdk.Tool{
        sdk.NewFunctionTool("kv.write", "Store data", map[string]any{
            "type": "object",
            "properties": map[string]any{
                "key":   map[string]any{"type": "string"},
                "value": map[string]any{"type": "string"},
            },
            "required": []string{"key", "value"},
        }),
        sdk.NewFunctionTool("kv.read", "Retrieve data", map[string]any{
            "type": "object",
            "properties": map[string]any{
                "key": map[string]any{"type": "string"},
            },
            "required": []string{"key"},
        }),
        // ... tasks.write tool
    },
})

# Create a session first
SESSION_ID=$(curl -s -X POST https://api.modelrelay.ai/api/v1/sessions \
  -H "Authorization: Bearer mr_sk_..." \
  -H "Content-Type: application/json" \
  -d '{}' | jq -r '.id')

# Use stateful tools with session context
curl -X POST https://api.modelrelay.ai/api/v1/responses \
  -H "Authorization: Bearer mr_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "session_id": "'$SESSION_ID'",
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": [{"type": "text", "text": "Analyze this codebase and track your progress"}]
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "kv.write",
          "description": "Store data",
          "parameters": {
            "type": "object",
            "properties": {
              "key": {"type": "string"},
              "value": {"type": "string"}
            },
            "required": ["key", "value"]
          }
        }
      },
      {
        "type": "function",
        "function": {
          "name": "kv.read",
          "description": "Retrieve data",
          "parameters": {
            "type": "object",
            "properties": {
              "key": {"type": "string"}
            },
            "required": ["key"]
          }
        }
      }
    ]
  }'

Session storage persists across multiple requests with the same session_id. Data is stored in session metadata under modelrelay.kv and modelrelay.tasks.

Option 2: Run Context

For workflow runs, KV data is automatically scoped to the run and persisted as run artifacts:

# Start a workflow run
curl -X POST https://api.modelrelay.ai/api/v1/runs \
  -H "Authorization: Bearer mr_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_id": "my-workflow",
    "input": {"query": "Analyze the codebase"}
  }'

Run-scoped KV data is stored in the run_memory.v0 artifact and emits kv_updated events that can be observed via run streaming.

Use Cases

Caching Expensive Results

User: "Analyze the security of our API endpoints"

Model calls kv.read(key: "security/api-analysis")
→ {"found": false}

Model performs analysis...

Model calls kv.write(key: "security/api-analysis", value: "...")
→ {"ok": true}

User: "What were the main issues you found?"

Model calls kv.read(key: "security/api-analysis")
→ {"found": true, "value": "..."}

Model responds using cached analysis

Multi-Step Task Tracking

User: "Refactor the authentication module"

Model calls tasks.write(tasks: [
  {content: "Review current implementation", status: "in_progress"},
  {content: "Identify refactoring opportunities", status: "pending"},
  {content: "Implement changes", status: "pending"},
  {content: "Write tests", status: "pending"}
])

... model completes first task ...

Model calls tasks.write(tasks: [
  {content: "Review current implementation", status: "completed"},
  {content: "Identify refactoring opportunities", status: "in_progress"},
  ...
])

Cross-Turn Context

User: "Remember that I prefer TypeScript examples"

Model calls kv.write(key: "user/preferences", value: "{\"language\": \"typescript\"}")

... later in conversation ...

User: "Show me how to use this API"

Model calls kv.read(key: "user/preferences")
→ {"found": true, "value": "{\"language\": \"typescript\"}"}

Model provides TypeScript examples

Best Practices

1. Use namespaced keys — Organize keys by category (cache/, user/, analysis/) to avoid collisions and make data easier to manage.

2. Keep values focused — Store structured JSON for complex data, but keep individual values under 32KB. Split large data across multiple keys.

3. Don’t over-persist — Use KV storage for data the model actually needs to reference. Don’t store conversation history (that’s what sessions are for).

4. Let the model decide — The power of tool-based state is that the model chooses when to save. Don’t force it via prompting unless necessary.

5. Check before writing — For expensive operations, have the model check if cached data exists before recomputing.

6. Clean up — Use kv.delete to remove temporary or stale data, especially in long-running sessions.

Limitations

• KV storage is scoped to either a session or a run—it cannot be shared across both
• Maximum total storage is 128KB per scope
• Keys must follow the namespaced format (no spaces, limited characters)
• Task lists are replaced entirely on each write (no incremental updates)

Next Steps

• Sessions — Multi-turn conversation management
• Workflows — Multi-step AI pipelines with run context
• Tool Use — Working with function calling