Stateful Tools

Stateful tools let AI models persist data across conversation turns without client-side infrastructure. Instead of managing state yourself, the model writes and reads data through built-in tools that ModelRelay executes server-side.

Why Tools?

Making state management a tool rather than a client-side concern has several advantages:

The model can store intermediate results, track progress on complex tasks, and maintain context across a multi-turn workflow—all by calling tools.

Available Tools

ModelRelay provides two categories of stateful tools:

KV Tools

KV tools provide persistent key-value storage scoped to a run or state handle. Use them for storing intermediate results, caching expensive computations, or maintaining context the model needs to reference later.

Writing Data

The kv_write tool stores a string value under a namespaced key:

{
  "type": "function",
  "function": {
    "name": "kv_write",
    "arguments": "{\"key\": \"analysis/summary\", \"value\": \"The codebase uses a clean architecture...\"}"
  }
}

Key format: Keys are namespaced paths using / as a separator (e.g., user/preferences, cache/api-response). This helps organize related data.

Constraints:

• Key: Max 128 bytes, pattern [A-Za-z0-9][A-Za-z0-9_.-]*(/[A-Za-z0-9][A-Za-z0-9_.-]*)*
• Value: Max 32KB
• Total storage: Max 128KB across all keys
• Max keys: 256

Reading Data

The kv_read tool retrieves a value by key:

{
  "type": "function",
  "function": {
    "name": "kv_read",
    "arguments": "{\"key\": \"analysis/summary\"}"
  }
}

Returns:

{"found": true, "value": "The codebase uses a clean architecture..."}

Or if the key doesn’t exist:

{"found": false}

Listing Keys

The kv_list tool returns all stored keys (sorted alphabetically):

{
  "type": "function",
  "function": {
    "name": "kv_list",
    "arguments": "{}"
  }
}

Returns:

{"keys": ["analysis/summary", "cache/api-response", "user/preferences"]}

Deleting Data

The kv_delete tool removes a key:

{
  "type": "function",
  "function": {
    "name": "kv_delete",
    "arguments": "{\"key\": \"cache/api-response\"}"
  }
}

Returns:

{"ok": true, "deleted": true}

Tasks Tool

The tasks_write tool lets the model track progress on multi-step work. This is useful for complex tasks where the model needs to show what it’s working on and what remains.

Writing Tasks

{
  "type": "function",
  "function": {
    "name": "tasks_write",
    "arguments": "{\"tasks\": [{\"content\": \"Analyze authentication module\", \"status\": \"completed\"}, {\"content\": \"Review database queries\", \"status\": \"in_progress\"}, {\"content\": \"Write documentation\", \"status\": \"pending\"}]}"
  }
}

Task structure:

The entire task list is replaced on each write, giving the model full control over task state.

Enabling Stateful Tools

Stateful tools require a context to store data. There are three options:

Option 1: State Context (Persistent)

Create a state handle and pass its id as state_id:

// Create a state handle first
const handleResp = await fetch("https://api.modelrelay.ai/api/v1/state-handles", {
  method: "POST",
  headers: {
    Authorization: "Bearer mr_sk_...",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ ttl_seconds: 86400 }),
});
const handle = await handleResp.json();

// Use stateful tools with state context
const response = await mr.responses.create({
  model: "claude-sonnet-4-5",
  state_id: handle.id,
  input: [
    {
      type: "message",
      role: "user",
      content: [{ type: "text", text: "Analyze this codebase and track your progress" }],
    },
  ],
  tools: [
    { type: "function", function: { name: "kv_write", description: "Store data", parameters: { type: "object", properties: { key: { type: "string" }, value: { type: "string" } }, required: ["key", "value"] } } },
    { type: "function", function: { name: "kv_read", description: "Retrieve data", parameters: { type: "object", properties: { key: { type: "string" } }, required: ["key"] } } },
    { type: "function", function: { name: "tasks_write", description: "Update task list", parameters: { type: "object", properties: { tasks: { type: "array", items: { type: "object", properties: { content: { type: "string" }, status: { type: "string", enum: ["pending", "in_progress", "completed"] } }, required: ["content", "status"] } } }, required: ["tasks"] } } },
  ],
});

// Create a state handle first
handleReq, _ := http.NewRequest(
    http.MethodPost,
    "https://api.modelrelay.ai/api/v1/state-handles",
    strings.NewReader(`{"ttl_seconds":86400}`),
)
handleReq.Header.Set("Authorization", "Bearer mr_sk_...")
handleReq.Header.Set("Content-Type", "application/json")

handleResp, _ := http.DefaultClient.Do(handleReq)
defer handleResp.Body.Close()
var handle struct {
    ID string `json:"id"`
}
_ = json.NewDecoder(handleResp.Body).Decode(&handle)
stateID, _ := uuid.Parse(handle.ID)

// Use stateful tools with state context
response, _ := client.Responses.Create(ctx, sdk.ResponseRequest{
    Model:     "claude-sonnet-4-5",
    StateID:   &stateID,
    Input: []sdk.InputItem{
        {
            Type: "message",
            Role: "user",
            Content: []sdk.ContentPart{
                {Type: "text", Text: "Analyze this codebase and track your progress"},
            },
        },
    },
    Tools: []sdk.Tool{
        sdk.NewFunctionTool("kv_write", "Store data", map[string]any{
            "type": "object",
            "properties": map[string]any{
                "key":   map[string]any{"type": "string"},
                "value": map[string]any{"type": "string"},
            },
            "required": []string{"key", "value"},
        }),
        sdk.NewFunctionTool("kv_read", "Retrieve data", map[string]any{
            "type": "object",
            "properties": map[string]any{
                "key": map[string]any{"type": "string"},
            },
            "required": []string{"key"},
        }),
        // ... tasks_write tool
    },
})

# Create a state handle first
STATE_ID=$(curl -s -X POST https://api.modelrelay.ai/api/v1/state-handles \
  -H "Authorization: Bearer mr_sk_..." \
  -H "Content-Type: application/json" \
  -d '{"ttl_seconds":86400}' | jq -r '.id')

# Use stateful tools with state context
curl -X POST https://api.modelrelay.ai/api/v1/responses \
  -H "Authorization: Bearer mr_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "state_id": "'$STATE_ID'",
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": [{"type": "text", "text": "Analyze this codebase and track your progress"}]
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "kv_write",
          "description": "Store data",
          "parameters": {
            "type": "object",
            "properties": {
              "key": {"type": "string"},
              "value": {"type": "string"}
            },
            "required": ["key", "value"]
          }
        }
      },
      {
        "type": "function",
        "function": {
          "name": "kv_read",
          "description": "Retrieve data",
          "parameters": {
            "type": "object",
            "properties": {
              "key": {"type": "string"}
            },
            "required": ["key"]
          }
        }
      }
    ]
  }'

State handles persist across multiple requests with the same state_id. Handles are validated server-side (ownership + optional TTL).

Option 2: Ephemeral Context

Omit state_id to keep tool state scoped to a single response:

curl -X POST https://api.modelrelay.ai/api/v1/responses \
  -H "Authorization: Bearer mr_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-5",
    "input": [{"type": "message", "role": "user", "content": [{"type": "text", "text": "Track progress"}]}],
    "tools": [{"type": "function", "function": {"name": "kv_write", "description": "Store data", "parameters": {"type": "object", "properties": {"key": {"type": "string"}, "value": {"type": "string"}}, "required": ["key", "value"]}}}]
  }'

Tool state does not persist beyond this response.

Option 3: Run Context

For workflow runs, KV data is automatically scoped to the run and persisted as run artifacts:

# Start a workflow run
curl -X POST https://api.modelrelay.ai/api/v1/runs \
  -H "Authorization: Bearer mr_sk_..." \
  -H "Content-Type: application/json" \
  -d '{
    "workflow_id": "my-workflow",
    "input": {"query": "Analyze the codebase"}
  }'

Run-scoped KV data is stored in the run_memory.v0 artifact and emits kv_updated events that can be observed via run streaming.

Use Cases

Caching Expensive Results

User: "Analyze the security of our API endpoints"

Model calls kv_read(key: "security/api-analysis")
→ {"found": false}

Model performs analysis...

Model calls kv_write(key: "security/api-analysis", value: "...")
→ {"ok": true}

User: "What were the main issues you found?"

Model calls kv_read(key: "security/api-analysis")
→ {"found": true, "value": "..."}

Model responds using cached analysis

Multi-Step Task Tracking

User: "Refactor the authentication module"

Model calls tasks_write(tasks: [
  {content: "Review current implementation", status: "in_progress"},
  {content: "Identify refactoring opportunities", status: "pending"},
  {content: "Implement changes", status: "pending"},
  {content: "Write tests", status: "pending"}
])

... model completes first task ...

Model calls tasks_write(tasks: [
  {content: "Review current implementation", status: "completed"},
  {content: "Identify refactoring opportunities", status: "in_progress"},
  ...
])

Cross-Turn Context

User: "Remember that I prefer TypeScript examples"

Model calls kv_write(key: "user/preferences", value: "{\"language\": \"typescript\"}")

... later in conversation ...

User: "Show me how to use this API"

Model calls kv_read(key: "user/preferences")
→ {"found": true, "value": "{\"language\": \"typescript\"}"}

Model provides TypeScript examples

Best Practices

1. Use namespaced keys — Organize keys by category (cache/, user/, analysis/) to avoid collisions and make data easier to manage.

2. Keep values focused — Store structured JSON for complex data, but keep individual values under 32KB. Split large data across multiple keys.

3. Don’t over-persist — Use KV storage for data the model actually needs to reference. Don’t store conversation history (that’s what sessions are for).

4. Let the model decide — The power of tool-based state is that the model chooses when to save. Don’t force it via prompting unless necessary.

5. Check before writing — For expensive operations, have the model check if cached data exists before recomputing.

6. Clean up — Use kv_delete to remove temporary or stale data, especially in long-running tool loops.

Limitations

• KV storage is scoped to either a state handle or a run—it cannot be shared across both
• Maximum total storage is 128KB per scope
• Keys must follow the namespaced format (no spaces, limited characters)
• Task lists are replaced entirely on each write (no incremental updates)

Next Steps

• Sessions — Multi-turn conversation management
• Workflows — Multi-step AI pipelines with run context
• Tool Use — Working with function calling