Skip to content

[Awareness] Idle Spacebot Consumed Entire Monthly Token Plan in ~2 Days — Cortex Timer-Driven Synthesis #593

@DogaOztuzun

Description

@DogaOztuzun

TL;DR

Spacebot was installed for practice/testing. With zero active usage (only 2 chat inputs over 11 days), the default cortex tick interval of 60s burned through a $100–$200 monthly token subscription in approximately a few days. The token waste came from timer-driven synthesis running on unchanged memories/conversations.


Environment

  • Spacebot version during incident: v0.4.x (previous version)
  • Spacebot version now: v0.5.0
  • Config: Mostly defaults, cortex tick_interval_secs = 60
  • Usage pattern: Installed for practice, not actively used

What Happened — Timeline

Date Event
Apr 27 Spacebot deployed
Apr 29 Brief usage (2 chat messages + 1 worker run)
Apr 30 – May 6 Zero user activity
May 7 API key returns 401 Unauthorized — limit exhausted
May 8 Investigation reveals ~2B tokens consumed while idle

Root Cause: Timer-Driven Synthesis

The default [defaults.cortex] configuration:

[defaults.cortex]
tick_interval_secs        = 60   # Every 60 seconds
worker_timeout_secs       = 1800
branch_timeout_secs       = 120
circuit_breaker_threshold = 3

On each tick, cortex attempts:

  1. generate_knowledge_synthesis
  2. generate_bulletin
  3. generate_profile
  4. Association backfill

Even with zero new activity, each tick re-processes the full agent context.

Result:

  • ~2B tokens consumed over ~11 days of idle operation
  • Provider's monthly subscription usage limit fully exhausted
  • Quick, simple setup — no active usage, no new memories or conversations

Impact

  • Financial: Consumed entire monthly token plan ($100–$200 estimated)
  • Operational: API key invalidated, Spacebot degraded to unusable state
  • Trust: User almost uninstalled after thinking it was a bug/misconfiguration

Suggestions (Non-Blaming)

1. Warning/Documentation

A prominent warning in docs would help:

⚠️ Default cortex interval is 60s. For metered/token-limited plans, increase to 86400 (daily) or disable until actively using Spacebot.

2. Configuration Guide

A dedicated doc for cortex configuration:

# Conservative (metered plans)
[defaults.cortex]
tick_interval_secs = 86400  # Daily

# Moderate (moderate usage)
[defaults.cortex]
tick_interval_secs = 3600   # Hourly

# Aggressive (unlimited plans)
[defaults.cortex]
tick_interval_secs = 300    # Every 5 min

3. Token Estimation

Could cortex estimate tokens per tick based on memory count and warn if it exceeds a threshold?

4. Idle Detection

Skip synthesis if no memories/conversations changed since last tick.


What's Already in v0.5.0 vs What's Planned

Note: The token consumption happened on v0.4.x (previous version). We updated to v0.5.0 after discovering the issue.

Shipped in v0.5.0 (great additions that helped us investigate):

  • token_usage table — tracks every API call with input/output tokens
  • cortex_events table — logs every synthesis attempt with errors
  • ✅ Config watcher — reloads config without restart
  • ✅ Working memory tables — foundation for the redesign

Still NOT in v0.5.0 (the root cause remains):

  • Change-driven synthesis — no version counter or skip logic found in DB schema or binary
  • ❌ Idle detection — cortex still runs on every tick regardless of changes

I see the working memory implementation plan (Phase 3) addresses this with change-driven synthesis:

let current_version = knowledge_synthesis_version.load();
let last_version = knowledge_synthesis_last_version.load();
if current_version == last_version {
    return;  // Skip - nothing changed!
}

Status check: The plan doc is ~3 weeks old. Is Phase 3 (change-driven synthesis) still in progress? Is there a timeline or branch we can follow?


Our Mitigation (Applied)

We changed the config to:

[defaults.cortex]
tick_interval_secs = 86400  # Once per day

This reduces consumption in my providers.

Also reconfigured routing to use a primary model for conversations and a cheaper model for background tasks.


Questions for the Team

  1. Is there an ETA for Phase 3 (change-driven synthesis)?
  2. Should the default tick_interval_secs be higher for new installations?
  3. Would a "first-run wizard" that asks about token plans be helpful?
  4. Can we help test or contribute to the working memory redesign?

Thank you for v0.5.0


Labels: awareness, documentation, cortex, token-usage, configuration

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions