Skip to content

orchestration: wake cursor keyed on wrapper_session_id but seq=env.message.line → silent message drop #4583

Description

@sanil-23

Problem

The orchestration wake path can silently drop an inbound agent message (persisted + acked but never delivered to the graph) because the wake cursor key and the message seq domain are now decoupled.

  • The session key (and thus the wake idempotence cursor) is wrapper_session_id — the shared per-pair conversation id (src/openhuman/orchestration/ingest.rs, SessionEnvelopeV1::session_key, introduced in fix(orchestration): key inbound DMs on the shared per-pair session id #4582).
  • seq is env.message.line (ingest.rs classify_message).
  • upsert_session clamps last_seq = MAX(old, new) (store.rs:169); the wake fires only when latest_seq > cursor (has_new_work, ops.rs:194) and ingest_cursor_lag counts last_seq > cursor (store.rs:384).

env.message.line is monotonic only within a single emitting harness session. Once the key is wrapper_session_id, a message whose seq <= last_seq is MAX-clamped away and the wake is skipped — the message is persisted and acked but never reaches the orchestration graph. Pre-#4582 this was impossible (key seq domain = harness_session_id).

When it bites

  1. Cross-harness reuse — a peer restarts/reopens the harness but the thread keeps the same wrapper_session_id; the new harness's line counter resets/lowers.
  2. Plugin-composed messages carry line: 0 — the tiny.place plugin's MCP send/auto_reply path emits message.line = 0 on every message (verified via encodeEnvelope). If that is the primary A2A path, seq is effectively degenerate (always 0), so the cursor cannot distinguish messages by seq at all → the 2nd+ inbound message on a session can skip the wake.

The failure mode is silent (no error, message acked), which is the exact class worth tracking.

Interim guard (shipped in #4582)

persist_message now logs a warn when a session window receives seq <= last_seq, so the invariant break surfaces in prod instead of dropping quietly. This does not fix the drop — it only makes it observable.

Robust fix (this issue)

Decouple the wake cursor from env.message.line. Options (in ops.rs / store.rs):

  • A per-wrapper monotonic ingest counter assigned at persist time (independent of the emitting harness), used as seq/cursor.
  • Or key the wake cursor on arrival order / message id / ingest timestamp rather than line.
  • Ensure ingest_cursor_lag and has_new_work use the same monotonic quantity.

Add tests for: (a) two messages with equal/lower line on one wrapper_session_id both wake the graph; (b) a plugin-style line: 0 stream advances the cursor per message.

Refs

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions