FIX: Keep the backend responsive while starting scenario runs

[varunj-msft]

Description

POST /runs initializes everything eagerly, on purpose, so that configuration errors reach the caller instead of disappearing into a background task. The problem is where that work ran.

It ran directly on the event loop, and it is slow and almost entirely synchronous — loading the default datasets alone takes minutes. For that whole window the server answered nothing at all. Health probes timed out, and the CLI reported the server as unavailable even though it was alive and simply busy. That is the failure mode behind the current End to End Tests failures, where the client gives up before the server has any chance to reply.

The main fix moves the eager initialization onto a worker thread with asyncio.to_thread, following the pattern initializer_service already uses for the same reason. The semaphore, the active-task registry and the create_task hand-off all deliberately stay on the server loop: asyncio.run cancels whatever is still pending when it closes its loop, so a background task created inside the worker thread would be destroyed the instant initialization finished. That shape silently cancels every run, and it is specifically avoided here.

Two concurrency-permit leaks are fixed along the way. Both are on paths the previous except Exception: release; raise could not reach:

• A client disconnecting during initialization raises CancelledError, which is a BaseException and so was never caught.
• The missing scenario_result_id check sat outside the try block entirely.

Either one leaked a permit, and three such failures exhausted the concurrency limit and wedged the server for the rest of the session. For the E2E suite that matters, because one session-scoped backend serves every scenario in the run. The permit is now released from a finally block until ownership transfers to the background task, tracked with an explicit release_on_exit flag so it is released exactly once and never twice.

The response is also built before the task hand-off, so a lookup failure can no longer leave a run executing that the caller never received an id for. The active_tasks entry is unwound on that path too.

Finally, the start_scenario_run route docstring said "Returns immediately", which was not true before this change and is still not true after it. It now describes what actually happens.

Part of the v1.1.0 release wave with #2510, #2511 and #2512.

Tests and Documentation

Six new tests in tests/unit/backend/test_scenario_run_service.py:

• test_start_run_keeps_event_loop_responsive counts heartbeats on the loop during a slow start. A blocked loop yields zero.
• test_start_run_background_task_survives_handoff asserts the run actually executes. This is the test that catches the "silently cancels every run" shape.
• test_start_run_releases_semaphore_when_cancelled_during_init covers CancelledError being a BaseException.
• test_start_run_releases_semaphore_when_result_id_missing covers the check that used to sit outside the try.
• test_start_run_cleans_up_when_response_lookup_fails asserts no stranded permit and no stranded active_tasks entry.
• test_start_run_releases_semaphore_exactly_once_on_success guards against the obvious over-correction of double-releasing.

The first two matter as a pair rather than individually: a responsiveness fix that cancels every run would pass the responsiveness test on its own, so the hand-off test is what makes the first one meaningful.

test_start_run_exceeds_concurrent_limit needed a fix. It was passing for the wrong reason: it relied on the event loop never yielding during start, so the mocked runs completed and handed their permits straight back before the limit could ever be reached. Now that start yields, the test holds its background runs open, which is what a real run does.

Ran pytest tests/unit/backend/test_scenario_run_service.py: 70 passed.

Documentation: the start_scenario_run route docstring is corrected in this PR. JupyText was not run and is not applicable: no notebooks or code samples are affected, and the public API is unchanged.

[Richard Lundeen (rlundeen2)]

(GHCP Generated): FYI only - this PR should not wait on the Scenario stack. Stacked PR #2376 introduces FIFO scheduling in ScenarioRunService and queues fully initialized runs, so it overlaps this service. The event-loop responsiveness fix here is still needed. If #2522 merges first, we will rebase the stack and preserve the worker-thread preparation while adapting the semaphore-specific cleanup that FIFO scheduling supersedes.

[hannahwestra25]

i think this can still race the same sqlite connection. this only serializes prep jobs, but active runs and status endpoints access it too. could we serialize this inside SQLiteMemory instead?

[hannahwestra25]

asyncio.run() closes this loop, so async stuff created during init can come back already cancelled. could we keep async init on the app loop and only send the blocking sync work to the executor?

[hannahwestra25]

I think we need to mark the scenario run as cancelled so we don't leave something in the created state here if the request gets cancelled

scenario = prepare_task.result()
if scenario._scenario_result_id:
    self._memory.update_scenario_run_state(
        scenario_result_id=scenario._scenario_result_id,
        scenario_run_state=ScenarioRunState.CANCELLED,
    )