Empty stream (NoOutputGeneratedError) should be retried by RetryManager

[buger]

Problem

When the AI provider returns an empty stream (zero steps recorded), the Vercel AI SDK throws NoOutputGeneratedError with message "No output generated. Check the stream for errors.". This error is not retried by ProbeAgent, causing the entire check to fail on what is usually a transient provider hiccup.

Production example: task 70d0b650 — a simple "hi!" greeting to gemini-3.1-pro-preview resulted in an empty stream. The user saw:

"The AI provider rate limit was reached before a response could be generated."

Root Cause

The retry boundary is in the wrong place:

1. RetryManager.executeWithRetry() wraps the streamText() call (ProbeAgent.js:4429)
2. streamText() returns immediately — it doesn't throw
3. The actual error surfaces later when await result.steps is called (ProbeAgent.js:4468)
4. This is outside the RetryManager scope — caught by the outer catch at line 4711
5. isRetryableError() checks the error message against DEFAULT_RETRYABLE_ERRORS, but "No output generated" doesn't match any pattern (429, rate_limit, timeout, etc.)

RetryManager.executeWithRetry(streamText(...))  ← retry wraps this
  → streamText() returns immediately (OK)       ← no error here
  → await result.steps                           ← NoOutputGeneratedError thrown HERE
  → caught at outer catch (line 4711)            ← NOT retried
  → "Error: Failed to get response from AI model. No output generated."

Suggested Fix

Two changes needed:

1. Add "No output generated" to DEFAULT_RETRYABLE_ERRORS

In RetryManager.js:

const DEFAULT_RETRYABLE_ERRORS = [
  'Overloaded',
  'overloaded',
  'rate_limit',
  'rate limit',
  '429',
  '500',
  '502',
  '503',
  '504',
  'timeout',
  'ECONNRESET',
  'ETIMEDOUT',
  'ENOTFOUND',
  'api_error',
  'No output generated',  // ← add this
];

2. Wrap await result.steps in a retry loop

In ProbeAgent.js around lines 4464-4488, the executeAIRequest function should catch NoOutputGeneratedError from await result.steps and re-invoke streamTextWithRetryAndFallback:

const executeAIRequest = async () => {
  const maxEmptyRetries = 2;
  for (let emptyRetry = 0; emptyRetry <= maxEmptyRetries; emptyRetry++) {
    const result = await this.streamTextWithRetryAndFallback(streamOptions);
    // ... timeout setup ...
    try {
      const steps = await result.steps;
      if (!steps || steps.length === 0) {
        if (emptyRetry < maxEmptyRetries) {
          console.log(`[WARN] Empty stream (0 steps), retrying (${emptyRetry + 1}/${maxEmptyRetries})...`);
          continue;
        }
      }
      // ... rest of text extraction ...
      return { finalText, result };
    } catch (err) {
      if (err?.name === 'AI_NoOutputGeneratedError' && emptyRetry < maxEmptyRetries) {
        console.log(`[WARN] NoOutputGeneratedError, retrying (${emptyRetry + 1}/${maxEmptyRetries})...`);
        continue;
      }
      throw err;
    }
  }
};

Impact

Any transient provider hiccup that returns an empty stream (common with Gemini under load) kills the entire task with no retry. This is especially bad for simple queries that would succeed on retry.

Trace Evidence

ai.request.started:  model=gemini-3.1-pro-preview, input_length=13316
ai.request.failed:   error="No output generated. Check the stream for errors."
visor.ai_check.failed: error="Error: Failed to get response from AI model. No output generated."
visor.check.generate-response.failed: error="The AI provider rate limit was reached..."

[probelabs]

Issue Analysis: Empty Stream Retry Handling

Summary

Thank you for this detailed analysis of the NoOutputGeneratedError retry issue. Your investigation correctly identifies a gap in Probe's retry logic where empty streams from AI providers are not retried despite being transient failures.

Issue Classification

• Category: bug
• Priority: medium
• Complexity: simple
• Recommended Labels: bug

Root Cause Confirmation

I've verified your analysis against the codebase:

1. Retry Boundary: RetryManager.executeWithRetry() wraps streamText() at ProbeAgent.js:1473-1478 and ProbeAgent.js:1523-1528. The streamText() call returns immediately with a result object containing Promises (steps, text, usage).

2. Error Surface: The actual error occurs when await result.steps is called at ProbeAgent.js:4468, which is outside the retry scope.

3. Missing Pattern: DEFAULT_RETRYABLE_ERRORS in RetryManager.js:14-27 does not include 'No output generated' or related patterns.

4. Error Handling: Errors are caught at ProbeAgent.js:4899 with the message "Failed to get response from AI model" but no retry logic exists for this specific error type.

Proposed Solution Assessment

Your suggested two-part fix is well-designed:

Part 1: Add Retryable Error Pattern ✅

Adding 'No output generated' to DEFAULT_RETRYABLE_ERRORS is straightforward and aligns with existing patterns. This is a simple, low-risk change.

Part 2: Wrap await result.steps in Retry Loop ⚠️

Your proposed retry loop around executeAIRequest is conceptually sound, but I'd suggest considering:

• Scope: The retry loop should be placed around the streamTextWithRetryAndFallback call, not just await result.steps, to ensure the entire stream generation is retried.
• Integration: This should leverage the existing RetryManager infrastructure rather than implementing a custom retry loop.
• Error Detection: The check for empty steps (steps.length === 0) should happen before awaiting result.text to catch the issue early.

Additional Considerations

1. Error Name: The Vercel AI SDK error is named AI_NoOutputGeneratedError (with AI_ prefix), which should be included in pattern matching.

2. Empty Steps Handling: Current code at ProbeAgent.js:4468-4475 already handles empty steps arrays by falling back to result.text, but this doesn't help if the error is thrown before resolution.

3. Test Coverage: The test suite in last-step-text.test.js includes tests for empty steps arrays, but may need additional coverage for the error-throwing scenario.

Next Steps

1. Verify Error Name: Confirm the exact error name/structure thrown by Vercel AI SDK for empty streams.

2. Implementation Priority: Start with Part 1 (adding the error pattern) as it's simpler and provides immediate benefit.

3. Design Review for Part 2: Consider whether to:

   • Extend RetryManager to handle post-stream errors
   • Add a wrapper around executeAIRequest that catches and retries empty stream errors
   • Modify streamTextWithRetryAndFallback to validate the result before returning

4. Testing: Add test cases that simulate NoOutputGeneratedError to verify retry behavior.

Questions for Clarification

1. Can you provide a stack trace or error object structure for the NoOutputGeneratedError to confirm the exact error name and properties?

2. Is the empty stream issue specific to certain providers (e.g., Gemini under load) or observed across multiple providers?

3. Would you prefer the retry to be provider-specific (retry same provider) or trigger fallback to a different provider?

References

• probe/npm/src/agent/RetryManager.js:14-27 – DEFAULT_RETRYABLE_ERRORS constant
• probe/npm/src/agent/RetryManager.js:90-279 – executeWithRetry implementation
• probe/npm/src/agent/ProbeAgent.js:1463-1532 – _executeWithVercelProvider with retry wrapper
• probe/npm/src/agent/ProbeAgent.js:4468-4475 – result.steps await and empty steps handling
• probe/npm/src/agent/ProbeAgent.js:4899 – error catch with "Failed to get response from AI model" message
• probe/npm/tests/unit/last-step-text.test.js – test coverage for empty steps scenarios

---

Powered by Visor from Probelabs

💡 TIP: You can chat with Visor using /visor ask <your question>