RFC: Adaptive Exponential Backoff and Reconnection Jitter for DDP/WebSocket Connections

[Shevilll]

RFC: Adaptive Exponential Backoff and Reconnection Jitter for DDP/WebSocket Connections

• Type: Resiliency & Infrastructure RFC
• Target Subsystem: Connection Service (app/lib/services/connect.ts) & SDK Initializer (app/lib/services/sdk.ts)
• Status: Proposed

1. Problem Statement & Root Cause Analysis

The client initializes its connection to the Rocket.Chat server via the Rocket.Chat JS SDK:

private initializeSdk(server: string): typeof Rocketchat {
	return new Rocketchat({ host: server, protocol: 'ddp', useSsl: isSsl(server), reopen: __DEV__ ? 20000 : 5000 });
}

The Architectural Defect: Fixed Interval Reconnection

• Reconnection Polling: In production, the reopen configuration parameter is set to a fixed 5000ms (5 seconds). When the connection is dropped, the app blindly retries establishing a connection every 5 seconds.
• Thundering Herd Problem: If a server experiences a temporary outage, or a network flap occurs, thousands of active mobile clients disconnect simultaneously. When they all attempt to reconnect at identical 5-second intervals, they hammer the server. This synchronized reconnection storm creates a distributed denial-of-service (DDoS) effect, preventing the server from recovering gracefully.
• Battery & Data Exhaustion: Mobile clients attempting aggressive WebSocket handshakes every 5 seconds on a flaky cellular network will rapidly deplete device battery and consume unnecessary user cellular bandwidth.

---

2. Proposed Architectural Solution

Replace the static reconnect interval with an Adaptive Exponential Backoff algorithm containing Randomized Jitter.

The Mathematical Model

The delay between reconnection attempts should follow this formula:

$$T_{\text{retry}} = \min(T_{\text{max}}, T_{\text{base}} \times M^{\text{attempt}}) + \text{random_jitter}$$

• $T_{\text{base}} = 1000,\text{ms}$ (Initial retry delay: 1 second)
• $M = 2.0$ (Backoff multiplier)
• $T_{\text{max}} = 60000,\text{ms}$ (Cap the delay at 60 seconds to guarantee connection within 1 minute of recovery)
• $\text{random_jitter} \in [0, 0.5 \times T_{\text{retry}}]$ (Add up to 50% randomized variation to desynchronize clients)

Expected Reconnection Delay Curve

Attempt	Base Delay ($2^n$)	Jitter Range (0–50%)	Total Delay Range
1	1.0s	0.0s – 0.5s	1.0s – 1.5s
2	2.0s	0.0s – 1.0s	2.0s – 3.0s
3	4.0s	0.0s – 2.0s	4.0s – 6.0s
4	8.0s	0.0s – 4.0s	8.0s – 12.0s
5	16.0s	0.0s – 8.0s	16.0s – 24.0s
6	32.0s	0.0s – 16.0s	32.0s – 48.0s
7 (Max)	60.0s	0.0s – 30.0s	60.0s – 90.0s

---

3. Proposed Blueprint & Implementation Strategy

Since the Rocket.Chat JS SDK under the hood uses a simple internal interval, we can either intercept the connection events or override the reconnection scheduler on the SDK driver directly.

Connection Resilience Controller Blueprint

// Proposed Reconnection Controller for app/lib/services/ReconnectionManager.ts

interface IBackoffConfig {
	baseDelay: number;    // 1000ms
	maxDelay: number;     // 60000ms
	multiplier: number;   // 2.0
	jitter: number;       // 0.5 (50%)
}

export class ReconnectionManager {
	private attempt = 0;
	private config: IBackoffConfig;
	private reconnectTimer: ReturnType<typeof setTimeout> | null = null;
	private onReconnectCallback: () => Promise<void>;

	constructor(onReconnect: () => Promise<void>, config?: Partial<IBackoffConfig>) {
		this.onReconnectCallback = onReconnect;
		this.config = {
			baseDelay: 1000,
			maxDelay: 60000,
			multiplier: 2.0,
			jitter: 0.5,
			...config
		};
	}

	public reset() {
		this.attempt = 0;
		if (this.reconnectTimer) {
			clearTimeout(this.reconnectTimer);
			this.reconnectTimer = null;
		}
	}

	/**
	 * Computes the next retry delay using exponential backoff with full jitter.
	 */
	public getNextDelay(): number {
		const rawDelay = this.config.baseDelay * Math.pow(this.config.multiplier, this.attempt);
		const cappedDelay = Math.min(this.config.maxDelay, rawDelay);
		
		// Add randomized jitter
		const jitterRange = cappedDelay * this.config.jitter;
		const randomJitter = Math.random() * jitterRange;
		
		this.attempt++;
		return cappedDelay + randomJitter;
	}

	/**
	 * Schedules the next reconnection attempt.
	 */
	public scheduleReconnection() {
		if (this.reconnectTimer) return;

		const delay = this.getNextDelay();
		console.log(`[Reconnection] Connection lost. Attempt ${this.attempt}. Retrying in ${(delay / 1000).toFixed(2)} seconds...`);

		this.reconnectTimer = setTimeout(async () => {
			this.reconnectTimer = null;
			try {
				await this.onReconnectCallback();
			} catch (error) {
				// If attempt fails, schedule the next one
				this.scheduleReconnection();
			}
		}, delay);
	}
}

Integration in app/lib/services/connect.ts

Modify the socket closure logic to delegate reconnection scheduling to the ReconnectionManager instead of relying on the SDK's built-in hardcoded loop:

// Integration Blueprint inside app/lib/services/connect.ts

import { ReconnectionManager } from './ReconnectionManager';
import { store } from '../store/auxStore';
import { connectRequest } from '../../actions/connect';

let reconnectionManager: ReconnectionManager | null = null;

const performReconnection = async () => {
	const activeServer = store.getState().server.server;
	if (!activeServer) return;
	
	store.dispatch(connectRequest());
	await sdk.current.connect();
};

// Initialize manager
reconnectionManager = new ReconnectionManager(performReconnection);

// Inside the connect() sequence:
connectedListener = sdk.current.onStreamData('connected', () => {
	reconnectionManager?.reset(); // Successful connect resets the backoff state
	// ... normal login logic ...
});

closeListener = sdk.current.onStreamData('close', () => {
	unsubscribeRooms();
	store.dispatch(disconnectAction());
	
	// Trigger the resilient backoff scheduling
	reconnectionManager?.scheduleReconnection();
});

---

4. Implementation Steps

1. Step 1: Add the ReconnectionManager file to the services directory.
2. Step 2: Modify initializeSdk in app/lib/services/sdk.ts to disable native auto-reopening or override it (setting reopen: false or extremely large values).
3. Step 3: Integrate ReconnectionManager into the close stream handler in app/lib/services/connect.ts.
4. Step 4: Test connection drop recovery:
   • Mock server downtime and observe that client reconnection attempts disperse along the backoff delay curve.
   • Verify that when the server comes back online, all clients reconnect without overloading the server.

[coderabbitai]

🔗 Related PRs

#7345 - fix: Drain lost hangups on WebSocket reconnect [merged]
#7362 - fix: re-subscribe rooms stream after forced socket reopen [merged]
#7380 - fix: keep live socket on deeplink login (prevent orphan WebSocket clobber) [merged]
#7426 - fix: re-subscribe room streams on DDP reconnect [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!