feat(op-reth): support multiple sequencer HTTP endpoints with automatic failover

[brendontan03]

Summary

• Support multiple sequencer RPC endpoints via comma-separated URLs in --rollup.sequencer-http for automatic failover
• Add --rollup.sequencer-dial-timeout and --rollup.sequencer-request-timeout CLI flags for configurable timeouts
• Implement failover logic that retries on transport-level errors (network failures, timeouts) but returns immediately on application-level errors (e.g., nonce too low)
• Use lock-free design with AtomicUsize preferred index and pre-built alloy clients sharing a single reqwest connection pool
• Add observability metrics: total calls, successes, failures, failovers, and preferred endpoint gauge

Mirrors the equivalent feature in op-geth (op-geth PR #764).

Changes

File	Change
op-reth/crates/rpc/src/sequencer.rs	Core refactor: multi-endpoint support, SequencerClientConfig, failover-aware request(), should_failover() classification
op-reth/crates/rpc/src/metrics.rs	Add counters (total_calls, successes, failures, failovers) and gauge (preferred_endpoint)
op-reth/crates/rpc/src/error.rs	Add AllEndpointsFailed variant to SequencerClientError
op-reth/crates/node/src/args.rs	Add --rollup.sequencer-dial-timeout (default 5s) and --rollup.sequencer-request-timeout (default 10s) CLI flags
op-reth/crates/node/src/node.rs	Thread timeout config through OpAddOnsBuilder, update both SequencerClient construction sites to use new_with_config
op-reth/crates/rpc/src/eth/mod.rs	Thread timeout config through OpEthApiBuilder, use SequencerClientConfig in build_eth_api()
op-reth/crates/rpc/src/lib.rs	Export SequencerClientConfig

Design

Failover logic: On each RPC call, try the preferred endpoint first, then cycle through remaining endpoints. Transport errors (network failures, null responses, deserialization errors) trigger failover. Application errors (ErrorResp from sequencer) return immediately.

Concurrency: Lock-free by design. All alloy Client instances are pre-built at startup and share a single reqwest::Client (connection pool). Only preferred_index: AtomicUsize is mutated — no mutex needed, unlike the Go implementation which requires sync.Mutex due to stateful rpc.Client connections.

Backward compatible: Single URL still works exactly as before. WebSocket is supported for single endpoints only.

Usage

# Single endpoint (backward compatible)
--rollup.sequencer-http http://seq:8545

# Multiple endpoints with failover
--rollup.sequencer-http "http://seq1:8545,http://seq2:8545,http://seq3:8545"

# With custom timeouts
--rollup.sequencer-http "http://seq1:8545,http://seq2:8545" \
--rollup.sequencer-dial-timeout 3s \
--rollup.sequencer-request-timeout 5s

Test plan

• Unit tests for comma-separated URL parsing (single, multiple, whitespace, trailing commas)
• Unit tests for empty/invalid URL rejection
• Unit tests for mixed HTTP+WS and multiple WS rejection
• Unit tests for should_failover() across all RpcError variants
• Unit tests for SequencerClientConfig with timeouts
• Unit tests for CLI flag parsing and defaults
• Manual testing: forwarding transactions through multi-endpoint setup
• cargo check -p reth-optimism-rpc -p reth-optimism-node passes
• cargo test -p reth-optimism-rpc -p reth-optimism-node passes (19 sequencer tests, 9 args tests)

🤖 Generated with Claude Code