[ruanwenjun]

Was this PR generated or assisted by AI?

YES. AI assisted with implementation, tests, documentation, and the PR description; the synchronization topology and behavior were agreed with the contributor.

Purpose of the pull request

Closes #18768.

Master, Worker and Alert Server currently rewrite full heartbeat data in registry nodes, coupling payload size and synchronization frequency to registry writes. Store stable process identity in ServerIdentity and fetch runtime heartbeats over RPC. Master polls peer Masters and Workers; API queries Master/Worker/Alert heartbeats directly on each request.

Brief change log

• Keep membership subscriptions and store registration identities separately from valid heartbeat metadata in each concrete cluster. Each of MasterClusters and WorkerClusters owns one timer thread and synchronously fetches its nodes in sequence, waiting master.heartbeat-fetch-interval (default 1s) after each round. Remove the additional RPC executors, in-flight sets and Worker expiration fields. Store only the local successful receipt time as heartbeatReceivedTime. ClusterManager only initializes members and wires subscriptions and listeners. Create metadata through parseFromHeartBeat only after a valid heartbeat.
• API discovers the registered nodes for each monitoring or Worker Group query and queries their typed RPCs concurrently. Each request obtains its own results without retaining a heartbeat cache. Failed node queries retain that registered node in the response with null heartbeat/time and metadataStale=true, while healthy nodes still return data. Validate each response against the discovered process identity.
• Generate Master/Worker/Alert snapshots on demand using a thread-safe one-second system-metrics cache. Alert registers stable identity at startup and restores registration on reconnect, without periodic heartbeat writes. Remove alert.max-heartbeat-interval and the obsolete configuration examples.
• Preserve Master/Worker membership ADD/REMOVE events for discovery and failover. Reject responses from removed or replaced registrations. Missing, failed or foreign-process heartbeats clear runtime metadata while preserving registration; Workers without valid metadata receive no new tasks. Master slots reflect only valid metadata. Failover events and liveness checks use ServerIdentity, including nodes that never returned a heartbeat. Runtime metadata removal does not trigger failover. Apply the Worker weight from its first valid heartbeat to the fixed-weight load balancer.
• Remove BaseHeartBeatTask and its Master/Worker/Alert subclasses. Use SelfFailoverMonitor to check whether the current Master/Worker process has been failed over and stop it independently of heartbeat RPC traffic.
• Show missing heartbeat data explicitly in monitoring. Use bounded address-based RPC connection locks, with authentication waits outside them and old-channel cleanup independent of reconnect creation.
• Keep configuration references focused on configuration entries, and document incompatible changes in the upgrade guide. Master, Worker, Alert and API require a coordinated upgrade. No new module dependencies or additions to dolphinscheduler-service.

Verify this pull request

• 148 focused tests pass across registration, concrete Master cluster polling, independent single-thread Master/Worker polling and Spring wiring with circular references disabled, instance validation, late responses, initial discovery/removal races, missing/failed heartbeats, registration-only failover checks, Master slots, Worker Groups, Alert registration, and RPC authentication/concurrency. API tests verify per-request discovery and fresh RPC queries, concurrent fan-out, partial failures, foreign process rejection, and real Netty Master/Worker/Alert queries. Master, Worker, API, Alert and their reactor dependencies compile successfully.
• Full-repository ./mvnw -o spotless:apply and the Spotless checks in the focused test reactor pass. git diff --check passes.
• Both documentation image checks (python3 img_utils.py -v dev-syntax and python3 img_utils.py -v check) pass. All three Issue Mermaid diagrams were rendered and inspected.
• UI vue-tsc --noEmit and ESLint for the changed monitoring files pass.

Full-suite tests, a complete cluster upgrade/failover exercise, and production performance have not been validated. API query latency now includes node RPCs. CI status is tracked separately.

Pull Request Notice

Pull Request Notice