Shared subscription: backlog drains and partition ack rate spikes on one pod restart, but consumer ack/QPS stay flat (Pulsar 3.0.14)

[raushantutu]

Environment

Pulsar / client: 3.0.14
Subscription type: Shared
DeadLetterPolicy enabled with maxRedeliverCount, failures via negativeAcknowledge
Multiple consumers (pods) on the same subscription; partitioned topic

What we observed
On some partitions, backlog (lag), unacked messages, and redelivery rate climb and stay high for a long time.
During that sticky period:

• Partition produce rate ≈ partition ack rate (no sustained produce−ack gap)
• Application consumption / request rate stays roughly constant (tracks normal traffic, not the redelivery storm)
• Consumer-side health looks fine: executor queue, threads, latency, app ack/nack rates look normal (nack stays low)

Then we restart one consumer pod (out of many). After that:

• Backlog drains sharply (near zero)
• Unacked drops only partially (e.g. ~80k → ~50k, not to ~0)
• Redelivery rate collapses
• Application consumption rate does not increase during the drain
• Partition broker msg_ack_rate rises while lag is draining
• Application ack rate does not rise in the same way (fleet and per-pod)
• So broker partition ack/backlog move at restart time, but our consumer metrics do not show a matching ack/consume spike.

Questions

• Who / what is draining the backlog in this scenario, if consumer QPS and app ack rate do not increase?
• Who is generating the extra acks on that partition (broker msg_ack_rate), if they are not reflected in our consumer ack metrics?
• Is this expected with Shared + ackTimeout / nack / consumer disconnect (e.g. cursor vs unacked accounting), or does it look like a known broker/client accounting issue on consumer remove/reconnect?

Happy to share metric graphs (backlog, unacked, redeliver, rate_out, broker ack vs app ack, consumer count) for the window.

Metrics:
1.) rate_in vs rate_out vs redeliver_rate vs ack rate on topic

mostly lag coming from partition-5 at this time.

2.) Ack rate by partition

During lag drain , we can see spike for only partition-5

3.) Unacked mssg for partition-5

4.) Backlog on partition-5

5.) pulsar_subscription_blocked_on_unacked_messages{subscription="$subscription", topic=~"$topic_regex"}

6.) Consumer fleet consume rate

No such increase in this graph during the period of lag drain

7.) consumer side [ ack vs request vs nack ]

No increase in ack rate from consumers also at that time

[lhotari]

Pulsar 3.0.x is no longer supported and doesn't get fixes. Please upgrade to a supported version (currently 4.0.x and 5.0.x).