proxy module monitoring indicators are improved

[socutes]

Current proxy module monitoring indicators are not perfect, need to improve the monitoring indicators for connection, time, thread several dimensions.add metrics for both the node and Topic dimensions

It is recommended to add the following indicators:

1. 【Node/Topic】Number of messages produced/consumed
2. 【Node/Topic】Production/consumption message size /
3. 【Node/Topic】Time spent on each interface request
4. 【Node/Topic】Statistics on error codes of each interface
5. 【Node】The number of calls to each interface
6. 【Node】TCP(GRPC/Remoting) connection count statistics
7. 【Node】Thread pool monitoring (total capacity, used capacity, free capacity of each thread pool) monitoring
8. 【Node】Process status: indicates that the current process is started or stopped

The following table describes the labels of the metrics that are related to the Message Queue for Apache RocketMQ Proxy.
cluster: RocketMQ cluster name.
node_type: the type of service node, whitch includes the following:proxy,broker,nameserver.
node_id:the ID of the service node.
topic: the topic of RocketMQ.
thread_type: The use of thread pools. eg: sendMessage、pullMessage
protocol_type: Type of protocol. eg: grpc, remoting
proxy_mode: cluster mode. eg: local, cluster

Indicators of node dimension:

Type	Name	Unit	Description	Label
counter	rocketmq_messages_in_total	count	The number of messages that are produced.	cluster,node_type,node_id,topic, protocol_type,proxy_mode
counter	rocketmq_messages_out_total	count	The number of messages that are consumed.	cluster,node_type,node_id,topic, protocol_type,proxy_mode
counter	rocketmq_throughput_in_total	byte	The write throughput that are produced.	cluster,node_type,node_id,topic, protocol_type,proxy_mode
counter	rocketmq_throughput_out_total	byte	The read throughput that are produced.	cluster,node_type,node_id,topic, protocol_type,proxy_mode
histogram	rocketmq_message_size	byte	The distribution of message sizes. This metric is counted only when messages are sent. The following shows the distribution ranges:le_1_kb: ≤ 1 KBle_4_kb: ≤ 4 KBle_512_kb: ≤ 512 KBle_1_mb: ≤ 1 MBle_2_mb: ≤ 2 MBle_4_mb: ≤ 4 MBle_overflow: > 4 MB	cluster,node_type,node_id,topic, protocol_type,proxy_mode
histogram	rocketmq_rpc_latency	millisecond	The rpc call latency.ranges:le_1_msle_5_msle_10_msle_100_msle_10000_msle_60000_msle_overflow	cluster,node_typ,node_id, request_code, response_code,proxy_mode
counter	rocketmq_producer_connections	count	Number of connections for the producer	cluster,node_typ,node_id,proxy_mode
counter	rocketmq_consumer_connections	count	The number of consumer connections	cluster,node_typ,node_id,proxy_mode
guage	rocketmq_processor_watermark	count	High watermark information for the thread	cluster,node_typ,node_id,thread_type,proxy_mode
gauge	rocketmq_proxy_up	value	Process running status	cluster,node_typ,node_id,proxy_mode

[socutes]

@xdkxlk @drpmma Take a look

[drpmma]

Could you please write a specification about metric as listed in the document, and based on my comprehension, the majority of cases can be described with the existing metric with different node id.

For example, Number of messages produced/consumed corresponds to rocketmq_messages_in_total/rocketmq_messages_out_total.

[ShadowySpirits]

@socutes For your reference, here's some previous work in #5350. I'd suggest you briefly outline your metrics specification and implementation approach. This will help the community understand your work better.

[socutes]

@ShadowySpirits @drpmma Take a look

[ShadowySpirits]

I'd like to give some suggestions for you.

1. keep the metrics name and label as same as that of the broker
   • rocketmq_response_code_total, rocketmq_request_total -> rocketmq_rpc_latency
   • rocketmq_connect_total -> rocketmq_producer_connections, rocketmq_consumer_connections
   • rocketmq_thread_total, rocketmq_thread_free_total -> rocketmq_processor_watermark
   • roceketmq_runtime_status -> rocketmq_proxy_up
2. remove metrics that should not be metrics
   • rocketmq_runtime_starttime
   • rocketmq_runtime_mode
   • rocketmq_runtime_config
3. Combine indicators for node dimension and topic dimension

[socutes]

@ShadowySpirits Thank you very much for your advice

rocketmq_response_code_total： The number of times each return code is counted
rocketmq_request_total：Indicates the number of requests for each interface
These two metrics are different from rocketmq_rpc_latency

I will follow your advice for the rest of 1 and 2.

I originally thought that these indicators in the second point might be used in troubleshooting problems. We use these two indicators in our internal troubleshooting.

I don't quite understand the third suggestion. Could you tell me more about it.

[ShadowySpirits]

The histogram metrics rocketmq_rpc_latency will be split into three separate metrics internally:

• rocketmq_rpc_latency_count
• rocketmq_rpc_latency_sum
• rocketmq_rpc_latency_bucket

rocketmq_rpc_latency_count and rocketmq_request_total have the same semantics. And rocketmq_response_code_total could be instead by label response_code in rocketmq_rpc_latency.

The rocketmq_runtime_mode and rocketmq_runtime_config configurations should not be utilized as metrics. Instead, they should be accessed through an API provided for troubleshooting purposes.

[socutes]

I understand. It has been adjusted. Thank you very much. I will develop based on the tweaked content.What do you think?

[ShadowySpirits]

It's good for me. Looking forward to your pr.