[Core] support limit pushdown with pk table

[wwj6591812]

Purpose

一、Background
Issue: #6847
Append table limit pushdown : #6848
This pr is to support limit pushdown with pk table.

二、Code Logic
We try best to filtering manifest entries before data reading.
Limit pushdown is only enabled for DEDUPLICATE/FIRST_ROW merge engines without deletion vectors, as accurate counting requires no merge operations or deleted rows.
We groups files by (partition, bucket) pairs and processes buckets sequentially. For each bucket, the algorithm checks if safe pushdown is possible: files must have no overlapping (same LSM level, excluding level 0) and no delete rows.
1、If safe, it accumulates row counts from file metadata until reaching the limit, then stops processing remaining buckets. 2、If unsafe (overlapping files or delete rows exist), all files in that bucket are included.

Linked issue: close #xxx

Tests

API and Format

Documentation

[JingsongLi]

You need to do performance test again.

[wwj6591812]

You are right, I will do it.

[wwj6591812]

I've conducted a performance comparison for the Limit PushDown feature. The results prove that it helps to improve the speed of OLAP queries on Paimon using a Flink Session Cluster.

一、Conclusion
1、Limit PushDown provides clear benefits for both PK and Append-Only tables.
2、Due to its complex implementation for PK tables, the feature is controlled by a configuration option.

二、Background
#6847

三、Append Only Table
1、Table Info And SQL

	No Limit Pushdown	With Limit Pushdown
Paimon Table Location	dfs://na61dfsalake1--cn-zhangjiakou/PanguVolume1/alake/omega_sample/s_holo_mainse_rank_xfc_all_features_swift_parsed
SQL	/*+ config(cluster=hongli-duibi-0528, job_name=hl_006) / select * from alake.omega_sample.s_holo_mainse_rank_xfc_all_features_swift_parsed /+ OPTIONS('scan.dedicated-split-generation' = 'true', 'limit-pushdown.with.pk.table.enabled' = 'false') */ limit 10;	/*+ config(cluster=hongli-duibi-0528, job_name=hl_007) / select * from alake.omega_sample.s_holo_mainse_rank_xfc_all_features_swift_parsed /+ OPTIONS('scan.dedicated-split-generation' = 'true', 'limit-pushdown.with.pk.table.enabled' = 'true') */ limit 10;

2、The First Test

3、The Second Test

四、PK Table
1、Table Info And SQL

	No Limit Pushdown	With Limit Pushdown
Paimon Table Location	dfs://ea119dfsalake1--cn-shanghai/PanguVolume2/alake/datalake_dt_rtcdm/s_atplog_base_hour_fi
SQL	/*+ config(cluster=hongli-duibi-0528, job_name=hl_008) / select * from alake.datalake_dt_rtcdm.s_atplog_base_hour_fi /+ OPTIONS('scan.dedicated-split-generation' = 'true', 'limit-pushdown.with.pk.table.enabled' = 'false') */ limit 10;	/*+ config(cluster=hongli-duibi-0528, job_name=hl_009) / select * from alake.datalake_dt_rtcdm.s_atplog_base_hour_fi /+ OPTIONS('scan.dedicated-split-generation' = 'true', 'limit-pushdown.with.pk.table.enabled' = 'true') */ limit 10;

2、The First Test

3、The Second Test

[wwj6591812]

@JingsongLi
Hi，Please CC, Thx.