Transaction.table_metadata deep-copies the whole table metadata on every read, making commits quadratic in snapshot count #4294

Apache Iceberg version

0.12.0 (latest release). main at e3564a5 has the same code.

Please describe the bug

Transaction.table_metadata recomputes on every read:

@property
def table_metadata(self) -> TableMetadata:
    return update_table_metadata(self._table.metadata, self._updates)

update_table_metadata ends in new_metadata.model_copy(deep=True), so every read deep-copies the whole TableMetadata, snapshot list included. One commit reads the property tens of times, so its cost grows linearly with the snapshots the table keeps, whatever rows it writes. Each commit also adds a snapshot, so N commits to one table cost O(N²) in total.

#2674 and #3301 hoisted repeated reads out of loops. The property itself is still uncached.

What we measured

One live table in a development environment: 126 partitions, 1.09M rows, snapshots expired after 24 h. Committed to about 225 times an hour, it reached thousands of snapshots.

cProfile puts most of the commit in that model_copy(deep=True). We did not time a whole commit at the 2,770 or 5,410 depths.

We work around it by batching commits and expiring snapshots. Neither removes the repeated copies.

Reproduction

Each timed commit appends one row. reads counts calls to update_table_metadata from pyiceberg.table.

uv run --isolated --no-project --python 3.12 --with 'pyiceberg[sql-sqlite,pyarrow]==0.12.0' python repro.py

import tempfile
import time
from unittest import mock

import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.table import update_table_metadata
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("repro", uri=f"sqlite:///{warehouse}/catalog.db", warehouse=f"file://{warehouse}")
catalog.create_namespace("ns")
table = catalog.create_table(
    "ns.t",
    schema=Schema(
        NestedField(1, "key", StringType(), required=False),
        NestedField(2, "value", LongType(), required=False),
    ),
    partition_spec=PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=IdentityTransform(), name="key")),
)


def rows(key: str, n: int) -> pa.Table:
    return pa.table({"key": pa.array([key] * n, pa.string()), "value": pa.array(range(n), pa.int64())})


for target in (0, 100, 200, 400, 800):
    while len(table.metadata.snapshots) < target:
        table.append(rows("other", 1))
    size = len(table.metadata.model_dump_json())
    snapshots = len(table.metadata.snapshots)
    with mock.patch("pyiceberg.table.update_table_metadata", wraps=update_table_metadata) as reads:
        start = time.perf_counter()
        table.append(rows("timed", 1))
        elapsed = time.perf_counter() - start
    print(f"snapshots={snapshots:4d}  metadata={size / 1e6:.2f} MB  append={elapsed * 1000:7.1f} ms  reads={reads.call_count}", flush=True)

One run, one timed append per depth, Python 3.12.11, pyarrow 26.0.1, on a laptop:

snapshots=   0  metadata=0.00 MB  append=   80.0 ms  reads=19
snapshots= 100  metadata=0.09 MB  append=   32.9 ms  reads=22
snapshots= 200  metadata=0.15 MB  append=   55.8 ms  reads=22
snapshots= 400  metadata=0.28 MB  append=  158.8 ms  reads=22
snapshots= 800  metadata=0.55 MB  append=  315.2 ms  reads=22

The fix exists

Two PRs fix this, and the stale bot closed both unmerged:

• perf: Cache Transaction.table_metadata between staged updates #3302, "Cache Transaction.table_metadata between staged updates". Per perf: cache Transaction.table_metadata between reads #3784, a maintainer approved it in April.
• perf: cache Transaction.table_metadata between reads #3784, "perf: cache Transaction.table_metadata between reads", which carries perf: Cache Transaction.table_metadata between staged updates #3302 forward. It reports 22 → 2 update_table_metadata calls per append, 4.8× at 500 snapshots, and a production writer back from 10.3 s to 2.9 s.

Could a maintainer reopen #3784, or say what path would land it?

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time