Steps to reproduce:
These are the steps I took on my existing database; I do not yet have a minimal reproducible dataset.
- Use an existing Parquet database with a large backlog of Gen1 files and working compaction into higher generations.
- Trigger cleanup using
POST /api/v3/configure/gen1_cleanup with body {}, following the recommendation in #27301.
- Wait for each background cleanup to finish before starting another. I completed seven sequential runs.
Expected behaviour:
Repeated cleanup runs should progress through the backlog and remove eligible obsolete files. Files that are still required should be retained without preventing later snapshot manifests from being processed.
Actual behaviour:
Cleanup deleted 750,930 Gen1 files in total, but the final run deleted nothing. 5,045,560 files remained.
A read-only investigation found a sharp boundary in the remaining snapshot manifests—the internal lists of Gen1 files:
- The 500 oldest remaining manifests each still contain at least one file listed in the current
leftover_gen1_files metadata: 5,771 files altogether.
- Manifest 500 has only 1 of 458 files remaining. The next manifest still has all 458 files, although none of them appears in the current leftover lists.
I suspect the remaining references keep the first 500 manifests in place, and each cleanup run selects those same manifests again instead of reaching later ones. This is a hypothesis, not a confirmed finding from the implementation. I am not assuming that every remaining file is safe to delete.
The 3.11.4 release notes describe a similar blockage caused by empty manifests. My instance already runs 3.11.5, but these manifests are not empty.
Is this a known issue, and is there a supported fix or procedure to let cleanup progress while preserving still-required files?
Environment info:
- InfluxDB 3 Enterprise 3.11.5, Home license, single node.
- Official prebuilt release; no local build or custom
RUSTFLAGS.
- Debian GNU/Linux 13 (trixie), running in a Proxmox LXC container.
uname -srm: Linux 7.0.14-17-pve x86_64.- Local XFS storage; Parquet engine, no storage-engine migration.
- I have not tested a build from
main.
Config:
--object-store is set to file.- I did not change compaction or cleanup settings for these runs.
Logs:
Relevant counter from the final, zero-deletion run:
total_files_already_deleted=587199
This exactly matches the 587,199 already-missing file references in the first 500 remaining manifests, supporting the suspicion that cleanup processed that same group again.
Steps to reproduce:
These are the steps I took on my existing database; I do not yet have a minimal reproducible dataset.
POST /api/v3/configure/gen1_cleanupwith body{}, following the recommendation in #27301.Expected behaviour:
Repeated cleanup runs should progress through the backlog and remove eligible obsolete files. Files that are still required should be retained without preventing later snapshot manifests from being processed.
Actual behaviour:
Cleanup deleted 750,930 Gen1 files in total, but the final run deleted nothing. 5,045,560 files remained.
A read-only investigation found a sharp boundary in the remaining snapshot manifests—the internal lists of Gen1 files:
leftover_gen1_filesmetadata: 5,771 files altogether.I suspect the remaining references keep the first 500 manifests in place, and each cleanup run selects those same manifests again instead of reaching later ones. This is a hypothesis, not a confirmed finding from the implementation. I am not assuming that every remaining file is safe to delete.
The 3.11.4 release notes describe a similar blockage caused by empty manifests. My instance already runs 3.11.5, but these manifests are not empty.
Is this a known issue, and is there a supported fix or procedure to let cleanup progress while preserving still-required files?
Environment info:
RUSTFLAGS.uname -srm:Linux 7.0.14-17-pve x86_64.main.Config:
--object-storeis set tofile.Logs:
Relevant counter from the final, zero-deletion run:
This exactly matches the 587,199 already-missing file references in the first 500 remaining manifests, supporting the suspicion that cleanup processed that same group again.