Repository navigation
ext4: use iomap for regular file's buffered I/O path - #2811
Open
vfsci-bot[bot] wants to merge 31 commits into
Open
vfsci-bot[bot] wants to merge 31 commits into
vfsci-bot[bot] wants to merge 31 commits into
Conversation
The logic for updating the file size in ext4_setattr() is currently somewhat messy. By directly entering the error-handling path after failing to add an orphan inode, the unnecessary recovery process involving old_disksize and the file size can be avoided. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Jan Kara <jack@suse.cz> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Refactor ext4_setattr() by introducing two helper functions, ext4_truncate_up() and ext4_truncate_down(), to handle size changes. The current ATTR_SIZE processing consolidates checks for both shrinking and non-shrinking cases, leading to cluttered code. Separating the truncation paths improves readability. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com> Reviewed-by: Jan Kara <jack@suse.cz>
Set EXT4_MAP_NEW in ext4_da_map_blocks() to properly indicate that a new delayed allocation block has been inserted, allowing callers to distinguish newly created delayed extents from existing ones. Currently, the buffer_head caller, ext4_da_get_block_prep(), does not consume this flag. It intercepts EXT4_MAP_DELAYED and returns early, and unconditionally calls set_buffer_new(bh) so EXT4_MAP_NEW is not used. The flag is prepared for the iomap buffered I/O path added later. Reported-by: Ojaswin Mujoo <ojaswin@linux.ibm.com> Link: https://lore.kernel.org/linux-ext4/cc05c17d-163e-4251-b2c9-aa3a6f9555d7@huaweicloud.com/ Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
After acquiring i_data_sem in write mode, recheck that the mapping found via the extent status tree or disk query has not changed. A racing truncate may have trimmed the extent between the earlier lookup and the write lock acquisition, since writeback does not hold i_rwsem or the folio locks covering the full extent. This could cause ext4_map_create_blocks() to allocate blocks beyond the truncated range, potentially leading to quota leaks in the upcomming iomap buffered writeback path, because the iomap writeback infrastructure caches extents beyond the folio range. Therefore, if we find a valid extent and the sequence number has changed, retry the entire lookup to obtain the correct trimmed mapping. Note that a retry only happens when i_es_seq has actually changed, which requires another thread to hold i_data_sem in write mode, so each retry implies real lock contention, and the writer yields on the rwsem slowpath rather than spinning. Moreover, the read-write lock also tries its best to guarantee fairness, so even though no retry limit is added here, livelock should not occur in theory. Also, the retry here mirrors the existing write_ops->iomap_valid() + IOMAP_F_STALE mechanism in iomap, which also rechecks the sequence number and retries without any upper bound, and has not shown livelock problems in practice. In addition, if ext4_map_query_blocks() fails, return the error immediately. Otherwise the following sequence number comparison would use a map->m_seq that may not have been updated, and silently continuing on an error is not advisable. Suggested-by: Jan Kara <jack@suse.cz> Link: https://lore.kernel.org/linux-ext4/b0781809-4759-4e12-be17-71555b764f48@gmail.com/ Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Save orig_mlen after clamping map->m_len to INT_MAX. Otherwise, the unclamped value may be passed below, bypassing its overflow protection. Fixes: 5bb12b1 ("ext4: Add support for EXT4_GET_BLOCKS_QUERY_LEAF_BLOCKS") Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Make ext4_map_blocks() start its own transaction handle when the caller does not provide one. The handle is started after the lookup path confirms that allocation is actually needed, and is stopped at the unified out exit path. This avoids unnecessarily starting a handle for pure mapping queries. This prepares for the buffered iomap writeback conversion, which improves performance for fragile overwrite cases. Suggested-by: Jan Kara <jack@suse.cz> Link: https://lore.kernel.org/linux-ext4/i536qqwj5eyulec3r2ki2ycnelqdd4bkpat2drn7t72t6p622k@ktueynysgo3j/ Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
…n extents When ext4_map_blocks() finds an unwritten extent in the extent cache and the caller is willing to accept unwritten extents without conversion, there is no need to start a journal transaction since no metadata update is required. This avoids unnecessary transaction overhead in the upcoming iomap writeback path when overwriting already-allocated unwritten extents. One thing to be careful about, as the comment in ext4_map_blocks() states, if the flags contain EXT4_GET_BLOCKS_CREATE, the function will mark @Map as mapped. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
When ext4_map_blocks() is called from the data submission path and I/O end extent conversion path (EXT4_GET_BLOCKS_IO_SUBMIT), it should not allocate blocks if the lookup returns a hole. The writeback path can legitimately encounter dirty ranges that map to holes. For example, when a folio straddles i_size and the tail beyond i_size is dirtied via a mmap write. Allocating blocks for such ranges is wrong because there is no data to write back, the dirty bits should simply be discarded without submitting I/O. This prepares for the buffered iomap writeback conversion, mirrors the existing buffer_head writeback path, where mpage_add_bh_to_extent() skip unmapped buffers and ext4_bio_write_folio() clears their dirty bits. Since ext4_map_blocks() can now return zero for a hole, mpage_map_one_extent() must handle it as well. However, because those holes are already filtered out by mpage_add_bh_to_extent(), it only maps buffers that are delayed or unwritten, so a hole can never occur in normal operation. Add a WARN_ON_ONCE() and bail out with -EINVAL rather than proceeding with an invalid physical block number of zero. In the ioend extent conversion path, holes are also not expected because we should wait for folio writeback before punching hole. If one is encountered, it likely indicates a failure in the concurrency protection, so ext4_map_blocks() returns zero, and we keep warning and bail out with -EINVAL to surface the failure rather than continuing conversion on torn data. Atomic writes in ext4_convert_unwritten_extents_atomic() already bail out similarly. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Introduce initial support for iomap in the buffered I/O path for regular
files on ext4.
- Add a new inode state flag EXT4_STATE_BUFFERED_IOMAP to indicate the
inode uses iomap instead of buffer_head for buffered I/O
- Add helper ext4_inode_buffered_iomap() to check the flag
- Add new address space operations ext4_iomap_aops with callbacks that
will use generic iomap implementations
- Add ext4_iomap_aops to ext4_set_aops() when the flag is set
The following callbacks(read_folio(), readahead(), writepages()) are
provided as placeholders and will be implemented in later patches.
Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Implement the iomap read path for ext4 by introducing a new ext4_iomap_buffered_read_ops instance. This provides the read_folio() and readahead() callbacks for ext4_iomap_aops. The implementation introduces: - ext4_iomap_map_blocks(): Helper function to query extent mappings for a given read range using ext4_map_blocks() and convert the mapping information to iomap type - ext4_iomap_buffered_read_begin(): The iomap_begin callbacks that maps blocks, validates filesystem state, and populates the iomap. It returns -ERANGE for inline data which is not yet supported. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Jan Kara <jack@suse.cz> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
The iomap buffered write path does not hold the folio lock between mapping the inode extent and copying data. Therefore, it can race with writeback that modifies the extent type (e.g., from unwritten to written). This can lead to data corruption on partial writes, as iomap_block_needs_zeroing() may return a false positive based on a stale extent. The iomap infrastructure uses the sequence counter stored in the inode to detect such stale mappings. Commit 07c440e ("ext4: pass out extent seq counter when mapping blocks") added the m_seq field to ext4_map_blocks to pass out extent sequence numbers, but it missed two callsites within ext4_da_map_blocks(). These callsites are on the delayed allocation path, which is needed in the iomap buffered write path. Pass out the sequence counter to ensure stale mappings can be detected. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Jan Kara <jack@suse.cz> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
The data=ordered mode introduces two fundamental conflicts with the
iomap buffered write path, leading to potential deadlocks.
1) Lock ordering conflict
In the iomap writeback path, each folio is processed sequentially:
the folio lock is acquired first, followed by starting a transaction
to create block mappings.
In data=ordered mode, writeback triggered by the journal commit
process may attempt to acquire a folio lock that is already held by
iomap background writeback process. Meanwhile, iomap, under that same
folio lock, may start a new transaction to map other blocks on this
folio and wait for the currently committing transaction to finish,
resulting in a deadlock.
Trans N commit background writeback(via iomap)
journal_submit_data_buffers()
ext4_journal_submit_inode_data_buffers()
iomap_writepages()
iomap_writepages()
folio_lock()
folio_lock() -- wait
iomap_writeback_folio()
iomap_writeback_range()
ext4_journal_start()
start new transaction
-- wait for trans N commit,
DEADLOCK
ext4_map_blocks()
Currently, in the buffer_head writeback path, this is handled by
starting the transaction before taking any folio locks for writeback.
2) Partial folio submission not supported
When block size < folio size, a folio may contain both mapped and
unmapped blocks. In data=ordered mode, a deadlock can occur if the
journal waits (pure JI_WAIT_DATA) for such a folio to be written back
while background writeback has already started on it (with the
writeback flag set). The problem is that mapping the remaining
delalloc blocks can deadlock because the writeback flag is not
cleared until the entire folio is processed and committed.
T0: Assume we have a folio contains four blocks, from front to back,
they are A, B, C, D. The block B and C are holes, and the last
block D is written in delalloc mode (the block is not allocated
yet).
T1: The background writeback process starts to write back data, set
writeback flag on the folio, allocates block D, and adds it to
transaction N's order list of jbd2 in pure JI_WAIT_DATA mode.
T2: This folio completes the writeback and clears the writeback flag.
T3: Before transaction N commit, we buffered write block A to C.
T4: Transaction N commit and folio writeback are running
concurrently.
Trans N commit background writeback(via iomap)
iomap_writeback_folio()
folio_start_writeback()
-- set writeback flag
jbd2_journal_finish_inode_data_buffers()
__filemap_fdatawait_range()
-- wait writeback flag to clear
iomap_writeback_range()
ext4_journal_start()
start new transaction
-- wait for trans N commit,
DEADLOCK
ext4_map_block() (B, C)
Currently, in the buffer_head writeback path, this is handled by:
1. Partial folio submission — already-allocated buffers can be
submitted first. The writeback flag is cleared after I/O
completes, preventing block allocation while the writeback flag is
set.
2. Allocation order — the transaction is started first, then blocks
are allocated, the writeback flag is set, and finally the
allocated buffers submission begins.
To support data=ordered mode, the iomap core would need two invasive
changes:
- Acquire the transaction handle before locking any folio for
writeback.
- Support partial folio submission.
Both changes are complicated and risk performance regressions.
Therefore, we must avoid using data=ordered mode when converting to the
iomap path.
Currently, data=ordered mode is used in three scenarios:
- Append write
- Post-EOF partial block truncate-up followed by append write
- Online defragmentation
We can address the first two without data=ordered mode:
- For append write: always allocate unwritten blocks (i.e. always
enable dioread_nolock), preserving the behavior of current
extent-type inodes.
- For post-EOF truncate-up + append write: postpone updating i_disksize
until after the zeroed partial block has been written back.
Online defragmentation does not yet support iomap; this can be resolved
separately in the future.
Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Introduce two new iomap_ops instances for ext4 buffered writes:
- ext4_iomap_buffered_da_write_ops: for delayed allocation mode, using
ext4_da_map_blocks() to map delalloc extents.
- ext4_iomap_buffered_write_ops: for non-delayed allocation mode, using
ext4_map_blocks() to directly allocate blocks.
Also add ext4_iomap_valid() for the iomap infrastructure to check extent
validity. This is useful for the iomap buffered write path because it
does not hold the folio lock between mapping the inode extent and
copying data. So it can race with writeback, which may modify or stale
the extent type.
Key changes and considerations:
- Unwritten extents for new blocks (dioread_nolock always on)
Since data=ordered mode is not used to prevent stale data exposure in
the non-delayed allocation path, new blocks are always allocated as
unwritten extents.
- Short write and write failure handling
a. Delalloc path: On short write or failure, the stale delalloc range
must be dropped and its space reservation released. Otherwise, a
clean folio may cover leftover delalloc extents, causing
inaccurate space reservation accounting.
b. Non-delalloc path: No cleanup of allocated blocks is needed on
short write since leaving the preallocated blocks behind doesn't
cause any real issue. These blocks are visible to users and are
not permanently leaked.
- Adjust the call frequency of ext4_nonda_switch() so that it is no
longer called once per folio in the buffer_head path, but instead
once per write. For code simplicity, ext4_iomap_buffered_da_write_ops
will be shared with the later page-fault path. Since
ext4_nonda_switch() -> try_to_writeback_inodes_sb() synchronously
waits for writeback, which could lead to deadlock under the same
folio lock acquired by iomap_page_mkwrite(), we can no longer call
ext4_nonda_switch() inside ext4_iomap_buffered_da_write_ops. This may
increase the probability of a writeback-time allocation failure, but
the expected real-world impact is small, so it can be implemented
this way for now.
- Lock ordering reversal
The folio lock and transaction start ordering is reversed compared to
the buffer_head buffered write path. To handle this, the journal
handle must be stopped in iomap_begin() callbacks. The lock ordering
documentation in super.c has been updated accordingly.
Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
While converting the buffered I/O path to iomap, we noticed that the writeback completion can hand over an ioend which was merged across an arbitrary number of extents because iomap_ioend_try_merge() doesn't limit the length of the merged ioends, while ext4_convert_unwritten_extents() converts all of them with the handle provided by the caller, and callers calculate the credits of that handle without knowing the exact number of extents. Converting more extents than that within one transaction can exhaust its credits. Prepare for the iomap path nevertheless by making the conversion loop check the remaining credits before each extent with ext4_journal_ensure_credits(), so that the transaction is extended, or restarted once it is full, as needed. Also estimate the credits with ext4_meta_trans_blocks(..., alloc_extents = 0), since no data blocks are allocated here, and manage an internally started transaction as a whole instead of starting and stopping one for each extent. The buffer_head writeback path is safe, as it passes a handle whose credits are reserved by ext4_meta_trans_blocks(inode, bpf, bpf, 0), which is enough to convert one extent per block of a folio, so the reservation is sufficient and the handle is never restarted there. This also helps the fallocate path, where ext4_alloc_file_blocks() sizes the handle for the conversion with ext4_chunk_trans_blocks() as well, i.e. for a single extent only, while a concurrent modification of the extent tree can make the conversion cover more extents than the handle has credits for. Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Add the iomap writeback path for ext4 buffered I/O. This introduces: - ext4_iomap_writepages(): the main writeback entry point. - ext4_writeback_ops: a new iomap_writeback_ops instance to handle block mapping and I/O submission. - A new end I/O work handler ext4_iomap_finish_ioend() is added for converting unwritten extents, updating file size, and handling DATA_ERR_ABORT after I/O completion. Core implementation details: - ->writeback_range() callback Calls ext4_iomap_map_writeback_range() to map and allocate blocks using the enhanced ext4_map_blocks(). ext4_map_blocks() now starts its own transaction internally when it needs to allocate blocks. For performance, when a block range is not yet allocated, it allocates based on the writeback length and delalloc extent length (ext4_map_blocks() will trim the allocated length), rather than allocating a single folio at a time. The folio is then added to an iomap_ioend instance. - ->writeback_submit() callback Registers ext4_iomap_end_bio() as the end bio callback. This callback schedules a worker to handle: - Unwritten extent conversion. - i_disksize update after data is written back. - Journal abort on writeback I/O failure. - The new end I/O work handler ext4_iomap_finish_ioend() This reuses i_rsv_conversion_work and i_rsv_conversion_list, which are now used in the buffer_head writeback path for memory saving, because an inode can never go through both buffer_head and iomap paths. Those are not initialized in this patch but will be when the iomap path is formally enabled later. Key changes and considerations: - Append write and unwritten extents Since data=ordered mode is not used to prevent stale data exposure during append writebacks, new blocks are always allocated as unwritten extents (i.e. always enable dioread_nolock), and i_disksize update is postponed until I/O completion. Additionally, the deadlock that the reserve handle was expected to resolve does not occur anymore. Therefore, the end I/O worker can start a normal journal handle instead of a reserve handle when converting unwritten extents. - Lock ordering The ->writeback_range() callback runs under the folio lock, requiring the journal handle to be started under that same lock. This reverses the order compared to the buffer_head writeback path. The lock ordering documentation in super.c has been updated accordingly. - Don't cache writes The iomap infrastructure sets the BIO_COMPLETE_IN_TASK flag when submitting I/O, so the ioend will be processed in task context. However, if a private defer worker is to be started, this flag must be cleared explicitly to avoid double deferral. In the future, all private defer work should be moved to the generic bio complete in task framework. Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Introduce ext4_iomap_page_mkwrite() to implement the mmap iomap path for ext4. The heavy lifting is delegated to iomap_page_mkwrite(), which only requires ext4_iomap_buffered_write_ops and ext4_iomap_buffered_da_write_ops to allocate and map blocks. Note that the lock ordering between folio lock and transaction start in this path is reversed compared to the buffer_head buffered write path. The lock ordering documentation in super.c has been updated accordingly. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com> Reviewed-by: Jan Kara <jack@suse.cz>
Introduce a new iomap_ops instance, ext4_iomap_zero_ops, along with
ext4_block_iomap_zero_range() to implement block zeroing via the iomap
infrastructure for ext4.
ext4_block_iomap_zero_range() calls iomap_zero_range() with
ext4_iomap_zero_begin() as the callback. The callback locates the
range and populates the iomap mapping. If the range is mapped,
iomap_zero_iter() in the iomap core zeros the partial block
directly. If the range is an unwritten extent within EOF, the
callback collects a dirty folio batch via iomap_fill_dirty_folios()
so that iomap_zero_iter() can zero those folios directly, bypassing
a separate slow flush operation that would otherwise be needed to
convert the unwritten extent.
Note that ext4_iomap_zero_begin() can race with concurrent writeback:
after it queries an unwritten extent, writeback may convert it to
written and complete on the folio before iomap_fill_dirty_folios() scans
the range. The empty batch then makes iomap_zero_iter() skip zeroing,
leaving stale on-disk data.
zero_range writeback
---------------------- ----------------------
ext4_block_zero_range()
iomap_zero_range()
iomap_iter()
ext4_iomap_zero_begin()
ext4_iomap_map_blocks()
-> extent is UNWRITTEN
ext4_convert_unwritten_extents_endio()
-> extent is converted to WRITTEN
-> folio is clean
iomap_fill_dirty_folios()
filemap_get_folios_dirty()
-> folio is clean, not added to batch
ext4_set_iomap() -> IOMAP_UNWRITTEN
iomap_zero_iter()
__iomap_get_folio() -> NULL (empty batch)
iomap_iter_advance_full() <-- zeroing skipped
[later read returns stale on-disk data] <-- CORRUPTION
Therefore, we retry the extent lookup when iomap_fill_dirty_folios()
adds nothing and the i_es_seq cookie captured at the first lookup has
advanced, indicating the race actually occurred.
Other important constraints:
Zeroing out under an active journal handle can cause deadlock, as the
lock/handle ordering is inconsistent with the iomap writeback path.
Therefore, ext4_block_iomap_zero_range() must not be called under an
active handle. In addition, for post-EOF zeroing, the caller cannot
rely on data=ordered mode to persist the zeroed data before
i_disksize is updated.
Subsequent patches will address this by deferring i_disksize update
to i_size until after the zeroed data has been written back.
Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Because the iomap infrastructure does not always create an ifs to
manage sub-folio state when folio size is larger than blocksize,
invalidating a partial dirty folio during punch hole may fail to
clear the dirty state of the affected range. As a result, writeback
of that folio may observe a hole. At writeback submit time,
ext4_map_blocks() already handles this case and will not allocate
blocks. However, when punch hole races with writeback, the
following scenario can cause I/O completion to encounter a hole.
punch hole writeback
---------- ---------
ext4_punch_hole()
ext4_truncate_page_cache_block_range()
iomap_invalidate_folio() [partial folio]
iomap_clear_range_dirty()
-- no ifs, sub-block dirty bits NOT cleared
ext4_iomap_writepages()
iomap_writepages()
ext4_iomap_writeback_submit()
ext4_iomap_map_writeback_range()
ext4_map_blocks(IO_SUBMIT)
-> extent exists, not a hole
submit_io() -> bio in flight
down_write(&i_data_sem)
ext4_es_remove_extent()
ext4_ext_remove_space()
-> extent removed, hole inserted
up_write(&i_data_sem)
[ transaction commits, freed blocks released to buddy ]
[ allocator reuses freed blocks for another inode ]
[bio completes]
ext4_iomap_finish_ioend()
ext4_convert_unwritten_extents()
ext4_map_blocks(IO_CONVERT_EXT)
-> returns 0 (hole found)
[bio writes to reallocated blocks ]
-> silent data corruption
If the bio completes after the freed blocks have been reallocated to
another inode, it silently overwrites them and causes data corruption.
Therefore, on the iomap path, drain writeback in ext4_punch_hole() after
partial zeroing and before starting the freeing transaction, so any bio
in flight on a partial folio completes against blocks still owned by
this inode before extent removal. Once extent removal begins, the ES
hole insert and the existing IO_SUBMIT short-circuit prevent further bio
submission.
This is a workaround. The proper fix is for the iomap infrastructure to
always maintain an ifs when folio size is larger than blocksize, so that
partial invalidate reliably clears sub-block dirty bits and no draining
is needed.
Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
Add tracepoints for iomap buffered read, write, partial block zeroing, and writeback operations to help debug the iomap buffered I/O path. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com> Reviewed-by: Jan Kara <jack@suse.cz>
Online defragmentation does not currently support inodes using the iomap buffered I/O path. The existing implementation relies on buffer_head for sub-folio block management and data=ordered mode for data consistency, both of which are incompatible with the iomap path. Signed-off-by: Zhang Yi <yi.zhang@huawei.com> Reviewed-by: Ojaswin Mujoo <ojaswin@linux.ibm.com> Reviewed-by: Jan Kara <jack@suse.cz>
Inodes using the iomap buffered I/O path do not use data=ordered mode, so the zeroed EOF block that straddles i_disksize needs explicit tracking to ensure it is written back before i_disksize is advanced. Add the EXT4_STATE_DISKSIZE_GROW_PENDING inode state bit and three helpers: ext4_iomap_clear_disksize_pending() to atomically clear the bit and wake waiters, ext4_iomap_wait_disksize_pending() to block until the bit is cleared, and ext4_iomap_get_disksize_pending_range() to compute the pending range from i_disksize. These will be used by subsequent patches to serialize i_disksize updates with the writeback of the zeroed EOF block. Suggested-by: Jan Kara <jack@suse.cz> Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
When the current writeback pass begins beyond the disksize-grow-pending zeroed EOF block, the ioend worker would otherwise have to wait for the pending EOF block to complete before it can advance i_disksize. Otherwise the old EOF block could be exposed as stale data once i_disksize advances past it. Therefore, introduce the ioend mechanism for the pending range, tag ioends that cover the pending zeroed EOF block which straddles i_disksize with EXT4_IOMAP_IOEND_DISKSIZE_GROW_IO in ext4_iomap_writeback_submit(), and clear the bit and wake up all waiters in ext4_iomap_end_bio() when such an ioend completes. Clearing EXT4_IOMAP_IOEND_DISKSIZE_GROW_IO does not depend on whether the disksize grow I/O succeeds. That is, even if the I/O fails, we still allow subsequent writes in the range to update i_disksize. This is consistent with the previous behavior, and we rely on data_err=abort to prevent metadata updates when data write failures occur. In order to avoid the ioend that passes the pending range waiting for a long time, proactively submit the pending range first in ext4_iomap_writepages() so it completes in parallel with the rest of the writeback. Note that the handling of discarding the zeroed EOF folio will be processed later, otherwise the bit will be set forever. EXT4_STATE_DISKSIZE_GROW_PENDING will be set after everthing is done. Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
When the disksize-grow-pending zeroed EOF I/O completes, i_disksize has to be advanced. Advancing it only to the end of that specific I/O would discard any later i_disksize updates from concurrent fallocate or similar operations, causing filesystem inconsistency. Scanning dirty or writeback folios beyond the current position to compute a safe advance target is expensive and racy with concurrent fallocate, so instead advance i_disksize directly to i_size. This may expose zeroed data (not stale data) after crash recovery when dirty data in the range is not yet on disk, but only for unaligned append writes, which is deemed acceptable. To support this, teach ext4_iomap_wb_update_disksize() to take an is_disksize_grow flag and advance i_disksize to i_size when set, and have ext4_iomap_finish_ioend() pass the flag based on the EXT4_IOMAP_IOEND_DISKSIZE_GROW_IO tag of the completing ioend. Note that EXT4_STATE_DISKSIZE_GROW_PENDING is not set for now, and it will be set after everthing is done. Suggested-by: Jan Kara <jack@suse.cz> Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Operations like append allocate, zero range, and truncate update i_disksize directly. If the new i_disksize exceeds the original value while the zeroed EOF block is still awaiting writeback, metadata may be persisted before the zeroed data, exposing stale data on crash. Defer i_disksize updates while EXT4_STATE_DISKSIZE_GROW_PENDING is set; the ioend worker for the pending block will advance i_disksize to i_size once the zeroed data is written back. The tradeoff is that i_disksize may lag i_size transiently, but this is observable only to callers that read i_disksize directly. Introduce __ext4_set_i_disksize() to centralize the bit check for callers already holding i_data_sem (ext4_ext_truncate and ext4_set_inode_size), and refactor ext4_update_inode_size() to take i_data_sem itself and check the bit atomically with i_size_write(), so the ioend worker observes the latest i_size under the same lock. Note that in the O_SYNC mode of ext4_do_fallocate() or ext4_zero_range(), we need to flush out pending blocks that can update i_disksize synchronously. This will be handled later, and EXT4_STATE_DISKSIZE_GROW_PENDING will also be set after everything is done. Suggested-by: Jan Kara <jack@suse.cz> Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Collapse range and insert range update i_disksize directly under i_data_sem. If the operation runs while the zeroed EOF block is still awaiting writeback, i_disksize could advance past the zeroed boundary before the zeroed data is persisted, exposing stale data on crash. Deferring i_disksize updates like fallocate and zero_range is not an option here because the shift would move written extents beyond the current i_disksize. So flush and wait for the pending zeroed EOF block before these operations advance i_disksize. Since these operations already perform writeback, the extra flush does not add significant overhead. In addition, for ext4_update_disksize_before_punch(), if the punch discards the pending block, the zeroed data will never be written back before advancing i_disksize, so it is also necessary to sync the pending EOF range there. Finally, for the SYNC variants of zero_range and fallocate, this also guarantees the i_disksize update is persisted on the synchronous return. Note that EXT4_STATE_DISKSIZE_GROW_PENDING is not set for now, and it will be set after everything is done. Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
The disksize-grow-pending state is set when a zeroed EOF block is queued for writeback and cleared by the ioend completion path once writeback finishes. However, the zeroed block may be discarded before writeback completes — through folio discard, truncate, or unlink and inode eviction. In any of these cases, leaving the bit set would block subsequent writeback indefinitely. Therefore, we must clear it on all paths that invalidate the pending block before writeback completes: - In ext4_iomap_discard_folio(), on folio discard. - In ext4_evict_inode(), when an unlinked inode is destroyed (don't bother with an inode that has nlink > 0, since it is only evicted after its dirty pages have been written out, and also don't bother with bad inodes, since they can never have the bit set). In ext4_truncate_down(), truncating past the pending zeroed EOF block also invalidates the pending disksize update, so the bit must be cleared there as well. Finally, add a WARN_ON in ext4_destroy_inode() to catch any inode destroyed with the bit still set. The check is skipped when the filesystem is in an error or emergency state, as the pending block may not have been written back in those cases. Note that EXT4_STATE_DISKSIZE_GROW_PENDING is not set for now, and it will be set after everthing is done. Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
In the iomap buffered I/O path, data=ordered mode is not used, so the zeroed EOF block has no implicit ordering with later i_disksize updates. Without the pending state being set, i_disksize can be advanced past the zeroed block before writeback completes, exposing stale data after a crash. Previous patches added the consumer side of the disksize-grow-pending mechanism: the state bit, clear and wait helpers, and ioend tagging. Now add ext4_iomap_mark_disksize_pending() and call it from ext4_block_zero_eof() after zeroing the tail of the block that straddles i_disksize. The helper locks the folio, waits for any in-flight writeback on it to complete, then sets EXT4_STATE_DISKSIZE_GROW_PENDING only if the folio is still dirty. Waiting for writeback prevents folio_test_dirty() from returning false mid-writeback, which would cause us to skip the pending state while zeroed data is still in flight. The dirty check then avoids setting the bit when the data has already been written back. Suggested-by: Jan Kara <jack@suse.cz> Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Add trace events ext4_iomap_mark_disksize_pending(), ext4_iomap_clear_disksize_pending(), and ext4_iomap_wait_disksize_pending() to track disksize-grow-pending state changes and waiting. Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Add tracepoints to track the disksize-grow-pending lifecycle in the writeback path and the block-zero-EOF entry point: - ext4_iomap_wb_disksize_pending_submit: ioend is marked as DISKSIZE_GROW_IO in ext4_iomap_writeback_submit(). - ext4_iomap_wb_disksize_pending_complete: ioend of type DISKSIZE_GROW_IO completes in ext4_iomap_finish_ioend(). - ext4_iomap_wb_disksize_pending_wait: ioend worker waits for the pending zeroed EOF block to complete. - ext4_iomap_wb_update_disksize: i_disksize is advanced in ext4_iomap_wb_update_disksize, including the new value and whether the update is a disksize-grow completion. - ext4_block_zero_eof: ext4_block_zero_eof is called with the range and zeroing outcome, capturing the producer-side entry point. Together with the previous mark/clear/wait tracepoints, these cover the full lifetime of the disksize-grow-pending state. Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Introduce ext4_enable_buffered_iomap() to determine whether a regular file inode should use the iomap buffered I/O path. We now support the default filesystem features, mount options, and the bigalloc feature. However, inline data, fsverity, fscrypt, indirect inode type, and data=journal mode are not fully supported. The decision is made at inode initialization time in __ext4_new_inode() and __ext4_iget() by setting the EXT4_STATE_BUFFERED_IOMAP state flag. If any of these unsupported features are met, the inode silently falls back to the traditional buffer_head path. Switching the buffered I/O path on an active inode is not supported, with the exception of changing a per-inode journal flag. For features like encryption, verity, and inline data that can be dynamically enabled at the superblock level, checking the global feature flag avoids the complexity of toggling the path on individual inodes. Additionally: - Extend ext4_inode_journal_mode() to force ordered mode for inodes using the iomap path under a data=journal mount. For the global data journal mode (EXT4_MOUNT_JOURNAL_DATA), dynamic enablement is deferred until the next inode re-initialization. For the per-inode data journal mode (EXT4_INODE_JOURNAL_DATA), dynamic changes take effect immediately, as it is safe to switch address_space operations and drop all page cache under i_rwsem and filemap_invalidate_lock. - Add a WARN_ON_ONCE() guard in _ext4_get_block() to catch inodes using the iomap path from accidentally entering the legacy buffer_head writeback path. - Place silent BUFFERED_IOMAP checks in ext4_do_writepages() and ext4_iomap_writepages() under the writepages rwsem. Unlike _ext4_get_block(), the writeback path does not hold i_rwsem or invalidate_lock and can race with ext4_change_inode_journal_flag() switching the inode's a_ops. - Reject extent-to-indirect migration via ext4_ind_migrate() for inodes on the iomap path. Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
Since the iomap buffered I/O path does not yet support all existing ext4 features, it cannot be enabled by default. Introduce the 'buffered_iomap' and 'nobuffered_iomap' mount options to explicitly enable or disable the iomap buffered I/O path for regular files. Toggling this option via remount is allowed. The change of I/O path will not take effect immediately. It will be deferred. The new setting will only take effect after the inode is re-initialized (i.e., after the last reference is dropped and the inode is re-read from disk). Signed-off-by: Zhang Yi <yi.zhang@huawei.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Series: https://patchwork.kernel.org/project/linux-fsdevel/list/?series=1182284
Submitter: Zhang Yi
Version: 7
Patches: 31/31
Message-ID:
<20261009103033.2920530-1-yi.zhang@huaweicloud.com>Base: vfs.base.ci
Lore: https://lore.kernel.org/linux-fsdevel/20261009103033.2920530-1-yi.zhang@huaweicloud.com
Automated by ml2pr