Repository navigation
[action] [PR:29464] [database] Publish database_config.json atomically - #29693
Merged
Merged
Conversation
<!--
Please make sure you've read and understood our contributing guidelines:
https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md
** Make sure all your commits include a signature generated with `git commit -s` **
If this is a bug fix, make sure your description includes "fixes #xxxx", or
"closes #xxxx" or "resolves #xxxx"
Please provide the following information:
-->
#### Why I did it
`database_config.json` is read concurrently during database startup. PR
[sonic-net#28343](sonic-net#28343) made the initial publication
atomic, but three later publication paths remained non-atomic and could truncate a live JSON file
before writing its replacement. This PR extends atomic publication to those remaining known paths.
Instrumentation on DUT measured a 285 us truncate-and-rewrite window in the live-file
`update_chassisdb_config` path. The controlled 202605 A/B test reproduced the empty-read failure
with the installed writer and eliminated it with the atomic writer.
##### Work item tracking
- Microsoft ADO **(number only)**: N/A
#### How I did it
- Copy `/etc/sonic/database_config*.json` to a sibling temporary file and rename it onto the live
path.
- Rename the already-generated modified configuration onto the live path instead of copying over
it, and tolerate the consumed temporary file during cleanup.
- Make `update_chassisdb_config` write a sibling temporary file and publish it with
`os.replace()`.
#### How to verify it
Run concurrent JSON readers while repeatedly invoking the installed in-place writer and the
patched atomic writer against the same scratch configuration. Use identical writer iterations and
reader counts for both arms, and verify that the live database configuration remains untouched.
<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest
backport release branch and provide tested image version on these two branches. For example, if
the PR is requested for master, 202211 and 202012, then the requester needs to provide test results
on master and 202012.
If you request a backport/cherry-pick, provide both:
1. A GitHub issue or Microsoft ADO work item tracking the change.
2. Test evidence from the target branch(es) requested below.
-->
#### Which release branch to backport (provide reason below if selected)
<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->
- [ ] 202305
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
- [ ] 202511
- [ ] 202512
- [x] 202605
- [ ] 202608
Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO):
N/A (internal tracking is not disclosed)
Failure type: other — existing non-atomic database configuration publication race
#### Tested branch
<!--
- Select each branch where the change was tested.
- If you request a backport/cherry-pick, select the base branch and the tested
target release branch(es).
-->
- [x] master
- [ ] 202305
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
- [ ] 202511
- [ ] 202512
- [x] 202605
- [ ] 202608
- [ ] N/A
#### Test result
<!--
- Provide the tested image version and test evidence for each selected branch.
- e.g.
- master: 20260716.01 - <test result or link>
- 202605: 20260531.42 - <test result or link>
-->
- master: — HEAD was exercised in three patched boots as part
of a broader startup-path test and produced zero parser errors. Because that arm contained
additional startup-path changes, it is integration coverage rather than isolated proof for this
PR. Restoring the control produced three errors.
- 202605: — controlled A/B with 150 writes and four concurrent readers per arm:
- Installed in-place writer: 118,554 successful reads and 701 empty reads.
- Patched temp-file plus `os.replace()` writer: 132,972 successful reads and zero empty reads.
- The experiment used a scratch copy; the live database configuration was not modified.
#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->
Publish `database_config.json` atomically to prevent readers from observing empty JSON during
normal updates.
<!--
Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo.
where, Generic Config and Update feature has been labelled as GCU.
-->
Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
Collaborator
Author
|
Original PR: #29464 |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
3 of 20 tasks
Collaborator
Author
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Contributor
|
/azpw retry |
Collaborator
Author
|
Retrying failed(or canceled) jobs... |
Collaborator
Author
|
Retrying failed(or canceled) stages in build 1227190: ✅Stage Test:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why I did it
database_config.jsonis read concurrently during database startup. PR#28343 made the initial publication
atomic, but three later publication paths remained non-atomic and could truncate a live JSON file
before writing its replacement. This PR extends atomic publication to those remaining known paths.
Instrumentation on DUT measured a 285 us truncate-and-rewrite window in the live-file
update_chassisdb_configpath. The controlled 202605 A/B test reproduced the empty-read failurewith the installed writer and eliminated it with the atomic writer.
Work item tracking
How I did it
/etc/sonic/database_config*.jsonto a sibling temporary file and rename it onto the livepath.
it, and tolerate the consumed temporary file during cleanup.
update_chassisdb_configwrite a sibling temporary file and publish it withos.replace().How to verify it
Run concurrent JSON readers while repeatedly invoking the installed in-place writer and the
patched atomic writer against the same scratch configuration. Use identical writer iterations and
reader counts for both arms, and verify that the live database configuration remains untouched.
Which release branch to backport (provide reason below if selected)
Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO):
N/A (internal tracking is not disclosed)
Failure type: other — existing non-atomic database configuration publication race
Tested branch
Test result
of a broader startup-path test and produced zero parser errors. Because that arm contained
additional startup-path changes, it is integration coverage rather than isolated proof for this
PR. Restoring the control produced three errors.
os.replace()writer: 132,972 successful reads and zero empty reads.Description for the changelog
Publish
database_config.jsonatomically to prevent readers from observing empty JSON duringnormal updates.
Signed-off-by: Sonic Build Admin sonicbld@microsoft.com