Skip to content

[action] [PR:29464] [database] Publish database_config.json atomically - #29693

Merged
mssonicbld merged 1 commit into
sonic-net:202605from
mssonicbld:cherry/202605/29464
Sep 23, 2026
Merged

mssonicbld merged 1 commit into
sonic-net:202605from
mssonicbld:cherry/202605/29464

Conversation

@mssonicbld

Copy link
Copy Markdown
Collaborator

Why I did it

database_config.json is read concurrently during database startup. PR
#28343 made the initial publication
atomic, but three later publication paths remained non-atomic and could truncate a live JSON file
before writing its replacement. This PR extends atomic publication to those remaining known paths.

Instrumentation on DUT measured a 285 us truncate-and-rewrite window in the live-file
update_chassisdb_config path. The controlled 202605 A/B test reproduced the empty-read failure
with the installed writer and eliminated it with the atomic writer.

Work item tracking
  • Microsoft ADO (number only): N/A

How I did it

  • Copy /etc/sonic/database_config*.json to a sibling temporary file and rename it onto the live
    path.
  • Rename the already-generated modified configuration onto the live path instead of copying over
    it, and tolerate the consumed temporary file during cleanup.
  • Make update_chassisdb_config write a sibling temporary file and publish it with
    os.replace().

How to verify it

Run concurrent JSON readers while repeatedly invoking the installed in-place writer and the
patched atomic writer against the same scratch configuration. Use identical writer iterations and
reader counts for both arms, and verify that the live database configuration remains untouched.

Which release branch to backport (provide reason below if selected)

  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608

Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO):
N/A (internal tracking is not disclosed)

Failure type: other — existing non-atomic database configuration publication race

Tested branch

  • master
  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605
  • 202608
  • N/A

Test result

  • master: — HEAD was exercised in three patched boots as part
    of a broader startup-path test and produced zero parser errors. Because that arm contained
    additional startup-path changes, it is integration coverage rather than isolated proof for this
    PR. Restoring the control produced three errors.
  • 202605: — controlled A/B with 150 writes and four concurrent readers per arm:
    • Installed in-place writer: 118,554 successful reads and 701 empty reads.
    • Patched temp-file plus os.replace() writer: 132,972 successful reads and zero empty reads.
    • The experiment used a scratch copy; the live database configuration was not modified.

Description for the changelog

Publish database_config.json atomically to prevent readers from observing empty JSON during
normal updates.

Signed-off-by: Sonic Build Admin sonicbld@microsoft.com

<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it

`database_config.json` is read concurrently during database startup. PR
[sonic-net#28343](sonic-net#28343) made the initial publication
atomic, but three later publication paths remained non-atomic and could truncate a live JSON file
before writing its replacement. This PR extends atomic publication to those remaining known paths.

Instrumentation on DUT measured a 285 us truncate-and-rewrite window in the live-file
`update_chassisdb_config` path. The controlled 202605 A/B test reproduced the empty-read failure
with the installed writer and eliminated it with the atomic writer.

##### Work item tracking
- Microsoft ADO **(number only)**: N/A

#### How I did it

- Copy `/etc/sonic/database_config*.json` to a sibling temporary file and rename it onto the live
  path.
- Rename the already-generated modified configuration onto the live path instead of copying over
  it, and tolerate the consumed temporary file during cleanup.
- Make `update_chassisdb_config` write a sibling temporary file and publish it with
  `os.replace()`.

#### How to verify it

Run concurrent JSON readers while repeatedly invoking the installed in-place writer and the
patched atomic writer against the same scratch configuration. Use identical writer iterations and
reader counts for both arms, and verify that the live database configuration remains untouched.

<!--
If PR needs to be backported, then the PR must be tested against the base branch and the earliest
backport release branch and provide tested image version on these two branches. For example, if
the PR is requested for master, 202211 and 202012, then the requester needs to provide test results
on master and 202012.

If you request a backport/cherry-pick, provide both:
1. A GitHub issue or Microsoft ADO work item tracking the change.
2. Test evidence from the target branch(es) requested below.
-->

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 202305
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
- [ ] 202511
- [ ] 202512
- [x] 202605
- [ ] 202608

Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO):
N/A (internal tracking is not disclosed)

Failure type: other — existing non-atomic database configuration publication race

#### Tested branch

<!--
- Select each branch where the change was tested.
- If you request a backport/cherry-pick, select the base branch and the tested
  target release branch(es).
-->

- [x] master
- [ ] 202305
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
- [ ] 202511
- [ ] 202512
- [x] 202605
- [ ] 202608
- [ ] N/A

#### Test result

<!--
- Provide the tested image version and test evidence for each selected branch.
- e.g.
- master: 20260716.01 - <test result or link>
- 202605: 20260531.42 - <test result or link>
-->

- master: — HEAD was exercised in three patched boots as part
  of a broader startup-path test and produced zero parser errors. Because that arm contained
  additional startup-path changes, it is integration coverage rather than isolated proof for this
  PR. Restoring the control produced three errors.
- 202605: — controlled A/B with 150 writes and four concurrent readers per arm:
  - Installed in-place writer: 118,554 successful reads and 701 empty reads.
  - Patched temp-file plus `os.replace()` writer: 132,972 successful reads and zero empty reads.
  - The experiment used a scratch copy; the live database configuration was not modified.

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

Publish `database_config.json` atomically to prevent readers from observing empty JSON during
normal updates.

<!--
 Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo.
 where, Generic Config and Update feature has been labelled as GCU.
-->

Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator Author

Original PR: #29464

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator Author

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@chartsai-nvidia

Copy link
Copy Markdown
Contributor

/azpw retry

@mssonicbld

Copy link
Copy Markdown
Collaborator Author

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator Author

Retrying failed(or canceled) stages in build 1227190:

✅Stage Test:

  • Job impacted-area-kvmtest-t1-lag-vpp by Elastictest: retried.

@mssonicbld
mssonicbld merged commit 10cb105 into sonic-net:202605 Sep 23, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants