Skip to content

DRA: allocate topology-aware claims - #821

Merged
askervin merged 2 commits into
mainfrom
stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims
Oct 12, 2026
Merged

askervin merged 2 commits into
mainfrom
stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims

Conversation

@bart0sh

@bart0sh bart0sh commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

Allocating and releasing CPUs for claims

The topology-aware policy implements AllocateClaim and ReleaseClaim now, so a claim against the NUMA devices #820 publishes gets CPUs of those nodes, and the test42-dra-devices e2e test is extended to claims.

What a claim gets

Each allocation result names a NUMA node device, and consumes some dra.cpu/cpu of it. AllocateClaim decodes that into a number of CPUs per node.

For each node it picks that many CPUs out of the node's free, grantable CPUs, isolated plus sharable less reserved, with the same cpuAllocator an exclusive grant uses. Two CPUs are therefore the two threads of one core, not one thread of each of two cores. Several results on one node get disjoint CPUs, and if one result cannot be filled, nothing stays claimed.

The container gets one CDI edit per result, each setting DRA_CPUSET_<claim UID>=<cpuset> to the whole claim's CPUs.

ReleaseClaim gives the CPUs back, and releasing an unknown claim is not an error, as the kubelet may unprepare a claim more than once.

A claim is a grant

A claim's CPUs are held as a grant, the policy's existing record of what is allocated: exclusive CPUs reserved from the deepest pool holding all of them, with no shared portion and no memory. Claim grants are kept beside the container grants in allocations, keyed by claim UID, and a claim grant has no container. So the NRI path cannot hand claimed CPUs to an exclusive or shared grant while the claim lives, and the claim is checked, dumped and saved with the other grants. Claim UIDs and container IDs stay separate keyspaces: in the saved state a claim grant is keyed claim:<uid>.

Reserving the grant refuses CPUs another grant holds, and refuses sharable CPUs whose loss would overcommit the shared portions already granted, as for any exclusive grant.

After a claim or release, updateSharedAllocations recomputes the shared cpusets, so shared containers move off claimed CPUs and back onto released ones.

Reconfigure rebuilds the pool tree and reinstates the claim grants before the container grants. A claim's CPUs cannot move, so a configuration which would take a claimed CPU away from us, by reserving it for example, is refused and the old configuration stays.

What is not here

These are known limitations, each lifted by a later PR in the series:

  • The claiming container is not pinned to its claim's CPUs.
  • Claims do not survive a restart. They are saved with the other grants, but the policy resets its saved state on start, so after the plugin restarts their CPUs are free again and can be granted twice.
  • The NodeResourceTopology CR goes stale after a claim, as nothing refreshes it when claimed CPUs leave the free supply.
  • No DeviceClass ships yet, so a claim needs one written by the user. The e2e test creates its own.

Verification

  • The new policy tests run on the recorded server sysfs. They cover one and several results on one or two nodes, a whole node, a node less its reserved CPUs, and each refused result, including a claim with no results. They also check picking by topology, re-preparing a claim, disjoint claims and exhaustion, a failed multi-result claim leaving nothing claimed, release, a Reconfigure which keeps claims and one refused for reserving a claimed CPU, and a claim and a container with the same ID both surviving a reconfiguration and a round trip through the cache.
  • Each of these mutations fails the tests: skipping the claim reinstatement in Reconfigure, skipping the release, and picking anew on a repeated prepare.
  • A test covers the shared overcommit check: on a node whose shared containers leave a little over one CPU's worth of shared capacity, reserving a claim grant of two sharable CPUs is refused and leaves nothing reserved, and one of one CPU succeeds. AllocateClaim already picks only what the shared containers can spare, so the test reserves directly.
  • The move of shared containers is not covered by the unit tests, since it needs pinned containers the test cache cannot supply. The e2e test below covers it.
  • test42-dra-devices passes on n4c16 with Kubernetes 1.37 and the DRAConsumableCapacity and DRANodeAllocatableResources gates. With CPU 15 reserved, the devices offer exactly 4, 4, 4 and 3 CPUs. A claim for all of node 3 gets 12-14. Two 2-CPU claims on node 1 get one core each, 4-5 and 6-7. A third stays unscheduled until the first is deleted, then gets 4-5. A besteffort container runs on no claimed CPU, gives up the two CPUs a claim takes of its node, and gets them back on release.

@bart0sh
bart0sh added this pull request to stack #807 September 30, 2026 13:31
@bart0sh bart0sh changed the title topology-aware: allocate CPUs to DRA claims. DRA: allocate topology-aware claims Sep 30, 2026
@bart0sh
bart0sh requested a balanced review from Copilot September 30, 2026 13:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Per-node idempotence, shared-capacity selection, and reconfiguration accounting can violate allocation guarantees.

Review effort: Balanced
Findings: 1 High severity · 3 Medium severity

Open (4)
What changed in this PR

Implements topology-aware DRA CPU claim allocation and release.

Changes:

  • Allocates claim CPUs by NUMA topology and returns CDI edits.
  • Excludes claimed CPUs from pool supplies and restores them on release.
  • Adds allocation, rollback, release, and reconfiguration tests.
File Description
topology-aware-policy.go Tracks claims and reapplies them during reconfiguration.
resources.go Adds claim/unclaim supply accounting.
dra.go Implements topology-aware claim lifecycle.
dra_test.go Tests DRA allocation and accounting behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cmd/plugins/topology-aware/policy/topology-aware-policy.go Outdated
Comment thread cmd/plugins/topology-aware/policy/dra.go Outdated
Comment thread cmd/plugins/topology-aware/policy/dra.go Outdated
Comment thread cmd/plugins/topology-aware/policy/topology-aware-policy.go
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch 4 times, most recently from a1bb176 to f4dd220 Compare October 1, 2026 08:42
@bart0sh
bart0sh requested a balanced review from Copilot October 1, 2026 08:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The allocation lifecycle, rollback paths, topology constraints, and supply accounting are consistently implemented and well tested.

Review effort: Balanced
Findings: None

Resolved since last review (4)

@bart0sh
bart0sh marked this pull request as ready for review October 1, 2026 08:55
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from f4dd220 to be437ab Compare October 1, 2026 15:22
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from be437ab to 751b471 Compare October 2, 2026 07:02
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from 751b471 to db70637 Compare October 2, 2026 07:11
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from db70637 to 645ec1a Compare October 2, 2026 07:27
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch 2 times, most recently from ec6eca1 to 32a15b5 Compare October 5, 2026 13:52
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch 2 times, most recently from 467f77b to 6fe1982 Compare October 6, 2026 08:45
@bart0sh
bart0sh requested a balanced review from Copilot October 6, 2026 10:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Allocation key collisions can lose grants, and the release assertion is vulnerable to asynchronous teardown.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)

Comment thread cmd/plugins/topology-aware/policy/topology-aware-policy.go Outdated
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from 6fe1982 to dae0c31 Compare October 6, 2026 10:56
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch 2 times, most recently from 7cb21ff to 59ea171 Compare October 6, 2026 13:37
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from 59ea171 to 8cc6dd0 Compare October 6, 2026 16:32
@bart0sh
bart0sh requested a balanced review from Copilot October 6, 2026 16:39

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation consistently integrates claims with allocation, persistence, reconfiguration, and comprehensive tests.

Review effort: Balanced
Findings: None

Resolved since last review (2)

@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from 8cc6dd0 to 1268fdd Compare October 7, 2026 07:41
Base automatically changed from stacked/bart0sh/DRA/005-topology-aware-devices to main October 7, 2026 09:00
@klihub
klihub force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from 1268fdd to 4586fa5 Compare October 7, 2026 09:00
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from 4586fa5 to 73180be Compare October 7, 2026 09:35
@bart0sh

bart0sh commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

@klihub @askervin This PR is ready for review now. PTAL when you have time, thanks!

@klihub
klihub requested review from askervin and klihub October 7, 2026 11:27
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
Signed-off-by: Ed Bartosh <eduard.bartosh@intel.com>
@bart0sh
bart0sh force-pushed the stacked/bart0sh/DRA/006-topology-aware-allocate-release-claims branch from 73180be to a22b69f Compare October 8, 2026 10:07
@klihub

klihub commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

@bart0sh @askervin This looks now pretty good to me and currently I think this is good to go in. But I'd like to go through it once more, with a bit more focus (and a clearer mind if I can achieve that, too).

@klihub klihub left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bart0sh There is one more thing related to this and DRA-claimed CPUs in general that I think we should add. However, since it could use a small addition to pkg/resmgr/policy and I've missed that earlier in the review, we can add that with a separate, self-contained, additional PR in this stack as well.

We should extend ExportResourceData() and the related existing commonly defined consts in resmgr/policy to expose per container the DRA-claimed CPU sets, in addition to the currently exposed isolated, exclusive, and shared CPU sets.

@bart0sh

bart0sh commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator Author

@klihub

We should extend ExportResourceData() and the related existing commonly defined consts in resmgr/policy to expose per container the DRA-claimed CPU sets, in addition to the currently exposed isolated, exclusive, and shared CPU sets.

Sounds good. I'll do this in a separate PR.

@askervin askervin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a tiny nit, that is so tiny that I'm perfectly fine this getting merged as is.

Comment thread cmd/plugins/topology-aware/policy/resources.go
@askervin
askervin merged commit 954688a into main Oct 12, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants