Report capacity and allocation per resource for each worker

[JeffLuoo]

As discussed in this issue comment #1664 (comment)

Problem

ate.workerpool.workers counts workers. A worker can now hold more than one actor, so a count of workers cannot show how full a pool is. A worker with 1 of 20 slots used and a worker with 19 of 20 are both partial (#1799).

The scheduler picks a random worker from the workers that have room. With more than one slot on each worker, the actors spread across the pool. At low load, most workers are partial, and at_capacity stays near 0. Under more load, at_capacity jumps suddenly. Neither state gives a good signal for an autoscaler.

No instrument reports the number of actors in operation. The count of assigned workers was a substitute for it, and that substitute is gone (#1664).

For discussion of HPA, follow this discussion in slack

[BenTheElder]

Slots isn't a very accurate way to put it I think. Think of workers more like Kubernetes nodes now. They have capacity both allocated and allocatable, while also have a limit on the maximum number of workloads to account for other resources not account for by the scheduler such as PIDs, FDs, etc.

The cap on number of actors is an upper bound, not a target. CPU / Memory / ... are more realistic.

See also the discussion about autoscaling in slack.

[krisztianfekete]

The format I proposed in #1664 already covers this (allocated/capacity per resource, like k8s.node.allocatable.*). Two more things:

• Leave unschedulable workers ((ateapi): report worker occupancy by actor slots in ate.worker.state #1854) out of capacity, otherwise a draining or not-yet-reported worker shows up as free.
• Allocated actors gives back the "actors in operation" count we lost with assigned.

[JeffLuoo]

Krisztian F (@krisztianfekete) are you proposing to add instrumentation for cpu, memory, and actor count grouped by "Allocated" and "Capacity" for each pool in this issue? Just want to make sure I understand it correctly.

[krisztianfekete]

Yes, but as separate instruments, e.g. ate.workerpool.cpu.capacity, ate.workerpool.cpu.allocated, etc., same for memory and actors. Allocated can exceed capacity, so they shouldn't be two values of one attribute. Same labels as ate.workerpool.workers. unschedulable workers count in allocated but not capacity, and a resource a worker doesn't report gets no series instead of 0.

[krisztianfekete]

Update after the Slack discussion here: https://cloud-native.slack.com/archives/C0B6M3E2J3D/p1790200752576789

Per worker, not per pool, like node metrics in k8s. Pool totals hide fragmentation (10 workers with 0.5 CPU free each can't fit a 1 CPU actor), and a pool total is just a sum over workers anyway.

So ate.worker.cpu.capacity / ate.worker.cpu.allocated, same for memory and actors, with a new ate.worker.name label on top of the pool labels. The rest of my comment above stays the same.

[JeffLuoo]

Updated