network: fix queue

[KurtWu10]

UPDATE: we are in the process of developing a general queue library, and therefore this PR may be abandoned.

---

This PR fixes the network queue implementation for functional correctness, especially under multicore.

Since there are many possible implementations for Arm that affects the CPU utilisation, this PR is in its early stages and is seeking feedback.

In summary, for maaxboard, using standard C11 memory barriers increases CPU utilisation from 63.9% to 69.7%. I also found some dangerous optimisations that reduce the number to 67.3%, for maaxboard only. There is no performance penalty on odroid-c4 and star64, some overhead on odroid-c2, but the penalty is more than 10% on raspberry pi 4B, and these optimisations are not effective on raspberry pi 4B.

TODO: given that no further performance optimsiation is required, the next step is to add model checking code for GenMC and comments.

Details

One design decision is that PDs should be location-transparent, meaning they are functionally correct regardless of whether other PDs are running on the same core. This will inevitably degrade unicore performance because explicit memory barriers are required on Arm and RISC-V. Therefore, the PR is currently optimised for unicore CPU utilisation for UDP using the echo-server example while ensuring location-transparency. Experiments achieve the highest available UDP throughput.

Settings

In early evaluations (at 100 Mb/s and 900 Mb/s request throughput only), I found that unicore CPU utilisation of maaxboard is more affected than odroid-c4, likely due to differences in processor micro-architecture implementations, so I chose the former for detailed analysis.

I used a custom seL4 and a custom sddf, which allows me to get PMU event data at the PD granularity (I have empirically validated that using the custom kernel does not affect total utilisation significantly). I use

make BENCH_PMU_EVENTS=INSTRUCTIONS,CHAIN,STALL_BACKEND_LD,CHAIN,STALL_BACKEND_ST,CHAIN

when building the echo-server example.

Findings

Firstly, for a controlled experiment, I hand-optimised common sub-expressions in the queue in the first commit. I found that inserting even only compiler barriers will affect the number of memory instructions in the current implementation. Although this does not significantly affect the CPU utilisation (likely due to the fast forwarding and L1 access of the processor; for reference, the L1 D-cache hit latency is measured to be about 3 cycles), this makes comparing PMU event data more difficult.

Secondly, for the available PMU events, I found that changes in the number of processor cycles stalled because of loads and stores (STALL_BACKEND_LD and STALL_BACKEND_ST) for the driver/virtualisers PDs represents changes in the CPU utilisation, especially under medium-to-high loads. These two processor stall events are the most significant among pipeline stalls, and when expensive memory barriers are used, the numbers also largely change, along with the CPU utilisation. If an optimised barrier implementation is adopted, the numbers change on a smaller scale, and the CPU utilisation is improved. I don't currently understand this empirical observation further at the processor micro-architectural level.

Thirdly, I also evaluated other possible implementations. There seems to be no performance overhead from using atomic load-acquire operations and acquire fences compared to non-atomic loads, but atomic store-release operations and release fences noticeably increase CPU util compared to non-atomic stores. These store operations are the target for optimisations in the following section.

Dangerous optimisations

For pancake, I prefer using these two optimisations. For C, the second of them could be dangerous because it results in data races at the language level (resulting in UB) and even without data races, it violates functional correctness at the language memory model level, but I am not able to clearly explain where it could actually fail.

The two optimisations bypass the C11 memory model and instead reasons about the hardware memory model directly. The first optimisation replaces the release fence in enqueue functions with wwmb() (on Arm, this changed the generated machine code from dmb ish to dmb st). This alone prevents reasoning the queue at the C language memory model level, because the corresponding instruction (dmb st) is not representable.

The second optimisation is the focus of this section, which replaces the release fence in dequeue functions with an acquire fence. This optimisation is not valid for all processor architectures, I believe, but it is correct for me for architectures supported by seL4.

In particular, the corresponding scenario is when the consumer performs a dequeue operation, followed by an enqueue from the producer. The C11 memory model allows the following test (written in a C-like pseudocode for CppMem) that characterises the key communication pattern:

int main() {
  atomic_int data = 0;
  atomic_int flag = 0;
  {{{ {
    // dequeue from consumer
    r1 = data.load(relaxed).readsvalue(2);
    fence(acquire);
    flag.store(1, relaxed);
  } ||| {
    // enqueue from producer
    r2 = flag.load(relaxed).readsvalue(1);
    fence(acquire);
    data.store(2, relaxed);
  }  }}}
}

but when the code is naively compiled to Arm (LB+dmb.lds), RISC-V (see the LB+fence.r.rws litmus test in riscv-tests) and also to x86 machine code, the execution is forbidden.

This is explained in the IMM paper. Informally, the execution is forbidden because there is a global memory order (in RISC-V's terminology, a.k.a. ordered-before in Arm) from P0's first load to the store, then from P0's store to P1's first load, and finally from P1's first load to P1's store. In the C11 memory model, however, the cross-core communication between P0's store and P1's first load is not a happens-before relation.

[gernotheiser]

I wouldn't be too paranoid about 10% CPU increase on unicore.
We'll normally run multicore, and comparing a unicore-optimised setup to other OSes isn't fair anyway

[KurtWu10]

OK, then I will stop worrying and optimising it further and instead benchmark other boards.