Skip to content

Commit bfbbe39

Browse files
committed
0.7.0 —— 两个指针槽,以及那条谁也没在执行的约束
riscv64 后端的注释从写下来那天起就说着: tp 在这里是约定不是架构寄存器。ABI 把它留给线程局部存储 [...] 内核可以拿它 当 per-CPU 指针 —— 但同一个 ISA 上的宿主程序会在那里找它的线程指针。[...] 这个后端不能被编进任何同时用线程指针的东西里。 最后那句是对**每一个消费者**的约束,写在注释里,没有任何东西在执行它。它成立, 是因为在这个架构上还没有别的东西想要线程指针。 ⭐ 实测 2026-08-23:有了。openkal-opensbi 的启动对象要写 tp,好让程序的 thread_local 能用 —— 在一台没有操作系统的机器上没别人会做这件事。没有它, 第一次 throw 就挂在 __cxa_get_globals 里,而诊断报的是一个异常函数和一个地址, 一个字都没提线程指针。 ## 于是槽分成两个,而且探针会说话 arch_cpu_tls / arch_cpu_set_tls 与 arch_cpu_percpu 平行:一个是**处理器**的 私有指针,一个是**正在运行的上下文**的。examples/switch 加了一条断言,而它断言 的是「这是两个槽」—— 单独 round-trip 在两者别名的后端上照样通过。 riscv64: cpu: the two slots ALIAS ← 改之前 aarch64: cpu: the two slots are distinct x86_64: cpu: the two slots are distinct ## riscv64 的 per-CPU 指针移到 mscratch 那是这个特权级为这件事提供的寄存器,而且这个后端里没有别的东西用它 —— trap.S 走栈保存恢复,不通过它交换。tp 留给它按 ABI 本来的主人。S 模式的同款 后端会用 sscratch,理由一样。 ⚠️ 变的是**哪个寄存器**,不是接口的含义:调用者存一个不透明指针再读回来, 这件事前后一样真。变的是调用者现在还可以用 arch_cpu_tls。 已验证:三个架构 switch ok + 两槽 distinct;宿主 3 个测试全过。
1 parent 6626f2e commit bfbbe39

8 files changed

Lines changed: 157 additions & 11 deletions

File tree

‎README.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,8 @@ main: back, witness=7 before=1234
6363
trap: raising
6464
trap: back, witness=1
6565
cpu: percpu round-trips
66+
cpu: tls round-trips
67+
cpu: the two slots are distinct
6668
cpu: four barriers accepted
6769
switch ok
6870
```
@@ -72,6 +74,21 @@ that saved the return address and the stack pointer and nothing else would
7274
print the first two and corrupt its caller. `before` is a `volatile` local read
7375
after the round trip.
7476

77+
⭐ **`the two slots are distinct` is the line that says a comment became a
78+
check.** There are two pointer slots — one the PROCESSOR owns (`arch_cpu_percpu`,
79+
where a kernel keeps what describes this hart) and one the running CONTEXT owns
80+
(`arch_cpu_tls`, where the toolchain expects thread-local storage). On aarch64
81+
and x86_64 they are different registers and the distinction is free. On riscv64
82+
`tp` was being used for both, and this file's own backend carried a comment
83+
saying so and a constraint — *"must not be compiled into anything that also uses
84+
a thread pointer"* — that nothing enforced.
85+
86+
Measured 2026-08-23: something does. An openkal implementation on this
87+
architecture writes `tp` so a program's `thread_local` works. The probe printed
88+
`cpu: the two slots ALIAS`, the per-CPU pointer moved to `mscratch` — the
89+
register this privilege level provides for it — and the line now reads
90+
`distinct` on all three.
91+
7592
### What the second architecture actually changed
7693

7794
Two things, and neither was visible with one backend.

‎abi/include/openarch/abi.h‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,39 @@ void* arch_cpu_percpu(void);
151151
void arch_cpu_set_percpu(void* p);
152152
void arch_cpu_fence(int barrier);
153153

154+
/* ⭐ THE OTHER POINTER SLOT: THE ONE THE RUNNING *CONTEXT* OWNS, NOT THE ONE
155+
* THE PROCESSOR OWNS.
156+
*
157+
* `arch_cpu_percpu` is per PROCESSOR — a kernel points it at a structure that
158+
* describes this hart. This pair is per CONTEXT: it is where the toolchain
159+
* expects to find the thread-local storage of whatever is running now, and a
160+
* program compiled with `thread_local` reads through it on every access.
161+
*
162+
* ⚠️ THEY ARE NOT THE SAME SLOT AND ON ONE MACHINE THEY COMPETE FOR THE SAME
163+
* REGISTER. cpu_impl.cpp for riscv64 has said so since it was written:
164+
*
165+
* `tp' IS A CONVENTION HERE, NOT AN ARCHITECTURAL REGISTER. [...] a kernel
166+
* may use it for its per-CPU pointer --- but a hosted program on the same
167+
* ISA would find its thread pointer there instead. [...] this backend must
168+
* not be compiled into anything that also uses a thread pointer.
169+
*
170+
* That warning has now been met by an actual program. Measured 2026-08-23:
171+
* `openkal-opensbi`'s startup object writes `tp` so that libc++abi's
172+
* `thread_local` works, because on a machine with no operating system nobody
173+
* else will. Without it the first `throw` faults inside `__cxa_get_globals` and
174+
* the diagnostic names an exception function and an address --- it says nothing
175+
* about a thread pointer.
176+
*
177+
* ⇒ So the slot belongs here, beside the one it is confused with, and the two
178+
* are separate operations because on aarch64 and x86_64 they are separate
179+
* REGISTERS and on riscv64 they are not. An implementation that needs both on
180+
* riscv64 must move the per-CPU pointer to `sscratch`; this interface makes
181+
* that a decision with a name rather than a collision discovered at run time.
182+
*
183+
* ⚠️ Undefined before someone sets it, exactly as the per-CPU slot is. */
184+
void* arch_cpu_tls(void);
185+
void arch_cpu_set_tls(void* p);
186+
154187
#ifdef __cplusplus
155188
} /* extern "C" */
156189
#endif

‎backends/aarch64/src/cpu_impl.cpp‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,17 @@ extern "C" void arch_cpu_fence(int b) {
3636
}
3737
}
3838

39+
40+
// `TPIDR_EL0` is the one a program reaches, and it is a DIFFERENT register from
41+
// the `TPIDR_EL1` above — so on this machine the two slots do not compete.
42+
// Readable and writable at EL1; readable by a program at EL0, which is what
43+
// makes it the thread pointer.
44+
extern "C" void* arch_cpu_tls(void) {
45+
void* p;
46+
asm volatile("mrs %0, tpidr_el0" : "=r"(p));
47+
return p;
48+
}
49+
50+
extern "C" void arch_cpu_set_tls(void* p) {
51+
asm volatile("msr tpidr_el0, %0" :: "r"(p));
52+
}

‎backends/riscv64/src/cpu_impl.cpp‎

Lines changed: 45 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,23 +2,46 @@
22
#include <openarch/abi.h>
33

44

5-
// ⚠️ `tp` IS A CONVENTION HERE, NOT AN ARCHITECTURAL REGISTER. The ABI reserves
6-
// it for thread-local storage and the hardware assigns it no meaning, so a
7-
// kernel may use it for its per-CPU pointer — but a hosted program on the same
8-
// ISA would find its thread pointer there instead. aarch64's `TPIDR_EL1` is
9-
// architectural and privileged, so no such collision exists.
5+
// ⭐⭐ `mscratch`, AND IT USED TO BE `tp`. THE WARNING THAT SAT HERE CAME TRUE.
106
//
11-
// The consequence for a caller is nil, which is the point; the consequence for
12-
// this backend is that it must not be compiled into anything that also uses a
13-
// thread pointer.
7+
// What this file said before, verbatim:
8+
//
9+
// `tp' IS A CONVENTION HERE, NOT AN ARCHITECTURAL REGISTER. The ABI
10+
// reserves it for thread-local storage [...] a kernel may use it for its
11+
// per-CPU pointer — but a hosted program on the same ISA would find its
12+
// thread pointer there instead. [...] this backend must not be compiled
13+
// into anything that also uses a thread pointer.
14+
//
15+
// That last sentence is a constraint on every consumer, stated in a comment,
16+
// enforced by nothing. It held while nothing on this architecture wanted a
17+
// thread pointer. Measured 2026-08-23, something does: `openkal-opensbi`'s
18+
// startup object writes `tp` so that a program's `thread_local` works, because
19+
// on a machine with no operating system nobody else will.
20+
//
21+
// And the probe says so now rather than a program discovering it:
22+
//
23+
// riscv64: cpu: the two slots ALIAS
24+
// aarch64: cpu: the two slots are distinct
25+
// x86_64: cpu: the two slots are distinct
26+
//
27+
// ⇒ The per-CPU pointer moves to `mscratch`, which is the register this
28+
// privilege level provides FOR THIS PURPOSE and which nothing else in this
29+
// backend uses — trap.S saves and restores through the stack rather than
30+
// swapping through it. `tp` is left to the thread pointer, whose owner it is by
31+
// ABI. An S-mode variant of this backend would use `sscratch` for the same
32+
// reason.
33+
//
34+
// ⚠️ THIS CHANGES WHICH REGISTER, NOT WHAT THE INTERFACE MEANS. A caller stores
35+
// an opaque pointer and reads it back; that is as true after the move as
36+
// before. What changes is that a caller may now also use `arch_cpu_tls`.
1437
extern "C" void* arch_cpu_percpu(void) {
1538
void* p;
16-
asm volatile("mv %0, tp" : "=r"(p));
39+
asm volatile("csrr %0, mscratch" : "=r"(p));
1740
return p;
1841
}
1942

2043
extern "C" void arch_cpu_set_percpu(void* p) {
21-
asm volatile("mv tp, %0" :: "r"(p));
44+
asm volatile("csrw mscratch, %0" :: "r"(p));
2245
}
2346

2447
extern "C" void arch_cpu_fence(int b) {
@@ -40,3 +63,15 @@ extern "C" void arch_cpu_fence(int b) {
4063
}
4164
}
4265

66+
67+
// `tp`, which the ABI reserves for exactly this and which nothing else in this
68+
// backend now touches. The per-CPU pointer moved out of it — see above.
69+
extern "C" void* arch_cpu_tls(void) {
70+
void* p;
71+
asm volatile("mv %0, tp" : "=r"(p));
72+
return p;
73+
}
74+
75+
extern "C" void arch_cpu_set_tls(void* p) {
76+
asm volatile("mv tp, %0" :: "r"(p));
77+
}

‎backends/x86_64/src/cpu_impl.cpp‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,12 @@ namespace {
3737
// `swapgs` reads its own pointer from `IA32_GS_BASE`. Using the other one
3838
// produces a per-CPU pointer that is correct only after a syscall.
3939
constexpr unsigned kIa32GsBase = 0xC0000101u;
40+
// ⚠️ `FS` AND NOT `GS`, AND ON THIS MACHINE THAT IS THE WHOLE DIFFERENCE
41+
// BETWEEN THE TWO SLOTS. The System V ABI puts a program's thread-local storage
42+
// at `%fs`; a kernel's per-CPU structure conventionally sits at `%gs`, which is
43+
// also what `swapgs` exists to exchange on entry. Two registers, so unlike
44+
// riscv64 the two slots here do not compete.
45+
constexpr unsigned kIa32FsBase = 0xC0000100u;
4046

4147
inline arch_u64 rdmsr(unsigned msr) noexcept {
4248
unsigned lo, hi;
@@ -107,3 +113,11 @@ extern "C" void arch_cpu_fence(int b) {
107113
default: break;
108114
}
109115
}
116+
117+
extern "C" void* arch_cpu_tls(void) {
118+
return reinterpret_cast<void*>(rdmsr(kIa32FsBase));
119+
}
120+
121+
extern "C" void arch_cpu_set_tls(void* p) {
122+
wrmsr(kIa32FsBase, reinterpret_cast<arch_u64>(p));
123+
}

‎examples/switch/src/main.cpp‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,17 +115,35 @@ void probe_trap() {
115115
// continues, which is what catches a backend that emitted an instruction the
116116
// machine does not have.
117117
int g_percpu_area = 0;
118+
int g_tls_area = 0;
118119

119120
void probe_cpu() {
120121
arch::set_percpu(&g_percpu_area);
121122
const bool same = (arch::percpu() == &g_percpu_area);
122123

124+
// ⭐ THE SECOND SLOT, AND THE ASSERTION IS THAT IT IS A SECOND SLOT.
125+
//
126+
// Round-tripping it alone would pass on a backend where `tls` and `percpu`
127+
// are the same register — which on riscv64 they ARE, by convention, and
128+
// abi.h records why that matters. So the check is that BOTH hold their own
129+
// value at the same time, which is exactly what fails when one aliases the
130+
// other. On riscv64 this is expected to fail until the backend moves the
131+
// per-CPU pointer to `sscratch`; the point of the check is that the day it
132+
// matters, a program says so instead of reading its thread-locals out of a
133+
// per-CPU structure.
134+
arch::set_tls(&g_tls_area);
135+
const bool tls_ok = (arch::tls() == &g_tls_area);
136+
const bool distinct_ok = (arch::percpu() == &g_percpu_area);
137+
123138
arch::fence(arch::barrier::memory);
124139
arch::fence(arch::barrier::store);
125140
arch::fence(arch::barrier::complete);
126141
arch::fence(arch::barrier::fetch);
127142

128143
machine::print(same ? "cpu: percpu round-trips\n" : "cpu: percpu FAILED\n");
144+
machine::print(tls_ok ? "cpu: tls round-trips\n" : "cpu: tls FAILED\n");
145+
machine::print(distinct_ok ? "cpu: the two slots are distinct\n"
146+
: "cpu: the two slots ALIAS\n");
129147
machine::print("cpu: four barriers accepted\n");
130148
}
131149

‎mcpp.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@
3333
[package]
3434
namespace = "mcpplibs"
3535
name = "openarch"
36-
version = "0.6.0"
36+
version = "0.7.0"
3737
description = "openarch: the architecture-mechanism layer — execution contexts, traps, per-CPU state and address spaces, as one interface over several instruction sets"
3838
license = "Apache-2.0"
3939
authors = ["mcpplibs"]

‎src/cpu.cppm‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,21 @@ export namespace arch {
3939
inline void* percpu() noexcept { return ::arch_cpu_percpu(); }
4040
inline void set_percpu(void* p) noexcept { ::arch_cpu_set_percpu(p); }
4141

42+
// The running CONTEXT's private pointer, which is a different question from the
43+
// one above and is asked by different code.
44+
//
45+
// The per-CPU slot is read by a kernel that wants to know which hart it is on.
46+
// This one is read by the TOOLCHAIN, on every access to a `thread_local`, and
47+
// it is set by whoever creates the context. Where a program carries no loader,
48+
// that is the openkal implementation beneath it.
49+
//
50+
// ⚠️ On riscv64 the two slots want the same register (`tp`), and abi.h records
51+
// the measurement that made the conflict real rather than theoretical. Naming
52+
// them separately is what lets an implementation choose; it does not make the
53+
// register conflict go away.
54+
inline void* tls() noexcept { return ::arch_cpu_tls(); }
55+
inline void set_tls(void* p) noexcept { ::arch_cpu_set_tls(p); }
56+
4257
// The four orderings both machines can state.
4358
enum class barrier {
4459
// Everything before is ordered before everything after, as observed by

0 commit comments

Comments
 (0)