diff --git a/.add/milestones/v3-3-vector-kv-polish/MILESTONE.md b/.add/milestones/v3-3-vector-kv-polish/MILESTONE.md index a3ef0538a..89d5a4c0f 100644 --- a/.add/milestones/v3-3-vector-kv-polish/MILESTONE.md +++ b/.add/milestones/v3-3-vector-kv-polish/MILESTONE.md @@ -18,8 +18,13 @@ In: - FT.SEARCH avoids the ~3.2 MB `key_hash_to_key` clone per query (borrow / `Arc` the map). - KV `INCR`/`DECR` write the integer via `itoa` to a buffer — no per-op `String` alloc on the hot path. `src/command/string/string_write.rs:312,317`. -- The command-dispatch path uses `parking_lot::RwLock`, not `std::sync::RwLock`, for the ACL table. - `src/shard/event_loop.rs:65`, `src/command/connection.rs:374,460`. +- The command-dispatch path uses `parking_lot::RwLock`, not `std::sync::RwLock`, for the ACL table + (`src/shard/event_loop.rs:65`, `src/command/connection.rs:374,460`) **and for the replica-role check + that gates inline dispatch** on the p=1 read hot path (`ctx.repl_state.try_read()`, + `src/server/conn/handler_monoio/mod.rs:524` — cache as `AtomicBool` flipped on role change, or `parking_lot`). +- KV inline `GET` hit writes the value **once**: frame the `$len\r\n…\r\n` reply straight from the borrowed + `&[u8]` into `write_buf`, dropping the intermediate `val.to_vec()` heap copy. + `src/server/conn/blocking.rs:1275` (on the p=1 read hot path; the wasted copy scales with value size). Out: - Re-quantization / new vector codecs — only the existing SQ8 decode length is in scope, not new @@ -51,8 +56,10 @@ Out: FT.SEARCH (borrow / `Arc`). - [ ] kv-incr-itoa depends-on: none — `INCR`/`DECR` via `itoa`-to-buffer; no `String` alloc on the hot path. -- [ ] kv-dispatch-lock-discipline depends-on: none — ACL / dispatch `std::sync::RwLock` -> - `parking_lot::RwLock`. +- [ ] kv-dispatch-lock-discipline depends-on: none — ACL / dispatch + inline-gate replica check + `std::sync::RwLock` -> `parking_lot::RwLock` / `AtomicBool` (incl. `handler_monoio/mod.rs:524`). +- [ ] kv-inline-get-nocopy depends-on: none — inline `GET` hit writes value straight from the + borrow into `write_buf`; drop the `val.to_vec()` double-copy (`blocking.rs:1275`, scales w/ value size). ## Exit criteria (observable; map each to the task that delivers it) - [ ] An SQ8 immutable segment decodes vectors at the correct length — recall parity with the @@ -61,5 +68,7 @@ Out: - [ ] FT.SEARCH performs no per-query 3 MB `key_hash` clone — allocation probe / throughput test shows the clone gone. (← vector-search-keyhash-noclone) - [ ] `INCR`/`DECR` allocate no `String` on the hot path — `itoa` path asserted (test / no-alloc check). (← kv-incr-itoa) -- [ ] No `std::sync::RwLock` remains on the command-dispatch path — ACL table is `parking_lot`; - audit/grep + test. (← kv-dispatch-lock-discipline) +- [ ] No `std::sync::RwLock` remains on the command-dispatch path — ACL table + inline-gate replica + check are `parking_lot`/atomic; audit/grep + test. (← kv-dispatch-lock-discipline) +- [ ] Inline `GET` performs no intermediate `Vec` copy of the value — the `$len…` reply is framed from + the borrowed slice; allocation probe / large-value throughput shows the copy gone. (← kv-inline-get-nocopy) diff --git a/.add/milestones/v3-4-kv-correctness/MILESTONE.md b/.add/milestones/v3-4-kv-correctness/MILESTONE.md new file mode 100644 index 000000000..b0083914c --- /dev/null +++ b/.add/milestones/v3-4-kv-correctness/MILESTONE.md @@ -0,0 +1,173 @@ +# MILESTONE: KV Write Correctness & Data-Integrity Parity + +goal: Moon's KV write commands honor Redis data-integrity contracts: no wrong-type command destroys data, integer commands reject overflow instead of panicking or wrapping, expire-in-the-past deletes the key, and MSETNX is atomic (cross-shard spans rejected). +rationale: new-major (split 3/3) — the KV-correctness sibling of the v3 "secondary-engine correctness & parity" theme. v3-1 (FTS) and v3-2 (graph) closed the search/graph defects; the 2026-07-02 KV deep review (`tmp/KV-DEEP-REVIEW.md`) surfaced a cluster of P0 data-integrity bugs in the *primary* KV engine (silent data loss on wrong-type GETDEL/GETSET/SET..GET, i64 overflow panics/wraps in DECRBY + the expire family, and a missing MSETNX). Correctness only — the p=1 throughput question from the same review is platform-driven (confound closed) and is NOT in scope here. +stage: production · status: active · created: 2026-07-02 + +> SDD living doc for this milestone. Keep it THIN: breadth, shared decisions, and +> exit criteria only — per-task detail lives in each `.add/tasks//TASK.md`, +> written just-in-time. Update this doc whenever a task reveals a milestone gap. + +## Scope +In: +- **Wrong-type data loss (P0):** GETDEL, GETSET, and `SET key val GET` must return WRONGTYPE and + leave the key intact when it holds a non-string — never delete/overwrite it. (`check-before-mutate` + parity with Redis.) `src/command/string/string_read.rs` (GETDEL), `string_write.rs` (SET..GET, GETSET). +- **Integer overflow (P0):** `DECRBY key i64::MIN` must not panic on `checked_neg`; the expire family + (`SET EX/EXAT`, `SETEX`, `EXPIRE`/`EXPIREAT`/`PEXPIRE`) must reject seconds/ms that overflow + `i64 * 1000 + now` with an "invalid expire time" error instead of wrapping to a bogus TTL. +- **Expire-in-the-past semantics (P0):** `EXPIRE`/`PEXPIRE`/`EXPIREAT` with a non-positive / already-past + time DELETES the key and returns 1 (Redis parity), and that deletion PROPAGATES through command-based + WAL replay to replicas (`DispatchReplayEngine::replay_command` re-dispatches the raw command). +- **MSETNX (P0, missing command):** add MSETNX as an atomic multi-key write (set all iff none exist). + Single-shard: two-phase check-then-set with no await (atomic). Multi-shard: the coordinator REJECTS a + cross-shard span with CROSSSLOT (by design — no 2PC), runs co-located keys atomically on the owner. + +Out: +- **P1 semantics polish (future milestone):** SET-option mutual-exclusion validation (EX+PX, NX+XX), + SCAN delete-stability under rehash, TOUCH access-time bump, stricter integer parse (leading `+`, + whitespace), TTL rounding parity. Advisor-endorsed P0/P1 split — these are correctness *refinements*, + not data-loss/crash bugs. +- **p=1 throughput vs Redis:** the KV deep review CLOSED this as platform/environment-driven (same + build wins p=1 on OrbStack, loses on GCloud shared-vCPU); not a KV-code defect, not in scope. +- **Cross-shard MSETNX atomicity via 2PC:** explicitly rejected (CROSSSLOT) rather than half-built. + +## Shared decisions & glossary deltas (living — every task must honor these) +- **check-before-mutate:** any command that could destroy data on a type mismatch must peek the entry + type and return WRONGTYPE BEFORE the mutation — never remove/overwrite first. +- **TDD red/green (CLAUDE.md Rule 3):** each defect lands a FAILING test first, then the fix. The + cross-shard MSETNX reject was red/green-proven by neutralizing only the CROSSSLOT branch. +- **Overflow is an error, not a panic/wrap:** integer/time arithmetic on the command path uses + `checked_*` and returns a `Frame::Error`, never `unwrap()`/silent wrap (CLAUDE.md error-handling). +- **Replay consistency:** semantic changes to write commands must be verified to propagate through + command-based WAL replay (a replay test), so master and replica converge. +- **New commands carry script coverage:** MSETNX added to `scripts/test-consistency.sh` (hash-tagged + for 1/4/12-shard parity with Redis) + `scripts/test-commands.sh` (CLAUDE.md New Commands rule). + +## Shared / risky contracts (freeze these first) +- **MSETNX cross-shard disposition** — reject (CROSSSLOT) vs best-effort scatter vs 2PC. Chosen: REJECT + (user decision). A wrong choice here re-does the coordinator + the command's whole test surface. + -> owning task `kv-msetnx-atomic` +- **Expire-past delete + replay** — deleting on past-time must replay identically or replicas diverge. + -> owning task `kv-expire-past-deletes` + +## Tasks (breadth-first decomposition; detail lives in each TASK.md) +- [x] kv-wrongtype-guard depends-on: none — GETDEL / GETSET / SET..GET check type + before mutate; wrong-type returns WRONGTYPE and preserves the key (no silent data loss). +- [x] kv-integer-overflow-guard depends-on: none — DECRBY i64::MIN + the expire family + (SET EX/EXAT, SETEX, EXPIRE/EXPIREAT/PEXPIRE) reject overflow with an error instead of panic/wrap. +- [x] kv-expire-past-deletes depends-on: none — EXPIRE/PEXPIRE/EXPIREAT with a past/ + non-positive time deletes the key (returns 1); verified to propagate through WAL replay. +- [x] kv-msetnx-atomic depends-on: none — add MSETNX; atomic on one shard, + CROSSSLOT-reject across shards, co-located keys atomic on the owner. + +## Exit criteria (observable; map each to the task that delivers it) +- [x] `GETDEL`/`GETSET`/`SET k v GET` on a key holding a list/hash returns WRONGTYPE and the key still + exists afterward (no data loss) — unit tests assert key preserved. (← kv-wrongtype-guard) +- [x] `DECRBY k -9223372036854775808` and `SETEX`/`SET … EX ` return an error, not a panic or a + wrapped TTL — unit tests for min-overflow + expire overflow. (← kv-integer-overflow-guard) +- [x] `EXPIRE k -1` (and past `EXPIREAT`) deletes k and returns 1; a WAL-replay test proves the delete + re-applies on replay (master/replica consistent). (← kv-expire-past-deletes) +- [x] Co-located `MSETNX` is atomic (all-new→1, any-exists→0 with no partial write); a cross-shard + `MSETNX` returns CROSSSLOT and writes nothing — unit + `--shards 4` integration test. (← kv-msetnx-atomic) + +## Close — ship review (AI fills when every task is done — the evidence behind the engine gate, read before the boxes are checked) +> Whole-milestone, cross-task review the AI fills in. It is the evidence behind the EXISTING engine +> gate (milestone-done / checking the Exit-criteria boxes) — NOT a new approval. Tool-agnostic. + +### Ship by domain (what changed, per bounded context) +- command : `src/command/string/{string_read,string_write,mod}.rs` (GETDEL/GETSET/SET..GET/DECRBY/ + SETEX/SET-EX guards + `msetnx` handler + unit tests), `src/command/key.rs` (expire-past delete + + overflow guards + tests), `src/command/metadata.rs` (MSETNX phf entry), `src/command/mod.rs` + ((6,'m') MSETNX dispatch). +- shard : `src/shard/coordinator.rs` (`coordinate_msetnx` — CROSSSLOT reject / owner-atomic), + `src/server/conn/shared.rs` (`is_multi_key_command` MSETNX arm). +- persistence : `src/persistence/replay.rs` (expire-past-delete replay propagation tests). +- scripts : `scripts/test-consistency.sh` + `scripts/test-commands.sh` (MSETNX entries). +- tests : `tests/msetnx_cross_shard_reject.rs` (new `--shards 4` regression suite). + +### Cross-task evidence (one row per task) +- kv-wrongtype-guard : gate=PASS · commits 4a2245b + 5044487 · tests=3 unit (key-preserved) green +- kv-integer-overflow-guard : gate=PASS · commits 56008df + 9a915d3 + d6b4136 (i64-domain bound — review Finding 2/3) · + tests=3 unit (min-overflow, SETEX, SET EX) + 9 unit (i64-bound reject + extreme-negative preserve) green +- kv-expire-past-deletes : gate=PASS · commit 56008df · tests=unit + 2 replay-propagation green +- kv-msetnx-atomic : gate=PASS · commit 3b2c7dc (atomicity) + coordinator local-leg AOF durability fix + (Finding 1) · tests=3 unit + 2 integration (--shards 4, red/green-proven) + 3 crash-recovery + (coordinator_local_leg_durability, red/green on monoio+tokio) green + +### Adversarial code review (whole-diff, senior-rust-engineer agent — **SHIP** verdict) +The five milestone commits deliver their stated fixes with no new regressions or reachable +panics. Three findings surfaced (all independently re-verified against source before acting): +- **Finding 2/3 (MEDIUM/LOW) — FIXED in `d6b4136`.** The overflow guards bounded against + `u64::MAX`, not Redis's effective `i64::MAX`; a huge-but-sub-`u64` TTL (e.g. `EXPIRE k + 15000000000000000`) was accepted and then read back as a **negative** `PTTL`/`PEXPIRETIME` + on a live key, and an extreme-negative `EXPIRE` deleted instead of erroring. Now bounded to + the i64 domain at all eight expiry setters via a shared `expiry_ms_in_range` helper (red/green, + 9 tests). This makes the "not a wrapped TTL" exit criterion hold on the READ path too. +- **Finding 1 (HIGH) — PRE-EXISTING, FIXED for MSET/MSETNX (this milestone).** The cross-shard + coordinator's LOCAL leg (`coordinate_mset` fast-path + scatter local slice; `coordinate_msetnx` + local branch) executed co-located multi-key writes in memory but never appended them to the owning + shard's AOF — while the REMOTE leg (`MultiExecute` in `spsc_handler`) does (`wal_append_and_fanout`). + So a co-located `MSET`/`MSETNX` was durable when the owner was a *remote* shard but silently + **non-durable** when the owner was the connection's *own* shard. **Blast radius:** multi-shard + (`--shards >1`) + appendonly + keys hashing to the connection's own shard; single-shard (the + recommended default) was unaffected (the coordinator is bypassed for `num_shards<=1` and normal + dispatch persists). **Not introduced by MSETNX** (it copied the existing MSET coordinator pattern; + MSETNX's own exit criterion is *atomicity*, delivered — not durability). + **Fix:** the local leg now persists to the owning shard's AOF via a new `persist_local_leg` helper — + the same `AofWriterPool::issue_append_lsn` + `try_send_append_durable` path **every local single-key + write already uses** (matching the local-write contract, NOT the SPSC remote path; `ChannelMesh` has + no self-send slot, so a local leg cannot route through `wal_append_and_fanout`). Granularity: the + **whole command** for a co-located owner (MSETNX; MSET fast path), and a **synthesized MSET over only + the local keys** for a scattered MSET's local slice (never the full scattered command — `my_shard` + does not own the remote keys, and replay re-dispatches raw commands). On AOF failure the leg returns + `AOF_FSYNC_ERR` instead of a false `+OK` (design-for-failure). Red/green crash-recovery TDD in + `tests/coordinator_local_leg_durability.rs` (`--shards 4 --appendonly yes`; one co-located group per + shard so exactly one is the local leg regardless of SO_REUSEPORT landing; SIGKILL → restart → + reload): RED before (the connection's-shard group vanished), GREEN after, on **both monoio and + tokio**. The fix strictly *adds* durability and changes nothing about replication (live fan-out via + `replica_txs` is SPSC-only and untouched — not independently re-verified for local writes here). + **Remaining (tracked follow-up — same mechanism, out of this KV milestone's command scope):** BITOP / + COPY (via `run_on_owner`'s local branch) and DEL / UNLINK (via `coordinate_multi_del_or_exists`) + carry the *identical* local-leg non-durability and are NOT yet fixed; a focused follow-up should + route their local legs through `persist_local_leg` too. + +### Performance validation (GCloud A/B — the Finding-1 durability fix) +The Finding-1 fix awaits an fsync on the local leg, so its cost was measured A/B on GCE +`c2d-standard-16` (16 vCPU AMD EPYC 7B13, x86_64): `moon-post` (`cd7c51c`, fix present) vs +`moon-pre` (`cd7c51c^`, fix absent), md5-verified distinct — isolating *the fix's own cost* from +Moon's baseline-vs-Redis gap (a real fix cost shows consistently-signed negative; noise centers on +zero). **Ship-clean:** +- **`everysec` (default fsync): free** — the A/B straddles zero on every co-located + scatter cell, + inside the ±5% noise band. +- **`always`, no pipeline (P1): clean −8.8%** (one extra awaited fsync) — and Moon still **beats + Redis 1.27×** on this cell. +- **`always` + pipeline + co-located:** a far-tail limitation confined to that (non-default) + intersection — < 0.5% of writes reach multi-second latency; p50/p95/**p99 stay healthy (~28 ms)**. + The tail is a **bounded fsync-await stacked on upstream pipeline queue wait** (redis-benchmark + times send→reply, so it measures their *sum*): the 2000 ms `--aof-fsync-timeout-ms` bound + (`config.rs:129`, `pool.rs:262-268`) + queue wait against the *one* hot shard — n=50k ≈ 2023 ms + (bound-dominated), n=100k ≈ 3000 ms (2000 ms bound + ~1000 ms queue). **No silent data loss:** the + tail-most writes (await reaches the bound) fail *loudly* with `AOF_FSYNC_ERR`, never a false + `+OK`; server verified healthy post-run (PING/MSET/MGET correct, no panic/diskfull). + +**Follow-up (filed — HIGH):** route the coordinator local-leg persist through group-commit / +`fsync_barrier` so the single hot shard batches fsyncs under pipeline (keeps the far tail well under +the 2000 ms bound); the tracked BITOP/COPY/DEL/UNLINK gap should land on that same batched path. + +Full method + raw matrix: `tmp/V3-4-GCLOUD-BENCH.md` (local artifact). This validates *performance* +only — durability correctness is proven by the crash-recovery tests above. + +### Goal met? (map the evidence back to this milestone's Exit criteria — read before the Exit-criteria boxes are checked) +- [x] each Exit criterion above is satisfied by a Cross-task evidence row or a Ship-by-domain change (cited inline) +- goal: KV writes now honor Redis data-integrity contracts — full default lib suite 3633 green + tokio + feature set green + both clippy gates + fmt clean; no wrong-type deletes, no overflow panics/wraps, + past-expire deletes (and replays), MSETNX atomic with cross-shard reject. + +## Release steps (AI-DEFINED — fill the ordered steps to ship this milestone; engine records, human gate) +> The AI writes the release steps for THIS milestone here (hints, not engine commands). MERGE is one +> small step among them. These feed the release scope (release.md) when the cut is bundled. +- [ ] Add a CHANGELOG `[Unreleased]` entry (### Fixed — KV data-integrity; ### Added — MSETNX) — CI Lint gate. +- [ ] Open a PR from `fix/v3-4-kv-correctness` using the Close ship-review above; human reviews + merges. +- [ ] `add.py milestone-done v3-4-kv-correctness` once the Exit-criteria boxes are verified. +- [ ] Fold milestone deltas into the foundation on merge (per project convention); bundle into the next release cut. diff --git a/.add/state.json b/.add/state.json index 1c7a86176..818339a39 100644 --- a/.add/state.json +++ b/.add/state.json @@ -2,7 +2,7 @@ "project": "moon", "stage": "production", "active_task": null, - "active_milestone": null, + "active_milestone": "v3-4-kv-correctness", "tasks": { "hotpath-lock-quickwins": { "title": "Eliminate per-command global locks & syscall-level quick wins", @@ -235,10 +235,18 @@ "status": "planned", "created": "2026-06-16T04:42:42+00:00", "updated": "2026-06-16T04:42:42+00:00" + }, + "v3-4-kv-correctness": { + "title": "KV Write Correctness & Data-Integrity Parity", + "goal": "Moon's KV write commands honor Redis data-integrity contracts: no wrong-type command destroys data, integer commands reject overflow instead of panicking or wrapping, expire-in-the-past deletes the key, and MSETNX is atomic (cross-shard spans rejected).", + "stage": "production", + "status": "active", + "created": "2026-07-02T13:45:23+00:00", + "updated": "2026-07-02T13:45:23+00:00" } }, "created": "2026-06-11T03:18:21+00:00", - "updated": "2026-06-23T08:25:16+00:00", + "updated": "2026-07-02T13:45:23+00:00", "setup": { "locked": true, "locked_at": "2026-06-11T03:28:00+00:00", diff --git a/BENCHMARK.md b/BENCHMARK.md index 1b61bb0dc..dc6201b82 100644 --- a/BENCHMARK.md +++ b/BENCHMARK.md @@ -1,6 +1,6 @@ # moon Benchmark Report -**Last Updated:** 2026-06-16 (4-feature concurrent-vs-competitor pass added — §10.5 vector vs RediSearch, §11.4 graph vs FalkorDB, **new §12 Full-Text Search** vs RediSearch; honestly records where Moon trails. §2.8 = v2-1/PR #189 `db61973` K=1024 re-measurement; v0.1.6 in §2.1–2.6; §2.7 on perf/shard-dispatch-hot-path) +**Last Updated:** 2026-07-04 (**new §6.6: multi-shard multi-connection WIN — reply-convoy fix + slot-unified replies take s4 c8 P1 from 0.44–0.64× to 1.57–2.0× Redis and c64 to 2.5×, both arches**; supersedes §6.5's c≥8 rows. Prior 2026-07-03: **§2.10: p=1 single-op WIN on both arches via `--io-busy-poll-us` poll-mode park** — ARM c4a 1.19–1.21×, x86 c3 1.65–1.66× vs Redis, n=3 instances/arch; supersedes the "loses p=1" rows in §2.7–2.9 when busy-poll is on. **New §6.5: shards × busy-poll sweep** — busy-poll recovers +22–42% of the cross-shard hop but shards>1 still loses non-pipelined; `--shards 1 --io-busy-poll-us 40` is the p=1 config. Prior: 2026-06-16 4-feature concurrent-vs-competitor pass — §10.5 vector vs RediSearch, §11.4 graph vs FalkorDB, §12 Full-Text Search; §2.8 = v2-1/PR #189 `db61973` K=1024 re-measurement; v0.1.6 in §2.1–2.6; §2.7 on perf/shard-dispatch-hot-path) **Platforms:** Linux (GCloud x86_64 + ARM64), macOS (Apple M4 Pro) **Redis:** 8.6.1 in §2.1–2.6; 7.0.15 in §2.7 **moon:** v0.1.6 in §2.1–2.6; perf/shard-dispatch-hot-path HEAD (commit `6582fa9`) in §2.7. Monoio runtime (io_uring on Linux, kqueue on macOS), fat LTO, codegen-units=1, target-cpu=native @@ -38,6 +38,8 @@ The SET absolute number can differ 3-4× between methodologies. Only strict-vs-s | Metric | moon vs Redis | Conditions | |--------|:-------------------:|------------| +| p=1 single-op GET/SET (x86) | **1.65-1.66x Redis** | `--io-busy-poll-us 40`, GCE c3 dedicated, n=3, §2.10 | +| p=1 single-op GET/SET (ARM) | **1.19-1.21x Redis** | `--io-busy-poll-us 40`, GCE c4a dedicated, n=3, §2.10 | | Peak GET (Linux x86_64) | **5.11M ops/s (1.72x)** | GCloud c3-standard-8, P=64 | | Peak GET (Linux ARM64) | **3.47M ops/s (2.20x)** | GCloud t2a-standard-8, P=64 | | Peak GET (macOS) | **7.94M ops/s (2.59x)** | OrbStack, Apple M4 Pro, P=64 | @@ -280,6 +282,41 @@ Moon-fair vs Redis (ratio = Moon/Redis), 8-vCPU c3/t2a, best-of-3: Within VM variance of §2.8 (x86 loose P64 1.87–1.90×). Moon wins at pipeline depth, loses p=1 (TCP-RTT bound). No regression. +### 2.10 2026-07-03 p=1 single-op WIN — `--io-busy-poll-us` poll-mode park (branch `perf/v3-3-p1-hotpath`, `6178a71`) + +The historical "loses p=1" rows (§2.7–2.9 P1 columns, 0.77–0.83×) are **superseded when busy-poll is +enabled**. Root cause of the p=1 deficit was attributed by perf tracepoints on GCE: both engines +sleep+wake the server thread on every non-pipelined op; Redis's 3-syscall epoll loop rode that path +slightly cheaper than Moon's drivers (io_uring's 2 enters cost more than 3 light syscalls on GCE; +Moon-epoll paid 4 syscalls/op from a speculative-read EAGAIN). `--io-busy-poll-us <µs>` removes the +sleep entirely: the shard thread busy-loops zero-timeout readiness polls for the budget before +blocking (vendored-monoio LegacyDriver patch; flag forces the epoll driver). Redis keeps paying the +wake on every op — the win is structural, not a tuning delta. + +**Same-instance A/B (Moon-spin vs Moon-tuned-uring vs Redis 8.x control), GCE dedicated 4-vCPU, +pinned disjoint cores, steal-gated, fresh-server best-of-5, loopback c=1 P=1, spin=40µs, n=3 +instances per arch:** + +| arch | cmd | Moon busy-poll | Redis | ratio (3 instances) | +|------|-----|:---:|:---:|:---:| +| **ARM c4a Axion** | SET | ~75.3k ops/s (13.3µs/op) | ~63.0k (15.9µs/op) | **1.193 / 1.196 / 1.198** | +| **ARM c4a Axion** | GET | ~77.2k ops/s (12.9µs/op) | ~63.9k (15.6µs/op) | **1.206 / 1.205 / 1.212** | +| **x86 c3 Intel** | SET | ~62.0k ops/s (16.1µs/op) | ~37.4k (26.7µs/op) | **1.657 / 1.663 / 1.647** | +| **x86 c3 Intel** | GET | ~63.2k ops/s (15.8µs/op) | ~38.2k (26.2µs/op) | **1.657 / 1.656 / 1.646** | + +Sub-1% ratio spread across instances on both arches. Without busy-poll, Moon's best p=1 stance is +`--io-driver epoll`: 0.95× ARM / 1.06× x86 (same-instance A/Bs, 2026-07-03). + +Multi-client (pinned-core VM probe): busy-poll also wins c=8 (+6.5%) and c=64 (+27.6%, p50 159→87µs) +— under load the loop rarely parks, so the spin only fires where it helps. Idle cost: ~3% of a core +(vs 1% stock) at the 40µs budget against 1ms timer parks. + +**Caveats:** (1) judge busy-poll ONLY on pinned/dedicated cores — on unpinned shared-core hosts +(laptops, OrbStack defaults) the spinning thread displaces its own client and shows as a regression; +(2) claim scope is shards=1, loopback; (3) io_uring cannot busy-poll this way (CQE posting requires +owner-task participation — DEFER_TASKRUN, TWA_SIGNAL, and SQPOLL variants all measured worse; +experiment ledger in `tmp/KV-FULLPROOF.md` Round 2). + --- ## 3. Memory Efficiency @@ -499,6 +536,68 @@ non-pipelined workloads; add shards only for pipelined / AOF / hash-tag-co-locat degrades with shard count (0.93×→0.61× as 1→12) as its keys scatter cross-shard — `{hash-tag}` co-location restores it. Detail: `docs/reviews/2026-06-17/WIDER-BENCH.md`. +> **Clarification (2026-07-02 KV deep review).** The **0.46–0.51×** p=1 figure above is from `bench-production.sh`, which changes three things at once *besides* shard count: distributed keys (`-r`, real cross-shard scatter — vs `bench-compare.sh`'s single hot `__rand_key__`), larger values (512B–4KB), and higher client counts (`-c 100/200` on the INCR rows). It is a **throughput artifact of {distributed keys × high concurrency × value size × multi-key scatter}, not a clean shard-count signal** — the cross-shard hop itself is ~10µs (v2-2 bare-metal), ~2% of the ~460µs p=1 baseline. The controlled `bench-compare.sh` sweep in the table above (only shard count varies) is **flat at p=1** (0.79→0.82×). Read 0.46× as "production-shaped multi-key under concurrency," not "the cross-shard hop costs 2×." + +### 6.5 2026-07-03 shards × busy-poll sweep (`--measure-shards`, build `74de849`) + +Same-instance rigor (pinned, steal-gated, fresh-server best-of-3, strict `-r 1M` keyspace so +shards=4 really scatters), `c4a-standard-8` + `c3-standard-8`: Moon shards {1, 4} × `--io-busy-poll-us` +{0, 40} vs one canonical single-threaded Redis; shards on cores 0–3, 3-thread client on 5–7. +Ratio = Moon/Redis: + +| cell (cmd ≈ SET/GET) | ARM stock | ARM spin40 | x86 stock | x86 spin40 | +|---|:---:|:---:|:---:|:---:| +| **s1 c1 P1** (single-op latency) | 1.02 | **1.21 / 1.25** | 1.06 | **1.42 / 1.43** | +| **s4 c1 P1** (cross-shard hop) | 0.70 | 0.85 / 0.88 | 0.64 | 0.88 / 0.91 | +| **s4 c8 P1** (scattered, no pipeline) | 0.44 | 0.50 / 0.58 | 0.41 / 0.45 | 0.64 / 0.56 | +| **s4 c8 P16 SET** (Redis server-bound) | **≥2.0×** | **≥2.0×** | ~1.0 (capped) | **≥2.0×** | + +Readings: +- **The §2.10 busy-poll p=1 win replicates on the 8-vCPU shape** (5th ARM + 4th x86 instance). +- **Busy-poll recovers a large fraction of the cross-shard hop** (the target shard normally sleeps; + spin removes that wake): s4 single-conn improves +22% ARM / +38–42% x86 — but shards=4 still + **loses** every non-pipelined cell. The SPSC dispatch cost dominates once keys scatter. +- **Guidance unchanged, now sharper:** `--shards 1 --io-busy-poll-us 40` is the p=1 configuration + that beats Redis outright; add shards only for pipelined / AOF / hash-tag-co-located workloads. +- ⚠ **Deep-pipeline cells (P16/P64) on this co-located pinned topology are mostly client-saturated** + — both engines plateau at ~1.2M ops/s with identical durations (the 3-thread pinned client is the + bottleneck), so their ≈1.00 "ties" are ceilings, not measurements. Readable exceptions: Redis SET + P16 is server-bound at ~599k (Moon ≥2.0× there, understated), and x86 s4 spin P64 SET burst past + the plateau to 1.74M (1.46×). For whole-machine peak throughput, §2.8/§6.4's methodology (client + gets the full core budget) remains the reference. +- ⚠ **SUPERSEDED for c≥8 by §6.6:** the s4 c8 P1 collapse in this table was a reply-spin convoy + bug, fixed on `perf/v3-3-p1-hotpath`. Multi-shard now **wins** every multi-connection cell. + +### 6.6 2026-07-04 multi-connection WIN — convoy fix + slot-unified replies (branch `perf/v3-3-p1-hotpath`, `fd13f03`) + +The §6.5 s4-c8 collapse (0.44–0.64×) was diagnosed as a **reply-spin convoy**: the cross-shard +reply busy-poll ran synchronously on the shard thread with a gate that admitted two waiters, so +at 2 conns/shard a spinning connection starved both its sibling and the shard's own SPSC drain +(circular cross-shard stall). Fix chain, each same-instance A/B-validated on GCE: +solo-conn spin gate (`795c4f0`, c8 P1 **2.75×** vs pre-fix) → monoio reply path unified on the +zero-allocation `ResponseSlotPool` (`fd13f03`, c8 P1 **+11–14%** on top). Ratio = Moon/Redis, +s4, P1, busy-poll 40, 4-thread pinned client, best-of-5 × 500k requests, n=2 instances on x86: + +| cell | ARM c4a-standard-8 | x86 c3-standard-8 | +|---|:---:|:---:| +| **c8 GET / SET** | **1.57× / 1.71×** | **1.9× / 2.0×** (same-instance; **≥1.6×** vs Redis's best client config) | +| **c64 GET / SET** | **2.50×** | **2.50×** (same-instance; ≥1.86× best-vs-best) | +| c1 (structural ceiling = s1 latency) | 0.93–0.96 | 0.91–0.94 | + +Readings: +- **Guidance updated:** `--shards 1` remains best for 1–4 unpipelined connections, but from + **8 concurrent connections up, `--shards 4` beats Redis 1.5–2.5×** even without pipelining. + The pipeline moat (§6.5, §7) is unchanged. See `docs/guides/tuning.md`. +- s4 c1 P1 is **hop-bound, not fixable by tuning**: a perfect message-passing multi-shard equals + s1 latency (its ceiling is §2.10's s1 ratio). Executing foreign reads locally (shared-read data + plane) is the only lever; deferred as future work. +- ⚠ **Instrument notes:** raw files `tmp/hp-{c4a,c3}-l3b-ab.txt`, `tmp/hp-c3-l3b-ab2.txt`, + `tmp/redis-threads-control.txt`. redis-benchmark `--threads 4` lowers *Redis's* x86 c8/c64 + readings 15–20% vs `--threads 3` (Moon is thread-insensitive; ARM unaffected) — the x86 row + therefore also states the conservative ratio against Redis's best client config. Unpipelined + c64 readings quantize to attractors (333,333 / 500,000 = requests ÷ ms-quantized duration); + values are best-of-5 per cell. + --- ## 7. Persistence (AOF) Performance diff --git a/CHANGELOG.md b/CHANGELOG.md index 680606b64..5021bc190 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,101 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +Two milestones since v0.4.1: + +- **v3-4 KV Write Correctness & Data-Integrity Parity** — the primary KV engine now honors + Redis's data-integrity contracts (no wrong-type data loss, no integer-overflow panics/wraps, + past-expire deletes, atomic MSETNX). +- **p=1 & multi-shard throughput** — Moon now beats Redis on non-pipelined (p=1) GET/SET on both + GCE arches (ARM 1.19–1.21×, x86 1.65–1.66×), and multi-shard wins from 8 concurrent connections + up (s4 c8 1.57–2.0×, c64 2.50×), via a poll-mode park (`--io-busy-poll-us`, backed by a vendored + monoio fork) plus a cross-shard reply-path convoy fix. All perf gains are bench-only knobs / + opt-in flags — the default runtime behavior is unchanged unless you set `--io-busy-poll-us`. + +### Added + +- **MSETNX** — atomic multi-key string write (set all pairs iff none of the keys exist; returns 1/0). + Single-shard atomic (two-phase check-then-set, no await between phases). The cross-shard coordinator + rejects a key span with `CROSSSLOT` (by design — no two-phase commit) and runs co-located `{hash-tag}` + keys atomically on the owning shard. New `--shards 4` regression suite `tests/msetnx_cross_shard_reject.rs`. +- **`--io-busy-poll-us <µs>`** — poll-mode park for the monoio legacy (epoll/kqueue) driver: the shard + thread busy-polls readiness for the given budget before blocking, deleting the per-op scheduler + sleep+wake. This is the lever that flips non-pipelined (p=1) GET/SET to a win vs Redis on both GCE + arches. Defaults to `0` (off); implies `--io-driver epoll`. Costs up to budget-µs CPU per idle park — + only a win on pinned, disjoint cores (a regression on shared-core hosts). Env equivalent + `MOON_EPOLL_SPIN_US`. +- **`--io-driver `** for the monoio runtime — force the epoll/kqueue LegacyDriver instead of + io_uring (some platforms, e.g. GCE ARM Axion, run KV faster on epoll). +- **Per-use-case tuning guide** at `docs/guides/tuning.md` (quick recipes, shard-count guidance, + busy-poll trade-offs, persistence cost, platform notes), linked from the docs-site Operations nav. + +### Performance + +- **Multi-shard now wins from 8 concurrent connections up, even without pipelining.** Fixed the C2 + reply-spin *convoy*: the cross-shard reply busy-poll was synchronous on the shard thread and, at + ≥2 connections per shard, a spinning connection starved its sibling and the shard's SPSC drain — the + root cause of the `--shards 4` c8-P1 0.45× collapse. A solo-conn gate now spins only when a connection + is alone on its shard. Result (s4, P1, `--io-busy-poll-us 40`, vs Redis): c8 GET/SET ARM 1.57/1.71×, + x86 1.9/2.0×; c64 2.50× both arches. (c1-P1 remains the structural single-hop ceiling, 0.91–0.96×.) +- **monoio cross-shard replies unified on the zero-allocation `ResponseSlotPool`** (L3b), removing the + per-op flume-oneshot allocation and chunked park from the hot reply path (+11–14% at c8). +- **Non-pipelined GET reply framed from a borrow** — dropped a per-GET `Vec` copy. +- Stripped four per-op lock/atomic costs from the p=1 dispatch hot path. +- Tuned the default monoio io_uring driver (`COOP_TASKRUN | SINGLE_ISSUER | DEFER_TASKRUN`) on Linux. + +### Changed + +- Corrected default-config documentation: `--shards` default is **1** (not "0/auto") and `--appendonly` + default is **yes** (not "no") — both config pages were wrong. + +### Fixed + +- **Cross-shard reply use-after-free on panic-unwind (memory safety).** The cross-shard reply path + sent a raw `*const ResponseSlot` into a pool living on the connection task's stack; a panic unwinding + through the reply dispatch/drain would drop that pool while a target shard still held the pointer, + making the target's later `slot.fill()` a use-after-free (heap corruption). `ResponseSlotPtr` now + carries an `Arc`, so the slot outlives whichever of {connection pool, in-flight message} + drops last — structurally closing the window on both dispatch and drain. This also **retroactively + fixes the same latent hazard on the tokio runtime** (shipped since v0.4.0) and removes four `unsafe` + blocks (net unsafe reduction). Follow-up: a shutdown-aware bound on the reply await (now safe to add) + to close a partial-shutdown liveness hang. +- **`MOON_NO_URING=1` now actually switches the monoio driver.** It was a silent no-op for the monoio + FusionDriver (io_uring was selected regardless); it now forces the epoll/kqueue LegacyDriver, matching + its documented contract and the new `--io-driver epoll`. + +- **KV data-integrity (P0):** wrong-type `GETDEL` / `GETSET` / `SET … GET` no longer silently delete or + overwrite a key holding a non-string — they return `WRONGTYPE` and preserve the value (check-before-mutate). +- **Integer overflow (P0):** `DECRBY key -9223372036854775808` no longer panics; every expiry-setting + command (`SET EX/PX/EXAT`, `SETEX`, `PSETEX`, `EXPIRE`/`EXPIREAT`/`PEXPIRE`) now rejects a time whose + absolute expiry falls outside the `i64` millisecond domain with an "invalid expire time" error — + matching Redis (`when > LLONG_MAX/1000`). This closes a latent wrap where an accepted-but-huge TTL + (e.g. `EXPIRE k 15000000000000000`) surfaced as a **negative** `PTTL`/`PEXPIRETIME` on a live key, and + makes an extreme-negative `EXPIRE`/`EXPIREAT` (`< i64::MIN/1000`) error instead of deleting. +- **Expire-in-the-past semantics (P0):** `EXPIRE`/`PEXPIRE`/`EXPIREAT` with a non-positive or already-past + time now deletes the key and returns 1 (Redis parity); verified to propagate through command-based WAL + replay so replicas stay consistent. +- **Cross-shard coordinator local-leg durability (P0, pre-existing):** a co-located `MSET`/`MSETNX` whose + keys hash to the **connection's own shard** now appends to that shard's AOF. The coordinator's local + leg previously executed the write in memory but never persisted it (the remote `MultiExecute` leg + always did), so with `--shards >1 --appendonly yes` an own-shard co-located `MSET`/`MSETNX` could be + lost on crash. It now persists via the same append path every local single-key write uses — the whole + command for a co-located owner, and a synthesized `MSET` over **only the local keys** for a scattered + `MSET`'s local slice — returning an AOF error instead of a false `+OK` on append failure. + Crash-recovery verified on both monoio and tokio (`tests/coordinator_local_leg_durability.rs`). + Single-shard (the default) was never affected. The same local-leg gap remains for coordinator + `BITOP` / `COPY` / `DEL` / `UNLINK` (tracked follow-up). + +### Dependencies + +- **Vendored `monoio` 0.2.4** (`vendor/monoio`, wired via `[patch.crates-io]`; upstream git sha + `f7827ddd`, recorded in `vendor/monoio/.cargo_vcs_info.json`). A fork of upstream monoio 0.2.4 with a + bounded "moon patch" (marked `// moon patch`, confined to 4 files — `lib.rs`, `driver/mod.rs`, + `driver/uring/mod.rs`, `driver/legacy/mod.rs` — verified against pristine upstream) that adds the + env/CLI-gated poll-mode park behind `--io-busy-poll-us` and a per-thread spin-hook handshake. Adds + **zero new `unsafe`**; the dependency list is byte-identical to upstream (no added transitive deps); + behavior is byte-identical to upstream when the `MOON_*` spin knobs are unset. Introduced to enable + the p=1 poll-mode park. + ## [0.4.1] — 2026-06-23 Measure-only validation release — **no server behavior change**. Bundles the closed diff --git a/CLAUDE.md b/CLAUDE.md index 7a96d6217..4c0a626dd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,8 +90,11 @@ orb run -m moon-dev bash -c 'sudo apt-get update -qq && sudo apt-get install -y ## Environment Variables - `RUST_LOG=moon=debug` — enable tracing output (uses `tracing-subscriber` with `env-filter`) -- `MOON_NO_URING=1` — force-disable io_uring everywhere (monoio runtime + tokio bridge); used in CI/containers/WSL where io_uring is unavailable +- `MOON_NO_URING=1` — force-disable io_uring everywhere (monoio runtime + tokio bridge); used in CI/containers/WSL where io_uring is unavailable. ⚠ Before 2026-07 this env was a silent NO-OP for the monoio driver (FusionDriver picked io_uring regardless); it now forces the epoll/kqueue LegacyDriver. CLI equivalent: `--io-driver epoll`. Empirically verify the driver via `ls -l /proc//fd | grep io_uring` — monoio logs neither choice. **GCE ARM (c4a Axion): epoll beats io_uring by 2-4% at ALL pipeline depths for KV** (same-instance A/B 2026-07-03); other platforms favor io_uring — bench per platform. - `MOON_URING=1` — opt **into** the tokio→io_uring bridge. The bridge is **default-off under the tokio runtime** (it floods errors under load and can hang the accept loop); tokio shards run plain epoll/kqueue unless this is set. No effect on the monoio runtime, which always uses io_uring unless `MOON_NO_URING` is set. +- `MOON_EPOLL_SPIN_US=<µs>` / CLI `--io-busy-poll-us <µs>` — poll-mode park (vendored-monoio patch, `vendor/monoio`, grep "moon patch"): the shard thread busy-loops zero-timeout readiness polls for the budget before blocking, deleting the per-op scheduler sleep+wake. Legacy (epoll/kqueue) driver only — the CLI flag forces it; io_uring CQEs are NOT observable from userspace without task participation (DEFER_TASKRUN posts only inside enter; TWA_SIGNAL kicks cost ~µs; SQPOLL doesn't run poll task_work on 6.17 — all measured 2026-07-03). **This is what flipped GCE p=1 c1 to a WIN vs Redis: ARM c4a 0.95→1.19/1.21, x86 c3 1.06→1.66** (same-instance A/Bs). Costs up to budget-µs CPU per idle park (~4%/core at 40µs vs 1ms timer parks). ⚠ Unpinned/shared-core environments (OrbStack default, laptop) show spin as a REGRESSION — only judge it on pinned disjoint cores. +- `MOON_URING_SPIN_US`, `MOON_URING_SQPOLL[_CPU]`, `MOON_URING_PLAIN` — io_uring-side experiment gates kept as documented diagnostics; all dead ends for the p=1 path (see `tmp/KV-FULLPROOF.md` Round 2). +- `MOON_XSHARD_SPIN_BUDGET` / `MOON_XSHARD_SPIN_GATE` / `MOON_XSHARD_SPIN_MAX_CONNS` — diagnostic overrides for the C2 reply-side spin (`src/shard/slice.rs`; defaults 4096 iters / gate 2 / **solo-conn 1**). Budget `0` disables the spin entirely (the same-instance A/B knob that proved the c8P1 convoy). ⚠ The solo-conn ceiling (spin only when the conn is ALONE on its shard thread) is the L1 convoy fix — raising `MAX_CONNS` re-creates the s4 c8P1 collapse (a spinning conn starves its sibling AND the shard's SPSC drain, 0.45× vs Redis; fixed = 2.75× better, see `tmp/MULTISHARD-REDESIGN.md`). Bench-only knobs: never set in production. - `RUSTFLAGS="-C target-cpu=native"` — enable CPU-specific optimizations for benchmarking ## Key Design Decisions @@ -140,7 +143,7 @@ orb run -m moon-dev bash -c 'sudo apt-get update -qq && sudo apt-get install -y - Never hold a lock across `.await` points. - Replace `.read().unwrap()` / `.write().unwrap()` with `.read()` / `.write()` (parking_lot doesn't poison). - Per-shard locks only — no global locks on the write path. -- **monoio cross-thread wakers:** `monoio::spawn` creates `!Send` tasks; `Waker::wake()` from another OS thread does NOT reach them. Use the `pending_wakers: Rc>>` relay — connection handler registers its waker, event loop drains and wakes locally after SPSC processing. For cross-thread signalling, prefer `flume::bounded(1)` over custom atomic oneshots. +- **monoio cross-thread wakers:** `monoio::spawn` creates `!Send` tasks; `Waker::wake()` from another OS thread does NOT reach them. The cross-shard **reply** path therefore has the connection **await a `flume` oneshot directly** (the target shard sends the reply on it after executing). A `pending_wakers: Rc>>` relay is still swept each event-loop iteration (pub/sub + backpressure paths thread it), but it is **no longer the reply-wake mechanism** — the old "register your waker, event loop drains it after SPSC" design was retired when test `swf0` disproved its premise (see `event_loop.rs` ~1730). For cross-thread signalling, prefer `flume::bounded(1)` over custom atomic oneshots. ### Error Handling - All command errors return `Frame::Error(Bytes)` — no `Result` types in dispatch paths. diff --git a/Cargo.lock b/Cargo.lock index 51ff7e66a..0a5d89e99 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1757,8 +1757,6 @@ dependencies = [ [[package]] name = "monoio" version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3bd0f8bcde87b1949f95338b547543fcab187bc7e7a5024247e359a5e828ba6a" dependencies = [ "auto-const-array", "bytes", @@ -1842,6 +1840,7 @@ dependencies = [ "hyper", "hyper-tungstenite", "hyper-util", + "io-uring 0.6.4", "io-uring 0.7.11", "itoa", "levenshtein_automata", diff --git a/Cargo.toml b/Cargo.toml index 97969d779..85a4381c9 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -124,8 +124,24 @@ text-index = [ "dep:levenshtein_automata", ] +# Vendored monoio 0.2.4 (upstream git sha f7827ddd, see vendor/monoio/.cargo_vcs_info.json) +# with a bounded "moon patch": env/CLI-gated poll-mode park that busy-polls readiness +# before blocking, removing the per-op scheduler wakeup at p=1 (io_uring: +# MOON_URING_SPIN_US; legacy epoll/kqueue: MOON_EPOLL_SPIN_US / --io-busy-poll-us), +# plus a per-thread spin-hook handshake. Diff vs upstream is confined to FOUR files — +# src/lib.rs, src/driver/mod.rs, src/driver/uring/mod.rs, src/driver/legacy/mod.rs +# (search "moon patch"); adds zero new unsafe. Default-off => byte-identical behavior +# when the MOON_* spin knobs are unset. +[patch.crates-io] +monoio = { path = "vendor/monoio" } + [target.'cfg(target_os = "linux")'.dependencies] io-uring = "0.7" +# Same crate, 0.6 line: monoio 0.2.4's `uring_builder()` takes an +# `io_uring 0.6::Builder`, incompatible with our 0.7 above. Used only to pass +# tuned setup flags (COOP_TASKRUN / SINGLE_ISSUER / DEFER_TASKRUN) to the +# monoio runtime; must track monoio's own io-uring version. +io-uring-06 = { package = "io-uring", version = "0.6" } nix = { version = "0.31", features = ["net", "socket"] } [target.'cfg(unix)'.dependencies] diff --git a/docs/configuration.md b/docs/configuration.md index bb51e439e..ec0fd26de 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -13,7 +13,7 @@ All options are available as command-line flags. Run `moon --help` for the full |------|---------|-------------| | `--bind` | `127.0.0.1` | Bind address | | `--port` / `-p` | `6379` | Port to listen on | -| `--shards` | `0` (auto) | Number of shards (0 = CPU count) | +| `--shards` | `1` | Number of shards (`0` = auto-detect CPU count) | | `--databases` | `16` | Number of databases | | `--requirepass` | *(none)* | Require password authentication | | `--protected-mode` | `yes` | Reject non-loopback when no password set | @@ -22,8 +22,9 @@ All options are available as command-line flags. Run `moon --help` for the full | Flag | Default | Description | |------|---------|-------------| -| `--appendonly` | `no` | Enable AOF persistence (`yes`/`no`) | +| `--appendonly` | `yes` | Enable AOF persistence (`yes`/`no`) — Moon is durable by default | | `--appendfsync` | `everysec` | AOF fsync policy (`always`/`everysec`/`no`) | +| `--aof-fsync-timeout-ms` | `2000` | Bound on a write's wait for its fsync barrier under `always` (0 = unbounded) | | `--appendfilename` | `appendonly.aof` | AOF filename | | `--save` | *(none)* | RDB auto-save rules (e.g., `"3600 1 300 100"`) | | `--dir` | `.` | Directory for persistence files | @@ -105,6 +106,10 @@ All options are available as command-line flags. Run `moon --help` for the full | `--tcp-keepalive` | `300` | TCP keepalive interval in seconds (0 = disabled) | | `--slowlog-log-slower-than` | `10000` | Slowlog threshold in microseconds | | `--slowlog-max-len` | `128` | Maximum slowlog entries | +| `--io-driver` | `auto` | I/O driver: `auto` (io_uring on Linux, kqueue on macOS) or `epoll` | +| `--io-busy-poll-us` | `0` (off) | Busy-poll the I/O driver for N µs before parking. Large single-op latency win on **dedicated, pinned cores**; a regression on shared/oversubscribed hosts. See the [tuning guide](guides/tuning.md) | +| `--initial-keyspace-hint` | `0` | Pre-size the keyspace (e.g. `1000000`) to avoid rehash pauses during bulk loads | +| `--memory-arenas-cap` | `8` | Cap jemalloc arenas — lower (e.g. `2`) for small containers | | `--uring-sqpoll` | *(disabled)* | io_uring SQPOLL idle timeout in ms. Requires CAP_SYS_NICE. Linux only | ## Disk offload (tiered storage) @@ -141,10 +146,12 @@ All options are available as command-line flags. Run `moon --help` for the full ## Tips !!! note - Use `--shards 0` to auto-detect CPU count. Use `--shards 1` for benchmarking or when comparing per-key memory against Redis. + The default `--shards 1` gives the best single-operation latency and is the right choice for most deployments. Add shards when you have **many concurrent connections (8+)** or **pipelined/batched traffic** — see the [tuning guide](guides/tuning.md) for measured guidance. !!! tip Hash tags like `{tag}` in key names (e.g., `user:{1234}:name`) route all tagged keys to the same shard, eliminating cross-shard dispatch for MGET/MSET operations. !!! warning Testing with more than 1,000 concurrent clients may require `ulimit -n 65536`. At 5,000 clients with pipelining, connection drops can occur without it. + +For workload-specific recipes (cache, high-concurrency API, durable store, vector search, containers), see the **[tuning guide](guides/tuning.md)**. diff --git a/docs/guides/configuration.md b/docs/guides/configuration.md index fc7d1e5a1..9f2f8f682 100644 --- a/docs/guides/configuration.md +++ b/docs/guides/configuration.md @@ -30,7 +30,7 @@ Moon is configured through command-line flags and an optional Redis-style config | Flag | Default | Description | |------|---------|-------------| -| `--appendonly` | `no` | Enable append-only file persistence (`yes`/`no`) | +| `--appendonly` | `yes` | Enable append-only file persistence (`yes`/`no`) — Moon is durable by default | | `--appendfsync` | `everysec` | AOF fsync policy: `always`, `everysec`, or `no` | | `--appendfilename` | `appendonly.aof` | AOF filename | | `--save` | *(none)* | RDB auto-save rules (e.g., `"3600 1 300 100"`) | diff --git a/docs/guides/tuning.md b/docs/guides/tuning.md new file mode 100644 index 000000000..bd9bf64eb --- /dev/null +++ b/docs/guides/tuning.md @@ -0,0 +1,139 @@ +# Tuning Guide + +Moon's defaults are chosen for the most common deployment: **one application, a moderate +number of connections, durability on**. Out of the box you get single-shard operation +(best per-operation latency), AOF persistence with `everysec` fsync (measured ≈ zero +throughput cost at the default shard count), and conservative I/O settings that are safe +on shared or virtualized hardware. + +This page tells you when to move away from those defaults, per workload. Every +recommendation here is backed by measurements on dedicated-vCPU GCE instances +(ARM `c4a` Axion and x86 `c3` Sapphire Rapids, 2026-07); your numbers will vary but the +*direction* of each knob is portable. + +## Quick recipes + +| Workload | Recipe | +|---|---| +| Pure cache (no durability) | `--appendonly no --maxmemory --maxmemory-policy allkeys-lru` | +| Sessions / rate limiting (few conns, latency-sensitive) | defaults; add `--io-busy-poll-us 40` on dedicated cores | +| High-concurrency API backend (8+ conns) | `--shards 4` (+ busy-poll on dedicated cores) | +| Pipelined / batch ingest | `--shards 4` or more; pipeline depth ≥ 16 | +| Durable primary store | defaults (`--appendonly yes --appendfsync everysec`); `always` only if you accept the write-latency cost | +| Bulk load | `--initial-keyspace-hint ` | +| Container / CI / WSL | `--io-driver epoll --memory-arenas-cap 2` | +| Vector search | see [Vector search guide](../vector-search-guide.md); match quantization to dimension | + +## Shard count: the most important knob + +Moon shards its keyspace across independent per-core threads. A key owned by another +shard costs a cross-thread hop (~10 µs round trip), so **more shards is not automatically +faster** — it depends on how much concurrency and pipelining your traffic has: + +| Traffic shape | Best setting | Measured (vs Redis, GCE) | +|---|---|---| +| 1–4 connections, no pipelining | `--shards 1` (default) | 1.2× (ARM) – 1.66× (x86) with busy-poll | +| 8+ connections, no pipelining | `--shards 4` | 1.3–1.5× at 8 conns; 1.67–1.86× at 64 conns | +| Pipelined (depth ≥ 16) | `--shards 4`+ | break-even at depth 16; up to 2.8× at depth 128 | +| 1 connection on many shards | avoid | a single unpipelined conn pays the hop on ~every op (0.85–0.99×) | + +Rules of thumb: + +- Start with the default `--shards 1`. It wins whenever concurrency is low and is the + fair configuration for memory comparisons. +- Move to `--shards 4` when you serve **8 or more concurrent connections** or any + pipelined traffic. Don't exceed the number of physical cores; very high shard counts + hurt shallow workloads (dispatch overhead dominates). +- `--shards 0` auto-detects the CPU count — use it only on hosts dedicated to Moon + with genuinely concurrent traffic. +- **Co-locate multi-key operations with hash tags**: `user:{1234}:name` and + `user:{1234}:session` land on the same shard, so MGET/MSET/transactions on them never + pay a hop. + +## Busy-polling: single-op latency on dedicated cores + +`--io-busy-poll-us 40` makes each shard thread poll for new I/O for up to 40 µs before +sleeping, deleting the wakeup latency that otherwise dominates shallow request/response +traffic. This is the flag that takes single-connection GET/SET from below Redis parity +to **1.2× (ARM) / 1.66× (x86)**, and it compounds with multi-shard concurrency (8-conn +throughput +25% on top of the shard win). + +The trade-offs are explicit: + +- Costs up to the budget in CPU per idle park (~4%/core at 40 µs) — you are trading + idle CPU for latency. +- **Only enable it on pinned, dedicated cores** (bare metal, dedicated-vCPU cloud + instances). On shared/oversubscribed hosts (laptops, burstable VMs, busy Kubernetes + nodes) it *regresses* performance — the spin fights neighbors for the core. +- Values 20–100 µs behave similarly; 40 is a good default. `0` (default) disables it. + +## Persistence: what durability costs + +Measured on GCE with the default single shard: + +- **`everysec` (default): free.** Throughput is indistinguishable from `--appendonly no` + in steady state. There is no reason to turn AOF off for speed alone. +- **`always`: −9% at depth 1** on unpipelined SET (still 1.27× Redis with busy-poll), but + under *pipelined* writes each batch waits for its fsync barrier and tail latencies grow + to the `--aof-fsync-timeout-ms` bound (2 s default) plus queue time. Use `always` only + for genuinely fsync-per-write requirements, and avoid deep pipelines on that path. +- **`--appendonly no`** for pure caches: saves the disk I/O entirely and removes recovery + time. Pair with `--maxmemory` + `--maxmemory-policy allkeys-lru` (or `allkeys-lfu`). +- **Multi-shard + AOF note:** at `--shards ≥ 2` Moon currently writes both the AOF and + the per-shard WAL (~2.7× the disk *volume* of the data ingested; throughput is + unaffected — the tax is disk bandwidth/wear). If you run multi-shard as a cache, turn + `--appendonly no`; if you need durability, budget the disk accordingly. + +## Memory + +- `--maxmemory` is a whole-instance budget; shards share it elastically (a hot shard can + borrow headroom from cold ones automatically — no per-shard tuning needed, even under + heavily skewed key distributions). +- For **bulk loads**, `--initial-keyspace-hint 1000000` (or your expected key count) + pre-sizes the tables and avoids rehash pauses mid-load. +- In **small containers**, cap allocator arenas: `--memory-arenas-cap 2` (default 8). + Also size `--vec-warm-mmap-budget` down if you use vector search under a cgroup limit. +- Comparing per-key memory against Redis? Use `--shards 1` and a fresh server; RSS is a + high-water mark, so measure by loading a known keyspace, not by deltas. + +## Platform notes + +- **Linux** is the production target. The default `--io-driver auto` picks io_uring; + on some platforms plain epoll measures 2–4% *faster* for key-value traffic (we saw + this on GCE ARM Axion) — if you're chasing the last few percent, A/B `--io-driver + epoll` on your own hardware. In containers/WSL/older kernels where io_uring is + unavailable or blocked by seccomp, set `--io-driver epoll` (or `MOON_NO_URING=1`). +- **macOS** runs the full feature set via kqueue but is a development platform — don't + benchmark on it. +- **Pinning**: for latency-critical deployments, pin Moon's shard threads and your + client/proxy to disjoint cores (`taskset`/cpuset). Every latency number above assumes + no core sharing between client and server. + +## Client-side checklist + +- More than ~1,000 connections needs `ulimit -n 65536` (5,000 clients with pipelining + will drop connections without it). +- Connection pools: with `--shards 1`, a handful of pooled connections is enough to + saturate the server; with `--shards 4`, size the pool at 8+ so all shards stay busy. +- Pipelining is Moon's strongest regime — batch what you can. The advantage over Redis + *grows* with depth (per-shard AOF removes Redis's single-file serialization). +- Leave `--tcp-keepalive 300` (default) on; set `--timeout` only if you have + leak-prone clients. + +## Observability + +`--admin-port 9100` enables `/metrics` (Prometheus), `/healthz`, `/readyz`, and the web +console at `/ui/`. It is off by default; enabling it adds a small per-batch accounting +cost on cross-shard traffic — negligible for most deployments, but leave it off on +single-purpose benchmark rigs. + +## Vector search (FT.*) + +- `EF_RUNTIME` (per index) trades recall for QPS at query time. +- Set `COMPACT_THRESHOLD` at or above your expected dataset size if you want a single + final compaction; explicit `FT.COMPACT` on a small mutable segment is a no-op below + the threshold. +- Match quantization to dimension: **SQ8** (or full-precision HNSW) for ≤ 384-d + embeddings; **TQ4** shines at 768-d and above. Validate recall with real embeddings, + not random vectors. +- Details: [Vector search guide](../vector-search-guide.md). diff --git a/mkdocs.yml b/mkdocs.yml index 796be97ba..bf3bb883a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -153,6 +153,7 @@ nav: - Operations: - Operator guide: OPERATOR-GUIDE.md - Production guide: production-guide.md + - Tuning guide: guides/tuning.md - Production contract: PRODUCTION-CONTRACT.md - Memory reclamation: - Overview: operations/README.md diff --git a/scripts/bench-compare.sh b/scripts/bench-compare.sh index 30f533e63..3a6f14651 100755 --- a/scripts/bench-compare.sh +++ b/scripts/bench-compare.sh @@ -16,6 +16,7 @@ PORT_MOON=6400 REQUESTS=100000 CLIENTS=50 SHARDS=1 +KEYSPACE="${KEYSPACE:-}" # -r random keyspace; empty = redis-benchmark default (1 hot key). Set for large-scale. RUST_BINARY="./target/release/moon" REDIS_PID="" @@ -64,7 +65,9 @@ parse_rps() { bench() { local port="$1" shift - redis-benchmark -p "$port" -n "$REQUESTS" -c "$CLIENTS" -q "$@" 2>/dev/null | parse_rps + local rflag=() + [[ -n "$KEYSPACE" ]] && rflag=(-r "$KEYSPACE") # large-scale: spread ops across a real keyspace + redis-benchmark -p "$port" -n "$REQUESTS" -c "$CLIENTS" -q "${rflag[@]}" "$@" 2>/dev/null | parse_rps } bench_cmd() { diff --git a/scripts/bench-resources.sh b/scripts/bench-resources.sh index 78163a2d4..65417ab67 100755 --- a/scripts/bench-resources.sh +++ b/scripts/bench-resources.sh @@ -83,7 +83,10 @@ start_servers() { --loglevel warning --daemonize no &>/dev/null & REDIS_PID=$! - "$RUST_BINARY" --port "$PORT_RUST" --shards "$SHARDS" &>/dev/null & + # Fair vs Redis (--save "" --appendonly no): Moon defaults appendonly=yes AND disk-offload=enable, + # which adds AOF buffers + cold-tier bookkeeping to RSS that Redis has no analog for — unfairly + # inflating Moon's memory in a per-key comparison. Match Redis: both persistence subsystems off. + "$RUST_BINARY" --port "$PORT_RUST" --shards "$SHARDS" --appendonly no --disk-offload disable &>/dev/null & RUST_PID=$! wait_for_port "$PORT_REDIS" diff --git a/scripts/gcloud-kv-p1-baseline.sh b/scripts/gcloud-kv-p1-baseline.sh new file mode 100644 index 000000000..2b4d6579d --- /dev/null +++ b/scripts/gcloud-kv-p1-baseline.sh @@ -0,0 +1,742 @@ +#!/usr/bin/env bash +# gcloud-kv-p1-baseline.sh — Moon vs Redis, shard=1, p=1 c1 single-op GET/SET latency, +# on GCloud DEDICATED-vCPU (steal-gated) instances, LOOPBACK, cross-arch (x86 + ARM). +# +# Goal (2026-07-03): ship the KV store at shard=1 that OUTPERFORMS Redis on p=1 +# single-op latency, proven on GCloud dedicated/bare-metal, on both ARM and x86. +# This is the BASELINE instrument: does clean shipped Moon already beat Redis at p=1 +# once shared-vCPU steal (the factor the KV deep review blamed) is removed? +# +# Topology = LOOPBACK (client + server co-located over lo). This is the ONLY topology +# where "our KV code beats Redis at p=1" is a code claim; cross-host p=1 measures TCP +# RTT (~50-150us), which both engines pay equally and which swamps the ~1.8us compute +# delta. Same topology as the OrbStack run where Moon won -> apples-to-apples, minus steal. +# +# Fair-config discipline (traps from prior GCloud runs, baked in): +# * Both engines AOF-OFF (Moon --appendonly no ; Redis --appendonly no --save ""). +# * Canonical builds: Moon `cargo build --release` (shipped profile), Redis `make` (-O2). +# No target-cpu=native (a bench-special flag, not shipped) -> generic vs generic, fair. +# * ONE redis-benchmark (from the Redis source build) drives BOTH engines -> same client. +# * Real keyspace (-r) not __rand_key__ ; SET-then-GET so GET is hit-path. +# * Fresh-server-per-rep, best-of-N, quiesce-per-rep, dedicated-core steal gate. +# * ELF-magic assert on the built moon binary (Mach-O trap). +# +# Measure-only: NEVER edits moon/src. Builds a clean committed ref read-only. +# +# Subcommands: +# --self-test gate + verdict logic against synthetic inputs (NO GCloud, NO cost). +# --measure INNER measurement; assumes it runs ON a suitably pinned Linux host. +# --gcloud (default) OUTER sweep: for each MACHINES entry provision/build/measure/teardown. +# --one OUTER for a single $GCE_MACHINE. +set -uo pipefail + +# ---------------------------------------------------------------------------- config +MOON_REF="${MOON_REF:-main}" # clean committed ref to baseline (NOT the dirty tree) +REDIS_VER="${REDIS_VER:-7.4.2}" +BEST_OF_N="${BEST_OF_N:-5}" +REQUESTS="${REQUESTS:-500000}" # redis-benchmark -n ; ~5-10s/rep at c1p1 +KEYSPACE="${KEYSPACE:-1000000}" # -r (real keys) +MOON_PORT="${MOON_PORT:-7501}" +REDIS_PORT="${REDIS_PORT:-7502}" +LOAD_MAX="${LOAD_MAX:-0.70}" +QUIESCE_DEADLINE="${QUIESCE_DEADLINE:-180}" +STEAL_MAX="${STEAL_MAX:-1}" # vCPU steal-% gate (~0 on dedicated cores) +NOISE_PCT="${NOISE_PCT:-8}" # tie-band + control flatness tolerance +SERVER_CORE="${SERVER_CORE:-0}" # single shard -> single core +CLIENT_CORE="${CLIENT_CORE:-2}" # disjoint client core (cores 1,3 idle buffer) + +# --- throughput mode (--measure-tput): rigorous pipeline-depth sweep, same pin/gate/best-of-N rigor as p=1 --- +# Motivation: unpinned single-run bench-compare cells are confounded (client/server core contention). +# This reuses the p=1 primitives (pinned disjoint cores, steal gate, fresh-server best-of-N) but sweeps +# pipeline depth so "faster per core at depth" is a CLEAN claim, not a scheduling artifact. +PIPES="${PIPES:-1}" # -P sweep (space-sep). p1 mode leaves this at 1 (unused). +CLIENTS_SWEEP="${CLIENTS_SWEEP:-1}" # -c sweep. c=1 = cleanest per-core saturation; add 50 for concurrency. +BENCH_PIPE="${BENCH_PIPE:-1}" # current -P for bench_pair (measure_tput sets per depth; p1 default 1) +BENCH_CLIENTS="${BENCH_CLIENTS:-1}" # current -c for bench_pair (measure_tput sets per client; p1 default 1) +# PERSIST=memory (pure in-memory, both AOF-off — the fair speed/latency bar) | aof (both --appendonly yes +# --appendfsync everysec, per-engine temp --dir — the "durable" comparison). At shards=1, Moon AOF is AOF-only +# (per_shard_aof_active needs >=2) so NO WAL double-write. Redis AOF-on everysec = the matching config. +PERSIST="${PERSIST:-memory}" + +# --- shard-scaling mode (--measure-shards): shards x busy-poll sweep on an 8-vCPU instance --- +SHARDS_SWEEP="${SHARDS_SWEEP:-1 4}" # moon --shards cells +SPIN_SWEEP="${SPIN_SWEEP:-0 40}" # --io-busy-poll-us cells (0 = off) +SHARD_CLIENT_CORES="${SHARD_CLIENT_CORES:-5-7}" # client core range (server takes 0..shards-1) +BENCH_THREADS="${BENCH_THREADS:-}" # client --threads for multi-conn cells (set per combo) + +# machines: x86 FIRST (cheapest decisive case), ARM second (config flip) +MACHINES="${MACHINES:-c3-standard-4}" # ARM run: MACHINES=c4a-standard-4 (or t2a-standard-4) +GCE_MACHINE="${GCE_MACHINE:-c3-standard-4}" +GCE_NAME="${GCE_NAME:-moon-kv-p1}" +GCE_ZONE="${GCE_ZONE:-us-central1-a}" +REPO_URL="${REPO_URL:-https://github.com/pilotspace/moon.git}" +OUT="${OUT:-tmp/KV-P1-BASELINE.md}" + +log() { printf '%s\n' "$*" >&2; } +die() { log "FATAL: $*"; exit 1; } + +# ============================================================================= pure gates + verdict +# Value-in / 0=ok so --self-test drives them with synthetic inputs (no GCloud). +gate_quiesce() { awk -v a="$1" -v b="$LOAD_MAX" 'BEGIN{exit !(a/dev/null 2>&1 && return 1 + pgrep -x 'redis-server' >/dev/null 2>&1 && return 1 + return 0 +} +require_floor() { [[ "${1:-0}" -ge "$BEST_OF_N" ]]; } # ok: full best-of-N sample +control_flat() { # ok: every control value within NOISE_PCT of the first + local ref="$1"; shift + [[ -z "$ref" || "$ref" -le 0 ]] 2>/dev/null && return 1 + local v + for v in "$@"; do + awk -v r="$ref" -v c="$v" -v n="$NOISE_PCT" 'BEGIN{ d=(r>c?r-c:c-r); exit !(d*100.0/r <= n) }' || return 1 + done + return 0 +} +us_per_op() { awk -v r="${1:-0}" 'BEGIN{ if(r>0) printf "%.2f", 1000000.0/r; else printf "n/a" }'; } +ratio() { awk -v m="${1:-0}" -v r="${2:-0}" 'BEGIN{ if(r>0) printf "%.3f", m/r; else printf "n/a" }'; } +# verdict -> WIN | LOSS | TIE (tie-band = +/- NOISE_PCT around parity) +verdict() { + awk -v m="${1:-0}" -v r="${2:-0}" -v n="$NOISE_PCT" 'BEGIN{ + if (r<=0 || m<=0) { print "n/a"; exit } + hi=1.0+n/100.0; lo=1.0-n/100.0; x=m/r; + if (x>=hi) print "WIN"; else if (x<=lo) print "LOSS"; else print "TIE"; + }' +} + +# ============================================================================= self-test +self_test() { + local fails=0 + _ok() { printf ' ok %s\n' "$1" >&2; } + _bad() { printf ' FAIL %s\n' "$1" >&2; fails=$((fails+1)); } + log "=== gcloud-kv-p1-baseline --self-test (gate + verdict logic; no GCloud) ===" + + gate_quiesce 1.40 && _bad "quiesce should VOID at load 1.40" || _ok "quiesce VOIDs high load" + gate_quiesce 0.20 && _ok "quiesce passes low load" || _bad "quiesce should pass load 0.20" + gate_steal 7 && _bad "steal should VOID at 7%" || _ok "steal VOIDs 7% (shared/contended)" + gate_steal 0 && _ok "steal passes ~0% (dedicated)" || _bad "steal should pass 0%" + gate_clean "planted-redis-server" && _bad "clean should refuse a planted proc" || _ok "clean refuses planted proc" + require_floor 1 && _bad "require_floor should reject 1 rep" || _ok "require_floor rejects single rep" + require_floor "$BEST_OF_N" && _ok "require_floor accepts full best-of-N" || _bad "require_floor should accept N" + control_flat 100000 101000 99000 && _ok "control_flat passes within noise" || _bad "control_flat should pass" + control_flat 100000 101000 70000 && _bad "control_flat should VOID a 30% drop" || _ok "control_flat VOIDs non-flat" + [[ "$(us_per_op 100000)" == "10.00" ]] && _ok "us_per_op 100000 -> 10.00us" || _bad "us_per_op 100000 wrong ($(us_per_op 100000))" + [[ "$(ratio 110000 100000)" == "1.100" ]] && _ok "ratio 110k/100k -> 1.100" || _bad "ratio wrong ($(ratio 110000 100000))" + # verdict: the goal predicate. WIN only beyond the noise band. + [[ "$(verdict 110000 100000)" == "WIN" ]] && _ok "verdict 110k vs 100k -> WIN" || _bad "verdict should WIN ($(verdict 110000 100000))" + [[ "$(verdict 100000 110000)" == "LOSS" ]] && _ok "verdict 100k vs 110k -> LOSS" || _bad "verdict should LOSS ($(verdict 100000 110000))" + [[ "$(verdict 103000 100000)" == "TIE" ]] && _ok "verdict 103k vs 100k -> TIE (within noise)" || _bad "verdict should TIE ($(verdict 103000 100000))" + [[ "$(verdict 100000 0)" == "n/a" ]] && _ok "verdict guards div-by-zero" || _bad "verdict should n/a on redis=0" + grep -q 'p=1' "$0" && grep -q 'LOOPBACK' "$0" && _ok "harness declares p=1 loopback intent" || _bad "harness must declare intent" + + if [[ "$fails" -eq 0 ]]; then log "=== self-test PASS (all gates + verdict fail-closed correctly) ==="; return 0; fi + log "=== self-test FAIL ($fails) ==="; return 1 +} + +# ============================================================================= measurement core +WORK="${WORK:-$HOME/moon-kv-p1}" +BINS="${BINS:-$HOME/moon-kv-p1-bins}" +REDIS_DIR="${REDIS_DIR:-$HOME/redis-$REDIS_VER}" +REDIS_SERVER="$REDIS_DIR/src/redis-server" +REDIS_BENCH="$REDIS_DIR/src/redis-benchmark" +REDIS_CLI="$REDIS_DIR/src/redis-cli" +MOON_PID=""; REDIS_PID="" +cleanup_moon() { [[ -n "$MOON_PID" ]] && { kill "$MOON_PID" 2>/dev/null||true; wait "$MOON_PID" 2>/dev/null||true; MOON_PID=""; }; return 0; } +cleanup_redis() { [[ -n "$REDIS_PID" ]] && { kill "$REDIS_PID" 2>/dev/null||true; wait "$REDIS_PID" 2>/dev/null||true; REDIS_PID=""; }; return 0; } +cleanup() { cleanup_moon; cleanup_redis; return 0; } +current_steal() { LC_ALL=C vmstat 1 2 2>/dev/null | tail -1 | awk '{print $(NF)}'; } # 'st' column +current_load() { awk '{print $1}' /proc/loadavg; } +wait_quiesced() { + local deadline=$((SECONDS+QUIESCE_DEADLINE)) l + while :; do + l=$(current_load) + gate_quiesce "$l" && return 0 + [[ $SECONDS -ge $deadline ]] && { log " WARN load $l >= $LOAD_MAX after ${QUIESCE_DEADLINE}s; proceeding"; return 0; } + sleep 2 + done +} + +prepare_repo() { + if [[ ! -d "$WORK/.git" ]]; then log "=== clone $REPO_URL -> $WORK ==="; git clone -q "$REPO_URL" "$WORK" || die "clone failed"; fi + git -C "$WORK" fetch --all -q 2>/dev/null || true +} + +build_moon() { # -> echoes binary path (read-only checkout; never edits src) + local out="$BINS/moon-${MOON_REF//\//_}" + [[ -x "$out" ]] && { echo "$out"; return 0; } + git -C "$WORK" checkout -q "$MOON_REF" || die "checkout $MOON_REF" + log " moon built-from: $(git -C "$WORK" rev-parse --short HEAD) $(git -C "$WORK" log -1 --format=%s | head -c 50)" + ( cd "$WORK" && CARGO_TARGET_DIR="$WORK/target" cargo build --release >/dev/null 2>&1 ) || die "moon build failed" + mkdir -p "$BINS"; cp "$WORK/target/release/moon" "$out" + # ELF-magic assert (Mach-O trap): first 4 bytes must be 7f 45 4c 46. + local magic; magic=$(od -An -tx1 -N4 "$out" | tr -d ' ') + [[ "$magic" == "7f454c46" ]] || die "moon binary is not ELF (magic=$magic) — stale Mach-O?" + echo "$out" +} + +build_redis() { # canonical Redis from source (redis-server + redis-benchmark + redis-cli) + [[ -x "$REDIS_SERVER" && -x "$REDIS_BENCH" ]] && return 0 + log "=== building Redis $REDIS_VER from source ===" + ( cd "$HOME" && curl -fsSL "https://download.redis.io/releases/redis-${REDIS_VER}.tar.gz" -o "redis-${REDIS_VER}.tar.gz" \ + && tar xzf "redis-${REDIS_VER}.tar.gz" && cd "redis-${REDIS_VER}" && make -j"$(nproc)" BUILD_TLS=no >/dev/null 2>&1 ) \ + || die "redis build failed" + [[ -x "$REDIS_SERVER" && -x "$REDIS_BENCH" ]] || die "redis binaries missing after build" +} + +start_moon() { # start_moon (shard=1, pinned, loopback; PERSIST=memory|aof) + local bin="$1"; cleanup_moon + local dir; dir="$(mktemp -d /tmp/moon-kvp1.XXXXXX)" + # Fair vs stock Redis. Moon defaults appendonly=yes AND disk-offload=enable; leaving offload on + # makes Moon do cold-tier bookkeeping Redis has no analog for (KV review §117) -> always disable it. + local pflags + if [[ "$PERSIST" == aof ]]; then + # AOF-on everysec, fair vs Redis --appendonly yes --appendfsync everysec. shards=1 => AOF-only + # (per_shard_aof_active needs >=2) => no WAL double-write. + pflags=(--appendonly yes --appendfsync everysec --disk-offload disable) + else + # pure in-memory: appendonly=no + no --save => persistence_dir=None => no WAL, no AOF (main.rs:765). + pflags=(--appendonly no --disk-offload disable) + fi + # MOON_START_ENV: optional space-separated KEY=VAL env for the server process + # (diagnostics, e.g. MOON_START_ENV="MOON_NO_URING=1" for an epoll-driver A/B). + # MOON_SHARDS / SERVER_CORES / MOON_EXTRA_FLAGS: multi-shard cells + # (--measure-shards) override shard count, core mask, and add flags such as + # --io-busy-poll-us; defaults preserve the original shards=1 single-core path. + # shellcheck disable=SC2086 + taskset -c "${SERVER_CORES:-$SERVER_CORE}" env ${MOON_START_ENV:-} "$bin" --port "$MOON_PORT" \ + --shards "${MOON_SHARDS:-1}" --dir "$dir" \ + "${pflags[@]}" ${MOON_EXTRA_FLAGS:-} --admin-port 0 >/dev/null 2>&1 & + MOON_PID=$! + local deadline=$((SECONDS+10)) + until "$REDIS_CLI" -p "$MOON_PORT" ping >/dev/null 2>&1; do + kill -0 "$MOON_PID" 2>/dev/null || { log " ERROR moon died on start"; return 1; } + [[ $SECONDS -ge $deadline ]] && { log " ERROR moon did not start"; return 1; } + sleep 0.1 + done + # io_uring self-report (once/run): monoio FusionDriver silently falls back to epoll if io_uring is + # unavailable and logs NEITHER choice -> assert the driver empirically via /proc//fd. A live + # io_uring driver shows as anon_inode:[io_uring]. Absence => epoll fallback => p=1 numbers are a + # driver confound, not an arch effect (ruled out on aarch64 proxy 2026-07-03; c4a self-confirms here). + if [[ -z "${URING_REPORTED:-}" ]]; then + URING_REPORTED=1 + if ls -l "/proc/$MOON_PID/fd" 2>/dev/null | grep -q 'anon_inode:\[io_uring\]'; then + log " moon io_uring driver: ACTIVE (anon_inode:[io_uring] fd present)" + else + log " moon io_uring driver: NOT FOUND (epoll fallback?) — treat p=1 as a driver confound" + fi + fi +} + +start_redis() { # canonical Redis, pinned, loopback; PERSIST=memory|aof + cleanup_redis + local pflags + if [[ "$PERSIST" == aof ]]; then + local rdir; rdir="$(mktemp -d /tmp/redis-kvp1.XXXXXX)" + pflags=(--appendonly yes --appendfsync everysec --dir "$rdir" --save '') + else + pflags=(--appendonly no --save '') + fi + taskset -c "$SERVER_CORE" "$REDIS_SERVER" --port "$REDIS_PORT" \ + "${pflags[@]}" --protected-mode no --daemonize no >/dev/null 2>&1 & + REDIS_PID=$! + local deadline=$((SECONDS+10)) + until "$REDIS_CLI" -p "$REDIS_PORT" ping >/dev/null 2>&1; do + kill -0 "$REDIS_PID" 2>/dev/null || { log " ERROR redis died on start"; return 1; } + [[ $SECONDS -ge $deadline ]] && { log " ERROR redis did not start"; return 1; } + sleep 0.1 + done +} + +bench_pair() { # bench_pair -> "SET_rps|GET_rps" (real keyspace, SET-then-GET hit-path) + # -c/-P come from BENCH_CLIENTS/BENCH_PIPE (default 1/1 = the p=1 single-op path). + # BENCH_THREADS (optional): client threads for high-throughput multi-shard + # cells where a single client thread would be the bottleneck. + # shellcheck disable=SC2086 + local out; out=$(taskset -c "$CLIENT_CORE" "$REDIS_BENCH" -p "$1" -c "$BENCH_CLIENTS" -P "$BENCH_PIPE" -n "$REQUESTS" \ + ${BENCH_THREADS:+--threads "$BENCH_THREADS"} \ + -r "$KEYSPACE" -t set,get --csv 2>/dev/null | tr '\r' '\n') + local s g + s=$(printf '%s\n' "$out" | grep '"SET"' | awk -F',' '{gsub(/"/,"",$2); printf "%.0f\n",$2}' | tail -1) + g=$(printf '%s\n' "$out" | grep '"GET"' | awk -F',' '{gsub(/"/,"",$2); printf "%.0f\n",$2}' | tail -1) + printf '%s|%s\n' "${s:-0}" "${g:-0}" +} + +# cell_engine -> emits "engine|SET|best|reps|n" and "engine|GET|best|reps|n" +cell_engine() { + local engine="$1" bin="$2" svals=() gvals=() n=0 attempts=0 pair s g + local max_attempts=$((BEST_OF_N*3)) + while [[ "$n" -lt "$BEST_OF_N" && "$attempts" -lt "$max_attempts" ]]; do + attempts=$((attempts+1)) + wait_quiesced + if [[ "$engine" == "moon" ]]; then start_moon "$bin" || { sleep 2; continue; } + else start_redis || { sleep 2; continue; }; fi + pair="$(bench_pair "$([[ "$engine" == moon ]] && echo "$MOON_PORT" || echo "$REDIS_PORT")")" + cleanup + s="${pair%%|*}"; g="${pair##*|}" + [[ -z "$s" || "$s" -le 0 || -z "$g" || "$g" -le 0 ]] 2>/dev/null && { log " retry $engine (bad rep: '$pair')"; continue; } + svals+=("$s"); gvals+=("$g"); n=$((n+1)) + done + local sbest=0 gbest=0 + [[ "$n" -gt 0 ]] && sbest=$(printf '%s\n' "${svals[@]}" | sort -n | tail -1) + [[ "$n" -gt 0 ]] && gbest=$(printf '%s\n' "${gvals[@]}" | sort -n | tail -1) + printf '%s|SET|%s|%s|%s\n' "$engine" "$sbest" "$(IFS=,;echo "${svals[*]:-}")" "$n" + printf '%s|GET|%s|%s|%s\n' "$engine" "$gbest" "$(IFS=,;echo "${gvals[*]:-}")" "$n" +} + +measure() { + command -v taskset >/dev/null || die "taskset not available" + prepare_repo + build_redis + trap cleanup EXIT INT TERM + + local steal; steal=$(current_steal); steal="${steal:-0}" + if ! gate_steal "$steal"; then + printf 'status: VOID\nreason: contended_instrument\ndetail: vCPU steal %%=%s > %s (not dedicated)\n' "$steal" "$STEAL_MAX" + die "VOID contended_instrument (steal=$steal%)" + fi + if ! gate_clean; then + printf 'status: VOID\nreason: dirty_instrument\ndetail: a stray moon/redis process is alive\n' + die "VOID dirty_instrument" + fi + log "=== gates OK (steal=$steal%, clean) — measuring on $(uname -srm) ===" + + local moonbin; moonbin=$(build_moon) || die "moon build" + log " settling compile heat..."; wait_quiesced + + echo "# kv-p1-baseline raw (loopback, shard=1, c1, P1, SET-then-GET hit-path)" + echo "# machine|engine|cmd|best_rps|us_per_op|reps|n" + declare -A BEST N + local engine line eng cmd best reps nn + for engine in moon redis; do + while IFS='|' read -r eng cmd best reps nn; do + BEST["$eng|$cmd"]="$best"; N["$eng|$cmd"]="$nn" + printf '%s|%s|%s|%s|%s|%s|%s\n' "$GCE_MACHINE" "$eng" "$cmd" "$best" "$(us_per_op "$best")" "$reps" "$nn" + done < <( cell_engine "$engine" "${moonbin:-"-"}" ) + done + + # validity: full sample on all four cells + local c + for c in "moon|SET" "moon|GET" "redis|SET" "redis|GET"; do + require_floor "${N[$c]:-0}" || { printf 'status: VOID\nreason: unstable_sample\ndetail: %s only %s reps\n' "$c" "${N[$c]:-0}"; die "VOID unstable_sample ($c)"; } + done + + echo "# verdict (Moon vs Redis, p=1 c1, dedicated loopback):" + echo "# machine|cmd|moon_rps|redis_rps|moon_us|redis_us|ratio|verdict" + for cmd in SET GET; do + local m="${BEST["moon|$cmd"]:-0}" r="${BEST["redis|$cmd"]:-0}" + printf '%s|%s|%s|%s|%s|%s|%s|%s\n' "$GCE_MACHINE" "$cmd" "$m" "$r" \ + "$(us_per_op "$m")" "$(us_per_op "$r")" "$(ratio "$m" "$r")" "$(verdict "$m" "$r")" + done +} + +# measure_driver: SAME-INSTANCE driver A/B — moon(io_uring) vs moon(epoll via MOON_NO_URING=1, +# requires the monoio LegacyDriver honor-fix) vs redis control. Same-instance kills the ~2.5% +# c4a instance-to-instance variance that makes two-instance driver deltas unreadable. +measure_driver() { + command -v taskset >/dev/null || die "taskset not available" + prepare_repo + build_redis + trap cleanup EXIT INT TERM + + local steal; steal=$(current_steal); steal="${steal:-0}" + gate_steal "$steal" || { printf 'status: VOID\nreason: contended_instrument\n'; die "VOID steal=$steal%"; } + gate_clean || { printf 'status: VOID\nreason: dirty_instrument\n'; die "VOID dirty"; } + log "=== gates OK (steal=$steal%, clean) — driver A/B on $(uname -srm) ===" + + local moonbin; moonbin=$(build_moon) || die "moon build" + log " settling compile heat..."; wait_quiesced + + echo "# kv-p1 driver A/B raw (same instance: moon-uring vs moon-epoll vs redis control)" + echo "# machine|variant|cmd|best_rps|us_per_op|reps|n" + declare -A BEST N + # Caller-provided MOON_START_ENV (e.g. MOON_URING_SPIN_US=40) composes into + # the moon cells instead of being clobbered by the per-cell driver forcing — + # the uring cell is "caller env as-is", the epoll cell adds MOON_NO_URING=1. + local base_env="${MOON_START_ENV:-}" + local eng cmd best reps nn variant vbin veng + for variant in uring epoll redis; do + case "$variant" in + uring) MOON_START_ENV="$base_env"; URING_REPORTED=""; veng=moon; vbin="$moonbin" ;; + epoll) MOON_START_ENV="$base_env MOON_NO_URING=1"; URING_REPORTED=""; veng=moon; vbin="$moonbin" ;; + redis) veng=redis; vbin="-" ;; + esac + while IFS='|' read -r eng cmd best reps nn; do + BEST["$variant|$cmd"]="$best"; N["$variant|$cmd"]="$nn" + printf '%s|%s|%s|%s|%s|%s|%s\n' "$GCE_MACHINE" "$variant" "$cmd" "$best" "$(us_per_op "$best")" "$reps" "$nn" + done < <( cell_engine "$veng" "$vbin" ) + done + + local c + for c in "uring|SET" "uring|GET" "epoll|SET" "epoll|GET" "redis|SET" "redis|GET"; do + require_floor "${N[$c]:-0}" || { printf 'status: VOID\nreason: unstable_sample\ndetail: %s only %s reps\n' "$c" "${N[$c]:-0}"; die "VOID unstable_sample ($c)"; } + done + + echo "# verdict (same-instance driver A/B):" + echo "# machine|cmd|uring_rps|epoll_rps|redis_rps|uring/redis|epoll/redis|epoll/uring" + for cmd in SET GET; do + local u="${BEST["uring|$cmd"]:-0}" e="${BEST["epoll|$cmd"]:-0}" r="${BEST["redis|$cmd"]:-0}" + printf '%s|%s|%s|%s|%s|%s|%s|%s\n' "$GCE_MACHINE" "$cmd" "$u" "$e" "$r" \ + "$(ratio "$u" "$r")" "$(ratio "$e" "$r")" "$(ratio "$e" "$u")" + done +} + +# measure_diag: on-instance perf ATTRIBUTION (not a verdict) — flat perf profiles, +# syscalls + ctx-switches for moon(epoll, debug-symbol build) vs redis under +# sustained p=1 c1 GET load. Answers "where do the remaining ~µs/op go on THIS +# machine". Requires linux-tools (installed by provisioning). +measure_diag() { + command -v taskset >/dev/null || die "taskset not available" + command -v perf >/dev/null || die "perf not installed (linux-tools)" + prepare_repo + build_redis + trap cleanup EXIT INT TERM + + local steal; steal=$(current_steal); steal="${steal:-0}" + gate_steal "$steal" || die "VOID steal=$steal%" + gate_clean || die "VOID dirty" + log "=== gates OK (steal=$steal%) — perf attribution on $(uname -srm) ===" + + # Debug-symbol moon build (frame pointers for perf -g); separate cache name so + # it never contaminates the verdict-mode generic build. + local dbin="$BINS/moon-diag-${MOON_REF//\//_}" + if [[ ! -x "$dbin" ]]; then + git -C "$WORK" checkout -q "$MOON_REF" || die "checkout $MOON_REF" + ( cd "$WORK" && CARGO_TARGET_DIR="$WORK/target-diag" CARGO_PROFILE_RELEASE_DEBUG=true \ + CARGO_PROFILE_RELEASE_STRIP=none RUSTFLAGS="-C force-frame-pointers=yes" \ + cargo build --release >/dev/null 2>&1 ) || die "diag moon build failed" + mkdir -p "$BINS"; cp "$WORK/target-diag/release/moon" "$dbin" + fi + local magic; magic=$(od -An -tx1 -N4 "$dbin" | tr -d ' ') + [[ "$magic" == "7f454c46" ]] || die "diag binary not ELF (magic=$magic)" + + echo "# p=1 perf attribution: moon(epoll driver, debug syms) vs redis, sustained c1 GET" + local engine pid port bpid rps + for engine in moon redis; do + wait_quiesced + if [[ "$engine" == moon ]]; then + MOON_START_ENV="MOON_NO_URING=1" URING_REPORTED="" start_moon "$dbin" || die "moon start" + pid=$MOON_PID; port=$MOON_PORT + else + start_redis || die "redis start" + pid=$REDIS_PID; port=$REDIS_PORT + fi + taskset -c "$CLIENT_CORE" "$REDIS_BENCH" -p "$port" -t set,get -n 30000 -c 1 -P 1 -r "$KEYSPACE" -q >/dev/null 2>&1 + taskset -c "$CLIENT_CORE" "$REDIS_BENCH" -p "$port" -t get -n 3000000 -c 1 -P 1 -r "$KEYSPACE" -q \ + > "/tmp/diagbench-$engine.txt" 2>&1 & + bpid=$! + sleep 2 + sudo perf stat -e task-clock,context-switches,raw_syscalls:sys_enter -p "$pid" \ + -o "/tmp/diagstat-$engine.txt" -- sleep 6 2>/dev/null || true + sudo perf record -F 997 -g -p "$pid" -o "/tmp/diagperf-$engine.data" -- sleep 8 2>/dev/null || true + kill "$bpid" 2>/dev/null; wait "$bpid" 2>/dev/null + cleanup + echo "== $engine ==" + echo "-- perf stat (6s window; ctx-switches ~= ops at p=1) --" + grep -E 'task-clock|context-switches|sys_enter' "/tmp/diagstat-$engine.txt" 2>/dev/null + echo "-- flat self% top 30 --" + sudo perf report -i "/tmp/diagperf-$engine.data" --stdio --no-children -g none \ + --percent-limit 0.4 2>/dev/null | grep -E "^\s+[0-9]" | head -30 + done +} + +# measure_tput: rigorous throughput sweep (clients x pipeline-depth), reusing the p=1 rigor +# (pinned disjoint cores, steal gate, fresh-server best-of-N). Emits ratio + verdict per cell so +# "faster per core at depth" is a clean claim. SET vs GET separated -> exposes read/write asymmetry. +measure_tput() { + command -v taskset >/dev/null || die "taskset not available" + prepare_repo + build_redis + trap cleanup EXIT INT TERM + + local steal; steal=$(current_steal); steal="${steal:-0}" + gate_steal "$steal" || { printf 'status: VOID\nreason: contended_instrument\ndetail: steal=%s > %s\n' "$steal" "$STEAL_MAX"; die "VOID contended (steal=$steal%)"; } + gate_clean || { printf 'status: VOID\nreason: dirty_instrument\n'; die "VOID dirty_instrument"; } + log "=== gates OK (steal=$steal%, clean) — throughput sweep on $(uname -srm) ===" + + local moonbin; moonbin=$(build_moon) || die "moon build" + log " settling compile heat..."; wait_quiesced + + echo "# kv-throughput raw (loopback, shard=1, pinned, best-of-$BEST_OF_N, real keyspace, SET-then-GET)" + echo "# persist=$PERSIST clients=[$CLIENTS_SWEEP] pipes=[$PIPES] requests=$REQUESTS keyspace=$KEYSPACE" + echo "# machine|clients|pipe|cmd|moon_rps|redis_rps|moon_us|redis_us|ratio|verdict" + declare -A BEST N + local C P engine eng cmd best reps nn cell + for C in $CLIENTS_SWEEP; do + for P in $PIPES; do + BENCH_CLIENTS="$C"; BENCH_PIPE="$P" + BEST=(); N=() + for engine in moon redis; do + while IFS='|' read -r eng cmd best reps nn; do + BEST["$eng|$cmd"]="$best"; N["$eng|$cmd"]="$nn" + done < <( cell_engine "$engine" "${moonbin:-"-"}" ) + done + for cell in "moon|SET" "moon|GET" "redis|SET" "redis|GET"; do + require_floor "${N[$cell]:-0}" || log " WARN c=$C P=$P $cell only ${N[$cell]:-0}/$BEST_OF_N reps" + done + for cmd in SET GET; do + local m="${BEST["moon|$cmd"]:-0}" r="${BEST["redis|$cmd"]:-0}" + printf '%s|%s|%s|%s|%s|%s|%s|%s|%s|%s\n' "$GCE_MACHINE" "$C" "$P" "$cmd" "$m" "$r" \ + "$(us_per_op "$m")" "$(us_per_op "$r")" "$(ratio "$m" "$r")" "$(verdict "$m" "$r")" + done + done + done +} + +# measure_shards: shard-scaling sweep with/without --io-busy-poll-us, same rigor +# (pinned, steal-gated, fresh-server best-of-N, strict keyspace so multi-shard +# actually scatters cross-shard). Needs an 8-vCPU instance: moon shards pin to +# cores 0..3, the client to 5-7, Redis control to core 0. Cells: +# shards x spin x { c1/P1 (single-conn latency, incl. the cross-shard-hop +# story at shards=4), c8/P1, c8/P16, c8/P64 }. +# Redis is measured once per client/pipe combo (single-threaded canonical). +measure_shards() { + command -v taskset >/dev/null || die "taskset not available" + prepare_repo + build_redis + trap cleanup EXIT INT TERM + + local steal; steal=$(current_steal); steal="${steal:-0}" + gate_steal "$steal" || { printf 'status: VOID\nreason: contended_instrument\n'; die "VOID contended (steal=$steal%)"; } + gate_clean || { printf 'status: VOID\nreason: dirty_instrument\n'; die "VOID dirty_instrument"; } + log "=== gates OK (steal=$steal%, clean) — shard sweep on $(uname -srm) ===" + + local moonbin; moonbin=$(build_moon) || die "moon build" + log " settling compile heat..."; wait_quiesced + + local combos=( "1|1" "8|1" "8|16" "8|64" ) + CLIENT_CORE="${SHARD_CLIENT_CORES:-5-7}" + echo "# kv-shard-scaling raw (loopback, pinned: client cores $CLIENT_CORE, best-of-$BEST_OF_N, strict keyspace)" + echo "# persist=$PERSIST requests=$REQUESTS keyspace=$KEYSPACE shards=[$SHARDS_SWEEP] spin=[$SPIN_SWEEP]" + echo "# machine|shards|spin_us|clients|pipe|cmd|moon_rps|redis_rps|ratio|verdict" + + declare -A RBEST MB + local combo C P eng cmd best reps nn S SPIN m r + # Redis control per combo (shards/spin don't apply to it) + for combo in "${combos[@]}"; do + C="${combo%%|*}"; P="${combo##*|}" + BENCH_CLIENTS="$C"; BENCH_PIPE="$P" + BENCH_THREADS=""; [[ "$C" -gt 1 ]] && BENCH_THREADS="${AB_BENCH_THREADS:-3}" + while IFS='|' read -r eng cmd best reps nn; do + RBEST["$C|$P|$cmd"]="$best" + done < <( cell_engine redis "-" ) + done + for S in $SHARDS_SWEEP; do + for SPIN in $SPIN_SWEEP; do + MOON_SHARDS="$S" + SERVER_CORES="0"; [[ "$S" -gt 1 ]] && SERVER_CORES="0-$((S-1))" + MOON_EXTRA_FLAGS=""; [[ "$SPIN" != 0 ]] && MOON_EXTRA_FLAGS="--io-busy-poll-us $SPIN" + for combo in "${combos[@]}"; do + C="${combo%%|*}"; P="${combo##*|}" + BENCH_CLIENTS="$C"; BENCH_PIPE="$P" + BENCH_THREADS=""; [[ "$C" -gt 1 ]] && BENCH_THREADS="${AB_BENCH_THREADS:-3}" + MB=() + while IFS='|' read -r eng cmd best reps nn; do + MB["$cmd"]="$best" + done < <( cell_engine moon "$moonbin" ) + for cmd in SET GET; do + m="${MB[$cmd]:-0}"; r="${RBEST["$C|$P|$cmd"]:-0}" + printf '%s|%s|%s|%s|%s|%s|%s|%s|%s|%s\n' "$GCE_MACHINE" "$S" "$SPIN" "$C" "$P" "$cmd" \ + "$m" "$r" "$(ratio "$m" "$r")" "$(verdict "$m" "$r")" + done + done + done + done + # reset overrides so any later mode in the same process gets shards=1 defaults + MOON_SHARDS=""; SERVER_CORES=""; MOON_EXTRA_FLAGS=""; BENCH_THREADS="" +} + +# measure_ab: SAME-INSTANCE commit A/B — MOON_REF (new) vs AB_BASE_REF (base) — +# at multi-shard busy-poll cells. Built for skip-notify validation: the delta +# of interest is a few percent, well inside the ~2.5% instance-to-instance +# variance, so both binaries MUST share one instance. spin=0 cells double as a +# no-regression control (skip-notify is gated off there → expect ~1.00). +# Ends with an engagement proof: INFO spsc_notify_skipped after a c1P1 burst. +measure_ab() { + command -v taskset >/dev/null || die "taskset not available" + prepare_repo + build_redis + trap cleanup EXIT INT TERM + + local steal; steal=$(current_steal); steal="${steal:-0}" + gate_steal "$steal" || { printf 'status: VOID\nreason: contended_instrument\n'; die "VOID contended (steal=$steal%)"; } + gate_clean || { printf 'status: VOID\nreason: dirty_instrument\n'; die "VOID dirty_instrument"; } + log "=== gates OK (steal=$steal%, clean) — commit A/B on $(uname -srm) ===" + + [[ -n "${AB_BASE_REF:-}" ]] || die "AB_BASE_REF not set" + local newbin basebin + newbin=$(build_moon) || die "moon build (new=$MOON_REF)" + basebin=$(MOON_REF="$AB_BASE_REF" build_moon) || die "moon build (base=$AB_BASE_REF)" + log " settling compile heat..."; wait_quiesced + + # AB_COMBOS: space-separated "clients|pipe" cells. AB_BASE_ENV / AB_NEW_ENV: + # per-side MOON_START_ENV (env-knob A/B on ONE binary — set AB_BASE_REF=MOON_REF). + local combos; read -r -a combos <<< "${AB_COMBOS:-1|1 8|1}" + CLIENT_CORE="${SHARD_CLIENT_CORES:-5-7}" + echo "# kv-commit-ab raw (same-instance, pinned: client cores $CLIENT_CORE, best-of-$BEST_OF_N, strict keyspace)" + echo "# base=$AB_BASE_REF new=$MOON_REF persist=$PERSIST requests=$REQUESTS shards=[${AB_SHARDS:-4}] spin=[${AB_SPINS:-0 40}]" + echo "# base_env='${AB_BASE_ENV:-}' new_env='${AB_NEW_ENV:-}' combos='${AB_COMBOS:-1|1 8|1}'" + echo "# machine|shards|spin_us|clients|pipe|cmd|base_rps|new_rps|redis_rps|new_vs_base|new_vs_redis" + + declare -A RBEST BB NB + local combo C P eng cmd best reps nn S SPIN b m r + for combo in "${combos[@]}"; do + C="${combo%%|*}"; P="${combo##*|}" + BENCH_CLIENTS="$C"; BENCH_PIPE="$P" + BENCH_THREADS=""; [[ "$C" -gt 1 ]] && BENCH_THREADS="${AB_BENCH_THREADS:-3}" + while IFS='|' read -r eng cmd best reps nn; do + RBEST["$C|$P|$cmd"]="$best" + done < <( cell_engine redis "-" ) + done + for S in ${AB_SHARDS:-4}; do + for SPIN in ${AB_SPINS:-0 40}; do + MOON_SHARDS="$S" + SERVER_CORES="0"; [[ "$S" -gt 1 ]] && SERVER_CORES="0-$((S-1))" + MOON_EXTRA_FLAGS=""; [[ "$SPIN" != 0 ]] && MOON_EXTRA_FLAGS="--io-busy-poll-us $SPIN" + for combo in "${combos[@]}"; do + C="${combo%%|*}"; P="${combo##*|}" + BENCH_CLIENTS="$C"; BENCH_PIPE="$P" + BENCH_THREADS=""; [[ "$C" -gt 1 ]] && BENCH_THREADS="${AB_BENCH_THREADS:-3}" + BB=(); NB=() + while IFS='|' read -r eng cmd best reps nn; do + BB["$cmd"]="$best" + done < <( MOON_START_ENV="${AB_BASE_ENV:-${MOON_START_ENV:-}}" cell_engine moon "$basebin" ) + while IFS='|' read -r eng cmd best reps nn; do + NB["$cmd"]="$best" + done < <( MOON_START_ENV="${AB_NEW_ENV:-${MOON_START_ENV:-}}" cell_engine moon "$newbin" ) + for cmd in SET GET; do + b="${BB[$cmd]:-0}"; m="${NB[$cmd]:-0}"; r="${RBEST["$C|$P|$cmd"]:-0}" + printf '%s|%s|%s|%s|%s|%s|%s|%s|%s|%s|%s\n' "$GCE_MACHINE" "$S" "$SPIN" "$C" "$P" "$cmd" \ + "$b" "$m" "$r" "$(ratio "$m" "$b")" "$(ratio "$m" "$r")" + done + done + done + done + + # Engagement proof: on the new binary at spin=40 the skip counter must move + # (near-total under sustained c1P1 load — the target shards spin constantly). + local eng_shards="${AB_SHARDS:-4}"; eng_shards="${eng_shards##* }" + MOON_SHARDS="$eng_shards"; SERVER_CORES="0-$((eng_shards-1))" + MOON_EXTRA_FLAGS="--io-busy-poll-us 40" + BENCH_CLIENTS=1; BENCH_PIPE=1; BENCH_THREADS="" + if start_moon "$newbin"; then + taskset -c "$CLIENT_CORE" "$REDIS_BENCH" -p "$MOON_PORT" -c 1 -P 1 -n 20000 \ + -r "$KEYSPACE" -t set,get >/dev/null 2>&1 || true + local skipped wakes + skipped=$("$REDIS_CLI" -p "$MOON_PORT" info stats 2>/dev/null | tr -d '\r' | awk -F: '/^spsc_notify_skipped/{print $2}') + wakes=$("$REDIS_CLI" -p "$MOON_PORT" info stats 2>/dev/null | tr -d '\r' | awk -F: '/^spsc_notify_wakes/{print $2}') + echo "# engagement: spsc_notify_skipped=${skipped:-?} spsc_notify_wakes=${wakes:-?} (new bin, shards=$eng_shards, spin=40, 40k c1P1 ops)" + cleanup_moon + else + echo "# engagement: SKIPPED (server failed to start)" + fi + MOON_SHARDS=""; SERVER_CORES=""; MOON_EXTRA_FLAGS=""; BENCH_THREADS="" +} + +# ============================================================================= gcloud orchestration +arch_image_family() { # ubuntu image family for the machine's arch + case "$1" in + c4a-*|t2a-*|*-arm*|*arm64*) echo "ubuntu-2404-lts-arm64" ;; + *) echo "ubuntu-2404-lts-amd64" ;; + esac +} + +disk_type_for() { # boot-disk type by machine family. c4a/c4/n4/*-metal are Hyperdisk-ONLY + case "$1" in # (pd-ssd is rejected at create); c3 & older still take pd-ssd. + c4a-*|c4-*|n4-*|*-metal) echo "hyperdisk-balanced" ;; + *) echo "pd-ssd" ;; + esac +} + +gcloud_run_one() { + command -v gcloud >/dev/null || die "gcloud CLI not found" + local imgfam; imgfam=$(arch_image_family "$GCE_MACHINE") + local disktype; disktype=$(disk_type_for "$GCE_MACHINE") + local GSSH=(gcloud compute ssh "$GCE_NAME" --zone="$GCE_ZONE" --quiet + --ssh-flag=-oStrictHostKeyChecking=no --ssh-flag=-oConnectTimeout=15) + # Leak backstop (design-for-failure): GCE auto-DELETEs the VM after MAX_RUN even if this + # orchestrator is killed mid-run (an unattended background job here can be reaped). Standard-VM + # limited-runtime form; no --provisioning-model needed. Toggle off with SELF_DESTRUCT=0. + local guard=(--max-run-duration="${MAX_RUN:-45m}" --instance-termination-action=DELETE) + [[ "${SELF_DESTRUCT:-1}" == 1 ]] || guard=() + log "=== provisioning $GCE_NAME ($GCE_MACHINE, $GCE_ZONE, $imgfam, $disktype; guard=${guard[*]:-off}) ===" + gcloud compute instances create "$GCE_NAME" \ + --machine-type="$GCE_MACHINE" --zone="$GCE_ZONE" \ + --image-family="$imgfam" --image-project=ubuntu-os-cloud \ + --boot-disk-size=50GB --boot-disk-type="$disktype" \ + "${guard[@]}" --quiet \ + || die "instance create failed" + + # Pin the actual name into the teardown trap NOW (env-prefix reverts GCE_NAME at fire time). + local _inst="$GCE_NAME" _zone="$GCE_ZONE" + trap "log '=== tearing down $_inst ==='; gcloud compute instances delete '$_inst' --zone='$_zone' -q 2>/dev/null || true" EXIT INT TERM + + log "=== waiting for SSH ===" + local tries=0 + until "${GSSH[@]}" --command='echo up' >/dev/null 2>&1; do + tries=$((tries+1)); [[ $tries -ge 40 ]] && die "SSH never came up"; sleep 5 + done + + log "=== provisioning toolchain + repo on the instance ===" + "${GSSH[@]}" --command=' + set -e + sudo apt-get update -qq + sudo apt-get install -y -qq build-essential pkg-config libssl-dev git curl ca-certificates + # perf for --measure-diag (kernel-matched first, gcp/generic fallback; non-fatal) + sudo apt-get install -y -qq linux-tools-$(uname -r) 2>/dev/null \ + || sudo apt-get install -y -qq linux-tools-gcp 2>/dev/null \ + || sudo apt-get install -y -qq linux-tools-generic 2>/dev/null || true + command -v cargo >/dev/null || curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.94.1 + ' || die "instance provisioning failed" + + log "=== pushing harness ===" + gcloud compute scp scripts/gcloud-kv-p1-baseline.sh "$GCE_NAME":~/gcloud-kv-p1-baseline.sh \ + --zone="$GCE_ZONE" --quiet --scp-flag=-oStrictHostKeyChecking=no || die "scp failed" + + local mcmd="${MEASURE_CMD:---measure}" + local rawfile="${RAWFILE:-tmp/kv-p1-${GCE_MACHINE}.txt}" + log "=== running $mcmd on $GCE_MACHINE (build + bench; the long part) -> $rawfile ===" + "${GSSH[@]}" --command=" + source \$HOME/.cargo/env + GCE_MACHINE='$GCE_MACHINE' MOON_REF='$MOON_REF' REDIS_VER='$REDIS_VER' BEST_OF_N='$BEST_OF_N' \ + REQUESTS='$REQUESTS' KEYSPACE='$KEYSPACE' PIPES='$PIPES' CLIENTS_SWEEP='$CLIENTS_SWEEP' PERSIST='$PERSIST' \ + SHARDS_SWEEP='$SHARDS_SWEEP' SPIN_SWEEP='$SPIN_SWEEP' SHARD_CLIENT_CORES='$SHARD_CLIENT_CORES' \ + AB_BASE_REF='${AB_BASE_REF:-}' AB_SHARDS='${AB_SHARDS:-}' AB_SPINS='${AB_SPINS:-}' \ + AB_BASE_ENV='${AB_BASE_ENV:-}' AB_NEW_ENV='${AB_NEW_ENV:-}' AB_COMBOS='${AB_COMBOS:-}' \ + AB_BENCH_THREADS='${AB_BENCH_THREADS:-}' \ + MOON_SHARDS='${MOON_SHARDS:-}' SERVER_CORES='${SERVER_CORES:-}' MOON_EXTRA_FLAGS='${MOON_EXTRA_FLAGS:-}' \ + CLIENT_CORE='${CLIENT_CORE_OVERRIDE:-$CLIENT_CORE}' \ + MOON_START_ENV='${MOON_START_ENV:-}' \ + bash ~/gcloud-kv-p1-baseline.sh $mcmd + " | tee "$rawfile" + + log "=== $GCE_MACHINE raw results in $rawfile; teardown follows (trap) ===" +} + +gcloud_sweep() { + log "=== KV-P1 SWEEP: $MACHINES ===" + local m short + for m in $MACHINES; do + short="${m%%-*}" + log ""; log "##################### machine: $m #####################" + ( GCE_MACHINE="$m"; GCE_NAME="moon-kv-p1-${short}"; gcloud_run_one ) \ + || log "WARN: machine $m run failed (see log); continuing" + done + log "=== sweep complete; per-machine raw files: tmp/kv-p1-.txt ===" +} + +# ============================================================================= dispatch +case "${1:---gcloud}" in + --self-test) self_test ;; + --measure) measure ;; + --measure-driver) measure_driver ;; + --measure-diag) measure_diag ;; + --measure-shards) measure_shards ;; + --measure-ab) measure_ab ;; + --measure-tput) measure_tput ;; + --gcloud) gcloud_sweep ;; + --one) gcloud_run_one ;; + *) die "unknown subcommand: $1 (use --self-test | --measure | --measure-driver | --measure-tput | --gcloud | --one)" ;; +esac diff --git a/scripts/test-commands.sh b/scripts/test-commands.sh index 2b865fcf4..e8297aff5 100755 --- a/scripts/test-commands.sh +++ b/scripts/test-commands.sh @@ -344,6 +344,8 @@ if should_run "string"; then assert_moon "INCRBYFLOAT" "6.5" INCRBYFLOAT str:cnt1 0.5 assert_moon "MSET" "OK" MSET str:m1 a str:m2 b str:m3 c assert_moon_ok "MGET" MGET str:m1 str:m2 str:m3 + assert_moon "MSETNX (new)" "(integer) 1" MSETNX "{mcm}n1" a "{mcm}n2" b + assert_moon "MSETNX (exists)" "(integer) 0" MSETNX "{mcm}n2" x "{mcm}n3" c assert_moon_ok "GETEX with EX" GETEX str:m1 EX 100 else assert_match "SET basic" SET str:k1 hello @@ -369,6 +371,8 @@ if should_run "string"; then assert_match "INCRBYFLOAT" INCRBYFLOAT str:cnt1 0.5 assert_match "MSET" MSET str:m1 a str:m2 b str:m3 c assert_match "MGET" MGET str:m1 str:m2 str:m3 + assert_match "MSETNX (new)" MSETNX "{mcc}n1" a "{mcc}n2" b + assert_match "MSETNX (exists)" MSETNX "{mcc}n2" x "{mcc}n3" c assert_match "GETEX with EX" GETEX str:m1 EX 100 fi fi diff --git a/scripts/test-consistency.sh b/scripts/test-consistency.sh index 67bf6613f..aca83d5bb 100755 --- a/scripts/test-consistency.sh +++ b/scripts/test-consistency.sh @@ -247,6 +247,13 @@ both MSET mk1 "val1" mk2 "val2" mk3 "val3" assert_both "MGET 3 keys" MGET mk1 mk2 mk3 assert_both "MGET with missing" MGET mk1 nonexistent mk3 +# MSETNX: hash-tagged ({mn}) so all keys co-locate on one shard -> atomic under +# Moon's 1/4/12 shard configs (cross-shard MSETNX is rejected CROSSSLOT by design). +assert_both "MSETNX all new" MSETNX "{mn}k1" "v1" "{mn}k2" "v2" +assert_both "MGET after MSETNX" MGET "{mn}k1" "{mn}k2" +assert_both "MSETNX one exists (0)" MSETNX "{mn}k2" "new2" "{mn}k3" "v3" +assert_both "MSETNX no partial write" GET "{mn}k3" + # =========================================================================== # 4. SET with options (EX, PX, NX, XX, KEEPTTL, GET) # =========================================================================== diff --git a/src/admin/metrics_setup.rs b/src/admin/metrics_setup.rs index 5d67f828a..2c55dda27 100644 --- a/src/admin/metrics_setup.rs +++ b/src/admin/metrics_setup.rs @@ -31,6 +31,11 @@ static CONNECTED_CLIENTS: AtomicU64 = AtomicU64::new(0); // per command — so plain (unsharded) atomics are fine here. static SPSC_NOTIFY_WAKES: AtomicU64 = AtomicU64::new(0); static SPSC_DRAIN_RENOTIFY: AtomicU64 = AtomicU64::new(0); +// Busy-poll skip-notify: cross-shard notifies elided because the target +// shard's driver advertised it is spin-polling (and will discover ringbuf +// items via its own probe). Per-dispatch rate, but only on the cross-shard +// path with busy-poll active — plain atomic is fine. +static SPSC_NOTIFY_SKIPPED: AtomicU64 = AtomicU64::new(0); // ── ft-search-off-eventloop (C5): cooperative-yield observability ──────── // Bumped once per cooperative yield taken by the FT.SEARCH local slice (the @@ -83,6 +88,19 @@ pub fn spsc_drain_renotify() -> u64 { SPSC_DRAIN_RENOTIFY.load(Ordering::Relaxed) } +/// Count a cross-shard notify elided because the target shard's busy-poll +/// driver advertised it will discover the ringbuf push via its spin probe. +#[inline] +pub fn bump_spsc_notify_skipped() { + SPSC_NOTIFY_SKIPPED.fetch_add(1, Ordering::Relaxed); +} + +/// Total skip-wake-elided cross-shard notifies (for INFO Stats). +#[inline] +pub fn spsc_notify_skipped() -> u64 { + SPSC_NOTIFY_SKIPPED.load(Ordering::Relaxed) +} + // ── QW4 (2026-06 review finding 1.6): sharded total-commands counter ──── // Previously a single `TOTAL_COMMANDS: AtomicU64` — one cache line bounced // across every shard core at full command rate (false sharing). Each OS @@ -1455,7 +1473,7 @@ mod tests { record_dispatch_cross_spsc(); let after = total_dispatch_cross_spsc(); assert!( - after >= before + 1, + after > before, "counter must have increased by at least 1; before={before} after={after}" ); } diff --git a/src/client_registry.rs b/src/client_registry.rs index c091997e4..65b2a5eaf 100644 --- a/src/client_registry.rs +++ b/src/client_registry.rs @@ -22,6 +22,8 @@ static REGISTRY: LazyLock>> = /// (writer, once per batch) and CLIENT LIST/INFO/KILL (occasional readers). pub struct ClientLiveState { pub connected_at: Instant, + /// Epoch ms at registration — baseline for `touch`'s caller-supplied clock. + pub connected_at_epoch_ms: u64, pub db: AtomicUsize, /// Milliseconds since `connected_at` of the last completed batch. pub last_cmd_ms: AtomicU64, @@ -32,12 +34,17 @@ pub struct ClientLiveState { } impl ClientLiveState { - /// Record batch-completion state. Three relaxed stores — no lock. + /// Record batch-completion state. Three relaxed stores — no lock, and no + /// clock read: `now_epoch_ms` comes from the caller (the shard-cached + /// clock on hot paths), keeping `Instant::now()` off the per-batch path + /// per the timestamp-caching invariant. Up to 1ms of cached-clock + /// staleness is fine for an idle-time stat; `saturating_sub` guards the + /// stale-clock-before-connect edge. #[inline] - pub fn touch(&self, db: usize, flags: ClientFlags) { + pub fn touch(&self, db: usize, flags: ClientFlags, now_epoch_ms: u64) { self.db.store(db, Ordering::Relaxed); self.last_cmd_ms.store( - self.connected_at.elapsed().as_millis() as u64, + now_epoch_ms.saturating_sub(self.connected_at_epoch_ms), Ordering::Relaxed, ); self.flags.store(flags.to_bits(), Ordering::Relaxed); @@ -106,6 +113,7 @@ impl ClientFlags { pub fn register(id: u64, addr: String, user: String, shard: usize) -> Arc { let live = Arc::new(ClientLiveState { connected_at: Instant::now(), + connected_at_epoch_ms: crate::storage::entry::current_time_ms(), db: AtomicUsize::new(0), last_cmd_ms: AtomicU64::new(0), flags: AtomicU8::new(ClientFlags::default().to_bits()), @@ -316,13 +324,32 @@ mod tests { update(id, |e| { e.name = Some("myconn".into()); }); - live.touch(3, ClientFlags::default()); + live.touch(3, ClientFlags::default(), live.connected_at_epoch_ms); let info = client_info(id).unwrap(); assert!(info.contains("name=myconn")); assert!(info.contains("db=3")); deregister(id); } + #[test] + fn test_touch_uses_caller_clock_not_instant_now() { + let id = 999_004; + let live = register(id, "10.0.0.6:9000".into(), "default".into(), 0); + // touch takes the caller's (shard-cached) epoch clock — no per-op + // Instant::now(). last_cmd_ms stays "ms since connect". + live.touch(2, ClientFlags::default(), live.connected_at_epoch_ms + 5000); + assert_eq!(live.last_cmd_ms.load(Ordering::Relaxed), 5000); + // A cached clock up to 1ms stale can lag the registration timestamp; + // must saturate to 0, never wrap. + live.touch( + 2, + ClientFlags::default(), + live.connected_at_epoch_ms.saturating_sub(10), + ); + assert_eq!(live.last_cmd_ms.load(Ordering::Relaxed), 0); + deregister(id); + } + #[test] fn test_flags_bits_roundtrip() { for bits in 0..8u8 { diff --git a/src/command/config.rs b/src/command/config.rs index ce18036a2..a6d78e8ba 100644 --- a/src/command/config.rs +++ b/src/command/config.rs @@ -98,7 +98,11 @@ pub fn config_set(runtime_config: &mut RuntimeConfig, args: &[Frame]) -> Frame { match param_name.as_str() { "maxmemory" => match value_str.parse::() { - Ok(v) => runtime_config.maxmemory = v, + Ok(v) => { + runtime_config.maxmemory = v; + // Keep the inline write path's lock-free pre-gate in sync. + crate::storage::eviction::publish_maxmemory_hints(&*runtime_config); + } Err(_) => { return Frame::Error(Bytes::from(format!( "ERR Invalid argument '{}' for CONFIG SET 'maxmemory'", diff --git a/src/command/connection.rs b/src/command/connection.rs index fde748e27..052ea6572 100644 --- a/src/command/connection.rs +++ b/src/command/connection.rs @@ -267,12 +267,14 @@ pub fn info(db: &Database, _args: &[Frame]) -> Frame { total_dispatch_cross_spsc:{}\r\n\ spsc_notify_wakes:{}\r\n\ spsc_drain_renotify:{}\r\n\ + spsc_notify_skipped:{}\r\n\ ft_search_cooperative_yields_total:{}\r\n", crate::admin::metrics_setup::total_commands_processed(), crate::admin::metrics_setup::total_connections_received(), crate::admin::metrics_setup::total_dispatch_cross_spsc(), crate::admin::metrics_setup::spsc_notify_wakes(), crate::admin::metrics_setup::spsc_drain_renotify(), + crate::admin::metrics_setup::spsc_notify_skipped(), crate::admin::metrics_setup::ft_search_cooperative_yields(), ); sections.push_str("\r\n"); diff --git a/src/command/helpers.rs b/src/command/helpers.rs index f55f93ddd..84392836e 100644 --- a/src/command/helpers.rs +++ b/src/command/helpers.rs @@ -27,3 +27,16 @@ pub fn ok() -> Frame { pub fn err(msg: &str) -> Frame { Frame::Error(Bytes::from(msg.to_string())) } + +/// Whether an absolute expiry (unix millis) is representable without a +/// client-visible wrap. +/// +/// Expiry is stored as `u64` millis, but `PTTL`/`PEXPIRETIME` cast it back to +/// `i64` — an expiry past `i64::MAX` surfaces as a NEGATIVE TTL on a key that is +/// very much alive. Redis rejects such out-of-range expiries outright +/// (`when > LLONG_MAX / 1000` → "invalid expire time"), so every expiry-setting +/// command must too. Returns `true` when `expires_at_ms` is safe to store. +#[inline] +pub fn expiry_ms_in_range(expires_at_ms: u64) -> bool { + expires_at_ms <= i64::MAX as u64 +} diff --git a/src/command/key.rs b/src/command/key.rs index 6c1b22bc4..685da0cae 100644 --- a/src/command/key.rs +++ b/src/command/key.rs @@ -6,7 +6,7 @@ use crate::storage::Database; use crate::storage::compact_key::CompactKey; use crate::storage::entry::current_time_ms; -use super::helpers::err_wrong_args; +use super::helpers::{err_wrong_args, expiry_ms_in_range}; /// Extract a key as &[u8] from a Frame argument. pub(crate) fn extract_key(frame: &Frame) -> Option<&[u8]> { @@ -64,8 +64,9 @@ pub fn exists(db: &mut Database, args: &[Frame]) -> Frame { /// EXPIRE key seconds /// -/// Set a timeout on key. Returns 1 if timeout was set, 0 if key does not exist. -/// Negative or zero seconds returns an error (modern Redis 7+ behavior). +/// Set a timeout on key. Returns 1 if the timeout was set (or the key was +/// deleted because of a non-positive/past TTL), 0 if the key does not exist. +/// A non-positive TTL deletes the key immediately (Redis past-time semantics). pub fn expire(db: &mut Database, args: &[Frame]) -> Frame { if args.len() != 2 { return err_wrong_args("EXPIRE"); @@ -82,12 +83,37 @@ pub fn expire(db: &mut Database, args: &[Frame]) -> Frame { )); } }; - if seconds <= 0 { + // Redis rejects an out-of-i64-range expiry (`seconds < LLONG_MIN/1000`) BEFORE + // the past-time delete, so an extreme negative errors rather than deleting. + if seconds < i64::MIN / 1000 { return Frame::Error(Bytes::from_static( b"ERR invalid expire time in 'EXPIRE' command", )); } - let expires_at_ms = current_time_ms() + (seconds as u64) * 1000; + // Redis parity: a non-positive TTL is a past-time expiry -> delete the key now + // (return 1 if it existed, 0 otherwise) rather than erroring. Mirrors EXPIREAT. + if seconds <= 0 { + return if db.remove(key).is_some() { + Frame::Integer(1) + } else { + Frame::Integer(0) + }; + } + // Guard the u64 arithmetic (seconds*1000 + now_ms can overflow) AND bound the + // result to the i64 domain so PTTL — which casts the stored u64 back to i64 — + // never wraps negative on a live key. Redis rejects an out-of-range expiry. + let expires_at_ms = match (seconds as u64) + .checked_mul(1000) + .and_then(|delta| current_time_ms().checked_add(delta)) + .filter(|ms| expiry_ms_in_range(*ms)) + { + Some(ms) => ms, + None => { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'EXPIRE' command", + )); + } + }; if db.set_expiry(key, expires_at_ms) { Frame::Integer(1) } else { @@ -97,7 +123,8 @@ pub fn expire(db: &mut Database, args: &[Frame]) -> Frame { /// PEXPIRE key milliseconds /// -/// Like EXPIRE but the timeout is specified in milliseconds. +/// Like EXPIRE but the timeout is specified in milliseconds. A non-positive TTL +/// deletes the key immediately (Redis past-time semantics). pub fn pexpire(db: &mut Database, args: &[Frame]) -> Frame { if args.len() != 2 { return err_wrong_args("PEXPIRE"); @@ -114,12 +141,27 @@ pub fn pexpire(db: &mut Database, args: &[Frame]) -> Frame { )); } }; + // Redis parity: a non-positive TTL is a past-time expiry -> delete the key now. if millis <= 0 { - return Frame::Error(Bytes::from_static( - b"ERR invalid expire time in 'PEXPIRE' command", - )); + return if db.remove(key).is_some() { + Frame::Integer(1) + } else { + Frame::Integer(0) + }; } - let expires_at_ms = current_time_ms() + millis as u64; + // Guard the u64 arithmetic against overflow AND bound the result to the i64 + // domain so PTTL never wraps negative on a live key (consistent with EXPIRE). + let expires_at_ms = match current_time_ms() + .checked_add(millis as u64) + .filter(|ms| expiry_ms_in_range(*ms)) + { + Some(ms) => ms, + None => { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'PEXPIRE' command", + )); + } + }; if db.set_expiry(key, expires_at_ms) { Frame::Integer(1) } else { @@ -236,6 +278,13 @@ pub fn expireat(db: &mut Database, args: &[Frame]) -> Frame { )); } }; + // Redis rejects an out-of-i64-range timestamp (`< LLONG_MIN/1000`) before the + // past-time delete, so an extreme negative errors rather than deleting. + if timestamp < i64::MIN / 1000 { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'EXPIREAT' command", + )); + } // Redis accepts 0 and negative timestamps as past-time expiry (deletes key immediately) if timestamp <= 0 { return if db.remove(key).is_some() { @@ -244,7 +293,19 @@ pub fn expireat(db: &mut Database, args: &[Frame]) -> Frame { Frame::Integer(0) }; } - let expires_at_ms = (timestamp as u64) * 1000; + // Guard the *1000 conversion against u64 overflow AND bound the result to the + // i64 domain so PEXPIRETIME never wraps negative on a live key. + let expires_at_ms = match (timestamp as u64) + .checked_mul(1000) + .filter(|ms| expiry_ms_in_range(*ms)) + { + Some(ms) => ms, + None => { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'EXPIREAT' command", + )); + } + }; if db.set_expiry(key, expires_at_ms) { Frame::Integer(1) } else { @@ -1383,10 +1444,130 @@ mod tests { } #[test] - fn test_expire_negative() { + fn test_expire_nonpositive_deletes() { + // Redis parity: EXPIRE with a non-positive TTL deletes the key immediately + // (past-time expiry) and returns 1 -- it must NOT error and leave the key. + let mut db = setup_db_with_key(b"foo", b"bar"); + assert_eq!(expire(&mut db, &[bs(b"foo"), bs(b"-1")]), Frame::Integer(1)); + assert!(!db.exists(b"foo"), "EXPIRE foo -1 must delete the key"); + + let mut db2 = setup_db_with_key(b"foo", b"bar"); + assert_eq!(expire(&mut db2, &[bs(b"foo"), bs(b"0")]), Frame::Integer(1)); + assert!(!db2.exists(b"foo"), "EXPIRE foo 0 must delete the key"); + } + + #[test] + fn test_expire_nonpositive_missing_key() { + let mut db = Database::new(); + assert_eq!( + expire(&mut db, &[bs(b"nope"), bs(b"-1")]), + Frame::Integer(0) + ); + } + + #[test] + fn test_expire_overflow_rejected() { + // now_ms + seconds*1000 overflows u64 -> error, no silent wrap; key untouched. + let mut db = setup_db_with_key(b"foo", b"bar"); + let huge = Frame::BulkString(Bytes::from(i64::MAX.to_string())); + let result = expire(&mut db, &[bs(b"foo"), huge]); + assert!( + matches!(result, Frame::Error(ref s) if s.starts_with(b"ERR invalid expire")), + "overflowing EXPIRE must error, got {result:?}" + ); + assert!( + db.exists(b"foo"), + "rejected EXPIRE must not disturb the key" + ); + } + + // --- i64-domain expiry bound (Finding 2/3): an absolute expiry past i64::MAX + // would surface as a NEGATIVE TTL on a live key, because PTTL/PEXPIRETIME cast + // the stored u64 back to i64. Redis rejects such expiries outright + // (`when > LLONG_MAX/1000` -> "invalid expire time"); Moon must too. --- + + #[test] + fn test_expire_i64_bound_rejected() { + // seconds*1000 fits u64 but now+that exceeds i64::MAX. Without the i64 bound + // Moon accepts it and PTTL wraps negative. Must error and leave the key's TTL + // untouched (no expiry set -> PTTL == -1, never a bogus large-negative). + let mut db = setup_db_with_key(b"foo", b"bar"); + let over = Frame::BulkString(Bytes::from("15000000000000000")); // 1.5e16 s + let result = expire(&mut db, &[bs(b"foo"), over]); + assert!( + matches!(result, Frame::Error(ref s) if s.starts_with(b"ERR invalid expire")), + "EXPIRE past i64::MAX must error, got {result:?}" + ); + assert!( + db.exists(b"foo"), + "rejected EXPIRE must not disturb the key" + ); + assert_eq!( + pttl(&mut db, &[bs(b"foo")]), + Frame::Integer(-1), + "rejected EXPIRE must leave the key with no expiry (PTTL -1, never wrapped-negative)" + ); + } + + #[test] + fn test_expire_extreme_negative_errors_not_deletes() { + // Redis rejects |seconds| > LLONG_MAX/1000 BEFORE the past-time delete + // (when < LLONG_MIN/1000). i64::MIN must ERROR and PRESERVE the key, unlike a + // normal small negative which deletes. + let mut db = setup_db_with_key(b"foo", b"bar"); + let min = Frame::BulkString(Bytes::from(i64::MIN.to_string())); + let result = expire(&mut db, &[bs(b"foo"), min]); + assert!( + matches!(result, Frame::Error(ref s) if s.starts_with(b"ERR invalid expire")), + "EXPIRE i64::MIN must error, got {result:?}" + ); + assert!( + db.exists(b"foo"), + "rejected extreme-negative EXPIRE must not delete the key" + ); + } + + #[test] + fn test_pexpire_i64_bound_rejected() { + // now_ms + i64::MAX ms exceeds i64::MAX -> reject (else PTTL wraps negative). let mut db = setup_db_with_key(b"foo", b"bar"); - let result = expire(&mut db, &[bs(b"foo"), bs(b"-1")]); - assert!(matches!(result, Frame::Error(ref s) if s.starts_with(b"ERR invalid expire"))); + let over = Frame::BulkString(Bytes::from(i64::MAX.to_string())); + let result = pexpire(&mut db, &[bs(b"foo"), over]); + assert!( + matches!(result, Frame::Error(ref s) if s.starts_with(b"ERR invalid expire")), + "PEXPIRE past i64::MAX must error, got {result:?}" + ); + assert!(db.exists(b"foo")); + assert_eq!(pttl(&mut db, &[bs(b"foo")]), Frame::Integer(-1)); + } + + #[test] + fn test_expireat_i64_bound_rejected() { + // absolute seconds*1000 exceeds i64::MAX -> reject (else PEXPIRETIME wraps negative). + let mut db = setup_db_with_key(b"foo", b"bar"); + let over = Frame::BulkString(Bytes::from("15000000000000000")); // 1.5e16 s + let result = expireat(&mut db, &[bs(b"foo"), over]); + assert!( + matches!(result, Frame::Error(ref s) if s.starts_with(b"ERR invalid expire")), + "EXPIREAT past i64::MAX must error, got {result:?}" + ); + assert!(db.exists(b"foo")); + assert_eq!(pexpiretime(&mut db, &[bs(b"foo")]), Frame::Integer(-1)); + } + + #[test] + fn test_expireat_extreme_negative_errors_not_deletes() { + let mut db = setup_db_with_key(b"foo", b"bar"); + let min = Frame::BulkString(Bytes::from(i64::MIN.to_string())); + let result = expireat(&mut db, &[bs(b"foo"), min]); + assert!( + matches!(result, Frame::Error(ref s) if s.starts_with(b"ERR invalid expire")), + "EXPIREAT i64::MIN must error, got {result:?}" + ); + assert!( + db.exists(b"foo"), + "rejected extreme-negative EXPIREAT must not delete the key" + ); } // --- PEXPIRE tests --- @@ -1403,6 +1584,17 @@ mod tests { } } + #[test] + fn test_pexpire_nonpositive_deletes() { + // Redis parity: PEXPIRE with a non-positive TTL deletes the key and returns 1. + let mut db = setup_db_with_key(b"foo", b"bar"); + assert_eq!( + pexpire(&mut db, &[bs(b"foo"), bs(b"-1")]), + Frame::Integer(1) + ); + assert!(!db.exists(b"foo"), "PEXPIRE foo -1 must delete the key"); + } + // --- TTL tests --- #[test] @@ -1881,6 +2073,22 @@ mod tests { assert!(db.get(b"k").unwrap().has_expiry()); } + #[test] + fn test_expireat_overflow_rejected() { + // (timestamp * 1000) overflows u64 for huge timestamps -> error, key untouched. + let mut db = setup_db_with_key(b"k", b"v"); + let huge = Frame::BulkString(Bytes::from(i64::MAX.to_string())); + let result = expireat(&mut db, &[bs(b"k"), huge]); + assert!( + matches!(result, Frame::Error(ref e) if e.starts_with(b"ERR invalid expire")), + "overflowing EXPIREAT must error, got {result:?}" + ); + assert!( + db.exists(b"k"), + "rejected EXPIREAT must not disturb the key" + ); + } + #[test] fn test_expireat_missing() { let mut db = Database::new(); diff --git a/src/command/metadata.rs b/src/command/metadata.rs index 935d27440..3923f108a 100644 --- a/src/command/metadata.rs +++ b/src/command/metadata.rs @@ -142,6 +142,7 @@ pub static COMMAND_META: phf::Map<&'static str, CommandMeta> = phf_map! { "SET" => CommandMeta { name: "SET", arity: -3, flags: WF, first_key: 1, last_key: 1, step: 1, acl_categories: STR }, "MGET" => CommandMeta { name: "MGET", arity: -2, flags: RF, first_key: 1, last_key: -1, step: 1, acl_categories: STR }, "MSET" => CommandMeta { name: "MSET", arity: -3, flags: WF, first_key: 1, last_key: -1, step: 2, acl_categories: STR }, + "MSETNX" => CommandMeta { name: "MSETNX", arity: -3, flags: WF, first_key: 1, last_key: -1, step: 2, acl_categories: STR }, "SETNX" => CommandMeta { name: "SETNX", arity: 3, flags: WF, first_key: 1, last_key: 1, step: 1, acl_categories: STR }, "SETEX" => CommandMeta { name: "SETEX", arity: 4, flags: WF, first_key: 1, last_key: 1, step: 1, acl_categories: STR }, "PSETEX" => CommandMeta { name: "PSETEX", arity: 4, flags: WF, first_key: 1, last_key: 1, step: 1, acl_categories: STR }, diff --git a/src/command/mod.rs b/src/command/mod.rs index ecface759..8d79fb609 100644 --- a/src/command/mod.rs +++ b/src/command/mod.rs @@ -506,10 +506,13 @@ fn dispatch_inner( } } (6, b'm') => { - // MEMORY (USAGE, STATS, DOCTOR, HELP) + // MEMORY (USAGE, STATS, DOCTOR, HELP), MSETNX if cmd.eq_ignore_ascii_case(b"MEMORY") { return resp(server_admin::memory(db, args)); } + if cmd.eq_ignore_ascii_case(b"MSETNX") { + return resp(string::msetnx(db, args)); + } } (6, b'o') => { // OBJECT diff --git a/src/command/string/mod.rs b/src/command/string/mod.rs index c573f87e7..97d699fcb 100644 --- a/src/command/string/mod.rs +++ b/src/command/string/mod.rs @@ -266,6 +266,44 @@ mod tests { assert!(matches!(result, Frame::Error(_))); } + #[test] + fn test_msetnx_all_new() { + // All keys absent -> set all, return 1. + let mut db = make_db(); + let r = msetnx(&mut db, &[bs(b"a"), bs(b"1"), bs(b"b"), bs(b"2")]); + assert_eq!(r, Frame::Integer(1)); + assert_eq!( + get(&mut db, &[bs(b"a")]), + Frame::BulkString(Bytes::from_static(b"1")) + ); + assert_eq!( + get(&mut db, &[bs(b"b")]), + Frame::BulkString(Bytes::from_static(b"2")) + ); + } + + #[test] + fn test_msetnx_one_exists_sets_none() { + // Atomic all-or-nothing: if ANY key exists, set NOTHING, return 0. + let mut db = make_db(); + db.set_string(Bytes::from_static(b"b"), Bytes::from_static(b"old")); + let r = msetnx(&mut db, &[bs(b"a"), bs(b"1"), bs(b"b"), bs(b"2")]); + assert_eq!(r, Frame::Integer(0)); + assert_eq!(get(&mut db, &[bs(b"a")]), Frame::Null, "a must not be set"); + assert_eq!( + get(&mut db, &[bs(b"b")]), + Frame::BulkString(Bytes::from_static(b"old")), + "b must be unchanged" + ); + } + + #[test] + fn test_msetnx_odd_args() { + let mut db = make_db(); + let r = msetnx(&mut db, &[bs(b"a"), bs(b"1"), bs(b"b")]); + assert!(matches!(r, Frame::Error(_))); + } + // --- INCR/DECR tests --- #[test] @@ -323,6 +361,80 @@ mod tests { assert_eq!(result, Frame::Integer(7)); } + #[test] + fn test_decrby_min_overflow() { + // DECRBY key i64::MIN negates to +2^63, which is unrepresentable -> must + // return an error, not panic (debug) or wrap (release). + let mut db = make_db(); + db.set_string(Bytes::from_static(b"n"), Bytes::from_static(b"0")); + let arg = Frame::BulkString(Bytes::from(i64::MIN.to_string())); + let result = decrby(&mut db, &[bs(b"n"), arg]); + match result { + Frame::Error(e) => assert!(e.ends_with(b"overflow"), "got {e:?}"), + other => panic!("expected overflow error, got {other:?}"), + } + } + + #[test] + fn test_setex_overflow_rejected() { + // now_ms + seconds*1000 overflows for huge seconds -> error, key not created. + let mut db = make_db(); + let huge = Frame::BulkString(Bytes::from(i64::MAX.to_string())); + let result = setex(&mut db, &[bs(b"k"), huge, bs(b"v")]); + assert!(matches!(result, Frame::Error(ref e) if e.starts_with(b"ERR invalid expire"))); + assert!(!db.exists(b"k"), "rejected SETEX must not create the key"); + } + + #[test] + fn test_set_ex_overflow_rejected() { + let mut db = make_db(); + let huge = Frame::BulkString(Bytes::from(i64::MAX.to_string())); + let result = set(&mut db, &[bs(b"k"), bs(b"v"), bs(b"EX"), huge]); + assert!(matches!(result, Frame::Error(ref e) if e.starts_with(b"ERR invalid expire"))); + assert!(!db.exists(b"k"), "rejected SET EX must not create the key"); + } + + // --- i64-domain expiry bound (Finding 2): reject expiries past i64::MAX so + // PTTL/PEXPIRETIME never wrap negative on a live key (Redis parity). --- + + #[test] + fn test_setex_i64_bound_rejected() { + // seconds*1000 fits u64 but exceeds i64::MAX -> reject (else PTTL wraps negative). + let mut db = make_db(); + let over = Frame::BulkString(Bytes::from("15000000000000000")); // 1.5e16 s + let result = setex(&mut db, &[bs(b"k"), over, bs(b"v")]); + assert!(matches!(result, Frame::Error(ref e) if e.starts_with(b"ERR invalid expire"))); + assert!(!db.exists(b"k"), "rejected SETEX must not create the key"); + } + + #[test] + fn test_set_ex_i64_bound_rejected() { + let mut db = make_db(); + let over = Frame::BulkString(Bytes::from("15000000000000000")); + let result = set(&mut db, &[bs(b"k"), bs(b"v"), bs(b"EX"), over]); + assert!(matches!(result, Frame::Error(ref e) if e.starts_with(b"ERR invalid expire"))); + assert!(!db.exists(b"k"), "rejected SET EX must not create the key"); + } + + #[test] + fn test_set_px_i64_bound_rejected() { + // now_ms + i64::MAX ms exceeds i64::MAX -> reject. + let mut db = make_db(); + let over = Frame::BulkString(Bytes::from(i64::MAX.to_string())); + let result = set(&mut db, &[bs(b"k"), bs(b"v"), bs(b"PX"), over]); + assert!(matches!(result, Frame::Error(ref e) if e.starts_with(b"ERR invalid expire"))); + assert!(!db.exists(b"k"), "rejected SET PX must not create the key"); + } + + #[test] + fn test_psetex_i64_bound_rejected() { + let mut db = make_db(); + let over = Frame::BulkString(Bytes::from(i64::MAX.to_string())); + let result = psetex(&mut db, &[bs(b"k"), over, bs(b"v")]); + assert!(matches!(result, Frame::Error(ref e) if e.starts_with(b"ERR invalid expire"))); + assert!(!db.exists(b"k"), "rejected PSETEX must not create the key"); + } + #[test] fn test_incrby() { let mut db = make_db(); @@ -557,6 +669,42 @@ mod tests { assert!(!entry.has_expiry()); } + #[test] + fn test_getset_wrongtype_preserves_key() { + // Regression (data-loss): GETSET on a wrong-type key must return WRONGTYPE + // and MUST NOT overwrite it. Previously db.set_string ran unconditionally. + let mut db = make_db(); + db.set(Bytes::from_static(b"myhash"), Entry::new_hash()); + let result = getset(&mut db, &[bs(b"myhash"), bs(b"newval")]); + match result { + Frame::Error(e) => assert!(e.starts_with(b"WRONGTYPE")), + other => panic!("Expected WRONGTYPE error, got {other:?}"), + } + // The hash must be intact: a plain GET still reports WRONGTYPE (not the new string). + match get(&mut db, &[bs(b"myhash")]) { + Frame::Error(e) => assert!(e.starts_with(b"WRONGTYPE")), + other => panic!("GETSET must not overwrite a wrong-type key, got {other:?}"), + } + } + + #[test] + fn test_set_get_option_wrongtype_preserves_key() { + // Regression (data-loss): SET key val GET on a wrong-type key must return + // WRONGTYPE and perform NO write (precedence over NX/XX). Previously db.set + // ran unconditionally, destroying the wrong-type value. + let mut db = make_db(); + db.set(Bytes::from_static(b"myhash"), Entry::new_hash()); + let result = set(&mut db, &[bs(b"myhash"), bs(b"newval"), bs(b"GET")]); + match result { + Frame::Error(e) => assert!(e.starts_with(b"WRONGTYPE")), + other => panic!("Expected WRONGTYPE error, got {other:?}"), + } + match get(&mut db, &[bs(b"myhash")]) { + Frame::Error(e) => assert!(e.starts_with(b"WRONGTYPE")), + other => panic!("SET..GET must not overwrite a wrong-type key, got {other:?}"), + } + } + // --- GETDEL tests --- #[test] @@ -575,6 +723,24 @@ mod tests { assert_eq!(result, Frame::Null); } + #[test] + fn test_getdel_wrongtype_preserves_key() { + // Regression (data-loss): GETDEL on a wrong-type key must return WRONGTYPE + // and MUST NOT delete the key. Previously db.remove() fired before the type + // check, destroying the key and then returning WRONGTYPE as if nothing happened. + let mut db = make_db(); + db.set(Bytes::from_static(b"myhash"), Entry::new_hash()); + let result = getdel(&mut db, &[bs(b"myhash")]); + match result { + Frame::Error(e) => assert!(e.starts_with(b"WRONGTYPE")), + other => panic!("Expected WRONGTYPE error, got {other:?}"), + } + assert!( + db.exists(b"myhash"), + "GETDEL on a wrong-type key must not delete it (data-loss regression)" + ); + } + // --- GETEX tests --- #[test] diff --git a/src/command/string/string_read.rs b/src/command/string/string_read.rs index 4cc4780a0..ca8187824 100644 --- a/src/command/string/string_read.rs +++ b/src/command/string/string_read.rs @@ -189,12 +189,25 @@ pub fn getdel(db: &mut Database, args: &[Frame]) -> Frame { Some(k) => k, None => return err_wrong_args("GETDEL"), }; + // Check-before-mutate (Redis parity): peek the type via a cheap borrow BEFORE + // removing. Removing a wrong-type key and only then returning WRONGTYPE would + // destroy the key (data-loss). as_bytes() borrows without cloning; the borrow + // ends before db.remove() so the mutable re-borrow is sound. + match db.get(key) { + None => return Frame::Null, + Some(entry) => { + if entry.value.as_bytes().is_none() { + return Frame::Error(Bytes::from_static( + b"WRONGTYPE Operation against a key holding the wrong kind of value", + )); + } + } + } + // Confirmed string — safe to remove and return its bytes. match db.remove(key) { Some(entry) => match entry.value.as_bytes_owned() { Some(v) => Frame::BulkString(v), - None => Frame::Error(Bytes::from_static( - b"WRONGTYPE Operation against a key holding the wrong kind of value", - )), + None => Frame::Null, }, None => Frame::Null, } diff --git a/src/command/string/string_write.rs b/src/command/string/string_write.rs index fab0f691a..8f35c33b8 100644 --- a/src/command/string/string_write.rs +++ b/src/command/string/string_write.rs @@ -5,7 +5,7 @@ use crate::storage::Database; use crate::storage::entry::{Entry, current_time_ms}; use super::{format_float, parse_f64, parse_i64, parse_positive_i64}; -use crate::command::helpers::{err_wrong_args, extract_bytes, ok}; +use crate::command::helpers::{err_wrong_args, expiry_ms_in_range, extract_bytes, ok}; /// SET command handler with EX/PX/EXAT/PXAT/NX/XX/KEEPTTL/GET options. pub fn set(db: &mut Database, args: &[Frame]) -> Frame { @@ -51,7 +51,20 @@ pub fn set(db: &mut Database, args: &[Frame]) -> Frame { return Frame::Error(Bytes::from_static(b"ERR syntax error")); } match parse_positive_i64(&args[i]) { - Some(secs) => expires_at_ms = current_time_ms() + (secs as u64) * 1000, + Some(secs) => { + expires_at_ms = match (secs as u64) + .checked_mul(1000) + .and_then(|d| current_time_ms().checked_add(d)) + .filter(|ms| expiry_ms_in_range(*ms)) + { + Some(ms) => ms, + None => { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'SET' command", + )); + } + } + } None => { return Frame::Error(Bytes::from_static( b"ERR value is not an integer or out of range", @@ -64,7 +77,19 @@ pub fn set(db: &mut Database, args: &[Frame]) -> Frame { return Frame::Error(Bytes::from_static(b"ERR syntax error")); } match parse_positive_i64(&args[i]) { - Some(ms) => expires_at_ms = current_time_ms() + ms as u64, + Some(ms) => { + expires_at_ms = match current_time_ms() + .checked_add(ms as u64) + .filter(|v| expiry_ms_in_range(*v)) + { + Some(v) => v, + None => { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'SET' command", + )); + } + }; + } None => { return Frame::Error(Bytes::from_static( b"ERR value is not an integer or out of range", @@ -78,7 +103,17 @@ pub fn set(db: &mut Database, args: &[Frame]) -> Frame { } match parse_positive_i64(&args[i]) { Some(ts) => { - expires_at_ms = (ts as u64) * 1000; + expires_at_ms = match (ts as u64) + .checked_mul(1000) + .filter(|ms| expiry_ms_in_range(*ms)) + { + Some(ms) => ms, + None => { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'SET' command", + )); + } + }; } None => { return Frame::Error(Bytes::from_static( @@ -127,6 +162,13 @@ pub fn set(db: &mut Database, args: &[Frame]) -> Frame { None }; + // Check-before-mutate (Redis parity): SET ... GET on a wrong-type key returns + // WRONGTYPE and performs NO write. This takes precedence over NX/XX handling. + // Previously db.set(...) ran unconditionally, overwriting (destroying) the value. + if matches!(old_value, Some(Frame::Error(_))) { + return old_value.unwrap_or(Frame::Null); + } + // NX + XX both set: contradictory, return nil (or old value if GET) if nx && xx { return if get_old { @@ -197,6 +239,43 @@ pub fn mset(db: &mut Database, args: &[Frame]) -> Frame { ok() } +/// MSETNX command handler (single-shard atomic). +/// +/// Sets all key/value pairs only if NONE of the keys already exist. Returns 1 if +/// all were set, 0 if at least one key already existed (and nothing was set). +/// +/// Atomic on a single shard/database (the two phases run without any await point). +/// Cross-shard atomicity is enforced by the coordinator, which rejects MSETNX when +/// keys span shards (CROSSSLOT); see `coordinate_msetnx`. +pub fn msetnx(db: &mut Database, args: &[Frame]) -> Frame { + if args.is_empty() || !args.len().is_multiple_of(2) { + return err_wrong_args("MSETNX"); + } + // Phase 1: verify NONE of the keys exist. + for pair in args.chunks(2) { + let key = match extract_bytes(&pair[0]) { + Some(k) => k, + None => return err_wrong_args("MSETNX"), + }; + if db.exists(key) { + return Frame::Integer(0); + } + } + // Phase 2: all keys absent -> set them all. + for pair in args.chunks(2) { + let key = match extract_bytes(&pair[0]) { + Some(k) => k.clone(), + None => return err_wrong_args("MSETNX"), + }; + let value = match extract_bytes(&pair[1]) { + Some(v) => v.clone(), + None => return err_wrong_args("MSETNX"), + }; + db.set_string(key, value); + } + Frame::Integer(1) +} + /// INCR command handler. pub fn incr(db: &mut Database, args: &[Frame]) -> Frame { if args.len() != 1 { @@ -258,7 +337,15 @@ pub fn decrby(db: &mut Database, args: &[Frame]) -> Frame { )); } }; - incrby_internal(db, key, -delta) + // Guard i64::MIN: -(i64::MIN) is unrepresentable. Redis returns an overflow + // error rather than negating with a debug panic / release wrap. + let neg = match delta.checked_neg() { + Some(n) => n, + None => { + return Frame::Error(Bytes::from_static(b"ERR decrement would overflow")); + } + }; + incrby_internal(db, key, neg) } /// Internal helper for INCR/DECR/INCRBY/DECRBY. @@ -580,7 +667,19 @@ pub fn setex(db: &mut Database, args: &[Frame]) -> Frame { Some(v) => v.clone(), None => return err_wrong_args("SETEX"), }; - db.set_string_with_expiry(key, value, current_time_ms() + (seconds as u64) * 1000); + let expires_at_ms = match (seconds as u64) + .checked_mul(1000) + .and_then(|d| current_time_ms().checked_add(d)) + .filter(|ms| expiry_ms_in_range(*ms)) + { + Some(ms) => ms, + None => { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'SETEX' command", + )); + } + }; + db.set_string_with_expiry(key, value, expires_at_ms); ok() } @@ -611,7 +710,18 @@ pub fn psetex(db: &mut Database, args: &[Frame]) -> Frame { Some(v) => v.clone(), None => return err_wrong_args("PSETEX"), }; - db.set_string_with_expiry(key, value, current_time_ms() + millis as u64); + let expires_at_ms = match current_time_ms() + .checked_add(millis as u64) + .filter(|ms| expiry_ms_in_range(*ms)) + { + Some(ms) => ms, + None => { + return Frame::Error(Bytes::from_static( + b"ERR invalid expire time in 'PSETEX' command", + )); + } + }; + db.set_string_with_expiry(key, value, expires_at_ms); ok() } @@ -636,8 +746,14 @@ pub fn getset(db: &mut Database, args: &[Frame]) -> Frame { )), }); - // GETSET removes TTL (sets new entry without expiry) - db.set_string(key, value); - - old.unwrap_or(Frame::Null) + // Check-before-mutate (Redis parity): a wrong-type key must return WRONGTYPE + // and stay untouched. Previously db.set_string ran unconditionally, destroying it. + match old { + Some(Frame::Error(e)) => Frame::Error(e), + other => { + // GETSET removes TTL (sets new entry without expiry) + db.set_string(key, value); + other.unwrap_or(Frame::Null) + } + } } diff --git a/src/config.rs b/src/config.rs index 8e0162856..0bca8fda3 100644 --- a/src/config.rs +++ b/src/config.rs @@ -177,11 +177,13 @@ pub struct ServerConfig { /// Number of shards (0 = auto-detect from CPU count). /// - /// Defaults to 1: single-shard gives the best throughput for - /// non-pipelined workloads (cross-shard SPSC dispatch dominates local - /// lookups otherwise) and a deterministic persistence layout across - /// hosts. Pass `--shards 0` to auto-detect from the CPU count, or pin - /// an explicit count for pipelined/AOF-heavy multi-core deployments. + /// Defaults to 1: single-shard gives the best per-op latency for + /// low-concurrency, non-pipelined workloads (a cross-shard hop costs + /// ~10µs) and a deterministic persistence layout across hosts. Pin an + /// explicit count (e.g. 4) for 8+ concurrent connections or pipelined + /// traffic — measured 1.3-1.9x Redis at 8-64 conns on 4 shards — or + /// pass `--shards 0` to auto-detect on a dedicated host. See + /// docs/guides/tuning.md. #[arg(long, default_value_t = 1)] pub shards: usize, @@ -258,6 +260,25 @@ pub struct ServerConfig { #[arg(long = "uring-sqpoll")] pub uring_sqpoll_ms: Option, + /// I/O driver for the monoio runtime. "auto" lets FusionDriver pick + /// (io_uring on Linux when available, else epoll/kqueue); "epoll" forces + /// the legacy poller. Measured on GCE ARM (c4a Axion, 2026-07): epoll is + /// 2-4% faster than io_uring across ALL pipeline depths for KV workloads, + /// while other platforms (e.g. OrbStack aarch64) favor io_uring — bench + /// per platform. Equivalent env kill-switch: MOON_NO_URING=1. + #[arg(long = "io-driver", default_value = "auto", value_parser = ["auto", "epoll"])] + pub io_driver: String, + + /// Busy-poll the shard event loop for N microseconds before sleeping + /// (0 = disabled). Implies `--io-driver epoll`. Poll-mode park: the shard + /// thread spins on readiness (zero-timeout polls) instead of blocking, so + /// the scheduler sleep+wake disappears from the request path. Best for + /// low-pipeline request/response workloads on dedicated cores; costs up to + /// ~N µs of CPU per idle park. Measured (GCE c1 GET p=1, 2026-07): + /// ARM c4a 0.95→1.21× vs Redis, x86 c3 1.06→1.66×. monoio runtime only. + #[arg(long = "io-busy-poll-us", default_value_t = 0)] + pub io_busy_poll_us: u64, + // ── MoonStore v2: Disk Offload ────────────────────────────────── /// Enable disk offload (tiered storage: RAM -> mmap -> NVMe) #[arg(long = "disk-offload", default_value = "enable")] @@ -1167,6 +1188,25 @@ mod tests { assert_eq!(config.shards, 0); } + #[test] + fn test_io_driver_flag_parses_and_rejects_unknown() { + let config = ServerConfig::parse_from::<[&str; 0], &str>([]); + assert_eq!(config.io_driver, "auto", "auto must stay the default"); + let config = ServerConfig::parse_from(["moon", "--io-driver", "epoll"]); + assert_eq!(config.io_driver, "epoll"); + // clap-level validation: anything outside auto|epoll is a parse error. + assert!(ServerConfig::try_parse_from(["moon", "--io-driver", "iouring"]).is_err()); + } + + #[test] + fn test_io_busy_poll_flag_parses_with_zero_default() { + let config = ServerConfig::parse_from::<[&str; 0], &str>([]); + assert_eq!(config.io_busy_poll_us, 0, "busy-poll must default OFF"); + let config = ServerConfig::parse_from(["moon", "--io-busy-poll-us", "40"]); + assert_eq!(config.io_busy_poll_us, 40); + assert!(ServerConfig::try_parse_from(["moon", "--io-busy-poll-us", "x"]).is_err()); + } + #[test] fn test_custom_port() { let config = ServerConfig::parse_from(["moon", "--port", "6380"]); diff --git a/src/main.rs b/src/main.rs index 9545e7587..922b6e7a6 100644 --- a/src/main.rs +++ b/src/main.rs @@ -803,6 +803,28 @@ fn main() -> anyhow::Result<()> { std::sync::Arc::new(std::sync::RwLock::new(table)) }; + // I/O driver selection — must land BEFORE any shard thread spawns so + // every monoio runtime observes it (clap restricts values to auto|epoll). + if config.io_driver == "epoll" { + moon::runtime::force_legacy_driver(); + tracing::info!("I/O driver: legacy poller (epoll/kqueue) forced via --io-driver epoll"); + } + if config.io_busy_poll_us > 0 { + // Busy-poll parks only exist in the legacy (readiness) driver: io_uring + // CQEs are not observable from userspace without task participation. + moon::runtime::force_legacy_driver(); + #[cfg(feature = "runtime-monoio")] + { + monoio::set_legacy_spin_budget_us(config.io_busy_poll_us); + tracing::info!( + "I/O busy-poll: {}µs readiness spin before park (epoll/kqueue driver forced)", + config.io_busy_poll_us + ); + } + #[cfg(not(feature = "runtime-monoio"))] + tracing::warn!("--io-busy-poll-us has no effect under the tokio runtime"); + } + // Build shared runtime config for sharded handlers let runtime_config_shared: std::sync::Arc> = { std::sync::Arc::new(parking_lot::RwLock::new(config.to_runtime_config())) }; @@ -810,6 +832,10 @@ fn main() -> anyhow::Result<()> { // whole-instance cap (per-shard budget = maxmemory / num_shards). Without // this, each shard would tolerate the full maxmemory → ~N× aggregate RSS. runtime_config_shared.write().num_shards = num_shards; + // Publish the lock-free maxmemory hints AFTER num_shards is resolved (the + // per-shard hint divides by it) — the inline write path's eviction + // pre-gate reads these instead of taking the runtime-config lock. + moon::storage::eviction::publish_maxmemory_hints(&runtime_config_shared.read()); moon::config::log_maxmemory_sharding(runtime_config_shared.read().maxmemory, num_shards); let server_config_shared: std::sync::Arc = { std::sync::Arc::new(config.clone()) }; diff --git a/src/persistence/replay.rs b/src/persistence/replay.rs index 0dd4d4f36..bd6fc9e7b 100644 --- a/src/persistence/replay.rs +++ b/src/persistence/replay.rs @@ -203,6 +203,56 @@ mod tests { ); } + // ── EXPIRE<=0 replay / propagation guard ────────────────────────────────── + + /// P0 propagation guard: WAL replay of `EXPIRE k -1` (non-positive TTL) must + /// DELETE the key, exactly as the live handler now does. Replay re-dispatches + /// the raw command, so a WAL-recovered replica stays consistent with the + /// master. Before the EXPIRE<=0 fix this failed (replay kept the key because + /// the handler returned an error and mutated nothing). + #[test] + fn replay_expire_nonpositive_deletes_key() { + let engine = DispatchReplayEngine::new(); + let mut databases = vec![make_db_with_key(b"k", b"v")]; + let mut selected = 0usize; + assert_eq!( + get_key(&mut databases[0], b"k").as_deref(), + Some(b"v".as_ref()), + "precondition: key present before replay" + ); + + let args = framevec![ + Frame::BulkString(bytes::Bytes::from_static(b"k")), + Frame::BulkString(bytes::Bytes::from_static(b"-1")), + ]; + engine.replay_command(&mut databases, b"EXPIRE", &args, &mut selected); + + assert_eq!( + get_key(&mut databases[0], b"k"), + None, + "replay of EXPIRE k -1 must delete the key (no master/replica divergence)" + ); + } + + /// Sibling anchor: EXPIREAT with a past timestamp already deletes on replay. + /// Confirms the EXPIRE fix matches the established EXPIREAT propagation. + #[test] + fn replay_expireat_past_deletes_key() { + let engine = DispatchReplayEngine::new(); + let mut databases = vec![make_db_with_key(b"k", b"v")]; + let mut selected = 0usize; + let args = framevec![ + Frame::BulkString(bytes::Bytes::from_static(b"k")), + Frame::BulkString(bytes::Bytes::from_static(b"-1")), + ]; + engine.replay_command(&mut databases, b"EXPIREAT", &args, &mut selected); + assert_eq!( + get_key(&mut databases[0], b"k"), + None, + "replay of EXPIREAT k -1 must delete the key" + ); + } + /// Same-index SWAPDB is a no-op during replay. #[test] fn replay_swapdb_same_index_noop() { diff --git a/src/runtime/channel.rs b/src/runtime/channel.rs index b9fee8fea..234f8b73d 100644 --- a/src/runtime/channel.rs +++ b/src/runtime/channel.rs @@ -169,18 +169,76 @@ impl WatchReceiver { pub struct Notify { tx: flume::Sender<()>, rx: flume::Receiver<()>, + // Busy-poll skip-notify handshake: while the owning shard's epoll driver + // spin-polls (probing its SPSC ringbufs itself — see the vendored monoio + // legacy driver "moon patch" and the hook registration in event_loop), + // it advertises `true` here and senders elide the entire cross-thread + // wake: the flume send, the foreign-waker relay, and the eventfd syscall. + // Always false unless the owning shard registered spin hooks. + skip_wake: std::sync::atomic::AtomicBool, +} + +// Process-global gate for the skip-wake fast path: flipped once at shard +// startup when busy-poll is configured with >1 shard. Keeps the Dekker fence +// out of notify_one for every other deployment shape. +static NOTIFY_SKIP_WAKE_ENABLED: std::sync::atomic::AtomicBool = + std::sync::atomic::AtomicBool::new(false); + +/// Enable the skip-wake fast path process-wide (idempotent; called per shard +/// at startup when `--io-busy-poll-us` is active with more than one shard). +pub fn enable_notify_skip_wake() { + NOTIFY_SKIP_WAKE_ENABLED.store(true, std::sync::atomic::Ordering::Relaxed); +} + +#[inline] +fn notify_skip_wake_enabled() -> bool { + NOTIFY_SKIP_WAKE_ENABLED.load(std::sync::atomic::Ordering::Relaxed) } impl Notify { pub fn new() -> Self { let (tx, rx) = flume::bounded(1); - Self { tx, rx } + Self { + tx, + rx, + skip_wake: std::sync::atomic::AtomicBool::new(false), + } } pub fn notify_one(&self) { + if notify_skip_wake_enabled() { + // Dekker handshake with set_skip_wake(false): the SeqCst fence + // orders the caller's ringbuf push before this flag load, so + // either the spinning target's final probe sees the push or this + // load sees the cleared flag — a wake is never lost (classic + // store-buffering prevention; both sides fence). + std::sync::atomic::fence(std::sync::atomic::Ordering::SeqCst); + if self.skip_wake.load(std::sync::atomic::Ordering::Relaxed) { + crate::admin::metrics_setup::bump_spsc_notify_skipped(); + return; + } + } let _ = self.tx.try_send(()); } + /// Deliver the token from the owning shard's own thread, bypassing the + /// skip-wake gate. Used by the driver spin probe: the registered waker is + /// task-local there, so the wake is a run-queue push, not a syscall. + pub fn notify_local(&self) { + let _ = self.tx.try_send(()); + } + + /// Advertise (or retract) that the owning shard's driver is busy-polling + /// and will discover ringbuf items via its spin probe. Retraction fences + /// SeqCst before the caller's final probe (pairs with notify_one). + pub fn set_skip_wake(&self, spinning: bool) { + self.skip_wake + .store(spinning, std::sync::atomic::Ordering::SeqCst); + if !spinning { + std::sync::atomic::fence(std::sync::atomic::Ordering::SeqCst); + } + } + pub async fn notified(&self) { let _ = self.rx.recv_async().await; } @@ -199,6 +257,31 @@ mod tests { use futures::executor::block_on; use futures::stream::FuturesUnordered; + #[test] + fn test_notify_skip_wake_gates_sender_but_not_local() { + let n = Notify::new(); + // Gate off (default): notify_one delivers even if the flag is set. + n.set_skip_wake(true); + n.notify_one(); + assert!(n.rx.try_recv().is_ok(), "gate off: token must deliver"); + + enable_notify_skip_wake(); + // Gate on + flag set: sender-side notify is elided entirely. + n.notify_one(); + assert!( + n.rx.try_recv().is_err(), + "gate on + spinning: notify_one must be skipped" + ); + // The driver probe's local delivery bypasses the flag. + n.notify_local(); + assert!(n.rx.try_recv().is_ok(), "notify_local must bypass the gate"); + + // Flag retracted: normal delivery resumes. + n.set_skip_wake(false); + n.notify_one(); + assert!(n.rx.try_recv().is_ok(), "flag clear: token must deliver"); + } + #[test] fn test_oneshot_send_then_recv() { let (tx, rx) = oneshot::(); diff --git a/src/runtime/mod.rs b/src/runtime/mod.rs index bfe0a8a5e..d84a097db 100644 --- a/src/runtime/mod.rs +++ b/src/runtime/mod.rs @@ -25,6 +25,44 @@ pub mod channel; pub mod race; pub mod traits; +/// Process-wide "force the epoll/kqueue LegacyDriver" switch for the monoio +/// runtime, set ONCE from `--io-driver epoll` in main BEFORE any shard thread +/// spawns (safe alternative to mutating `MOON_NO_URING` via unsafe `set_var`). +/// Read by `MonoioRuntimeFactory::block_on_local` alongside the env var. +static FORCE_LEGACY_DRIVER: std::sync::atomic::AtomicBool = + std::sync::atomic::AtomicBool::new(false); + +/// Request the legacy (epoll/kqueue) monoio driver for all shards. Must be +/// called before shard threads spawn; later calls still apply to any runtime +/// built afterwards but never to already-running shards. +pub fn force_legacy_driver() { + FORCE_LEGACY_DRIVER.store(true, std::sync::atomic::Ordering::Release); +} + +/// True when `--io-driver epoll` (or `MOON_NO_URING=1`) forces the legacy driver. +pub fn legacy_driver_forced() -> bool { + FORCE_LEGACY_DRIVER.load(std::sync::atomic::Ordering::Acquire) + || std::env::var_os("MOON_NO_URING").is_some() +} + +/// True when the epoll busy-poll park is configured — via the +/// `--io-busy-poll-us` flag (the caller passes the config value) or the +/// `MOON_EPOLL_SPIN_US` env fallback the vendored driver also honors. Gates +/// the skip-notify hook registration in the shard event loop. +pub fn epoll_spin_configured(flag_us: u64) -> bool { + if flag_us > 0 { + return true; + } + static ENV_SPIN: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENV_SPIN.get_or_init(|| { + std::env::var("MOON_EPOLL_SPIN_US") + .ok() + .and_then(|v| v.parse::().ok()) + .unwrap_or(0) + > 0 + }) +} + /// Cooperatively relinquish to the shard event loop, letting co-located /// connections + the 1ms tick make progress, then resume. /// diff --git a/src/runtime/monoio_impl.rs b/src/runtime/monoio_impl.rs index 7596f9137..4ef3514c1 100644 --- a/src/runtime/monoio_impl.rs +++ b/src/runtime/monoio_impl.rs @@ -58,6 +58,89 @@ pub struct MonoioRuntimeFactory; impl RuntimeFactory for MonoioRuntimeFactory { fn block_on_local + 'static>(name: String, f: F) { + // Honor the documented MOON_NO_URING contract (and `--io-driver + // epoll`) for the monoio runtime too: FusionDriver silently + // auto-picks io_uring on Linux and offers no runtime introspection, + // so the kill-switch must force the epoll/kqueue LegacyDriver + // explicitly. (Previously MOON_NO_URING only gated the tokio bridge + // + uring_active(); the monoio driver choice ignored it — discovered + // during the c4a p=1 driver A/B.) + if crate::runtime::legacy_driver_forced() { + let mut rt = monoio::RuntimeBuilder::::new() + .enable_timer() + .build() + .unwrap_or_else(|e| { + panic!("failed to build monoio legacy runtime '{}': {}", name, e) + }); + rt.block_on(f); + return; + } + // Tuned io_uring: COOP_TASKRUN + SINGLE_ISSUER + DEFER_TASKRUN slash + // io_uring_enter cost for a thread-per-core ring (created and entered + // only on this shard thread; moon's cross-thread signalling is fd-based + // — flume/eventfd/UnixStream — never a ring op from another thread, + // which DEFER_TASKRUN forbids). Measured need: on GCE (c4a/c3) the + // stock ring's enter cost made io_uring LOSE to epoll at p=1. + // Kernels <6.1 reject the flags -> warn once and fall back to the + // stock FusionDriver below. MOON_URING_PLAIN=1 skips the tuning. + #[cfg(target_os = "linux")] + if std::env::var_os("MOON_URING_PLAIN").is_none() { + let mut urb = io_uring_06::IoUring::builder(); + // SQPOLL experiment gate: MOON_URING_SQPOLL= starts a + // kernel-side submission-polling thread — submits become shared- + // memory writes (no io_uring_enter), dropping p=1 to ~1 syscall/op + // vs Redis's 3. Costs a busy core while active. SQPOLL is mutually + // exclusive with DEFER_TASKRUN (kernel rejects the combo), so the + // tuned taskrun flags are skipped in this mode. + // MOON_URING_SQPOLL_CPU= pins the poll thread (IORING_SETUP_SQ_AFF); + // REQUIRED when the server process is taskset-pinned — the kthread + // inherits the process affinity and would otherwise fight the shard + // thread for its core. + let sqpoll_idle_ms = std::env::var("MOON_URING_SQPOLL") + .ok() + .and_then(|v| v.parse::().ok()); + // CQ spin (vendored-monoio MOON_URING_SPIN_US) requires completions + // to be observable from userspace WITHOUT an io_uring_enter: + // DEFER_TASKRUN posts CQEs only inside enter(GETEVENTS) and + // COOP_TASKRUN only at kernel-entry boundaries, so either flag makes + // the spin burn its whole budget every park (measured: 49µs/op on + // c4a, a 3× regression). Spin mode therefore forces a plain ring; + // combined with SQPOLL the sqpoll kthread posts CQEs continuously + // and the spinning shard thread reaps with zero per-op syscalls. + let spin_active = std::env::var_os("MOON_URING_SPIN_US").is_some(); + if let Some(idle_ms) = sqpoll_idle_ms { + urb.setup_sqpoll(idle_ms); + if let Some(cpu) = std::env::var("MOON_URING_SQPOLL_CPU") + .ok() + .and_then(|v| v.parse::().ok()) + { + urb.setup_sqpoll_cpu(cpu); + } + } else if !spin_active { + urb.setup_coop_taskrun() + .setup_single_issuer() + .setup_defer_taskrun(); + } + match monoio::RuntimeBuilder::::new() + .uring_builder(urb) + .enable_timer() + .build() + { + Ok(mut rt) => { + rt.block_on(f); + return; + } + Err(e) => { + tracing::warn!( + "tuned io_uring (COOP_TASKRUN|SINGLE_ISSUER|DEFER_TASKRUN) \ + unavailable for '{}' ({}); falling back to stock driver", + name, + e + ); + } + } + } + let mut rt = monoio::RuntimeBuilder::::new() .enable_timer() .build() diff --git a/src/server/conn/blocking.rs b/src/server/conn/blocking.rs index b47022100..38cee6322 100644 --- a/src/server/conn/blocking.rs +++ b/src/server/conn/blocking.rs @@ -1261,41 +1261,48 @@ pub(crate) fn try_inline_dispatch( let consumed = key_end_crlf; let key_bytes = &buf[key_start..key_end]; // Hot-key sampling + lookup via thread-local slice — no lock needed. - enum GetResult { - Found(Vec), - WrongType, + // Frame the reply straight from the borrowed `&[u8]` INSIDE the closure so + // the value is copied exactly once (borrow -> write_buf), never via an + // intermediate `Vec` (`val.to_vec()`). The closure writes response bytes + + // itoa ONLY; it must not re-enter `with_shard*` (thread-local RefCell + // reentrancy) and takes no `.await`. + enum GetOutcome { + // Hit or wrong-type: the response is already framed into `write_buf`. + Handled, + // Absent locally: consult the cold tier below (`cold_loc` carries where). Miss, } - let (inline_result, cold_loc) = crate::shard::slice::with_shard_db(selected_db, |db| { + let (outcome, cold_loc) = crate::shard::slice::with_shard_db(selected_db, |db| { if db.hot_keys().tick() { db.hot_keys().observe(key_bytes); } match db.get_if_alive(key_bytes, now_ms) { Some(entry) => match entry.value.as_bytes() { - Some(val) => (GetResult::Found(val.to_vec()), None), - None => (GetResult::WrongType, None), + Some(val) => { + write_buf.extend_from_slice(b"$"); + let mut itoa_buf = itoa::Buffer::new(); + write_buf.extend_from_slice(itoa_buf.format(val.len()).as_bytes()); + write_buf.extend_from_slice(b"\r\n"); + write_buf.extend_from_slice(val); + write_buf.extend_from_slice(b"\r\n"); + (GetOutcome::Handled, None) + } + None => { + write_buf.extend_from_slice( + b"-WRONGTYPE Operation against a key holding the wrong kind of value\r\n", + ); + (GetOutcome::Handled, None) + } }, None => { let loc = db.cold_lookup_location(key_bytes); - (GetResult::Miss, loc) + (GetOutcome::Miss, loc) } } }); - match inline_result { - GetResult::Found(val) => { - write_buf.extend_from_slice(b"$"); - let mut itoa_buf = itoa::Buffer::new(); - write_buf.extend_from_slice(itoa_buf.format(val.len()).as_bytes()); - write_buf.extend_from_slice(b"\r\n"); - write_buf.extend_from_slice(&val); - write_buf.extend_from_slice(b"\r\n"); - } - GetResult::WrongType => { - write_buf.extend_from_slice( - b"-WRONGTYPE Operation against a key holding the wrong kind of value\r\n", - ); - } - GetResult::Miss => { + match outcome { + GetOutcome::Handled => {} + GetOutcome::Miss => { let cold = cold_loc.and_then(|(loc, shard_dir)| { crate::storage::tiered::cold_read::read_cold_entry_at(&shard_dir, loc, now_ms) }); @@ -1315,8 +1322,6 @@ pub(crate) fn try_inline_dispatch( } else { write_buf.extend_from_slice(b"$-1\r\n"); } - let _ = read_buf.split_to(consumed); - return 1; } } let _ = read_buf.split_to(consumed); @@ -1367,30 +1372,39 @@ pub(crate) fn try_inline_dispatch( // We must not index into `buf` after this point — use `frozen` instead. let frozen = read_buf.split_to(consumed).freeze(); - // Eviction check + write via the thread-local slice (no lock needed). + // Eviction check + write via the thread-local slice. The lock-free + // pre-gate proves the common case (no memory pressure / no limit) without + // the per-SET `runtime_config.read()` lock pair; only under pressure — + // or before the hints are published — does the full locked path run. { - let rt = runtime_config.read(); let budget = shard_databases.elastic_budget(shard_id); - let oom = crate::shard::slice::with_shard_db(selected_db, |db| { - crate::storage::eviction::try_evict_if_needed_budget(db, &rt, budget).is_err() - }); - drop(rt); - if oom { - write_buf - .extend_from_slice(b"-OOM command not allowed when used memory > 'maxmemory'\r\n"); - return 1; + let est = crate::shard::slice::with_shard_db(selected_db, |db| db.estimated_memory()); + if !crate::storage::eviction::inline_write_can_skip_eviction(est, budget) { + let rt = runtime_config.read(); + let oom = crate::shard::slice::with_shard_db(selected_db, |db| { + crate::storage::eviction::try_evict_if_needed_budget(db, &rt, budget).is_err() + }); + drop(rt); + if oom { + write_buf.extend_from_slice( + b"-OOM command not allowed when used memory > 'maxmemory'\r\n", + ); + return 1; + } } let key = frozen.slice(key_start..key_end); let value = frozen.slice(val_start..val_end); - crate::shard::slice::with_shard_db(selected_db, |db| { + // `move` closure: `key` and `value` are consumed here (last use), so the + // entry/set take ownership — no Bytes refcount bump+drop pair per SET. + crate::shard::slice::with_shard_db(selected_db, move |db| { if db.hot_keys().tick() { db.hot_keys().observe(&key); } - let mut entry = crate::storage::entry::Entry::new_string(value.clone()); + let mut entry = crate::storage::entry::Entry::new_string(value); entry.set_last_access(db.now()); entry.set_access_counter(5); - db.set(key.clone(), entry); + db.set(key, entry); }); } diff --git a/src/server/conn/handler_monoio/dispatch.rs b/src/server/conn/handler_monoio/dispatch.rs index 44708e05e..f1c7368cd 100644 --- a/src/server/conn/handler_monoio/dispatch.rs +++ b/src/server/conn/handler_monoio/dispatch.rs @@ -835,6 +835,7 @@ pub(super) fn try_handle_client_admin( in_multi: conn.in_multi, blocked: false, }, + crate::storage::entry::current_time_ms(), ); }); let list = crate::client_registry::client_list(); @@ -852,6 +853,7 @@ pub(super) fn try_handle_client_admin( in_multi: conn.in_multi, blocked: false, }, + crate::storage::entry::current_time_ms(), ); }); let info = crate::client_registry::client_info(client_id).unwrap_or_default(); @@ -1286,6 +1288,8 @@ pub(super) async fn try_handle_cross_shard_commands( &ctx.dispatch_tx, &ctx.spsc_notifiers, &ctx.cached_clock, + ctx.aof_pool.as_ref(), + &ctx.repl_state, &(), // monoio: coordinator uses oneshot, not response_pool ) .await; diff --git a/src/server/conn/handler_monoio/mod.rs b/src/server/conn/handler_monoio/mod.rs index 166019efc..dc0fa2922 100644 --- a/src/server/conn/handler_monoio/mod.rs +++ b/src/server/conn/handler_monoio/mod.rs @@ -11,7 +11,6 @@ mod txn; mod write; use crate::runtime::cancel::CancellationToken; -use crate::runtime::channel; use bytes::{Bytes, BytesMut}; use ringbuf::traits::Producer; use std::cell::RefCell; @@ -39,32 +38,15 @@ use super::{ use crate::framevec; use crate::pubsub::subscriber::Subscriber; use crate::server::codec::RespCodec; +use crate::server::response_slot::ResponseSlotPool; use crate::shard::dispatch::ShardMessage; -// ResponseSlotPool is not used on monoio (yet): this handler predates the -// proof that cross-thread wakes DO reach monoio tasks (the `sync` feature's -// waker channel + driver unpark — see tests/spsc_wake_floor_red.rs::swf0). -// The flume oneshot per batch works the same way; unifying on the -// zero-allocation ResponseSlotPool is a candidate follow-up. - -// ── F3: cross-shard dispatch backpressure / response-wait bounds ── -// Design-for-failure: a wedged or saturated target shard must surface a -// bounded error, never park the connection (holding its buffers) forever. -// Push-retry bounds live in `crate::shard::dispatch` (shared with the tokio -// handler); the response-wait bounds below are monoio-specific (the tokio -// path awaits via `ResponseSlotPool`, not a flume oneshot). - -/// Chunk length for the bounded cross-shard reply wait (M2, spsc-wake-floor). -/// Replies normally wake the task directly (cross-thread waker via monoio's -/// `sync` feature); the chunking only bounds how long a shutdown can go -/// unnoticed when the reply never arrives. -#[cfg(feature = "runtime-monoio")] -const CROSS_SHARD_RESPONSE_CHUNK_MS: u64 = 100; - -/// Total reply-wait budget before declaring the response lost. The batch was -/// already dispatched, so this is an *uncertain write* backstop (the command -/// may have applied on the target) — set generously (~30s). -#[cfg(feature = "runtime-monoio")] -const CROSS_SHARD_RESPONSE_TIMEOUT_MS: u64 = 30_000; +// L3b: the Phase 2b cross-shard batch path awaits replies via the +// zero-allocation `ResponseSlotPool` (tokio parity, handler_sharded), not a +// per-batch flume oneshot. Cross-thread wakes DO reach monoio tasks (the +// `sync` feature's waker channel + driver unpark — proven at runtime by +// tests/spsc_wake_floor_red.rs::swf0 on both drivers); the slot's +// AtomicWaker rides that mechanism. Transaction (txn.rs) and blocking-write +// (write.rs) paths remain on oneshots — they are off the hot path. /// Result of `handle_connection_sharded_monoio` execution. /// @@ -140,6 +122,12 @@ pub(crate) async fn handle_connection_sharded_monoio< ) -> (MonoioHandlerResult, Option) { use monoio::io::AsyncWriteRentExt; + // Solo-conn spin gate (L1 convoy fix): register this connection on the + // shard thread so the C2 reply-spin's sibling check (`xshard_may_spin`) + // sees it. RAII — decrements when the handler returns, including the + // migration hand-off (the conn re-registers on its new shard's thread). + let _conn_guard = crate::shard::slice::ShardConnGuard::new(); + // NOTE: do NOT call record_connection_opened() here — the caller // (conn_accept.rs) already increments via try_accept_connection(). @@ -205,6 +193,14 @@ pub(crate) async fn handle_connection_sharded_monoio< let mut reply_futures: Vec<(Vec<(usize, Option, Bytes)>, usize)> = Vec::with_capacity(ctx.num_shards); + // Pre-allocated response slots for zero-allocation cross-shard dispatch + // (L3b, tokio parity — handler_sharded/mod.rs). One slot per target shard; + // Phase 2b sends at most one slotted batch per target per round and drains + // every pushed slot before the round ends, so a slot is always EMPTY when + // reused. The pool lives on this task's stack — see the await-side SAFETY + // note in Phase 2b for the lifetime contract. + let response_pool = ResponseSlotPool::new(ctx.num_shards, ctx.shard_id); + // Pre-allocate frames Vec outside the loop; reused via .clear() each iteration. let mut frames: Vec = Vec::with_capacity(64); @@ -520,15 +516,15 @@ pub(crate) async fn handle_connection_sharded_monoio< // Skip when unauthenticated or workspace-bound (prefix injection in normal path only). if conn.authenticated && conn.workspace_id.is_none() { // Inline writes safe only when: ACL unrestricted, !in_multi, !tracking, - // !is_replica, no spill_sender. Replica check is non-blocking try_read. - let is_replica = ctx.repl_state.as_ref().is_some_and(|rs| { - rs.try_read().is_ok_and(|g| { - matches!( - g.role, - crate::replication::state::ReplicationRole::Replica { .. } - ) - }) - }); + // !is_replica, no spill_sender. Replica check reads the lock-free + // `is_replica_mirror` (kept in sync by `ReplicationState::set_role`) + // instead of `repl_state.try_read()` — the RwLock CAS was a measured + // per-op cost on ARM (see S3.5a note in dispatch.rs), and unlike + // try_read the mirror stays accurate while the lock is held. + let is_replica = ctx + .is_replica_mirror + .as_ref() + .is_some_and(|m| m.load(std::sync::atomic::Ordering::Acquire)); let can_inline_writes = conn.acl_skip_allowed() && !conn.in_multi && !conn.tracking_state.enabled @@ -1636,21 +1632,17 @@ pub(crate) async fn handle_connection_sharded_monoio< } } - // Phase 2b: Dispatch all deferred remote commands as batched PipelineBatch - // messages (one per target shard), await all in parallel. + // Phase 2b: Dispatch all deferred remote commands as batched + // PipelineBatchSlotted messages (one per target shard), await all in parallel. if !remote_groups.is_empty() { reply_futures.clear(); - // Capture `target` per batch so the cross-shard AOF write at the bottom - // of the loop can route to the owning shard's pool (not ctx.shard_id — - // mirrors the load-bearing fix at handler_sharded/mod.rs:1651). - let mut oneshot_futures: Vec<( - usize, // target shard — owner for AOF append - Vec<(usize, Option, Bytes)>, - channel::OneshotReceiver>, - )> = Vec::new(); + // L3b: dispatch via the pre-allocated ResponseSlotPool (no per-batch + // flume oneshot alloc). `target` is captured per batch so the H1 + // fsync barrier at the bottom of the loop can route to the owning + // shard's pool (not ctx.shard_id — mirrors handler_sharded). for (target, entries) in remote_groups.drain() { - let (reply_tx, reply_rx) = channel::oneshot(); + let slot_arc = response_pool.slot_arc(target); let (meta, commands): ( Vec<(usize, Option, Bytes)>, Vec>, @@ -1659,10 +1651,10 @@ pub(crate) async fn handle_connection_sharded_monoio< .map(|(idx, arc_frame, aof, cmd)| ((idx, aof, cmd), arc_frame)) .unzip(); - let msg = ShardMessage::PipelineBatch { + let msg = ShardMessage::PipelineBatchSlotted { db_index: conn.selected_db, commands, - reply_tx, + response_slot: crate::shard::dispatch::ResponseSlotPtr(slot_arc), }; let target_idx = ChannelMesh::target_index(ctx.shard_id, target); // F3: bounded backpressure retry. The closure retains the @@ -1693,17 +1685,19 @@ pub(crate) async fn handle_connection_sharded_monoio< match outcome { crate::shard::dispatch::PushOutcome::Pushed => { tracing::trace!( - "Shard {}: pushed PipelineBatch to shard {}, notifying", + "Shard {}: pushed PipelineBatchSlotted to shard {}, notifying", ctx.shard_id, target ); ctx.spsc_notifiers[target].notify_one(); + reply_futures.push((meta, target)); } crate::shard::dispatch::PushOutcome::Backpressure | crate::shard::dispatch::PushOutcome::Cancelled => { // Target shard not draining (saturated/wedged) or - // shutting down. The PipelineBatch was NEVER accepted, - // so this is a clean reject — fail this batch's entries + // shutting down. The batch was NEVER accepted, so this + // is a clean reject — `slot_ptr` had no side effect + // (the slot stays EMPTY); fail this batch's entries // instead of parking the connection forever. tracing::warn!( "Shard {}: cross-shard push to shard {} gave up ({:?}); rejecting batch", @@ -1716,124 +1710,56 @@ pub(crate) async fn handle_connection_sharded_monoio< b"ERR cross-shard dispatch backpressure", )); } - continue; } } - oneshot_futures.push((target, meta, reply_rx)); } - // M2 (spsc-wake-floor): await each reply oneshot DIRECTLY. The executing - // shard's `send` wakes this task cross-thread — monoio 0.2.4's `sync` - // feature routes a remote Waker::wake() through a per-thread waker - // channel + driver unpark (eventfd on io_uring, kqueue wake on the - // legacy driver), proven at runtime by tests/spsc_wake_floor_red.rs:: - // swf0 on both drivers. The previous pending_wakers relay assumed this - // was impossible and polled on the shard loop's ~1ms tick — the relay - // sweep still runs in the event loop, but this path no longer uses it. + // L3b: await each response slot directly (tokio parity — + // handler_sharded/mod.rs). Cross-thread wakes reach this task via + // the slot's AtomicWaker + monoio's `sync`-feature waker channel + // (proven by tests/spsc_wake_floor_red.rs::swf0 on both drivers). + // + // DROP-SAFETY (ResponseSlotPtr is Arc-owned): every batch pushed above + // carries an `Arc` clone, so the slot outlives BOTH this + // connection's `response_pool` AND the in-flight message. Abandoning + // this await (drop, panic-unwind, or a future shutdown break) can no + // longer dangle the target shard's late `slot.fill()` — the refcount + // keeps the slot alive until the last handle drops. (This replaced the + // old raw-pointer-into-stack-pool design, whose contract required the + // await to run to completion to avoid a panic-unwind UAF; see the + // `ResponseSlotPtr` doc + PR review.) The await is still unbounded here + // for simplicity — the target shard always fills every message it + // drains — but a shutdown-aware bound is now a safe, tracked follow-up. + // The tokio handler carries the identical await. + // // C2 pipeline guard (see XSHARD_SPIN_MAX_BATCH_REMOTE): total cross-shard // commands in THIS batch. The reply-side spin may engage only for a singleton // foreign read; >1 means a pipeline / multi-key fan-out where a synchronous // spin would serialize the reads and starve pipelined throughput (s4-P16 −27%). - let batch_remote_total: usize = - oneshot_futures.iter().map(|(_, meta, _)| meta.len()).sum(); - for (target, meta, reply_rx) in oneshot_futures.drain(..) { - tracing::trace!( - "Shard {}: awaiting cross-shard response (direct oneshot)", - ctx.shard_id - ); - // F3: bound the response wait. The batch was already dispatched, - // so a missing reply is an *uncertain write* (the target shard may - // have applied it) — the error must NOT imply rejection. Break on - // disconnect (sender gone), shutdown, or the generous ~30s cap. - // - // The wait is CHUNKED (race2 against a short sleep + is_cancelled - // check) instead of racing `shutdown.cancelled()`: CancelledFuture - // pushes a waker clone into the token's wakers Vec on every - // registration and never drains it until cancel fires — racing it - // per batch would accumulate wakers for the server's lifetime. - let shard_responses = match reply_rx.try_recv() { - // Fast path: pipelined batches often have the reply queued by - // the time this target is awaited — no future, no allocation. - Ok(value) => Ok(value), - Err(flume::TryRecvError::Disconnected) => { - Err("ERR cross-shard dispatch failed") - } - Err(flume::TryRecvError::Empty) => { - // C2 (xshard-read-fastpath): adaptive idle-gated reply-side spin. - // When this shard is near-idle (xshard_may_spin), busy-poll the - // reply for a bounded budget to skip the reply-side cross-thread - // wake (the c1 win). The poll is synchronous — it holds no borrow - // across `.await`; on miss it falls through to the EXISTING chunked - // race2 park loop UNCHANGED. When the gate is closed (busy shard) - // the path is byte-identical to before (immediate park). - let _wait_guard = crate::shard::slice::XshardWaitGuard::new(); - let mut spun = None; - if crate::shard::slice::xshard_should_spin(batch_remote_total) { - for _ in 0..crate::shard::slice::XSHARD_SPIN_BUDGET { - match reply_rx.try_recv() { - Ok(value) => { - spun = Some(Ok(value)); - break; - } - Err(flume::TryRecvError::Disconnected) => { - spun = Some(Err("ERR cross-shard dispatch failed")); - break; - } - Err(flume::TryRecvError::Empty) => core::hint::spin_loop(), - } - } - } - match spun { - Some(result) => result, - None => { - // OneshotReceiver is itself a Future with a cached inner - // recv future — pin it ONCE so its waker registration - // persists across chunk boundaries. - let mut recv = std::pin::pin!(reply_rx); - let mut waited_ms: u64 = 0; - loop { - if shutdown.is_cancelled() { - break Err("ERR cross-shard response aborted (shutdown)"); - } - let chunk = std::pin::pin!(monoio::time::sleep( - std::time::Duration::from_millis( - CROSS_SHARD_RESPONSE_CHUNK_MS - ) - )); - match crate::runtime::race::race2(recv.as_mut(), chunk).await { - crate::runtime::race::Arm::First(Ok(value)) => { - break Ok(value); - } - crate::runtime::race::Arm::First(Err(_)) => { - break Err("ERR cross-shard dispatch failed"); - } - crate::runtime::race::Arm::Second(()) => { - waited_ms += CROSS_SHARD_RESPONSE_CHUNK_MS; - if waited_ms >= CROSS_SHARD_RESPONSE_TIMEOUT_MS { - tracing::warn!( - "Shard {}: cross-shard response wait exhausted; \ - target may have applied the write", - ctx.shard_id - ); - break Err( - "ERR cross-shard response timeout (write may have applied)", - ); - } - } - } - } + let batch_remote_total: usize = reply_futures.iter().map(|(meta, _)| meta.len()).sum(); + for (meta, target) in reply_futures.drain(..) { + // C2 (xshard-read-fastpath): adaptive idle-gated reply-side spin. + // When this shard is near-idle (xshard_may_spin) AND this batch holds + // a single cross-shard read, busy-poll the response slot for a bounded + // budget to skip the reply-side cross-thread wake (the c1 win). The + // poll is synchronous — it holds no borrow across `.await`; on miss it + // falls through to the slot's park path. When the gate is closed + // (busy/pipelined shard) the path is an immediate park. + let _wait_guard = crate::shard::slice::XshardWaitGuard::new(); + let shard_responses = { + let mut spun = None; + if crate::shard::slice::xshard_should_spin(batch_remote_total) { + for _ in 0..crate::shard::slice::xshard_spin_budget() { + if let Some(r) = response_pool.slot_for(target).try_take() { + spun = Some(r); + break; } + core::hint::spin_loop(); } } - }; - let shard_responses = match shard_responses { - Ok(r) => r, - Err(err_msg) => { - for (resp_idx, _, _) in &meta { - responses[*resp_idx] = - Frame::Error(Bytes::from_static(err_msg.as_bytes())); - } - continue; + match spun { + Some(r) => r, + None => response_pool.future_for(target).await, } }; // H1-BARRIER: collect write resp_idxs before consuming meta @@ -1842,8 +1768,8 @@ pub(crate) async fn handle_connection_sharded_monoio< for ((resp_idx, aof_bytes, cmd_name), resp) in meta.into_iter().zip(shard_responses) { // C4-FOLD-FIX: AOF append for cross-shard writes is now done - // inside the SPSC arm (PipelineBatch), BEFORE the oneshot reply - // is sent. Appending here (after awaiting the oneshot response) + // inside the SPSC arm (PipelineBatchSlotted), BEFORE the response + // slot is filled. Appending here (after awaiting the response) // defers the append until after drain_spsc_shared returns, which // makes AofFold's pending_aof_count undercount it → escape to // new incr → double-apply on restart. The SPSC arm now owns the @@ -1896,7 +1822,8 @@ pub(crate) async fn handle_connection_sharded_monoio< } // Update live state after each batch — lock-free (QW8, 2026-06 - // review: this was a global registry write lock per batch). + // review: this was a global registry write lock per batch), and + // clock-free (shard-cached ms, not Instant::now()). client_live.touch( conn.selected_db, crate::client_registry::ClientFlags { @@ -1904,6 +1831,7 @@ pub(crate) async fn handle_connection_sharded_monoio< in_multi: conn.in_multi, blocked: false, }, + ctx.cached_clock.ms(), ); // Check if migration was triggered during frame processing. diff --git a/src/server/conn/handler_sharded/dispatch.rs b/src/server/conn/handler_sharded/dispatch.rs index 8fc46076e..6207cd467 100644 --- a/src/server/conn/handler_sharded/dispatch.rs +++ b/src/server/conn/handler_sharded/dispatch.rs @@ -112,6 +112,7 @@ pub(super) fn try_handle_client_command( in_multi: conn.in_multi, blocked: false, }, + crate::storage::entry::current_time_ms(), ); }); let list = crate::client_registry::client_list(); @@ -129,6 +130,7 @@ pub(super) fn try_handle_client_command( in_multi: conn.in_multi, blocked: false, }, + crate::storage::entry::current_time_ms(), ); }); let info = crate::client_registry::client_info(client_id).unwrap_or_default(); diff --git a/src/server/conn/handler_sharded/mod.rs b/src/server/conn/handler_sharded/mod.rs index 03d16e231..a05ddae06 100644 --- a/src/server/conn/handler_sharded/mod.rs +++ b/src/server/conn/handler_sharded/mod.rs @@ -231,6 +231,11 @@ pub(crate) async fn handle_connection_sharded_inner< ) -> (HandlerResult, Option) { use tokio::io::{AsyncReadExt, AsyncWriteExt}; + // Solo-conn spin gate (L1 convoy fix): register this connection on the + // shard thread so the C2 reply-spin's sibling check (`xshard_may_spin`) + // sees it. RAII — decrements when the handler returns (incl. migration). + let _conn_guard = crate::shard::slice::ShardConnGuard::new(); + // Direct buffer I/O: bypass Framed/codec for the hot path. let mut stream = stream; let mut read_buf = if initial_read_buf.is_empty() { @@ -1035,7 +1040,7 @@ pub(crate) async fn handle_connection_sharded_inner< // --- Multi-key commands --- if is_multi_key_command(cmd, cmd_args) { - let response = crate::shard::coordinator::coordinate_multi_key(cmd, cmd_args, ctx.shard_id, ctx.num_shards, conn.selected_db, &ctx.shard_databases, &ctx.dispatch_tx, &ctx.spsc_notifiers, &ctx.cached_clock, &()).await; + let response = crate::shard::coordinator::coordinate_multi_key(cmd, cmd_args, ctx.shard_id, ctx.num_shards, conn.selected_db, &ctx.shard_databases, &ctx.dispatch_tx, &ctx.spsc_notifiers, &ctx.cached_clock, ctx.aof_pool.as_ref(), &ctx.repl_state, &()).await; responses.push(response); continue; } @@ -1575,13 +1580,13 @@ pub(crate) async fn handle_connection_sharded_inner< if !remote_groups.is_empty() { let mut reply_futures: Vec<(Vec<(usize, Option, Bytes)>, usize)> = Vec::with_capacity(remote_groups.len()); for (target, entries) in remote_groups { - let slot_ptr = response_pool.slot_ptr(target); + let slot_arc = response_pool.slot_arc(target); // Use the db_index captured with the first command (all commands in a // pipeline batch targeting the same shard share the same db_index). let batch_db = entries.first().map(|(_, _, _, _, db)| *db).unwrap_or(conn.selected_db); let (meta, commands): (Vec<(usize, Option, Bytes)>, Vec>) = entries.into_iter().map(|(idx, arc_frame, aof, cmd, _db)| ((idx, aof, cmd), arc_frame)).unzip(); - let msg = ShardMessage::PipelineBatchSlotted { db_index: batch_db, commands, response_slot: crate::shard::dispatch::ResponseSlotPtr(slot_ptr) }; + let msg = ShardMessage::PipelineBatchSlotted { db_index: batch_db, commands, response_slot: crate::shard::dispatch::ResponseSlotPtr(slot_arc) }; let target_idx = ChannelMesh::target_index(ctx.shard_id, target); // F3: bounded backpressure retry (shared helper). The // closure retains the message on a full ring; the helper @@ -1647,7 +1652,7 @@ pub(crate) async fn handle_connection_sharded_inner< let shard_responses = { let mut spun = None; if crate::shard::slice::xshard_should_spin(batch_remote_total) { - for _ in 0..crate::shard::slice::XSHARD_SPIN_BUDGET { + for _ in 0..crate::shard::slice::xshard_spin_budget() { if let Some(r) = response_pool.slot_for(target).try_take() { spun = Some(r); break; @@ -1758,7 +1763,8 @@ pub(crate) async fn handle_connection_sharded_inner< } // Update live state after each batch — lock-free (QW8, 2026-06 - // review: this was a global registry write lock per batch). + // review: this was a global registry write lock per batch), and + // clock-free (shard-cached ms, not Instant::now()). client_live.touch( conn.selected_db, crate::client_registry::ClientFlags { @@ -1766,6 +1772,7 @@ pub(crate) async fn handle_connection_sharded_inner< in_multi: conn.in_multi, blocked: false, }, + ctx.cached_clock.ms(), ); // Check if migration was triggered during frame processing. diff --git a/src/server/conn/shared.rs b/src/server/conn/shared.rs index 5b2215803..e8ef4fa5d 100644 --- a/src/server/conn/shared.rs +++ b/src/server/conn/shared.rs @@ -376,6 +376,9 @@ pub(crate) fn is_multi_key_command(cmd: &[u8], args: &[Frame]) -> bool { let b0 = cmd[0] | 0x20; match (len, b0) { (4, b'm') => cmd.eq_ignore_ascii_case(b"MGET") || cmd.eq_ignore_ascii_case(b"MSET"), + // MSETNX: atomic multi-key write; the coordinator rejects it (CROSSSLOT) when + // keys span shards, and runs it atomically when they are co-located. + (6, b'm') => cmd.eq_ignore_ascii_case(b"MSETNX"), // DEL, UNLINK, EXISTS with multiple keys (3, b'd') => args.len() > 1 && cmd.eq_ignore_ascii_case(b"DEL"), (6, b'u') => args.len() > 1 && cmd.eq_ignore_ascii_case(b"UNLINK"), diff --git a/src/server/conn/tests.rs b/src/server/conn/tests.rs index f4d7a1634..a8595f118 100644 --- a/src/server/conn/tests.rs +++ b/src/server/conn/tests.rs @@ -60,6 +60,64 @@ fn test_inline_get_hit() { assert_eq!(&write_buf[..], b"$3\r\nbar\r\n"); } +/// Byte-parity guard for the inline GET hit across the `CompactValue` SSO +/// boundary (12B inline / 13B heap) and up to a large value. The reply must be +/// exactly `$\r\n\r\n` for every size — this pins the response +/// framing so the no-copy refactor (writing the value straight from the borrow +/// into `write_buf` instead of via an intermediate `Vec`) cannot change a byte. +#[test] +fn test_inline_get_hit_byte_parity_sizes() { + for &size in &[0usize, 1, 12, 13, 65536] { + let dbs = make_dbs(); + let value: Vec = (0..size).map(|i| (i % 251) as u8).collect(); + crate::shard::slice::with_shard_db(0, |db| { + db.set( + Bytes::from_static(b"k"), + Entry::new_string(Bytes::from(value.clone())), + ); + }); + let mut read_buf = BytesMut::from(&b"*2\r\n$3\r\nGET\r\n$1\r\nk\r\n"[..]); + let mut write_buf = BytesMut::new(); + let aof_pool: Option> = None; + let rt_config = make_rt_config(); + + let result = try_inline_dispatch( + &mut read_buf, + &mut write_buf, + &dbs, + 0, + 0, + &aof_pool, + &None, + 0, + 1, + false, + &rt_config, + ); + + let mut expected = Vec::new(); + expected.extend_from_slice(b"$"); + expected.extend_from_slice(size.to_string().as_bytes()); + expected.extend_from_slice(b"\r\n"); + expected.extend_from_slice(&value); + expected.extend_from_slice(b"\r\n"); + + assert_eq!( + result, 1, + "size {size}: expected exactly one command inlined" + ); + assert!( + read_buf.is_empty(), + "size {size}: read_buf not fully consumed" + ); + assert_eq!( + &write_buf[..], + &expected[..], + "size {size}: reply byte mismatch" + ); + } +} + #[test] fn test_inline_get_miss() { let dbs = make_dbs(); diff --git a/src/server/response_slot.rs b/src/server/response_slot.rs index 82eec1a37..8b1e1361c 100644 --- a/src/server/response_slot.rs +++ b/src/server/response_slot.rs @@ -7,8 +7,8 @@ use std::cell::UnsafeCell; use std::future::Future; -use std::marker::PhantomData; use std::pin::Pin; +use std::sync::Arc; use std::sync::atomic::{AtomicU8, Ordering}; use std::task::{Context, Poll}; @@ -129,12 +129,9 @@ impl ResponseSlot { /// (xshard-read-fastpath C2): the caller is busy-polling, not parking, so it /// must not register a waker. Single consumer (connection owner thread). /// - /// Live under the tokio runtime (handler_sharded's `ResponseSlotPool` reply - /// path). Dead under monoio, which replies via a flume oneshot rather than a - /// response slot — hence the cfg-gated allow keeps the dead-code check active - /// where the method is actually used. + /// Live under BOTH runtimes: handler_sharded (tokio) and handler_monoio + /// (L3b) reply paths spin on this before parking on `poll_take`. #[inline] - #[cfg_attr(not(feature = "runtime-tokio"), allow(dead_code))] pub(crate) fn try_take(&self) -> Option> { // Same FILLED-confirmed take as poll_take's fast path, minus the waker // registration — the spin caller is busy-polling, not parking. @@ -163,26 +160,24 @@ impl Default for ResponseSlot { } } -/// Future that wraps a `ResponseSlot` reference for async `.await` usage. +/// Future that owns a shared handle to a `ResponseSlot` for async `.await` usage. /// -/// The lifetime `'a` ties this future to the `ResponseSlotPool` that owns -/// the slot, preventing use-after-free at compile time. -pub struct ResponseSlotFuture<'a> { - slot: *const ResponseSlot, - _marker: PhantomData<&'a ResponseSlot>, +/// Holds an `Arc` clone (not a borrow or raw pointer), so the slot +/// stays alive for as long as EITHER this future OR any in-flight `ResponseSlotPtr` +/// on a target shard references it. Abandoning this future (drop, panic-unwind, +/// shutdown break) can never dangle the target shard's late `fill()` — the refcount +/// keeps the slot allocated until the last handle drops. This is the structural +/// replacement for the old raw-pointer-into-stack-pool design (see PR review: +/// panic-unwind cross-shard reply UAF). +pub struct ResponseSlotFuture { + slot: Arc, } -// SAFETY: The underlying ResponseSlot is Send+Sync, and the lifetime parameter -// ensures the pool (which owns the slot) outlives this future. -unsafe impl Send for ResponseSlotFuture<'_> {} - -impl Future for ResponseSlotFuture<'_> { +impl Future for ResponseSlotFuture { type Output = Vec; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { - // SAFETY: Pointer validity guaranteed by lifetime bound to ResponseSlotPool. - let slot = unsafe { &*self.slot }; - slot.poll_take(cx) + self.slot.poll_take(cx) } } @@ -191,7 +186,7 @@ impl Future for ResponseSlotFuture<'_> { /// Eliminates oneshot channel allocation on the cross-shard dispatch hot path. /// Created once per connection, reused across all dispatches for that connection. pub struct ResponseSlotPool { - slots: Vec, + slots: Vec>, /// The shard this connection is assigned to (for diagnostics/debugging). #[allow(dead_code)] my_shard: usize, @@ -202,34 +197,36 @@ impl ResponseSlotPool { pub fn new(num_shards: usize, my_shard: usize) -> Self { let mut slots = Vec::with_capacity(num_shards); for _ in 0..num_shards { - slots.push(ResponseSlot::new()); + slots.push(Arc::new(ResponseSlot::new())); } Self { slots, my_shard } } - /// Get a reference to the slot for the given target shard. + /// Get a reference to the slot for the given target shard (for the reply-side + /// spin `try_take()`; borrows the pool, no refcount bump). #[inline] pub fn slot_for(&self, target_shard: usize) -> &ResponseSlot { &self.slots[target_shard] } - /// Get a raw pointer to the slot for the given target shard. - /// - /// This pointer is stable for the lifetime of the pool and can be - /// safely sent across threads in ShardMessage variants. + /// Clone the shared handle to the slot for the given target shard, to send + /// cross-thread in a `ShardMessage`. One refcount bump per batch — the clone + /// keeps the slot alive until BOTH this connection's pool AND the in-flight + /// message drop, so the target shard's `fill()` can never dangle even if the + /// connection task is dropped mid-flight (panic-unwind / shutdown). #[inline] - pub fn slot_ptr(&self, target_shard: usize) -> *const ResponseSlot { - &self.slots[target_shard] as *const ResponseSlot + pub fn slot_arc(&self, target_shard: usize) -> Arc { + Arc::clone(&self.slots[target_shard]) } /// Create a future that resolves when the target shard fills the slot. /// - /// The returned future borrows the pool, ensuring the slot outlives the future. + /// The future OWNS an `Arc` clone of the slot, so it is drop-safe: abandoning + /// it (shutdown break / panic-unwind) cannot dangle a concurrent `fill()`. #[inline] - pub fn future_for(&self, target_shard: usize) -> ResponseSlotFuture<'_> { + pub fn future_for(&self, target_shard: usize) -> ResponseSlotFuture { ResponseSlotFuture { - slot: self.slot_ptr(target_shard), - _marker: PhantomData, + slot: Arc::clone(&self.slots[target_shard]), } } } @@ -327,16 +324,44 @@ mod tests { } #[test] - fn test_pool_slot_for_returns_correct_slot() { + fn test_pool_slot_arc_returns_correct_slot() { let pool = ResponseSlotPool::new(4, 0); - // Verify slot_ptr and slot_for return the same address + // Verify slot_arc and slot_for reference the same slot object. for i in 0..4 { let slot_ref = pool.slot_for(i) as *const ResponseSlot; - let slot_ptr = pool.slot_ptr(i); - assert_eq!(slot_ref, slot_ptr); + let slot_arc = pool.slot_arc(i); + assert_eq!(slot_ref, Arc::as_ptr(&slot_arc)); } } + /// Drop-safety regression (PR review: panic-unwind cross-shard reply UAF). + /// The slot handle sent cross-thread MUST outlive the connection's pool being + /// dropped while a reply is still in flight — otherwise the target shard's late + /// `fill()` is a use-after-free. With Arc-owned slots the allocation survives + /// (strong_count >= 1) until the last handle drops; the raw-pointer design this + /// replaced could not express this and would dangle. Provable without a + /// sanitizer via `Arc::strong_count`. + #[test] + fn test_slot_arc_outlives_pool_drop_and_is_fillable() { + let pool = ResponseSlotPool::new(4, 0); + // Simulate the cross-thread send: clone the handle out of the pool. + let in_flight = pool.slot_arc(2); + assert_eq!(Arc::strong_count(&in_flight), 2, "pool + in-flight handle"); + + // Connection task drops its pool (e.g. panic-unwind / shutdown) while the + // message is still queued on the target shard. + drop(pool); + + // The slot is still alive and safe to fill — no dangling pointer. + assert_eq!( + Arc::strong_count(&in_flight), + 1, + "handle keeps the slot allocated after pool drop" + ); + in_flight.fill(vec![Frame::Integer(7)]); + assert_eq!(in_flight.try_take().map(|v| v.len()), Some(1)); + } + #[test] fn test_future_resolves_after_fill() { let pool = ResponseSlotPool::new(4, 0); @@ -354,17 +379,14 @@ mod tests { #[test] fn test_concurrent_fill_from_another_thread() { let pool = ResponseSlotPool::new(4, 0); - let addr = pool.slot_ptr(2) as usize; + // The cross-thread send is now an Arc clone (always Send) — no raw pointer. + let in_flight = pool.slot_arc(2); let future = pool.future_for(2); // Spawn a thread that fills the slot after a brief yield. - // We transmit the address as usize (always Send) and reconstruct the pointer. let handle = std::thread::spawn(move || { std::thread::yield_now(); - // SAFETY: addr points to a valid ResponseSlot in the pool - // which outlives this thread (we join before pool is dropped). - let slot = unsafe { &*(addr as *const ResponseSlot) }; - slot.fill(vec![Frame::Integer(123)]); + in_flight.fill(vec![Frame::Integer(123)]); }); let result = block_on(future); diff --git a/src/shard/coordinator.rs b/src/shard/coordinator.rs index a3222687d..ea16bc544 100644 --- a/src/shard/coordinator.rs +++ b/src/shard/coordinator.rs @@ -40,6 +40,12 @@ pub async fn coordinate_multi_key( dispatch_tx: &Rc>>>, spsc_notifiers: &[Arc], cached_clock: &CachedClock, + // Local-leg persistence context (review Finding 1): the coordinator's + // in-process local legs for MSET/MSETNX append to the owning shard's AOF + // via these, matching the local single-key write contract. `None` disables + // persistence (tests / no-AOF deployments). + aof_pool: Option<&Arc>, + repl_state: ReplStateRef<'_>, _response_pool: &(), // placeholder — coordinator uses oneshot internally ) -> Frame { if cmd.eq_ignore_ascii_case(b"MGET") { @@ -65,6 +71,23 @@ pub async fn coordinate_multi_key( dispatch_tx, spsc_notifiers, cached_clock, + aof_pool, + repl_state, + _response_pool, + ) + .await + } else if cmd.eq_ignore_ascii_case(b"MSETNX") { + coordinate_msetnx( + args, + my_shard, + num_shards, + db_index, + shard_databases, + dispatch_tx, + spsc_notifiers, + cached_clock, + aof_pool, + repl_state, _response_pool, ) .await @@ -196,6 +219,62 @@ async fn run_on_owner( } } +/// Type of the replication-state handle threaded into the coordinator's local +/// persistence path (same shape `AofWriterPool::issue_append_lsn` expects). +type ReplStateRef<'a> = + &'a Option>>; + +/// Persist a coordinator LOCAL-leg write to the owning shard's AOF, matching the +/// local single-key write contract (the `is_write` block in +/// `handler_monoio`/`handler_sharded`): issue an LSN off `repl_state`, then +/// durable-append. Under `appendfsync=always` this awaits the writer's fsync ack; +/// under everysec/no it is fire-and-forget. WAL append is external to +/// `cmd_dispatch`, so the coordinator's in-process local legs (`run_local`, +/// `coordinate_mset` fast path / local slice) MUST call this or their writes are +/// lost on restart while the remote legs (via `wal_append_and_fanout`) survive. +/// +/// `serialized` MUST cover only keys OWNED by `my_shard`: +/// - co-located command (MSETNX; MSET fast path) → the whole command, +/// - scattered MSET local slice → a synthesized MSET over +/// just the local keys (never the full scattered command — `my_shard` does +/// not own the remote keys and replay would misapply them on this shard). +/// +/// Returns `Err(())` on AOF failure so the caller surfaces `AOF_FSYNC_ERR` +/// instead of a false `+OK` (design-for-failure; matches the handler). +async fn persist_local_leg( + aof_pool: Option<&Arc>, + repl_state: ReplStateRef<'_>, + my_shard: usize, + serialized: Bytes, +) -> Result<(), ()> { + let Some(pool) = aof_pool else { return Ok(()) }; + let lsn = crate::persistence::aof::AofWriterPool::issue_append_lsn( + repl_state, + my_shard, + serialized.len(), + ); + match pool + .try_send_append_durable(my_shard, lsn, serialized) + .await + { + Ok(()) => Ok(()), + Err(_) => Err(()), + } +} + +/// Serialize an `MSET k v ...` command over `pairs` for AOF logging of a local +/// MSET leg. Used for both the fast path (all keys local) and a scattered MSET's +/// local slice (only the local keys) — never the full scattered command. +fn serialize_local_mset(pairs: &[(Bytes, Bytes)]) -> Bytes { + let mut parts: Vec = Vec::with_capacity(pairs.len() * 2 + 1); + parts.push(Frame::BulkString(Bytes::from_static(b"MSET"))); + for (k, v) in pairs { + parts.push(Frame::BulkString(k.clone())); + parts.push(Frame::BulkString(v.clone())); + } + crate::persistence::aof::serialize_command(&Frame::Array(parts.into())) +} + fn bulk(b: &Bytes) -> Frame { Frame::BulkString(b.clone()) } @@ -728,6 +807,8 @@ async fn coordinate_mset( dispatch_tx: &Rc>>>, spsc_notifiers: &[Arc], cached_clock: &CachedClock, + aof_pool: Option<&Arc>, + repl_state: ReplStateRef<'_>, _response_pool: &(), // placeholder — coordinator uses oneshot internally ) -> Frame { if args.is_empty() || !args.len().is_multiple_of(2) { @@ -761,10 +842,22 @@ async fn coordinate_mset( // Fast path: all keys on local shard if groups.len() == 1 && groups.contains_key(&my_shard) { - return crate::shard::slice::with_shard_db(db_index, |db| { + let resp = crate::shard::slice::with_shard_db(db_index, |db| { db.refresh_now_from_cache(cached_clock); crate::command::string::mset(db, args) }); + // Local leg (review Finding 1): persist the whole MSET — every key is + // owned by my_shard — matching the local single-key write contract. + if let Some(pairs) = groups.get(&my_shard) { + let serialized = serialize_local_mset(pairs); + if persist_local_leg(aof_pool, repl_state, my_shard, serialized) + .await + .is_err() + { + return Frame::Error(Bytes::from_static(crate::persistence::aof::AOF_FSYNC_ERR)); + } + } + return resp; } let mut pending_shards: Vec>> = Vec::new(); @@ -804,9 +897,114 @@ async fn coordinate_mset( let _ = reply_rx.recv().await; } + // Local leg (review Finding 1): persist a synthesized MSET over ONLY the + // local keys. The remote slices persisted themselves on their owner shards + // via MultiExecute -> wal_append_and_fanout; my_shard must not log their keys + // (replay re-dispatches raw commands, so a full-command log here would try to + // write keys this shard doesn't own). + if let Some(pairs) = groups.get(&my_shard) { + let serialized = serialize_local_mset(pairs); + if persist_local_leg(aof_pool, repl_state, my_shard, serialized) + .await + .is_err() + { + return Frame::Error(Bytes::from_static(crate::persistence::aof::AOF_FSYNC_ERR)); + } + } + Frame::SimpleString(Bytes::from_static(b"OK")) } +/// Coordinate MSETNX across shards. +/// +/// MSETNX is atomic by contract: set every pair iff *none* of the keys already +/// exist. Moon cannot honor that atomically across shards (like MSET, cross-shard +/// writes scatter with no two-phase commit or rollback), so — by design — MSETNX +/// is rejected with a CROSSSLOT error when its keys hash to more than one shard. +/// When all keys are co-located on a single shard (including via `{hash-tag}`), +/// the whole command runs atomically on that shard's owner. +#[allow(clippy::too_many_arguments)] +async fn coordinate_msetnx( + args: &[Frame], + my_shard: usize, + num_shards: usize, + db_index: usize, + shard_databases: &Arc, + dispatch_tx: &Rc>>>, + spsc_notifiers: &[Arc], + cached_clock: &CachedClock, + aof_pool: Option<&Arc>, + repl_state: ReplStateRef<'_>, + _response_pool: &(), // placeholder — coordinator uses oneshot internally +) -> Frame { + if args.is_empty() || !args.len().is_multiple_of(2) { + return Frame::Error(Bytes::from_static( + b"ERR wrong number of arguments for 'MSETNX' command", + )); + } + + // Every key must hash to the same shard; otherwise MSETNX cannot be atomic. + let first_key = match extract_key(&args[0]) { + Some(k) => k, + None => { + return Frame::Error(Bytes::from_static( + b"ERR wrong number of arguments for 'MSETNX' command", + )); + } + }; + let owner = key_to_shard(&first_key, num_shards); + for pair in args.chunks(2) { + let key = match extract_key(&pair[0]) { + Some(k) => k, + None => { + return Frame::Error(Bytes::from_static( + b"ERR wrong number of arguments for 'MSETNX' command", + )); + } + }; + if key_to_shard(&key, num_shards) != owner { + return Frame::Error(Bytes::from_static( + b"CROSSSLOT Keys in MSETNX request don't hash to the same shard", + )); + } + } + + // All keys co-located -> run the whole MSETNX atomically on the owning shard. + // Branch on ownership explicitly (rather than via run_on_owner) so the LOCAL + // leg can persist to my_shard's AOF on a successful write (review Finding 1); + // the REMOTE leg persists on the owner via MultiExecute -> wal_append_and_fanout. + let mut command_parts: Vec = Vec::with_capacity(args.len() + 1); + command_parts.push(Frame::BulkString(Bytes::from_static(b"MSETNX"))); + command_parts.extend_from_slice(args); + if owner == my_shard { + let resp = run_local(shard_databases, db_index, cached_clock, b"MSETNX", args); + // Persist only on an actual write (:1). A :0 means some key already + // existed and MSETNX wrote nothing — there is nothing to log. + if matches!(resp, Frame::Integer(1)) { + let serialized = + crate::persistence::aof::serialize_command(&Frame::Array(command_parts.into())); + if persist_local_leg(aof_pool, repl_state, my_shard, serialized) + .await + .is_err() + { + return Frame::Error(Bytes::from_static(crate::persistence::aof::AOF_FSYNC_ERR)); + } + } + resp + } else { + run_remote( + owner, + &first_key, + Frame::Array(command_parts.into()), + my_shard, + db_index, + dispatch_tx, + spsc_notifiers, + ) + .await + } +} + /// Coordinate DEL/UNLINK/EXISTS with multiple keys across shards using VLL pattern. /// /// Groups keys by shard in ascending order (BTreeMap), dispatches sub-commands @@ -2185,6 +2383,8 @@ mod tests { &dispatch_tx, ¬ifiers, &cached_clock, + None, + &None, &response_pool, ) .await; diff --git a/src/shard/dispatch.rs b/src/shard/dispatch.rs index 9ee6e132c..3d71f10ca 100644 --- a/src/shard/dispatch.rs +++ b/src/shard/dispatch.rs @@ -1,3 +1,4 @@ +use std::sync::Arc; use std::sync::atomic::{AtomicI64, AtomicU32, Ordering}; use atomic_waker::AtomicWaker; @@ -8,18 +9,23 @@ use crate::protocol::Frame; use crate::runtime::channel; use crate::server::response_slot::ResponseSlot; -/// Newtype wrapper for `*const ResponseSlot` to isolate the `Send` unsafety. +/// Shared handle to a connection's `ResponseSlot`, carried cross-thread in a +/// `ShardMessage` so the target shard can reply into it. /// -/// Raw pointers are `!Send` by default. This newtype provides a localized -/// `unsafe impl Send` with a clear safety contract, instead of requiring a -/// blanket `unsafe impl Send for ShardMessage`. -#[derive(Debug, Clone, Copy)] -pub struct ResponseSlotPtr(pub *const ResponseSlot); - -// SAFETY: The pointed-to ResponseSlot is Send+Sync (enforced by its own unsafe impls). -// The pointer remains valid for the lifetime of the connection's ResponseSlotPool, -// which outlives all dispatched ShardMessage values. -unsafe impl Send for ResponseSlotPtr {} +/// Holds an `Arc` (was a raw `*const`). The refcount keeps the slot +/// alive until BOTH the connection's `ResponseSlotPool` AND this in-flight message +/// drop, so the target shard's `fill()` can never dangle even if the connection task +/// is dropped mid-flight (panic-unwind / shutdown). This closes the panic-unwind +/// cross-shard reply use-after-free found in review, and — being an ordinary `Arc` +/// over a `Send + Sync` slot — needs no `unsafe impl Send` and adds no `unsafe`. +#[derive(Clone)] +pub struct ResponseSlotPtr(pub Arc); + +impl std::fmt::Debug for ResponseSlotPtr { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "ResponseSlotPtr({:p})", Arc::as_ptr(&self.0)) + } +} /// One batched cross-shard READ message carrying N independent single-key foreign /// reads from DIFFERENT connections on the ORIGIN shard to ONE owner shard. Each @@ -721,9 +727,9 @@ pub struct AofFoldSnapshot { pub pending_aof_count: usize, } -// ShardMessage is Send because all fields are Send. The raw pointer in -// ResponseSlotPtr is the only non-auto-Send field, and it has its own -// localized unsafe impl Send with documented safety invariants. +// ShardMessage is Send because all fields are Send. ResponseSlotPtr wraps an +// Arc over a Send + Sync slot, so it is auto-Send — no unsafe impl +// or raw pointer is involved. /// Compile-time guard: keep `ShardMessage` under 256 bytes to prevent /// hot-path allocator pressure in the SPSC ring buffer. Large variants diff --git a/src/shard/event_loop.rs b/src/shard/event_loop.rs index 15664b76f..b04a747db 100644 --- a/src/shard/event_loop.rs +++ b/src/shard/event_loop.rs @@ -53,7 +53,7 @@ impl super::Shard { &mut self, conn_rx: channel::MpscReceiver<(crate::runtime::TcpStream, bool)>, tls_config: Option, - mut consumers: Vec>, + consumers: Vec>, producers: Vec>, shutdown: CancellationToken, aof_pool: Option>, @@ -766,6 +766,47 @@ impl super::Shard { #[cfg(feature = "runtime-monoio")] let _ = &spsc_notify_local; + // tokio drains through the select! arms below and mutates the Vec + // directly; monoio re-wraps it in Rc> for the spin probe. + #[cfg(feature = "runtime-tokio")] + let mut consumers = consumers; + + // Busy-poll skip-notify handshake (monoio only; tokio spawns Send + // tasks so its consumers stay a plain Vec). The driver's spin probe + // needs shared read access to this shard's SPSC consumers, so wrap + // them in Rc>; the probe runs only while the event-loop + // task is parked in race2, so it never observes an active borrow + // (drain_spsc_shared's borrow_mut is not held across an await). + #[cfg(feature = "runtime-monoio")] + let consumers = Rc::new(RefCell::new(consumers)); + #[cfg(feature = "runtime-monoio")] + if num_shards > 1 && crate::runtime::epoll_spin_configured(server_config.io_busy_poll_us) { + // While this shard's driver spin-polls, remote producers elide + // their cross-thread wake (flume send + foreign-waker relay + + // eventfd syscall) — the probe below discovers their ringbuf + // pushes instead and re-arms the race2 Notify from the local + // thread. set_skip_wake carries the SeqCst Dekker fences. + crate::runtime::channel::enable_notify_skip_wake(); + let adv_notify = spsc_notify_local.clone(); + let probe_notify = spsc_notify_local.clone(); + let probe_consumers = Rc::clone(&consumers); + monoio::set_legacy_spin_hooks( + Box::new(move |spinning| adv_notify.set_skip_wake(spinning)), + Box::new(move || { + use ringbuf::traits::Observer as _; + let has_pending = probe_consumers.borrow().iter().any(|c| !c.is_empty()); + if has_pending { + probe_notify.notify_local(); + } + has_pending + }), + ); + info!( + "Shard {}: busy-poll skip-notify hooks registered ({} shards)", + shard_id, num_shards + ); + } + // Per-shard cached clock: updated once per 1ms tick. let cached_clock = CachedClock::new(); @@ -1771,7 +1812,7 @@ impl super::Shard { // No outer with_shard — each arm takes its own flat borrow. let hit_cap = spsc_handler::drain_spsc_shared( &shard_databases, - &mut consumers, + &mut consumers.borrow_mut(), &mut *pubsub_arc.write(), &blocking_rc, &mut pending_snapshot, diff --git a/src/shard/slice.rs b/src/shard/slice.rs index a7c8dc472..4cb18a9dd 100644 --- a/src/shard/slice.rs +++ b/src/shard/slice.rs @@ -226,6 +226,64 @@ thread_local! { /// Connections on this shard thread currently blocked in a cross-shard /// reply-wait. Incremented on entry, decremented on exit (see `XshardWaitGuard`). static XSHARD_INFLIGHT: Cell = const { Cell::new(0) }; + /// Live client connections handled by THIS shard thread (fresh + migrated). + /// Maintained by `ShardConnGuard` at the top of both connection handlers. + static SHARD_CONN_COUNT: Cell = const { Cell::new(0) }; +} + +/// Max live connections on this shard thread for which the reply-side spin is +/// still allowed. Default 1: spin only when the requesting connection is ALONE +/// on its shard thread. The spin is synchronous — with ANY sibling connection +/// present it starves that sibling AND stalls this shard's SPSC drain for other +/// shards' requests, a circular cross-shard convoy (measured s4 c8P1 on GCE +/// c4a same-instance A/B 2026-07-03: spin-on 72.7k vs spin-off 200k ops/s, +/// 2.75×). The `XSHARD_INFLIGHT` gate cannot see a sibling whose readable +/// event is still parked in the driver — it only counts conns already INSIDE a +/// reply-wait — so the executor-level starvation needs this conn-count check. +pub const XSHARD_SPIN_MAX_CONNS: u32 = 1; + +/// Effective solo-conn spin ceiling: `MOON_XSHARD_SPIN_MAX_CONNS` env override +/// (diagnostic — a large value reproduces the pre-fix convoy for A/Bs), else +/// the const. Read once. +#[inline] +fn xshard_spin_max_conns() -> u32 { + static V: std::sync::OnceLock = std::sync::OnceLock::new(); + *V.get_or_init(|| { + std::env::var("MOON_XSHARD_SPIN_MAX_CONNS") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(XSHARD_SPIN_MAX_CONNS) + }) +} + +/// RAII guard registering a live client connection on this shard thread for +/// the solo-conn spin gate. Construct at connection-handler entry; the count +/// decrements when the handler returns (including migration hand-off, where +/// the connection re-registers on its new shard's thread). +#[must_use = "the guard must live for the duration of the connection"] +pub struct ShardConnGuard { + _priv: (), +} + +impl ShardConnGuard { + #[inline] + pub fn new() -> Self { + SHARD_CONN_COUNT.with(|c| c.set(c.get().saturating_add(1))); + Self { _priv: () } + } +} + +impl Default for ShardConnGuard { + fn default() -> Self { + Self::new() + } +} + +impl Drop for ShardConnGuard { + #[inline] + fn drop(&mut self) { + SHARD_CONN_COUNT.with(|c| c.set(c.get().saturating_sub(1))); + } } /// Upper bound (INCLUSIVE) on concurrent cross-shard reply-waiters for which the @@ -238,6 +296,34 @@ pub const XSHARD_SPIN_GATE: u32 = 2; /// parking. Caps the busy-loop so a slow owner shard cannot starve this thread. pub const XSHARD_SPIN_BUDGET: u32 = 4_096; +/// Effective reply-side spin budget: `MOON_XSHARD_SPIN_BUDGET` env override +/// (diagnostic — same-instance A/Bs of the C2 spin without a rebuild; 0 fully +/// disables the reply-side spin), else the tuned const. Read once; after init +/// this is a lock-free load, cheap enough for the reply-wait path. +#[inline] +pub fn xshard_spin_budget() -> u32 { + static V: std::sync::OnceLock = std::sync::OnceLock::new(); + *V.get_or_init(|| { + std::env::var("MOON_XSHARD_SPIN_BUDGET") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(XSHARD_SPIN_BUDGET) + }) +} + +/// Effective reply-side spin gate: `MOON_XSHARD_SPIN_GATE` env override +/// (diagnostic), else the tuned const. Read once. +#[inline] +fn xshard_spin_gate() -> u32 { + static V: std::sync::OnceLock = std::sync::OnceLock::new(); + *V.get_or_init(|| { + std::env::var("MOON_XSHARD_SPIN_GATE") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(XSHARD_SPIN_GATE) + }) +} + /// Max cross-shard commands in the CURRENT connection batch for which the reply-side /// spin is allowed — the call site gates on `batch_remote <= this && xshard_may_spin()`. /// @@ -259,7 +345,8 @@ pub const XSHARD_SPIN_MAX_BATCH_REMOTE: usize = 1; /// Reads a thread-local `Cell`: no atomic, no lock, no syscall. #[inline] pub fn xshard_may_spin() -> bool { - XSHARD_INFLIGHT.with(|c| c.get() <= XSHARD_SPIN_GATE) + XSHARD_INFLIGHT.with(|c| c.get() <= xshard_spin_gate()) + && SHARD_CONN_COUNT.with(|c| c.get() <= xshard_spin_max_conns()) } /// The full reply-side spin decision for the current batch: spin only when the batch diff --git a/src/shard/spsc_handler.rs b/src/shard/spsc_handler.rs index d821b8d41..250cba19f 100644 --- a/src/shard/spsc_handler.rs +++ b/src/shard/spsc_handler.rs @@ -709,17 +709,22 @@ pub(crate) fn handle_shard_message_shared( }; if is_write && !matches!(frame, crate::protocol::Frame::Error(_)) { - let serialized = aof::serialize_command(cmd_frame); - wal_append_and_fanout( - &serialized, - wal_writer, - wal_v3_writer, - repl_backlog, - replica_txs, - repl_state, - shard_id, - aof_pool, // FIX-W1-2 - ); + // Skip the serialization alloc when the fanout would + // no-op (persistence + replication all off) — it was + // pure waste on every cross-shard write. + if wal_fanout_has_work(wal_writer, wal_v3_writer, replica_txs, aof_pool) { + let serialized = aof::serialize_command(cmd_frame); + wal_append_and_fanout( + &serialized, + wal_writer, + wal_v3_writer, + repl_backlog, + replica_txs, + repl_state, + shard_id, + aof_pool, // FIX-W1-2 + ); + } let needs_wake = cmd.eq_ignore_ascii_case(b"LPUSH") || cmd.eq_ignore_ascii_case(b"RPUSH") @@ -798,27 +803,31 @@ pub(crate) fn handle_shard_message_shared( }; if is_write && !matches!(frame, crate::protocol::Frame::Error(_)) { - let serialized = aof::serialize_command(cmd_frame); - wal_append_and_fanout( - &serialized, - wal_writer, - wal_v3_writer, - repl_backlog, - replica_txs, - repl_state, - shard_id, - // C4-FOLD-FIX: AOF append MUST happen here (in the SPSC arm, - // before the response is sent) so the append is already in the - // AOF channel when AofFold reads sender.len(). Moving the append - // to the connection handler (after awaiting the response) defers - // it until AFTER drain_spsc_shared returns, so AofFold's - // pending_aof_count undercount by ≥1 and that append escapes - // into the NEW incr → double-apply on restart (+1 after - // restart observed in test_ssm4a_fold_4shard_experimental). - // The handler_monoio cross-shard AOF write is removed to avoid - // the double-write that was the original reason for None. - aof_pool, // FIX-C4-FOLD - ); + // See `wal_fanout_has_work` — skip the serialization alloc + // entirely when the fanout would no-op (persistence off). + if wal_fanout_has_work(wal_writer, wal_v3_writer, replica_txs, aof_pool) { + let serialized = aof::serialize_command(cmd_frame); + wal_append_and_fanout( + &serialized, + wal_writer, + wal_v3_writer, + repl_backlog, + replica_txs, + repl_state, + shard_id, + // C4-FOLD-FIX: AOF append MUST happen here (in the SPSC arm, + // before the response is sent) so the append is already in the + // AOF channel when AofFold reads sender.len(). Moving the append + // to the connection handler (after awaiting the response) defers + // it until AFTER drain_spsc_shared returns, so AofFold's + // pending_aof_count undercount by ≥1 and that append escapes + // into the NEW incr → double-apply on restart (+1 after + // restart observed in test_ssm4a_fold_4shard_experimental). + // The handler_monoio cross-shard AOF write is removed to avoid + // the double-write that was the original reason for None. + aof_pool, // FIX-C4-FOLD + ); + } } // Auto-index: if HSET succeeded, check for vector index match. @@ -891,9 +900,8 @@ pub(crate) fn handle_shard_message_shared( let (cmd, args) = match extract_command_static(&command) { Some(pair) => pair, None => { - // SAFETY: response_slot points to a valid ResponseSlot owned by the - // connection's ResponseSlotPool, which outlives all dispatched messages. - let slot = unsafe { &*response_slot.0 }; + // Arc-owned slot: deref is safe, refcount keeps it alive. + let slot = &*response_slot.0; slot.fill(vec![crate::protocol::Frame::Error( bytes::Bytes::from_static(b"ERR invalid command format"), )]); @@ -972,8 +980,8 @@ pub(crate) fn handle_shard_message_shared( frame }) }; - // SAFETY: response_slot points to a valid ResponseSlot (see above). - let slot = unsafe { &*response_slot.0 }; + // Arc-owned slot: deref is safe, refcount keeps it alive. + let slot = &*response_slot.0; slot.fill(vec![frame]); } } @@ -1011,17 +1019,22 @@ pub(crate) fn handle_shard_message_shared( }; if is_write && !matches!(frame, crate::protocol::Frame::Error(_)) { - let serialized = aof::serialize_command(cmd_frame); - wal_append_and_fanout( - &serialized, - wal_writer, - wal_v3_writer, - repl_backlog, - replica_txs, - repl_state, - shard_id, - aof_pool, // FIX-W1-2 - ); + // Skip the serialization alloc when the fanout would + // no-op (persistence + replication all off) — it was + // pure waste on every cross-shard write. + if wal_fanout_has_work(wal_writer, wal_v3_writer, replica_txs, aof_pool) { + let serialized = aof::serialize_command(cmd_frame); + wal_append_and_fanout( + &serialized, + wal_writer, + wal_v3_writer, + repl_backlog, + replica_txs, + repl_state, + shard_id, + aof_pool, // FIX-W1-2 + ); + } let needs_wake = cmd.eq_ignore_ascii_case(b"LPUSH") || cmd.eq_ignore_ascii_case(b"RPUSH") @@ -1061,8 +1074,8 @@ pub(crate) fn handle_shard_message_shared( results.push(frame); } }); - // SAFETY: response_slot points to a valid ResponseSlot (see ExecuteSlotted). - let slot = unsafe { &*response_slot.0 }; + // Arc-owned slot: deref is safe, refcount keeps it alive. + let slot = &*response_slot.0; slot.fill(results); } ShardMessage::PipelineBatchSlotted { @@ -1101,28 +1114,32 @@ pub(crate) fn handle_shard_message_shared( }; if is_write && !matches!(frame, crate::protocol::Frame::Error(_)) { - let serialized = aof::serialize_command(cmd_frame); - wal_append_and_fanout( - &serialized, - wal_writer, - wal_v3_writer, - repl_backlog, - replica_txs, - repl_state, - shard_id, - // C4-FOLD-FIX: AOF append MUST happen here, before the - // response_slot is filled, so the append is already in the - // AOF channel when AofFold reads sender.len(). Deferring to - // the connection handler (after slot.fill wakes the handler - // task) means the append arrives AFTER drain_spsc_shared - // returns and AFTER AofFold's sender.len() snapshot, so - // pending_aof_count undercounts by ≥1 → that append escapes - // into the NEW incr → double-apply on restart (+1 observed - // in test_ssm4a_fold_4shard_experimental). The handler's - // cross-shard AOF write (handler_sharded/mod.rs) is removed - // to avoid the double-write this None guard was preventing. - aof_pool, // FIX-C4-FOLD - ); + // See `wal_fanout_has_work` — skip the serialization alloc + // entirely when the fanout would no-op (persistence off). + if wal_fanout_has_work(wal_writer, wal_v3_writer, replica_txs, aof_pool) { + let serialized = aof::serialize_command(cmd_frame); + wal_append_and_fanout( + &serialized, + wal_writer, + wal_v3_writer, + repl_backlog, + replica_txs, + repl_state, + shard_id, + // C4-FOLD-FIX: AOF append MUST happen here, before the + // response_slot is filled, so the append is already in the + // AOF channel when AofFold reads sender.len(). Deferring to + // the connection handler (after slot.fill wakes the handler + // task) means the append arrives AFTER drain_spsc_shared + // returns and AFTER AofFold's sender.len() snapshot, so + // pending_aof_count undercounts by ≥1 → that append escapes + // into the NEW incr → double-apply on restart (+1 observed + // in test_ssm4a_fold_4shard_experimental). The handler's + // cross-shard AOF write (handler_sharded/mod.rs) is removed + // to avoid the double-write this None guard was preventing. + aof_pool, // FIX-C4-FOLD + ); + } } // Auto-index: if HSET succeeded, check for vector index match. @@ -1182,8 +1199,8 @@ pub(crate) fn handle_shard_message_shared( results.push(frame); } }); - // SAFETY: response_slot points to a valid ResponseSlot (see ExecuteSlotted). - let slot = unsafe { &*response_slot.0 }; + // Arc-owned slot: deref is safe, refcount keeps it alive. + let slot = &*response_slot.0; slot.fill(results); } ShardMessage::PubSubPublish(payload) => { @@ -2398,6 +2415,26 @@ pub(crate) fn cow_intercept( /// through the per-shard AOF pool. The SPSC drain is synchronous so we use /// `try_send_append` (fire-and-forget). The `appendfsync=always` rendezvous is /// handled by the connection handler (async context), not here. +/// True when `wal_append_and_fanout` has any consumer for the serialized +/// command (S3.5b criterion: WAL v2/v3, live replicas, or the AOF pool). +/// ARM perf annotate showed the pre-S3.5b locks were ~21% of CPU on 8-shard +/// SET p=64 with everything off; the criterion is fully derivable from the +/// inputs — no flags or shared state. Skipping leaves shard_offset +/// un-advanced, which is fine: with no WAL and no replicas the offsets are +/// dead bytes (no consumer exists). Callers on the cross-shard write arms +/// check THIS before `aof::serialize_command` so the serialization alloc + +/// copy is also skipped when the fanout would no-op (it was pure waste on +/// every cross-shard write with persistence off). +#[inline] +pub(crate) fn wal_fanout_has_work( + wal_writer: &Option, + wal_v3_writer: &Option, + replica_txs: &[(u64, channel::MpscSender)], + aof_pool: Option<&std::sync::Arc>, +) -> bool { + wal_writer.is_some() || wal_v3_writer.is_some() || !replica_txs.is_empty() || aof_pool.is_some() +} + pub(crate) fn wal_append_and_fanout( data: &[u8], wal_writer: &mut Option, @@ -2409,20 +2446,9 @@ pub(crate) fn wal_append_and_fanout( aof_pool: Option<&std::sync::Arc>, ) { // S3.5b (2026-04-27): hot-path bypass when nothing actually has work. - // ARM perf annotate showed `repl_backlog.lock()` (caslb/casab) and - // `repl_state.read()` (RwLock CAS) were ~21% of CPU on 8-shard SET p=64 - // even with `--appendonly no` and zero replicas connected. The criterion - // is fully derivable from the inputs — no flags or shared state needed. - // Skipping leaves shard_offset un-advanced; that is fine since with no - // WAL and no replicas the offsets are dead bytes (no consumer exists). - // - // FIX-W1-2: also require `aof_pool.is_none()` so that per-shard AOF - // entries are not skipped when WAL/replication are off but AOF is on. - if wal_writer.is_none() - && wal_v3_writer.is_none() - && replica_txs.is_empty() - && aof_pool.is_none() - { + // See `wal_fanout_has_work` — callers use the same predicate to skip the + // `aof::serialize_command` alloc entirely when the fanout would no-op. + if !wal_fanout_has_work(wal_writer, wal_v3_writer, replica_txs, aof_pool) { return; } // WAL v3 supersedes v2 — skip v2 append when v3 is active to avoid diff --git a/src/storage/db.rs b/src/storage/db.rs index 85f5fa1ff..c5254732d 100644 --- a/src/storage/db.rs +++ b/src/storage/db.rs @@ -783,7 +783,7 @@ impl Database { // a None value. Annotated for the hot-path unwrap ratchet. #[allow(clippy::expect_used)] let result = self.data.insert_or_update( - CompactKey::from(key.clone()), // Bytes::clone is a refcount bump, not deep copy + CompactKey::from(key.as_ref()), // borrow: CompactKey copies the bytes either way |existing: &mut Entry| { // Hit path: replace existing entry, bump version. let new_entry = entry_cell.take().expect("update closure called once"); diff --git a/src/storage/eviction.rs b/src/storage/eviction.rs index 97b6adf26..72f0d0eb5 100644 --- a/src/storage/eviction.rs +++ b/src/storage/eviction.rs @@ -23,6 +23,74 @@ use crate::storage::tiered::spill_thread::SpillRequest; /// the user-tunable `maxmemory-samples` (Redis default 5; we accept up to 16). const MAX_VICTIM_SAMPLES: usize = 16; +/// Lock-free mirrors of `RuntimeConfig::{maxmemory, maxmemory_per_shard()}` +/// for the inline write fast path, which previously paid a +/// `parking_lot::RwLock` read pair per SET just to discover eviction had +/// nothing to do. `usize::MAX` = not yet published — fail safe: callers must +/// take the config lock and run the full path. +static MAXMEMORY_HINT: std::sync::atomic::AtomicUsize = + std::sync::atomic::AtomicUsize::new(usize::MAX); +static MAXMEMORY_PER_SHARD_HINT: std::sync::atomic::AtomicUsize = + std::sync::atomic::AtomicUsize::new(usize::MAX); + +/// Publish the maxmemory hints. MUST be called wherever `maxmemory` (or the +/// shard count it is divided by) changes: server startup after the resolved +/// `num_shards` is written, and `CONFIG SET maxmemory`. A missed publish is +/// safe-but-slow for lowered limits only if stale-high — so this is kept to +/// a single helper to make the write sites greppable. +pub fn publish_maxmemory_hints(config: &RuntimeConfig) { + MAXMEMORY_PER_SHARD_HINT.store( + config.maxmemory_per_shard(), + std::sync::atomic::Ordering::Relaxed, + ); + MAXMEMORY_HINT.store(config.maxmemory, std::sync::atomic::Ordering::Relaxed); +} + +/// Lock-free pre-gate for the inline write path: `true` iff the slow path +/// (`try_evict_if_needed_*`) would PROVABLY no-op, i.e. `maxmemory == 0` or +/// `estimated_memory <= budget` with the budget computed exactly as the slow +/// path does (`elastic_budget.min(maxmemory)`, else per-shard split). Any +/// uncertainty (unpublished hints) returns `false` → caller takes the lock +/// and runs the existing full path. Hint staleness is bounded by the same +/// one-write-behind window the 100ms elastic-budget tick already accepts. +#[inline] +pub fn inline_write_can_skip_eviction(estimated_memory: usize, elastic_budget: usize) -> bool { + can_skip_eviction( + MAXMEMORY_HINT.load(std::sync::atomic::Ordering::Relaxed), + MAXMEMORY_PER_SHARD_HINT.load(std::sync::atomic::Ordering::Relaxed), + estimated_memory, + elastic_budget, + ) +} + +/// Pure decision core of [`inline_write_can_skip_eviction`], separated from +/// the process-global hint statics so it can be tested deterministically +/// (the statics race with `CONFIG SET maxmemory` tests in the parallel +/// test binary). `usize::MAX` in either hint = unpublished → fail safe. +#[inline] +fn can_skip_eviction( + mm: usize, + per_shard: usize, + estimated_memory: usize, + elastic_budget: usize, +) -> bool { + if mm == usize::MAX { + return false; // unpublished — fail safe + } + if mm == 0 { + return true; // unlimited: slow path early-returns + } + let budget = if elastic_budget > 0 { + elastic_budget.min(mm) + } else { + if per_shard == usize::MAX { + return false; + } + per_shard + }; + estimated_memory <= budget +} + /// Reservoir-sample up to `samples` random keys from the database without /// materializing the entire keyspace. /// @@ -1352,4 +1420,47 @@ mod tests { std::fs::write(data.join("base-000999.rdb"), b"x").unwrap(); assert_eq!(next_spill_file_id_seed(Some(tmp.path())), 514); } + + /// The lock-free inline-write pre-gate must skip the runtime-config lock + /// ONLY when the slow path (`try_evict_if_needed_*`, no-op iff + /// `total <= budget`) would provably do nothing. Tests the pure decision + /// core — the public wrapper only adds two Relaxed loads of the + /// process-global hints, which race with `CONFIG SET maxmemory` tests in + /// the parallel test binary and are therefore not asserted on here. + #[test] + fn test_inline_write_can_skip_eviction_gate() { + const UNPUB: usize = usize::MAX; + + // Unpublished sentinel (either hint needed) → fail safe: slow path. + assert!(!can_skip_eviction(UNPUB, UNPUB, 0, 0)); + assert!(!can_skip_eviction(UNPUB, 1000, 0, 0)); + assert!(!can_skip_eviction(1000, UNPUB, 0, 0)); // per-shard needed, missing + + // maxmemory == 0 (unlimited): slow path early-returns → provable skip, + // even if the per-shard hint is missing. + assert!(can_skip_eviction(0, UNPUB, usize::MAX, 0)); + + // maxmemory 1000, per-shard 1000 (1 shard), no elastic budget: + // skip iff est <= 1000 (slow-path loop condition is `total > budget`). + assert!(can_skip_eviction(1000, 1000, 1000, 0)); + assert!(!can_skip_eviction(1000, 1000, 1001, 0)); + + // Elastic budget overrides per-shard, capped at instance maxmemory — + // mirror of `budget_override.min(config.maxmemory)` in the slow path. + // (per-shard hint irrelevant when elastic > 0, even unpublished.) + assert!(can_skip_eviction(1000, UNPUB, 700, 800)); + assert!(!can_skip_eviction(1000, UNPUB, 900, 800)); + assert!(can_skip_eviction(1000, UNPUB, 999, 1500)); // cap: min(1500,1000) + assert!(!can_skip_eviction(1000, UNPUB, 1400, 1500)); + + // Multi-shard: per-shard budget = ceil(maxmemory / num_shards), as + // published from `maxmemory_per_shard()` (its rounding has its own + // tests in config.rs). 1000 across 4 shards → 250. + let mut rt = RuntimeConfig::default(); + rt.maxmemory = 1000; + rt.num_shards = 4; + let per_shard = rt.maxmemory_per_shard(); + assert!(can_skip_eviction(1000, per_shard, 250, 0)); + assert!(!can_skip_eviction(1000, per_shard, 251, 0)); + } } diff --git a/tests/coordinator_local_leg_durability.rs b/tests/coordinator_local_leg_durability.rs new file mode 100644 index 000000000..378a9681c --- /dev/null +++ b/tests/coordinator_local_leg_durability.rs @@ -0,0 +1,482 @@ +//! ADD milestone `v3-4-kv-correctness` — cross-shard coordinator LOCAL-leg WAL +//! durability (review Finding 1). +//! +//! The cross-shard coordinator splits a multi-key write (MSET / MSETNX) into a +//! LOCAL leg (keys owned by the connection's own shard, executed in-process) and +//! REMOTE legs (keys owned by other shards, executed via `ShardMessage:: +//! MultiExecute`). The remote legs persist through `wal_append_and_fanout`; the +//! local leg historically executed the write in memory but NEVER appended it to +//! the owning shard's AOF. So a co-located MSET/MSETNX was durable when its owner +//! was a *remote* shard but silently **non-durable** when its owner was the +//! connection's *own* shard — the write survived until the next SIGKILL, then +//! vanished on restart. +//! +//! ## Why this is deterministic despite SO_REUSEPORT +//! +//! The connection lands on some shard `C` (SO_REUSEPORT picks it, we cannot +//! predict which). Each test writes ONE co-located group per shard (via a +//! `{hash-tag}` that provably routes to that shard). Whichever shard `C` is, +//! exactly ONE group is the LOCAL leg and the rest are remote. We assert ALL +//! groups survive a crash+restart: +//! - pre-fix: the group whose owner == `C` is gone (local leg never persisted) +//! → the assertion fails (RED) no matter which shard `C` turned out to be. +//! - post-fix: the local leg persists via the same `aof_pool` path every local +//! write uses → every group recovers (GREEN). +//! +//! All writes go through a SINGLE reused connection so `my_shard` (== `C`) is +//! stable across the whole write phase. +//! +//! Harness: `CARGO_BIN_EXE_moon` + a hand-rolled RESP2 client (no redis-cli), so +//! this runs as a live gate under both runtimes. Fsync policy + 1.5s quiescing +//! sleep before SIGKILL mirror the proven `crash_matrix_per_shard_aof.rs` +//! everysec pattern (the sleep drains the fire-and-forget remote-leg appends so +//! the ONLY thing a crash can drop pre-fix is the un-persisted local leg). +//! +//! Run alone with: cargo test --test coordinator_local_leg_durability + +use std::io::{Read, Write}; +use std::net::{TcpStream, ToSocketAddrs}; +use std::process::{Child, Command}; +use std::time::{Duration, Instant}; + +use moon::shard::dispatch::key_to_shard; + +// --------------------------------------------------------------------------- +// Harness (CARGO_BIN_EXE pattern, mirrors msetnx_cross_shard_reject.rs + +// crash_matrix_per_shard_aof.rs) +// --------------------------------------------------------------------------- + +const SHARDS: u32 = 4; +/// Keys per co-located group (a multi-key MSET/MSETNX, not a single SET). +const GROUP_SIZE: usize = 3; + +fn moon_binary() -> std::path::PathBuf { + std::path::PathBuf::from(env!("CARGO_BIN_EXE_moon")) +} + +fn free_port() -> u16 { + let l = std::net::TcpListener::bind("127.0.0.1:0").expect("bind :0"); + let p = l.local_addr().expect("local_addr").port(); + drop(l); + p +} + +/// Spawn moon with per-shard AOF enabled (`--appendonly yes`). `everysec` + +/// a 1.5s quiescing sleep before the kill gives 100% durability for everything +/// that was actually appended — so the only pre-fix loss is the local leg that +/// was never appended at all. +fn spawn_moon_aof(port: u16, dir: &std::path::Path, shards: u32) -> Child { + Command::new(moon_binary()) + .args([ + "--port", + &port.to_string(), + "--dir", + &dir.to_string_lossy(), + "--shards", + &shards.to_string(), + "--appendonly", + "yes", + "--appendfsync", + "everysec", + ]) + .stdout(std::fs::File::create(dir.join("moon.stdout.log")).expect("stdout log")) + .stderr(std::fs::File::create(dir.join("moon.stderr.log")).expect("stderr log")) + .spawn() + .expect("spawn moon (CARGO_BIN_EXE_moon)") +} + +struct ServerGuard(Child); +impl Drop for ServerGuard { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +/// SIGKILL round-1 and reap it so the port + AOF file handles are released +/// before round-2 spawns. `Child::kill()` sends SIGKILL on Unix. +fn sigkill(mut child: Child) { + let _ = child.kill(); + let _ = child.wait(); +} + +fn connect(port: u16, deadline: Duration) -> TcpStream { + let addr = format!("127.0.0.1:{port}") + .to_socket_addrs() + .expect("addr") + .next() + .expect("one addr"); + let start = Instant::now(); + loop { + match TcpStream::connect_timeout(&addr, Duration::from_millis(200)) { + Ok(s) => { + s.set_read_timeout(Some(Duration::from_secs(5))).ok(); + s.set_write_timeout(Some(Duration::from_secs(5))).ok(); + return s; + } + Err(_) if start.elapsed() < deadline => { + std::thread::sleep(Duration::from_millis(50)); + } + Err(e) => panic!("server never accepted on {port}: {e}"), + } + } +} + +fn wait_ready(port: u16) { + let mut s = connect(port, Duration::from_secs(30)); + let start = Instant::now(); + loop { + s.write_all(b"PING\r\n").expect("write PING"); + let mut buf = [0u8; 64]; + if let Ok(n) = s.read(&mut buf) + && n > 0 + && buf[..n].windows(4).any(|w| w == b"PONG") + { + return; + } + assert!( + start.elapsed() < Duration::from_secs(10), + "server accepted TCP but never answered PING" + ); + std::thread::sleep(Duration::from_millis(100)); + s = connect(port, Duration::from_secs(5)); + } +} + +// --------------------------------------------------------------------------- +// Minimal RESP2 client +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, PartialEq)] +enum Resp { + Simple(String), + Error(String), + Int(i64), + Bulk(Option>), + Array(Option>), +} + +struct Conn { + s: TcpStream, + buf: Vec, + pos: usize, +} + +impl Conn { + fn open(port: u16) -> Self { + Conn { + s: connect(port, Duration::from_secs(10)), + buf: Vec::with_capacity(16 * 1024), + pos: 0, + } + } + + fn cmd(&mut self, parts: &[&str]) -> Resp { + let mut req = Vec::with_capacity(64); + req.extend_from_slice(format!("*{}\r\n", parts.len()).as_bytes()); + for p in parts { + req.extend_from_slice(format!("${}\r\n", p.len()).as_bytes()); + req.extend_from_slice(p.as_bytes()); + req.extend_from_slice(b"\r\n"); + } + self.s.write_all(&req).expect("write cmd"); + self.frame() + } + + fn fill(&mut self) { + let mut chunk = [0u8; 16 * 1024]; + let n = self.s.read(&mut chunk).expect("read"); + assert!(n > 0, "connection closed mid-frame"); + self.buf.extend_from_slice(&chunk[..n]); + } + + fn line(&mut self) -> String { + loop { + if let Some(rel) = self.buf[self.pos..].windows(2).position(|w| w == b"\r\n") { + let line = + String::from_utf8_lossy(&self.buf[self.pos..self.pos + rel]).into_owned(); + self.pos += rel + 2; + return line; + } + self.fill(); + } + } + + fn exact(&mut self, n: usize) -> Vec { + while self.buf.len() - self.pos < n + 2 { + self.fill(); + } + let out = self.buf[self.pos..self.pos + n].to_vec(); + self.pos += n + 2; + out + } + + fn frame(&mut self) -> Resp { + if self.pos > 0 && self.pos == self.buf.len() { + self.buf.clear(); + self.pos = 0; + } + let line = self.line(); + let (tag, rest) = line.split_at(1); + match tag { + "+" => Resp::Simple(rest.to_string()), + "-" => Resp::Error(rest.to_string()), + ":" => Resp::Int(rest.parse().unwrap_or(0)), + "$" => { + let n: i64 = rest.parse().unwrap_or(-1); + if n < 0 { + Resp::Bulk(None) + } else { + Resp::Bulk(Some(self.exact(n as usize))) + } + } + "*" => { + let n: i64 = rest.parse().unwrap_or(-1); + if n < 0 { + Resp::Array(None) + } else { + let mut items = Vec::with_capacity(n as usize); + for _ in 0..n { + items.push(self.frame()); + } + Resp::Array(Some(items)) + } + } + other => panic!("unexpected RESP tag {other:?} (line {line:?})"), + } + } +} + +/// Find, for each shard, a `{hash-tag}` value whose tagged keys provably route +/// to that shard (via moon's own `key_to_shard`, so co-location is proven, not +/// assumed). `out[s]` is a tag whose `{tag}...` keys land on shard `s`. +fn tags_per_shard(num_shards: usize) -> Vec { + let mut out: Vec> = vec![None; num_shards]; + let mut found = 0; + for i in 0..1_000_000 { + let tag = format!("s{i}"); + // Only the tag inside `{}` is hashed, so any suffix routes identically. + let probe = format!("{{{}}}k", tag); + let s = key_to_shard(probe.as_bytes(), num_shards); + if out[s].is_none() { + out[s] = Some(tag); + found += 1; + if found == num_shards { + break; + } + } + } + out.into_iter() + .map(|o| o.expect("found a tag for every shard")) + .collect() +} + +/// The `(key, value)` pairs of the co-located group on shard `s`. +fn group_pairs(tag: &str) -> Vec<(String, String)> { + (0..GROUP_SIZE) + .map(|j| (format!("{{{}}}:k{}", tag, j), format!("{}-v{}", tag, j))) + .collect() +} + +/// A moon `MOONERR diskfull` write-pause reply. When the data volume dips below +/// the 5%-free guard, writes are refused and this durability test cannot run — +/// callers SKIP (not fail), so a full dev/CI disk doesn't masquerade as a +/// regression. On a healthy host (ample `/tmp`, as on CI) this never fires. +fn is_diskfull(r: &Resp) -> bool { + matches!(r, Resp::Error(m) if m.contains("diskfull")) +} + +/// Kill round-1, restart, and assert EVERY written pair recovered. The failure +/// message identifies the missing group so a RED run points straight at the +/// un-persisted local leg. +fn assert_all_survive_restart( + port: u16, + dir: &std::path::Path, + child1: Child, + expected: &[(String, String)], + what: &str, +) { + // > 1s so the everysec fsync window flushed every append that WAS made. + std::thread::sleep(Duration::from_millis(1500)); + sigkill(child1); + + let _guard = ServerGuard(spawn_moon_aof(port, dir, SHARDS)); + wait_ready(port); + + let mut c = Conn::open(port); + let mut missing: Vec = Vec::new(); + let mut mismatched: Vec = Vec::new(); + for (k, v) in expected { + match c.cmd(&["GET", k]) { + Resp::Bulk(Some(got)) if got == v.as_bytes() => {} + Resp::Bulk(Some(got)) => mismatched.push(format!( + "{k}: want={v} got={}", + String::from_utf8_lossy(&got) + )), + Resp::Bulk(None) => missing.push(k.clone()), + other => panic!("unexpected GET reply for {k}: {other:?}"), + } + } + + assert!( + missing.is_empty() && mismatched.is_empty(), + "{what}: coordinator LOCAL leg lost writes across restart — {} missing, {} mismatched. \ + Missing (the co-located group whose owner == the connection's own shard, whose local \ + leg never hit the AOF): {:?}; mismatched: {:?}", + missing.len(), + mismatched.len(), + missing.iter().take(GROUP_SIZE + 1).collect::>(), + mismatched.iter().take(5).collect::>(), + ); +} + +// --------------------------------------------------------------------------- +// MSET — fast path (all keys of a group co-located on one shard). +// The group whose owner == the connection's shard takes the `groups.len() == 1 +// && contains_key(&my_shard)` fast path → `string::mset(db, args)` with no AOF. +// --------------------------------------------------------------------------- + +#[test] +fn mset_colocated_local_leg_persists_across_restart() { + let port = free_port(); + let dir = tempfile::tempdir().expect("tempdir"); + let child1 = spawn_moon_aof(port, dir.path(), SHARDS); + wait_ready(port); + + let tags = tags_per_shard(SHARDS as usize); + let mut expected: Vec<(String, String)> = Vec::new(); + + // ONE connection for the whole write phase → my_shard is stable. + let mut c = Conn::open(port); + for tag in &tags { + let pairs = group_pairs(tag); + let mut argv: Vec<&str> = vec!["MSET"]; + for (k, v) in &pairs { + argv.push(k); + argv.push(v); + } + let resp = c.cmd(&argv); + if is_diskfull(&resp) { + eprintln!( + "SKIP mset_colocated: MOONERR diskfull — durability untestable on a \ + <5%-free filesystem; needs ample /tmp (CI has it)" + ); + return; + } + assert_eq!( + resp, + Resp::Simple("OK".to_string()), + "co-located MSET on tag {tag} should return OK" + ); + expected.extend(pairs); + } + drop(c); + + assert_all_survive_restart( + port, + dir.path(), + child1, + &expected, + "MSET co-located fast-path", + ); +} + +// --------------------------------------------------------------------------- +// MSET — scatter path (ONE MSET spanning every shard). The slice owned by the +// connection's shard is the local leg (`db.set_string` per key, no AOF); the +// other slices scatter remotely (persisted). +// --------------------------------------------------------------------------- + +#[test] +fn mset_scatter_local_slice_persists_across_restart() { + let port = free_port(); + let dir = tempfile::tempdir().expect("tempdir"); + let child1 = spawn_moon_aof(port, dir.path(), SHARDS); + wait_ready(port); + + let tags = tags_per_shard(SHARDS as usize); + // One key per shard, all in a single MSET → guaranteed multi-shard scatter + // with a local slice on whichever shard the connection landed on. + let expected: Vec<(String, String)> = tags + .iter() + .map(|tag| (format!("{{{}}}:scatter", tag), format!("{}-scatter", tag))) + .collect(); + + let mut argv: Vec<&str> = vec!["MSET"]; + for (k, v) in &expected { + argv.push(k); + argv.push(v); + } + let mut c = Conn::open(port); + let resp = c.cmd(&argv); + if is_diskfull(&resp) { + eprintln!( + "SKIP mset_scatter: MOONERR diskfull — durability untestable on a \ + <5%-free filesystem; needs ample /tmp (CI has it)" + ); + return; + } + assert_eq!( + resp, + Resp::Simple("OK".to_string()), + "scatter MSET should return OK" + ); + drop(c); + + assert_all_survive_restart( + port, + dir.path(), + child1, + &expected, + "MSET scatter local-slice", + ); +} + +// --------------------------------------------------------------------------- +// MSETNX — co-located (all-new → :1). The group whose owner == the connection's +// shard runs via `run_on_owner` → `run_local` with no AOF append. +// --------------------------------------------------------------------------- + +#[test] +fn msetnx_colocated_local_leg_persists_across_restart() { + let port = free_port(); + let dir = tempfile::tempdir().expect("tempdir"); + let child1 = spawn_moon_aof(port, dir.path(), SHARDS); + wait_ready(port); + + let tags = tags_per_shard(SHARDS as usize); + let mut expected: Vec<(String, String)> = Vec::new(); + + let mut c = Conn::open(port); + for tag in &tags { + let pairs = group_pairs(tag); + let mut argv: Vec<&str> = vec!["MSETNX"]; + for (k, v) in &pairs { + argv.push(k); + argv.push(v); + } + let resp = c.cmd(&argv); + if is_diskfull(&resp) { + eprintln!( + "SKIP msetnx_colocated: MOONERR diskfull — durability untestable on a \ + <5%-free filesystem; needs ample /tmp (CI has it)" + ); + return; + } + assert_eq!( + resp, + Resp::Int(1), + "all-new co-located MSETNX on tag {tag} should return 1" + ); + expected.extend(pairs); + } + drop(c); + + assert_all_survive_restart( + port, + dir.path(), + child1, + &expected, + "MSETNX co-located local leg", + ); +} diff --git a/tests/mq_integration.rs b/tests/mq_integration.rs index 5c3ab5784..24034449a 100644 --- a/tests/mq_integration.rs +++ b/tests/mq_integration.rs @@ -83,6 +83,8 @@ async fn start_mq_server(num_shards: usize) -> (u16, CancellationToken) { vec_diskann_beam_width: 8, vec_diskann_cache_levels: 3, uring_sqpoll_ms: None, + io_driver: "auto".to_string(), + io_busy_poll_us: 0, admin_port: 0, slowlog_log_slower_than: 10000, slowlog_max_len: 128, diff --git a/tests/msetnx_cross_shard_reject.rs b/tests/msetnx_cross_shard_reject.rs new file mode 100644 index 000000000..262d4860a --- /dev/null +++ b/tests/msetnx_cross_shard_reject.rs @@ -0,0 +1,311 @@ +//! ADD milestone `v3-4-kv-correctness` — MSETNX cross-shard contract. +//! +//! MSETNX is atomic by contract (set every pair iff none of the keys exist). +//! Moon cannot honor that atomically across shards, so — by deliberate design +//! decision — a MSETNX whose keys span more than one shard is REJECTED with a +//! CROSSSLOT error and writes nothing. When the keys are co-located on a single +//! shard (naturally or via a `{hash-tag}`) the whole command runs atomically on +//! that shard's owner. +//! +//! These are green-after-fix regression tests. They assert BOTH sides so the +//! suite cannot pass vacuously: an "always reject" bug fails the co-located +//! case; a "never reject / route-by-connection" bug fails the cross-shard case. +//! +//! Run alone with: cargo test --test msetnx_cross_shard_reject + +use std::io::{Read, Write}; +use std::net::{TcpStream, ToSocketAddrs}; +use std::process::{Child, Command}; +use std::time::{Duration, Instant}; + +use moon::shard::dispatch::key_to_shard; + +// --------------------------------------------------------------------------- +// Harness (CARGO_BIN_EXE pattern, mirrors cross_shard_consistency_red.rs) +// --------------------------------------------------------------------------- + +fn moon_binary() -> std::path::PathBuf { + std::path::PathBuf::from(env!("CARGO_BIN_EXE_moon")) +} + +fn free_port() -> u16 { + let l = std::net::TcpListener::bind("127.0.0.1:0").expect("bind :0"); + let p = l.local_addr().expect("local_addr").port(); + drop(l); + p +} + +fn spawn_moon(port: u16, dir: &std::path::Path, shards: u32) -> Child { + Command::new(moon_binary()) + .args([ + "--port", + &port.to_string(), + "--dir", + &dir.to_string_lossy(), + "--shards", + &shards.to_string(), + ]) + .stdout(std::fs::File::create(dir.join("moon.stdout.log")).expect("stdout log")) + .stderr(std::fs::File::create(dir.join("moon.stderr.log")).expect("stderr log")) + .spawn() + .expect("spawn moon (CARGO_BIN_EXE_moon)") +} + +struct ServerGuard(Child); +impl Drop for ServerGuard { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +fn connect(port: u16, deadline: Duration) -> TcpStream { + let addr = format!("127.0.0.1:{port}") + .to_socket_addrs() + .expect("addr") + .next() + .expect("one addr"); + let start = Instant::now(); + loop { + match TcpStream::connect_timeout(&addr, Duration::from_millis(200)) { + Ok(s) => { + s.set_read_timeout(Some(Duration::from_secs(5))).ok(); + s.set_write_timeout(Some(Duration::from_secs(5))).ok(); + return s; + } + Err(_) if start.elapsed() < deadline => { + std::thread::sleep(Duration::from_millis(50)); + } + Err(e) => panic!("server never accepted on {port}: {e}"), + } + } +} + +fn wait_ready(port: u16) -> TcpStream { + let mut s = connect(port, Duration::from_secs(30)); + let start = Instant::now(); + loop { + s.write_all(b"PING\r\n").expect("write PING"); + let mut buf = [0u8; 64]; + if let Ok(n) = s.read(&mut buf) + && n > 0 + && buf[..n].windows(4).any(|w| w == b"PONG") + { + return s; + } + assert!( + start.elapsed() < Duration::from_secs(10), + "server accepted TCP but never answered PING" + ); + std::thread::sleep(Duration::from_millis(100)); + s = connect(port, Duration::from_secs(5)); + } +} + +// --------------------------------------------------------------------------- +// Minimal RESP2 reader +// --------------------------------------------------------------------------- + +#[derive(Debug, Clone, PartialEq)] +enum Resp { + Simple(String), + Error(String), + Int(i64), + Bulk(Option>), + Array(Option>), +} + +struct Conn { + s: TcpStream, + buf: Vec, + pos: usize, +} + +impl Conn { + fn open(port: u16) -> Self { + Conn { + s: connect(port, Duration::from_secs(10)), + buf: Vec::with_capacity(16 * 1024), + pos: 0, + } + } + + fn cmd_s(&mut self, parts: &[&str]) -> Resp { + let mut req = Vec::with_capacity(64); + req.extend_from_slice(format!("*{}\r\n", parts.len()).as_bytes()); + for p in parts { + req.extend_from_slice(format!("${}\r\n", p.len()).as_bytes()); + req.extend_from_slice(p.as_bytes()); + req.extend_from_slice(b"\r\n"); + } + self.s.write_all(&req).expect("write cmd"); + self.frame() + } + + fn fill(&mut self) { + let mut chunk = [0u8; 16 * 1024]; + let n = self.s.read(&mut chunk).expect("read"); + assert!(n > 0, "connection closed mid-frame"); + self.buf.extend_from_slice(&chunk[..n]); + } + + fn line(&mut self) -> String { + loop { + if let Some(rel) = self.buf[self.pos..].windows(2).position(|w| w == b"\r\n") { + let line = + String::from_utf8_lossy(&self.buf[self.pos..self.pos + rel]).into_owned(); + self.pos += rel + 2; + return line; + } + self.fill(); + } + } + + fn exact(&mut self, n: usize) -> Vec { + while self.buf.len() - self.pos < n + 2 { + self.fill(); + } + let out = self.buf[self.pos..self.pos + n].to_vec(); + self.pos += n + 2; + out + } + + fn frame(&mut self) -> Resp { + if self.pos > 0 && self.pos == self.buf.len() { + self.buf.clear(); + self.pos = 0; + } + let line = self.line(); + let (tag, rest) = line.split_at(1); + match tag { + "+" => Resp::Simple(rest.to_string()), + "-" => Resp::Error(rest.to_string()), + ":" => Resp::Int(rest.parse().unwrap_or(0)), + "$" => { + let n: i64 = rest.parse().unwrap_or(-1); + if n < 0 { + Resp::Bulk(None) + } else { + Resp::Bulk(Some(self.exact(n as usize))) + } + } + "*" => { + let n: i64 = rest.parse().unwrap_or(-1); + if n < 0 { + Resp::Array(None) + } else { + let mut items = Vec::with_capacity(n as usize); + for _ in 0..n { + items.push(self.frame()); + } + Resp::Array(Some(items)) + } + } + other => panic!("unexpected RESP tag {other:?} (line {line:?})"), + } + } +} + +/// Return one key per shard (index = shard id), generated deterministically via +/// moon's own hash so cross-shard-ness is provable, not assumed. +fn keys_per_shard(prefix: &str, num_shards: usize) -> Vec { + let mut out: Vec> = vec![None; num_shards]; + let mut found = 0; + for i in 0..10_000 { + let k = format!("{prefix}{i}"); + let s = key_to_shard(k.as_bytes(), num_shards); + if out[s].is_none() { + out[s] = Some(k); + found += 1; + if found == num_shards { + break; + } + } + } + out.into_iter() + .map(|o| o.expect("found a key for every shard")) + .collect() +} + +const SHARDS: u32 = 4; + +// --------------------------------------------------------------------------- +// Cross-shard MSETNX is rejected (CROSSSLOT) and writes NOTHING. +// --------------------------------------------------------------------------- + +#[test] +fn msetnx_cross_shard_rejected_no_partial_write() { + let port = free_port(); + let dir = tempfile::tempdir().expect("tempdir"); + let _guard = ServerGuard(spawn_moon(port, dir.path(), SHARDS)); + drop(wait_ready(port)); + + // Two keys that PROVABLY land on different shards. + let ks = keys_per_shard("msnx:", SHARDS as usize); + let (k0, k1) = (&ks[0], &ks[1]); + assert_ne!( + key_to_shard(k0.as_bytes(), SHARDS as usize), + key_to_shard(k1.as_bytes(), SHARDS as usize), + "test precondition: keys must span shards" + ); + + let mut c = Conn::open(port); + let r = c.cmd_s(&["MSETNX", k0, "v0", k1, "v1"]); + match &r { + Resp::Error(m) => assert!( + m.starts_with("CROSSSLOT"), + "cross-shard MSETNX must be a CROSSSLOT error (got {r:?})" + ), + other => panic!("cross-shard MSETNX must be rejected, got {other:?}"), + } + + // Reject is total: NEITHER key was written (no partial side effects). + assert_eq!( + c.cmd_s(&["GET", k0]), + Resp::Bulk(None), + "rejected MSETNX must not write k0" + ); + assert_eq!( + c.cmd_s(&["GET", k1]), + Resp::Bulk(None), + "rejected MSETNX must not write k1" + ); +} + +// --------------------------------------------------------------------------- +// Co-located MSETNX runs atomically on the owner shard (all-or-nothing). +// --------------------------------------------------------------------------- + +#[test] +fn msetnx_colocated_is_atomic() { + let port = free_port(); + let dir = tempfile::tempdir().expect("tempdir"); + let _guard = ServerGuard(spawn_moon(port, dir.path(), SHARDS)); + drop(wait_ready(port)); + + let mut c = Conn::open(port); + + // All keys share the {t} hash-tag -> one shard, even at --shards 4. + assert_eq!( + c.cmd_s(&["MSETNX", "{t}a", "v1", "{t}b", "v2"]), + Resp::Int(1), + "all-new co-located MSETNX returns 1" + ); + assert_eq!(c.cmd_s(&["GET", "{t}a"]), Resp::Bulk(Some(b"v1".to_vec()))); + assert_eq!(c.cmd_s(&["GET", "{t}b"]), Resp::Bulk(Some(b"v2".to_vec()))); + + // One key already exists -> whole command is a no-op, returns 0. + assert_eq!( + c.cmd_s(&["MSETNX", "{t}b", "vX", "{t}c", "v3"]), + Resp::Int(0), + "MSETNX with any existing key returns 0" + ); + // Atomic: the new key {t}c must NOT have been written. + assert_eq!( + c.cmd_s(&["GET", "{t}c"]), + Resp::Bulk(None), + "MSETNX no-op must not write {{t}}c" + ); + // And the pre-existing value is unchanged. + assert_eq!(c.cmd_s(&["GET", "{t}b"]), Resp::Bulk(Some(b"v2".to_vec()))); +} diff --git a/tests/txn_kv_wiring.rs b/tests/txn_kv_wiring.rs index dd10c4322..6569bf8be 100644 --- a/tests/txn_kv_wiring.rs +++ b/tests/txn_kv_wiring.rs @@ -88,6 +88,8 @@ async fn start_txn_server(num_shards: usize, persistence_dir: &str) -> (u16, Can vec_diskann_beam_width: 8, vec_diskann_cache_levels: 3, uring_sqpoll_ms: None, + io_driver: "auto".to_string(), + io_busy_poll_us: 0, admin_port: 0, slowlog_log_slower_than: 10000, slowlog_max_len: 128, diff --git a/tests/workspace_integration.rs b/tests/workspace_integration.rs index 569bbc952..dda2a6492 100644 --- a/tests/workspace_integration.rs +++ b/tests/workspace_integration.rs @@ -76,6 +76,8 @@ async fn start_workspace_server(num_shards: usize) -> (u16, CancellationToken) { vec_diskann_beam_width: 8, vec_diskann_cache_levels: 3, uring_sqpoll_ms: None, + io_driver: "auto".to_string(), + io_busy_poll_us: 0, admin_port: 0, slowlog_log_slower_than: 10000, slowlog_max_len: 128, @@ -301,6 +303,8 @@ async fn start_workspace_server_with_auth( vec_diskann_beam_width: 8, vec_diskann_cache_levels: 3, uring_sqpoll_ms: None, + io_driver: "auto".to_string(), + io_busy_poll_us: 0, admin_port: 0, slowlog_log_slower_than: 10000, slowlog_max_len: 128, diff --git a/tests/xshard_fastpath_api.rs b/tests/xshard_fastpath_api.rs index 32fbed558..1df03988d 100644 --- a/tests/xshard_fastpath_api.rs +++ b/tests/xshard_fastpath_api.rs @@ -15,10 +15,42 @@ //! Running: cargo test --test xshard_fastpath_api use moon::shard::slice::{ - XSHARD_SPIN_GATE, XSHARD_SPIN_MAX_BATCH_REMOTE, XshardWaitGuard, xshard_may_spin, - xshard_should_spin, + ShardConnGuard, XSHARD_SPIN_GATE, XSHARD_SPIN_MAX_BATCH_REMOTE, XshardWaitGuard, + xshard_may_spin, xshard_should_spin, }; +/// L1 convoy fix — the reply-side spin must engage ONLY when the requesting +/// connection is ALONE on its shard thread. With any sibling connection present +/// the synchronous spin starves that sibling AND stalls this shard's SPSC drain +/// for other shards' requests — the measured s4 c8P1 convoy (GCE c4a +/// same-instance A/B 2026-07-03: spin-on 72.7k vs spin-off 200k ops/s). The +/// `XSHARD_INFLIGHT` gate cannot see a sibling whose readable event is still +/// parked in the driver; the per-shard connection count can. +#[test] +fn solo_conn_gate_blocks_spin_with_sibling_connection() { + // Test threads start with 0 registered conns — gate open (idle thread). + assert!(xshard_may_spin(), "no registered conns → gate open"); + + let _me = ShardConnGuard::new(); + assert!( + xshard_may_spin(), + "a solo connection may spin — this is the c1 latency win and must be preserved" + ); + + { + let _sibling = ShardConnGuard::new(); + assert!( + !xshard_may_spin(), + "a sibling connection on the shard thread must force the park path (anti-convoy)" + ); + } + + assert!( + xshard_may_spin(), + "sibling disconnected (RAII decrement) → the solo conn may spin again" + ); +} + /// xrf1 — a near-idle shard (no in-flight cross-shard reply-waiters on this thread) /// MUST allow the reply-side spin. This is the entire c1 latency win: when the /// requesting connection is effectively alone, polling its reply skips the reply-side diff --git a/vendor/monoio/.cargo-ok b/vendor/monoio/.cargo-ok new file mode 100644 index 000000000..5f8b79583 --- /dev/null +++ b/vendor/monoio/.cargo-ok @@ -0,0 +1 @@ +{"v":1} \ No newline at end of file diff --git a/vendor/monoio/.cargo_vcs_info.json b/vendor/monoio/.cargo_vcs_info.json new file mode 100644 index 000000000..e7e301535 --- /dev/null +++ b/vendor/monoio/.cargo_vcs_info.json @@ -0,0 +1,7 @@ +{ + "git": { + "sha1": "f7827ddd54e0a6c7e9d1805109df041fd101994b", + "dirty": true + }, + "path_in_vcs": "monoio" +} \ No newline at end of file diff --git a/vendor/monoio/Cargo.toml b/vendor/monoio/Cargo.toml new file mode 100644 index 000000000..6de3d74df --- /dev/null +++ b/vendor/monoio/Cargo.toml @@ -0,0 +1,260 @@ +# THIS FILE IS AUTOMATICALLY GENERATED BY CARGO +# +# When uploading crates to the registry Cargo will automatically +# "normalize" Cargo.toml files for maximal compatibility +# with all versions of Cargo and also rewrite `path` dependencies +# to registry (e.g., crates.io) dependencies. +# +# If you are reading this file be aware that the original Cargo.toml +# will likely look very different (and much more reasonable). +# See Cargo.toml.orig for the original contents. + +[package] +edition = "2021" +name = "monoio" +version = "0.2.4" +authors = [ + "ChiHai ", + "XuShuai ", +] +build = false +autobins = false +autoexamples = false +autotests = false +autobenches = false +description = "A thread per core runtime based on iouring." +readme = "README.md" +keywords = [ + "runtime", + "iouring", + "async", +] +categories = [ + "asynchronous", + "network-programming", +] +license = "MIT OR Apache-2.0" +repository = "https://github.com/bytedance/monoio" + +[lib] +name = "monoio" +path = "src/lib.rs" + +[[test]] +name = "buf_writter" +path = "tests/buf_writter.rs" + +[[test]] +name = "ctrlc_legacy" +path = "tests/ctrlc_legacy.rs" + +[[test]] +name = "ctrlc_uring" +path = "tests/ctrlc_uring.rs" + +[[test]] +name = "fs_create_dir" +path = "tests/fs_create_dir.rs" + +[[test]] +name = "fs_file" +path = "tests/fs_file.rs" + +[[test]] +name = "fs_metadata" +path = "tests/fs_metadata.rs" + +[[test]] +name = "fs_rename" +path = "tests/fs_rename.rs" + +[[test]] +name = "fs_unlink" +path = "tests/fs_unlink.rs" + +[[test]] +name = "tcp_accept" +path = "tests/tcp_accept.rs" + +[[test]] +name = "tcp_connect" +path = "tests/tcp_connect.rs" + +[[test]] +name = "tcp_echo" +path = "tests/tcp_echo.rs" + +[[test]] +name = "tcp_into_split" +path = "tests/tcp_into_split.rs" + +[[test]] +name = "tcp_split" +path = "tests/tcp_split.rs" + +[[test]] +name = "udp" +path = "tests/udp.rs" + +[[test]] +name = "uds_cred" +path = "tests/uds_cred.rs" + +[[test]] +name = "uds_split" +path = "tests/uds_split.rs" + +[[test]] +name = "uds_stream" +path = "tests/uds_stream.rs" + +[[test]] +name = "unix_datagram" +path = "tests/unix_datagram.rs" + +[[test]] +name = "unix_seqpacket" +path = "tests/unix_seqpacket.rs" + +[[test]] +name = "zero_copy" +path = "tests/zero_copy.rs" + +[dependencies.auto-const-array] +version = "0.2" + +[dependencies.bytes] +version = "1" +optional = true + +[dependencies.ctrlc] +version = "3" +optional = true + +[dependencies.flume] +version = "0.11" +optional = true + +[dependencies.fxhash] +version = "0.2" + +[dependencies.lazy_static] +version = "1" +optional = true + +[dependencies.libc] +version = "0.2" + +[dependencies.memchr] +version = "2.7" + +[dependencies.mio] +version = "0.8" +features = [ + "net", + "os-poll", + "os-ext", +] +optional = true + +[dependencies.monoio-macros] +version = "0.1.0" +optional = true + +[dependencies.once_cell] +version = "1.19.0" +optional = true + +[dependencies.pin-project-lite] +version = "0.2" + +[dependencies.socket2] +version = "0.5" +features = ["all"] + +[dependencies.threadpool] +version = "1" +optional = true + +[dependencies.tokio] +version = "1" +optional = true +default-features = false + +[dependencies.tracing] +version = "0.1" +features = ["std"] +optional = true +default-features = false + +[dev-dependencies.futures] +version = "0.3" + +[dev-dependencies.local-sync] +version = "0.0.5" + +[dev-dependencies.tempfile] +version = "3.2" + +[features] +async-cancel = [] +debug = ["tracing"] +default = [ + "async-cancel", + "bytes", + "iouring", + "legacy", + "macros", + "utils", +] +iouring = ["io-uring"] +legacy = ["mio"] +macros = ["monoio-macros"] +mkdirat = [] +poll-io = [ + "tokio", + "mio", +] +renameat = [] +signal = [ + "ctrlc", + "sync", +] +signal-termination = [ + "signal", + "ctrlc/termination", +] +splice = [] +sync = [ + "flume", + "threadpool", + "once_cell", +] +tokio-compat = ["tokio"] +unlinkat = [] +unstable = [] +utils = ["nix"] +zero-copy = [] + +[target.'cfg(target_os = "linux")'.dependencies.io-uring] +version = "0.6" +optional = true + +[target."cfg(unix)".dependencies.nix] +version = "0.26" +optional = true + +[target."cfg(windows)".dependencies.windows-sys] +version = "0.48.0" +features = [ + "Win32_Foundation", + "Win32_Networking_WinSock", + "Win32_System_IO", + "Win32_Storage_FileSystem", + "Win32_Security", +] + +[lints.rust.unexpected_cfgs] +level = "warn" +priority = 0 +check-cfg = ["cfg(loom)"] diff --git a/vendor/monoio/Cargo.toml.orig b/vendor/monoio/Cargo.toml.orig new file mode 100644 index 000000000..6d1c34a51 --- /dev/null +++ b/vendor/monoio/Cargo.toml.orig @@ -0,0 +1,101 @@ +[package] +authors = ["ChiHai ", "XuShuai "] +categories = ["asynchronous", "network-programming"] +description = "A thread per core runtime based on iouring." +edition = "2021" +keywords = ["runtime", "iouring", "async"] +license = "MIT OR Apache-2.0" +name = "monoio" +readme = "../README.md" +repository = "https://github.com/bytedance/monoio" +version = "0.2.4" + +# common dependencies +[dependencies] +monoio-macros = { version = "0.1.0", path = "../monoio-macros", optional = true } + +auto-const-array = "0.2" +fxhash = "0.2" +libc = "0.2" +pin-project-lite = "0.2" +socket2 = { version = "0.5", features = ["all"] } +memchr = "2.7" + +bytes = { version = "1", optional = true } +flume = { version = "0.11", optional = true } +mio = { version = "0.8", features = [ + "net", + "os-poll", + "os-ext", +], optional = true } +threadpool = { version = "1", optional = true } +tokio = { version = "1", default-features = false, optional = true } +tracing = { version = "0.1", default-features = false, features = [ + "std", +], optional = true } +ctrlc = { version = "3", optional = true } +lazy_static = { version = "1", optional = true } +once_cell = { version = "1.19.0", optional = true } + +# windows dependencies(will be added when windows support finished) +[target.'cfg(windows)'.dependencies] +windows-sys = { version = "0.48.0", features = [ + "Win32_Foundation", + "Win32_Networking_WinSock", + "Win32_System_IO", + "Win32_Storage_FileSystem", + "Win32_Security" +] } + +# unix dependencies +[target.'cfg(unix)'.dependencies] +nix = { version = "0.26", optional = true } + +[target.'cfg(target_os = "linux")'.dependencies] +io-uring = { version = "0.6", optional = true } + +[dev-dependencies] +futures = "0.3" +local-sync = "0.0.5" +tempfile = "3.2" + +[features] +# use nightly only feature flags +unstable = [] +# async-cancel will push a async-cancel entry into sq when op is canceled +async-cancel = [] +# enanle zero copy(enable SOCK_ZEROCOPY + MSG_ZEROCOPY flag) +# WARNING: this feature may cause performance degradation +zero-copy = [] +# splice op(requires kernel 5.7+) +splice = [] +# mkdirat2 op(requires kernel 5.15+) +mkdirat = [] +# unlinkat op(requires kernel 5.11+) +unlinkat = [] +# renameat op(requires kernel 5.11+) +renameat = [] +# enable `async main` macros support +macros = ["monoio-macros"] +# allow waker to be sent across threads +sync = ["flume", "threadpool", "once_cell"] +# enable bind cpu set +utils = ["nix"] +# enable debug if you want to know what runtime does +debug = ["tracing"] +# enable legacy driver support(will make monoio available for older kernel and macOS) +legacy = ["mio"] +# iouring support +iouring = ["io-uring"] +# tokio-compatible(only have effect when legacy is enabled and iouring is not) +tokio-compat = ["tokio"] +# (experimental)enable poll-io to convert structs to structs that impl tokio's poll io +poll-io = ["tokio", "mio"] +# signal enables setting ctrl_c handler +signal = ["ctrlc", "sync"] +signal-termination = ["signal", "ctrlc/termination"] +# by default both iouring and legacy are enabled +default = ["async-cancel", "bytes", "iouring", "legacy", "macros", "utils"] + +[lints.rust] +unexpected_cfgs = { level = "warn", check-cfg = ['cfg(loom)'] } diff --git a/vendor/monoio/LICENSE-APACHE b/vendor/monoio/LICENSE-APACHE new file mode 100644 index 000000000..b773ef56f --- /dev/null +++ b/vendor/monoio/LICENSE-APACHE @@ -0,0 +1,202 @@ +Copyright (c) 2021 ihciah, dyxushuai and other Monoio Contributors + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + +Copyright [yyyy] [name of copyright owner] + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. \ No newline at end of file diff --git a/vendor/monoio/LICENSE-MIT b/vendor/monoio/LICENSE-MIT new file mode 100644 index 000000000..81ab2910b --- /dev/null +++ b/vendor/monoio/LICENSE-MIT @@ -0,0 +1,25 @@ +Copyright (c) 2021 ihciah, dyxushuai and other Monoio Contributors + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/vendor/monoio/LICENSE-THIRD-PARTY b/vendor/monoio/LICENSE-THIRD-PARTY new file mode 100644 index 000000000..edbde19de --- /dev/null +++ b/vendor/monoio/LICENSE-THIRD-PARTY @@ -0,0 +1,360 @@ +Third party project code used by this project: Mio, Tokio, Tokio-uring, Futures-rs, Slab. + +=============================================================================== + +Mio +https://github.com/tokio-rs/mio/blob/master/LICENSE + +Copyright (c) 2014 Carl Lerche and other MIO contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. + + +=============================================================================== + +Tokio +https://github.com/tokio-rs/tokio/blob/master/LICENSE + +Copyright (c) 2021 Tokio Contributors + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + + +=============================================================================== + +Tokio-Uring +https://github.com/tokio-rs/tokio-uring/blob/master/LICENSE + +Copyright (c) 2021 Carl Lerche + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + + +=============================================================================== + +Futures-rs +https://github.com/rust-lang/futures-rs/blob/master/LICENSE-MIT + +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. + +https://github.com/rust-lang/futures-rs/blob/master/LICENSE-APACHE + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + +4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + +5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + +6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + +7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + +8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + +9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + +END OF TERMS AND CONDITIONS + +APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + +Copyright (c) 2016 Alex Crichton +Copyright (c) 2017 The Tokio Authors + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + + +=============================================================================== + +Slab +https://github.com/tokio-rs/slab/blob/master/LICENSE + +Copyright (c) 2019 Carl Lerche + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. diff --git a/vendor/monoio/README.md b/vendor/monoio/README.md new file mode 100644 index 000000000..0083a3f9c --- /dev/null +++ b/vendor/monoio/README.md @@ -0,0 +1,111 @@ +# Monoio +A thread-per-core Rust runtime with io_uring/epoll/kqueue. + +[![Crates.io][crates-badge]][crates-url] +[![MIT/Apache-2 licensed][license-badge]][license-url] +[![Build Status][actions-badge]][actions-url] +[![Codecov][codecov-badge]][codecov-url] +[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fbytedance%2Fmonoio.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fbytedance%2Fmonoio?ref=badge_shield) +[中文说明][zh-readme-url] + +[crates-badge]: https://img.shields.io/crates/v/monoio.svg +[crates-url]: https://crates.io/crates/monoio +[license-badge]: https://img.shields.io/crates/l/monoio.svg +[license-url]: LICENSE-MIT +[actions-badge]: https://github.com/bytedance/monoio/actions/workflows/ci.yml/badge.svg +[actions-url]: https://github.com/bytedance/monoio/actions +[codecov-badge]: https://codecov.io/gh/bytedance/monoio/branch/master/graph/badge.svg?token=3MSAMJ6X3E +[codecov-url]: https://codecov.io/gh/bytedance/monoio +[zh-readme-url]: README-zh.md + +## Design Goal +Monoio is a pure io_uring/epoll/kqueue Rust async runtime. Part of the design has been borrowed from Tokio and Tokio-uring. However, unlike Tokio-uring, Monoio does not run on top of another runtime, rendering it more efficient. + +Moreover, Monoio is designed with a thread-per-core model in mind. Users do not need to worry about tasks being `Send` or `Sync`, as thread local storage can be used safely. In other words, the data does not escape the thread on await points, unlike on work-stealing runtimes such as Tokio. This is because for some use cases, specifically those targeted by this runtime, it is not necessary to make task schedulable between threads. For example, if we were to write a load balancer like NGINX, we would write it in a thread-per-core way. The thread local data does not need to be shared between threads, so the `Sync` and `Send` do not need to be implemented in the first place. + +As you may have guessed, this runtime is primarily targeted at servers, where operations are io-bound on network sockets, and therefore the use of native asynchronous I/O APIs maximizes the throughput of the server. In order for Monoio to be as efficient as possible, we've enabled some unstable Rust features, and we've designed a whole new IO abstraction, which unfortunately may cause some compatibility problems. [Our benchmarks](https://github.com/bytedance/monoio/blob/master/docs/en/benchmark.md) prove that, for our use-cases, Monoio has a better performance than other Rust runtimes. + +## Quick Start +To use monoio, you need rust 1.75. If you already installed it, please make sure it is the latest version. + +Also, if you want to use io_uring, you must make sure your kernel supports it([5.6+](docs/en/platform-support.md)). And, memlock is [configured as a proper number](docs/en/memlock.md). If your kernel version does not meet the requirements, you can try to use the legacy driver to start, currently supports Linux and macOS([ref here](/docs/en/use-legacy-driver.md)). + +🚧Experimental windows support is on the way. + +Here is a basic example of how to use Monoio. + +```rust,no_run +/// A echo example. +/// +/// Run the example and `nc 127.0.0.1 50002` in another shell. +/// All your input will be echoed out. +use monoio::io::{AsyncReadRent, AsyncWriteRentExt}; +use monoio::net::{TcpListener, TcpStream}; + +#[monoio::main] +async fn main() { + let listener = TcpListener::bind("127.0.0.1:50002").unwrap(); + println!("listening"); + loop { + let incoming = listener.accept().await; + match incoming { + Ok((stream, addr)) => { + println!("accepted a connection from {}", addr); + monoio::spawn(echo(stream)); + } + Err(e) => { + println!("accepted connection failed: {}", e); + return; + } + } + } +} + +async fn echo(mut stream: TcpStream) -> std::io::Result<()> { + let mut buf: Vec = Vec::with_capacity(8 * 1024); + let mut res; + loop { + // read + (res, buf) = stream.read(buf).await; + if res? == 0 { + return Ok(()); + } + + // write all + (res, buf) = stream.write_all(buf).await; + res?; + + // clear + buf.clear(); + } +} +``` + +You can find more example code in `examples` of this repository. + +## Limitations +1. On Linux 5.6 or newer, Monoio can use uring or epoll as io driver. On lower versions of Linux, it can only run in epoll mode. On macOS, kqueue can be used. Other platforms are currently not supported. +2. Monoio can not solve all problems. If the workload is very unbalanced, it may cause performance degradation than Tokio since CPU cores may not be fully utilized. + +## Contributors + + +Thanks for their contributions! + +## Community +Monoio is a subproject of [CloudWeGo](https://www.cloudwego.io/). We are committed to building a cloud native ecosystem. + +## Associated Projects +- [local-sync](https://github.com/monoio-rs/local-sync): A thread local channel. +- [monoio-tls](https://github.com/monoio-rs/monoio-tls): TLS wrapper for Monoio. +- [monoio-codec](https://github.com/monoio-rs/monoio-codec): Codec utility for Monoio. + +HTTP framework and RPC framework are on the way. + +## Licenses +Monoio is licensed under the MIT license or Apache license. + +During developing we referenced a lot from Tokio, Mio, Tokio-uring and other related projects. We would like to thank the authors of these projects. + + +[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fbytedance%2Fmonoio.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fbytedance%2Fmonoio?ref=badge_large) diff --git a/vendor/monoio/src/blocking.rs b/vendor/monoio/src/blocking.rs new file mode 100644 index 000000000..d28d0dbf9 --- /dev/null +++ b/vendor/monoio/src/blocking.rs @@ -0,0 +1,328 @@ +//! Blocking tasks related. + +use std::{future::Future, task::Poll}; + +use threadpool::{Builder as ThreadPoolBuilder, ThreadPool as ThreadPoolImpl}; + +use crate::{ + task::{new_task, JoinHandle}, + utils::thread_id::DEFAULT_THREAD_ID, +}; + +/// Users may implement a ThreadPool and attach it to runtime. +/// We also provide an implementation based on threadpool crate, you can use DefaultThreadPool. +pub trait ThreadPool { + /// Monoio runtime will call `schedule_task` on `spawn_blocking`. + /// ThreadPool impl must execute it now or later. + fn schedule_task(&self, task: BlockingTask); +} + +/// Error on waiting blocking task. +#[derive(Debug, Clone, Copy)] +pub enum JoinError { + /// Task is canceled. + Canceled, +} + +/// BlockingTask is contrusted by monoio, ThreadPool impl +/// will execute it with `.run()`. +pub struct BlockingTask { + task: Option>, + blocking_vtable: &'static BlockingTaskVtable, +} + +unsafe impl Send for BlockingTask {} + +struct BlockingTaskVtable { + pub(crate) drop: unsafe fn(&mut crate::task::Task), +} + +fn blocking_vtable() -> &'static BlockingTaskVtable { + &BlockingTaskVtable { + drop: blocking_task_drop::, + } +} + +fn blocking_task_drop(task: &mut crate::task::Task) { + let mut opt: Option> = Some(Err(JoinError::Canceled)); + unsafe { task.finish((&mut opt) as *mut _ as *mut ()) }; +} + +impl Drop for BlockingTask { + fn drop(&mut self) { + if let Some(task) = self.task.as_mut() { + unsafe { (self.blocking_vtable.drop)(task) }; + } + } +} + +impl BlockingTask { + /// Run task. + #[inline] + pub fn run(mut self) { + let task = self.task.take().unwrap(); + task.run(); + // // if we are within a runtime, just run it. + // if crate::runtime::CURRENT.is_set() { + // task.run(); + // return; + // } + // // if we are on a standalone thread, we will use thread local ctx as Context. + // crate::runtime::DEFAULT_CTX.with(|ctx| { + // crate::runtime::CURRENT.set(ctx, || task.run()); + // }); + } +} + +/// BlockingStrategy can be set if there is no ThreadPool attached. +/// It controls how to handle `spawn_blocking` without thread pool. +#[derive(Clone, Copy, Debug)] +pub enum BlockingStrategy { + /// Panic when `spawn_blocking`. + Panic, + /// Execute with current thread when `spawn_blocking`. + ExecuteLocal, +} + +/// `spawn_blocking` is used for executing a task(without async) with heavy computation or blocking +/// io. To used it, users may initialize a thread pool and attach it on creating runtime. +/// Users can also set `BlockingStrategy` for a runtime when there is no thread pool. +/// WARNING: DO NOT USE THIS FOR ASYNC TASK! Async tasks will not be executed but only built the +/// future! +pub fn spawn_blocking(func: F) -> JoinHandle> +where + F: FnOnce() -> R + Send + 'static, + R: Send + 'static, +{ + let fut = BlockingFuture(Some(func)); + let (task, join) = new_task(DEFAULT_THREAD_ID, fut, NoopScheduler); + crate::runtime::CURRENT.with(|inner| { + let handle = &inner.blocking_handle; + match handle { + BlockingHandle::Attached(shared) => shared.schedule_task(BlockingTask { + task: Some(task), + blocking_vtable: blocking_vtable::(), + }), + BlockingHandle::Empty(BlockingStrategy::ExecuteLocal) => task.run(), + BlockingHandle::Empty(BlockingStrategy::Panic) => { + // For users: if you see this panic, you have 2 choices: + // 1. attach a shared thread pool to execute blocking tasks + // 2. set runtime blocking strategy to `BlockingStrategy::ExecuteLocal` + // Note: solution 2 will execute blocking task on current thread and may block other + // tasks This may cause other tasks high latency. + panic!("execute blocking task without thread pool attached") + } + } + }); + + join +} + +/// DefaultThreadPool is a simple wrapped `threadpool::ThreadPool` that implement +/// `monoio::blocking::ThreadPool`. You may use this implementation, or you can use your own thread +/// pool implementation. +#[derive(Clone)] +pub struct DefaultThreadPool { + pool: ThreadPoolImpl, +} + +impl DefaultThreadPool { + /// Create a new DefaultThreadPool. + pub fn new(num_threads: usize) -> Self { + let pool = ThreadPoolBuilder::default() + .num_threads(num_threads) + .build(); + Self { pool } + } +} + +impl ThreadPool for DefaultThreadPool { + #[inline] + fn schedule_task(&self, task: BlockingTask) { + self.pool.execute(move || task.run()); + } +} + +pub(crate) struct NoopScheduler; + +impl crate::task::Schedule for NoopScheduler { + fn schedule(&self, _task: crate::task::Task) { + unreachable!() + } + + fn yield_now(&self, _task: crate::task::Task) { + unreachable!() + } +} + +pub(crate) enum BlockingHandle { + Attached(Box), + Empty(BlockingStrategy), +} + +impl From for BlockingHandle { + fn from(value: BlockingStrategy) -> Self { + Self::Empty(value) + } +} + +struct BlockingFuture(Option); + +impl Unpin for BlockingFuture {} + +impl Future for BlockingFuture +where + F: FnOnce() -> R + Send + 'static, + R: Send + 'static, +{ + type Output = Result; + + fn poll( + mut self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll { + let me = &mut *self; + let func = me.0.take().expect("blocking task ran twice."); + Poll::Ready(Ok(func())) + } +} + +#[cfg(test)] +mod tests { + use super::DefaultThreadPool; + + /// NaiveThreadPool always create a new thread on executing tasks. + struct NaiveThreadPool; + + impl super::ThreadPool for NaiveThreadPool { + fn schedule_task(&self, task: super::BlockingTask) { + std::thread::spawn(move || { + task.run(); + }); + } + } + + /// FakeThreadPool always drop tasks. + struct FakeThreadPool; + + impl super::ThreadPool for FakeThreadPool { + fn schedule_task(&self, _task: super::BlockingTask) {} + } + + #[test] + fn hello_blocking() { + let shared_pool = Box::new(NaiveThreadPool); + let mut rt = crate::RuntimeBuilder::::new() + .attach_thread_pool(shared_pool) + .enable_timer() + .build() + .unwrap(); + rt.block_on(async { + let begin = std::time::Instant::now(); + let join = crate::spawn_blocking(|| { + // Simulate a heavy computation. + std::thread::sleep(std::time::Duration::from_millis(400)); + "hello spawn_blocking!".to_string() + }); + let sleep_async = crate::time::sleep(std::time::Duration::from_millis(400)); + let (result, _) = crate::join!(join, sleep_async); + let eps = begin.elapsed(); + assert!(eps < std::time::Duration::from_millis(800)); + assert!(eps >= std::time::Duration::from_millis(400)); + assert_eq!(result.unwrap(), "hello spawn_blocking!"); + }); + } + + #[test] + #[should_panic] + fn blocking_panic() { + let mut rt = crate::RuntimeBuilder::::new() + .with_blocking_strategy(crate::blocking::BlockingStrategy::Panic) + .enable_timer() + .build() + .unwrap(); + rt.block_on(async { + let join = crate::spawn_blocking(|| 1); + let _ = join.await; + }); + } + + #[test] + fn blocking_current() { + let mut rt = crate::RuntimeBuilder::::new() + .with_blocking_strategy(crate::blocking::BlockingStrategy::ExecuteLocal) + .enable_timer() + .build() + .unwrap(); + rt.block_on(async { + let begin = std::time::Instant::now(); + let join = crate::spawn_blocking(|| { + // Simulate a heavy computation. + std::thread::sleep(std::time::Duration::from_millis(100)); + "hello spawn_blocking!".to_string() + }); + let sleep_async = crate::time::sleep(std::time::Duration::from_millis(100)); + let (result, _) = crate::join!(join, sleep_async); + let eps = begin.elapsed(); + assert!(eps > std::time::Duration::from_millis(200)); + assert_eq!(result.unwrap(), "hello spawn_blocking!"); + }); + } + + #[test] + fn drop_task() { + let shared_pool = Box::new(FakeThreadPool); + let mut rt = crate::RuntimeBuilder::::new() + .attach_thread_pool(shared_pool) + .enable_timer() + .build() + .unwrap(); + rt.block_on(async { + let ret = crate::spawn_blocking(|| 1).await; + assert!(matches!(ret, Err(super::JoinError::Canceled))); + }); + } + + #[test] + fn default_pool() { + let shared_pool = Box::new(DefaultThreadPool::new(3)); + let mut rt = crate::RuntimeBuilder::::new() + .attach_thread_pool(shared_pool) + .enable_timer() + .build() + .unwrap(); + rt.block_on(async { + let begin = std::time::Instant::now(); + let join1 = crate::spawn_blocking(|| { + // Simulate a heavy computation. + std::thread::sleep(std::time::Duration::from_millis(150)); + "hello spawn_blocking1!".to_string() + }); + let join2 = crate::spawn_blocking(|| { + // Simulate a heavy computation. + std::thread::sleep(std::time::Duration::from_millis(150)); + "hello spawn_blocking2!".to_string() + }); + let join3 = crate::spawn_blocking(|| { + // Simulate a heavy computation. + std::thread::sleep(std::time::Duration::from_millis(150)); + "hello spawn_blocking3!".to_string() + }); + let join4 = crate::spawn_blocking(|| { + // Simulate a heavy computation. + std::thread::sleep(std::time::Duration::from_millis(150)); + "hello spawn_blocking4!".to_string() + }); + let sleep_async = crate::time::sleep(std::time::Duration::from_millis(150)); + let (result1, result2, result3, result4, _) = + crate::join!(join1, join2, join3, join4, sleep_async); + let eps = begin.elapsed(); + assert!(eps < std::time::Duration::from_millis(590)); + assert!(eps >= std::time::Duration::from_millis(150)); + assert_eq!(result1.unwrap(), "hello spawn_blocking1!"); + assert_eq!(result2.unwrap(), "hello spawn_blocking2!"); + assert_eq!(result3.unwrap(), "hello spawn_blocking3!"); + assert_eq!(result4.unwrap(), "hello spawn_blocking4!"); + }); + } +} diff --git a/vendor/monoio/src/buf/io_buf.rs b/vendor/monoio/src/buf/io_buf.rs new file mode 100644 index 000000000..8ba914757 --- /dev/null +++ b/vendor/monoio/src/buf/io_buf.rs @@ -0,0 +1,519 @@ +use std::{ops, rc::Rc, sync::Arc}; + +use super::Slice; +use crate::buf::slice::SliceMut; + +/// An `io_uring` compatible buffer. +/// +/// The `IoBuf` trait is implemented by buffer types that can be passed to +/// io_uring operations. Users will not need to use this trait directly, except +/// for the [`slice`] method. +/// +/// # Slicing +/// +/// Because buffers are passed by ownership to the runtime, Rust's slice API +/// (`&buf[..]`) cannot be used. Instead, `monoio` provides an owned slice +/// API: [`slice()`]. The method takes ownership of the buffer and returns a +/// `Slice` type that tracks the requested offset. +/// +/// [`slice()`]: IoBuf::slice +/// # Safety +/// impl it safely +pub unsafe trait IoBuf: Unpin + 'static { + /// Returns a raw pointer to the vector's buffer. + /// + /// This method is to be used by the `monoio` runtime and it is not + /// expected for users to call it directly. + /// + /// `monoio` Runtime will `Box::pin` the buffer. Runtime makes sure + /// the buffer will not be moved, and the implement must ensure + /// `as_ptr` returns the same valid address. + /// Kernel will read `bytes_init`-length data from the pointer. + fn read_ptr(&self) -> *const u8; + + /// Number of initialized bytes. + /// + /// This method is to be used by the `monoio` runtime and it is not + /// expected for users to call it directly. + /// + /// For `Vec`, this is identical to `len()`. + fn bytes_init(&self) -> usize; + + /// Returns a view of the buffer with the specified range. + #[inline] + fn slice(self, range: impl ops::RangeBounds) -> Slice + where + Self: Sized, + { + let (begin, end) = parse_range(range, self.bytes_init()); + Slice::new(self, begin, end) + } + + /// Returns a view of the buffer with the specified range without boundary + /// checking. + /// + /// # Safety + /// Range must be within the bounds of the buffer. + #[inline] + unsafe fn slice_unchecked(self, range: impl ops::RangeBounds) -> Slice + where + Self: Sized, + { + let (begin, end) = parse_range(range, self.bytes_init()); + Slice::new_unchecked(self, begin, end) + } +} + +unsafe impl IoBuf for Vec { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + self.len() + } +} + +unsafe impl IoBuf for Box<[u8]> { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + self.len() + } +} + +unsafe impl IoBuf for &'static [u8] { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + <[u8]>::len(self) + } +} + +unsafe impl IoBuf for Box<[u8; N]> { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + self.len() + } +} + +unsafe impl IoBuf for &'static [u8; N] { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + self.len() + } +} + +unsafe impl IoBuf for &'static mut [u8; N] { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + self.len() + } +} + +unsafe impl IoBuf for &'static str { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + ::len(self) + } +} + +#[cfg(feature = "bytes")] +unsafe impl IoBuf for bytes::Bytes { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + self.len() + } +} + +#[cfg(feature = "bytes")] +unsafe impl IoBuf for bytes::BytesMut { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + self.len() + } +} + +unsafe impl IoBuf for Rc +where + T: IoBuf, +{ + #[inline] + fn read_ptr(&self) -> *const u8 { + ::read_ptr(self) + } + + #[inline] + fn bytes_init(&self) -> usize { + ::bytes_init(self) + } +} + +unsafe impl IoBuf for Arc +where + T: IoBuf, +{ + #[inline] + fn read_ptr(&self) -> *const u8 { + ::read_ptr(self) + } + + #[inline] + fn bytes_init(&self) -> usize { + ::bytes_init(self) + } +} + +/// A mutable `io_uring` compatible buffer. +/// +/// The `IoBufMut` trait is implemented by buffer types that can be passed to +/// io_uring operations. Users will not need to use this trait directly. +/// +/// # Safety +/// See the safety note of the methods. +pub unsafe trait IoBufMut: Unpin + 'static { + /// Returns a raw mutable pointer to the vector's buffer. + /// + /// `monoio` Runtime will `Box::pin` the buffer. Runtime makes sure + /// the buffer will not be moved, and the implement must ensure + /// `as_ptr` returns the same valid address. + /// Kernel will write `bytes_init`-length data to the pointer. + fn write_ptr(&mut self) -> *mut u8; + + /// Total size of the buffer, including uninitialized memory, if any. + /// + /// This method is to be used by the `monoio` runtime and it is not + /// expected for users to call it directly. + /// + /// For `Vec`, this is identical to `capacity()`. + fn bytes_total(&mut self) -> usize; + + /// Updates the number of initialized bytes. + /// + /// The specified `pos` becomes the new value returned by + /// `IoBuf::bytes_init`. + /// + /// # Safety + /// + /// The caller must ensure that all bytes starting at `stable_mut_ptr()` up + /// to `pos` are initialized and owned by the buffer. + unsafe fn set_init(&mut self, pos: usize); + + /// Returns a view of the buffer with the specified range. + /// + /// This method is similar to Rust's slicing (`&buf[..]`), but takes + /// ownership of the buffer. + /// + /// # Examples + /// + /// ``` + /// use monoio::buf::{IoBuf, IoBufMut}; + /// + /// let buf = b"hello world".to_vec(); + /// buf.slice(5..10); + /// ``` + #[inline] + fn slice_mut(mut self, range: impl ops::RangeBounds) -> SliceMut + where + Self: Sized, + Self: IoBuf, + { + let (begin, end) = parse_range(range, self.bytes_total()); + SliceMut::new(self, begin, end) + } + + /// Returns a view of the buffer with the specified range. + /// + /// # Safety + /// Begin must within the initialized bytes, end must be within the + /// capacity. + #[inline] + unsafe fn slice_mut_unchecked(mut self, range: impl ops::RangeBounds) -> SliceMut + where + Self: Sized, + { + let (begin, end) = parse_range(range, self.bytes_total()); + SliceMut::new_unchecked(self, begin, end) + } +} + +unsafe impl IoBufMut for Vec { + #[inline] + fn write_ptr(&mut self) -> *mut u8 { + self.as_mut_ptr() + } + + #[inline] + fn bytes_total(&mut self) -> usize { + self.capacity() + } + + #[inline] + unsafe fn set_init(&mut self, init_len: usize) { + self.set_len(init_len); + } +} + +unsafe impl IoBufMut for Box<[u8]> { + #[inline] + fn write_ptr(&mut self) -> *mut u8 { + self.as_mut_ptr() + } + + #[inline] + fn bytes_total(&mut self) -> usize { + self.len() + } + + #[inline] + unsafe fn set_init(&mut self, _: usize) {} +} + +unsafe impl IoBufMut for Box<[u8; N]> { + #[inline] + fn write_ptr(&mut self) -> *mut u8 { + self.as_mut_ptr() + } + + #[inline] + fn bytes_total(&mut self) -> usize { + self.len() + } + + #[inline] + unsafe fn set_init(&mut self, _: usize) {} +} + +unsafe impl IoBufMut for &'static mut [u8; N] { + #[inline] + fn write_ptr(&mut self) -> *mut u8 { + self.as_mut_ptr() + } + + #[inline] + fn bytes_total(&mut self) -> usize { + self.len() + } + + #[inline] + unsafe fn set_init(&mut self, _: usize) {} +} + +#[cfg(feature = "bytes")] +unsafe impl IoBufMut for bytes::BytesMut { + #[inline] + fn write_ptr(&mut self) -> *mut u8 { + self.as_mut_ptr() + } + + #[inline] + fn bytes_total(&mut self) -> usize { + self.capacity() + } + + #[inline] + unsafe fn set_init(&mut self, init_len: usize) { + if self.len() < init_len { + self.set_len(init_len); + } + } +} + +fn parse_range(range: impl ops::RangeBounds, end: usize) -> (usize, usize) { + use core::ops::Bound; + + let begin = match range.start_bound() { + Bound::Included(&n) => n, + Bound::Excluded(&n) => n + 1, + Bound::Unbounded => 0, + }; + + let end = match range.end_bound() { + Bound::Included(&n) => n.checked_add(1).expect("out of range"), + Bound::Excluded(&n) => n, + Bound::Unbounded => end, + }; + (begin, end) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn io_buf_vec() { + let mut buf = Vec::with_capacity(10); + buf.extend_from_slice(b"0123"); + let ptr = buf.as_mut_ptr(); + + assert_eq!(buf.read_ptr(), ptr); + assert_eq!(buf.bytes_init(), 4); + + assert_eq!(buf.write_ptr(), ptr); + assert_eq!(buf.bytes_total(), 10); + + unsafe { buf.set_init(8) }; + assert_eq!(buf.bytes_init(), 8); + assert_eq!(buf.len(), 8); + } + + #[test] + fn io_buf_str() { + let s = "hello world"; + let ptr = s.as_ptr(); + + assert_eq!(s.read_ptr(), ptr); + assert_eq!(s.bytes_init(), 11); + } + + #[test] + fn io_buf_n() { + let mut buf = Box::new([1, 2, 3, 4, 5]); + let ptr = buf.as_mut_ptr(); + + assert_eq!(buf.read_ptr(), ptr); + assert_eq!(buf.bytes_init(), 5); + assert_eq!(buf.write_ptr(), ptr); + assert_eq!(buf.bytes_total(), 5); + } + + #[test] + fn io_buf_n_boxed() { + let mut buf = Box::new([1, 2, 3, 4, 5]); + let ptr = buf.as_mut_ptr(); + + assert_eq!(buf.read_ptr(), ptr); + assert_eq!(buf.bytes_init(), 5); + assert_eq!(buf.write_ptr(), ptr); + assert_eq!(buf.bytes_total(), 5); + } + + #[test] + fn io_buf_n_static() { + let buf = &*Box::leak(Box::new([1, 2, 3, 4, 5])); + let ptr = buf.as_ptr(); + + assert_eq!(buf.read_ptr(), ptr); + assert_eq!(buf.bytes_init(), 5); + } + + #[test] + fn io_buf_n_mut_static() { + let mut buf = Box::leak(Box::new([1, 2, 3, 4, 5])); + let ptr = buf.as_mut_ptr(); + + assert_eq!(buf.read_ptr(), ptr); + assert_eq!(buf.bytes_init(), 5); + assert_eq!(buf.write_ptr(), ptr); + assert_eq!(buf.bytes_total(), 5); + } + + #[test] + fn io_buf_rc_str() { + let s = Rc::new("hello world"); + let ptr = s.as_ptr(); + + assert_eq!(s.read_ptr(), ptr); + assert_eq!(s.bytes_init(), 11); + } + + #[test] + fn io_buf_arc_str() { + let s = Arc::new("hello world"); + let ptr = s.as_ptr(); + + assert_eq!(s.read_ptr(), ptr); + assert_eq!(s.bytes_init(), 11); + } + + #[test] + fn io_buf_slice_ref() { + let s: &[u8] = &[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]; + let ptr = s.as_ptr(); + + assert_eq!(s.read_ptr(), ptr); + assert_eq!(s.bytes_init(), 10); + } + + #[test] + fn io_buf_slice() { + let mut buf = Vec::with_capacity(10); + buf.extend_from_slice(b"0123"); + let ptr = buf.as_mut_ptr(); + + let slice = buf.slice(1..3); + assert_eq!((slice.begin(), slice.end()), (1, 3)); + assert_eq!(slice.read_ptr(), unsafe { ptr.add(1) }); + assert_eq!(slice.bytes_init(), 2); + let buf = slice.into_inner(); + + let mut slice = buf.slice_mut(1..8); + assert_eq!((slice.begin(), slice.end()), (1, 8)); + assert_eq!(slice.write_ptr(), unsafe { ptr.add(1) }); + assert_eq!(slice.bytes_total(), 7); + unsafe { slice.set_init(5) }; + assert_eq!(slice.bytes_init(), 5); + assert_eq!(slice.into_inner().len(), 6); + } + + #[test] + fn io_buf_arc_slice() { + let mut buf = Vec::with_capacity(10); + buf.extend_from_slice(b"0123"); + let buf = Arc::new(buf); + let ptr = buf.as_ptr(); + + let slice = buf.slice(1..3); + assert_eq!((slice.begin(), slice.end()), (1, 3)); + assert_eq!(slice.read_ptr(), unsafe { ptr.add(1) }); + assert_eq!(slice.bytes_init(), 2); + let buf = Arc::into_inner(slice.into_inner()).unwrap(); + + let mut slice = buf.slice_mut(1..8); + assert_eq!((slice.begin(), slice.end()), (1, 8)); + assert_eq!(slice.bytes_total(), 7); + unsafe { slice.set_init(5) }; + assert_eq!(slice.bytes_init(), 5); + assert_eq!(slice.into_inner().len(), 6); + } +} diff --git a/vendor/monoio/src/buf/io_vec_buf.rs b/vendor/monoio/src/buf/io_vec_buf.rs new file mode 100644 index 000000000..82a6433b4 --- /dev/null +++ b/vendor/monoio/src/buf/io_vec_buf.rs @@ -0,0 +1,279 @@ +// use super::shared_buf::Shared; + +#[cfg(windows)] +use windows_sys::Win32::Networking::WinSock::WSABUF; + +/// An `io_uring` compatible iovec buffer. +/// +/// # Safety +/// See the safety note of the methods. +#[allow(clippy::unnecessary_safety_doc)] +pub unsafe trait IoVecBuf: Unpin + 'static { + /// Returns a raw pointer to iovec struct. + /// struct iovec { + /// void *iov_base; /* Starting address */ + /// size_t iov_len; /* Number of bytes to transfer */ + /// }; + /// \[iovec1\]\[iovec2\]\[iovec3\]... + /// ^ The pointer + /// + /// # Safety + /// The implementation must ensure that, while the runtime owns the value, + /// the pointer returned by `stable_mut_ptr` **does not** change. + /// Also, the value pointed must be a valid iovec struct. + #[cfg(unix)] + fn read_iovec_ptr(&self) -> *const libc::iovec; + + /// Returns the count of iovec struct behind the pointer. + /// + /// # Safety + /// There must be really that number of iovec here. + #[cfg(unix)] + fn read_iovec_len(&self) -> usize; + + /// Returns a raw pointer to WSABUF struct. + #[cfg(windows)] + fn read_wsabuf_ptr(&self) -> *const WSABUF; + + /// Returns the count of WSABUF struct behind the pointer. + #[cfg(windows)] + fn read_wsabuf_len(&self) -> usize; +} + +/// A intermediate struct that impl IoVecBuf and IoVecBufMut. +#[derive(Clone)] +pub struct VecBuf { + #[cfg(unix)] + iovecs: Vec, + #[cfg(windows)] + wsabufs: Vec, + raw: Vec>, +} + +#[cfg(unix)] +unsafe impl IoVecBuf for VecBuf { + fn read_iovec_ptr(&self) -> *const libc::iovec { + self.iovecs.read_iovec_ptr() + } + fn read_iovec_len(&self) -> usize { + self.iovecs.read_iovec_len() + } +} + +#[cfg(unix)] +unsafe impl IoVecBuf for Vec { + fn read_iovec_ptr(&self) -> *const libc::iovec { + self.as_ptr() + } + + fn read_iovec_len(&self) -> usize { + self.len() + } +} + +#[cfg(windows)] +unsafe impl IoVecBuf for VecBuf { + fn read_wsabuf_ptr(&self) -> *const WSABUF { + self.wsabufs.read_wsabuf_ptr() + } + fn read_wsabuf_len(&self) -> usize { + self.wsabufs.read_wsabuf_len() + } +} + +#[cfg(windows)] +unsafe impl IoVecBuf for Vec { + fn read_wsabuf_ptr(&self) -> *const WSABUF { + self.as_ptr() + } + + fn read_wsabuf_len(&self) -> usize { + self.len() + } +} + +impl From>> for VecBuf { + fn from(vs: Vec>) -> Self { + #[cfg(unix)] + { + let iovecs = vs + .iter() + .map(|v| libc::iovec { + iov_base: v.as_ptr() as _, + iov_len: v.len(), + }) + .collect(); + Self { iovecs, raw: vs } + } + #[cfg(windows)] + { + let wsabufs = vs + .iter() + .map(|v| WSABUF { + buf: v.as_ptr() as _, + len: v.len() as _, + }) + .collect(); + Self { wsabufs, raw: vs } + } + } +} + +impl From for Vec> { + fn from(vb: VecBuf) -> Self { + vb.raw + } +} + +// /// SliceVec impl IoVecBuf and IoVecBufMut. +// pub struct SliceVec { +// iovecs: Vec, +// indices: Vec<(usize, usize)>, +// buf: T, +// } + +// impl SliceVec { +// /// New SliceVec. +// pub fn new(buf: T) -> Self { +// Self { +// iovecs: Default::default(), +// indices: Default::default(), +// buf, +// } +// } + +// /// New SliceVec with given indices. +// pub fn new_with_indices(buf: T, indices: Vec<(usize, usize)>) -> Self { +// Self { +// iovecs: Default::default(), +// indices, +// buf, +// } +// } +// } + +// unsafe impl IoVecBuf for SliceVec +// where +// T: Shared, +// { +// fn stable_iovec_ptr(&self) -> *const libc::iovec { +// self.iovecs.as_ptr() + +// // self.iovecs.clear(); +// // self.iovecs.reserve(self.indices.len()); +// // let base = self.buf.stable_ptr(); +// // for (begin, end) in self.indices.iter() { +// // self.iovecs.push(libc::iovec { +// // iov_base: unsafe { base.add(*begin) as *mut libc::c_void +// }, // iov_len: end - begin, +// // }); +// // } +// // self.iovecs.as_ptr() +// } + +// fn iovec_len(&self) -> usize { +// self.indices.len() +// } +// } + +// impl SliceVec where T: Shared { +// pub fn write_all(&mut self, data: &[u8]) -> Result<(), std::io::Error> { +// unimplemented!() +// } +// } + +/// A mutable `io_uring` compatible iovec buffer. +/// +/// # Safety +/// See the safety note of the methods. +#[allow(clippy::unnecessary_safety_doc)] +pub unsafe trait IoVecBufMut: Unpin + 'static { + /// Returns a raw mutable pointer to iovec struct. + /// struct iovec { + /// void *iov_base; /* Starting address */ + /// size_t iov_len; /* Number of bytes to transfer */ + /// }; + /// \[iovec1\]\[iovec2\]\[iovec3\]... + /// ^ The pointer + /// + /// # Safety + /// The implementation must ensure that, while the runtime owns the value, + /// the pointer returned by `write_iovec_ptr` **does not** change. + /// Also, the value pointed must be a valid iovec struct. + #[cfg(unix)] + fn write_iovec_ptr(&mut self) -> *mut libc::iovec; + + /// Returns the count of iovec struct behind the pointer. + #[cfg(unix)] + fn write_iovec_len(&mut self) -> usize; + + /// Returns a raw mutable pointer to WSABUF struct. + #[cfg(windows)] + fn write_wsabuf_ptr(&mut self) -> *mut WSABUF; + + /// Returns the count of WSABUF struct behind the pointer. + #[cfg(windows)] + fn write_wsabuf_len(&mut self) -> usize; + + /// Updates the number of initialized bytes. + /// + /// The specified `pos` becomes the new value returned by + /// `IoBuf::bytes_init`. + /// + /// # Safety + /// + /// The caller must ensure that there are really pos data initialized. + unsafe fn set_init(&mut self, pos: usize); +} + +#[cfg(unix)] +unsafe impl IoVecBufMut for VecBuf { + fn write_iovec_ptr(&mut self) -> *mut libc::iovec { + self.read_iovec_ptr() as *mut _ + } + + fn write_iovec_len(&mut self) -> usize { + self.read_iovec_len() + } + + unsafe fn set_init(&mut self, mut len: usize) { + for (idx, iovec) in self.iovecs.iter_mut().enumerate() { + if iovec.iov_len <= len { + // set_init all + self.raw[idx].set_len(iovec.iov_len); + len -= iovec.iov_len; + } else { + if len > 0 { + self.raw[idx].set_len(len); + } + break; + } + } + } +} + +#[cfg(windows)] +unsafe impl IoVecBufMut for VecBuf { + fn write_wsabuf_ptr(&mut self) -> *mut WSABUF { + self.read_wsabuf_ptr() as *mut _ + } + + fn write_wsabuf_len(&mut self) -> usize { + self.read_wsabuf_len() + } + + unsafe fn set_init(&mut self, mut len: usize) { + for (idx, wsabuf) in self.wsabufs.iter_mut().enumerate() { + if wsabuf.len as usize <= len { + // set_init all + self.raw[idx].set_len(wsabuf.len as _); + len -= wsabuf.len as usize; + } else { + if len > 0 { + self.raw[idx].set_len(len); + } + break; + } + } + } +} diff --git a/vendor/monoio/src/buf/mod.rs b/vendor/monoio/src/buf/mod.rs new file mode 100644 index 000000000..20c54bce8 --- /dev/null +++ b/vendor/monoio/src/buf/mod.rs @@ -0,0 +1,31 @@ +//! Utilities for working with buffers. +//! +//! `io_uring` APIs require passing ownership of buffers to the runtime. The +//! crate defines [`IoBuf`] and [`IoBufMut`] traits which are implemented by +//! buffer types that respect the `io_uring` contract. +// Heavily borrowed from tokio-uring. +// Copyright (c) 2021 Tokio-uring Contributors, licensed under the MIT license. + +mod io_buf; +pub use io_buf::{IoBuf, IoBufMut}; + +mod io_vec_buf; +pub use io_vec_buf::{IoVecBuf, IoVecBufMut, VecBuf}; + +mod slice; +pub use slice::{IoVecWrapper, IoVecWrapperMut, Slice, SliceMut}; + +mod raw_buf; +pub use raw_buf::{RawBuf, RawBufVectored}; + +mod vec_wrapper; +pub(crate) use vec_wrapper::{read_vec_meta, write_vec_meta, IoVecMeta}; + +mod msg; +pub use msg::{MsgBuf, MsgBufMut, MsgMeta}; + +pub(crate) fn deref(buf: &impl IoBuf) -> &[u8] { + // Safety: the `IoBuf` trait is marked as unsafe and is expected to be + // implemented correctly. + unsafe { std::slice::from_raw_parts(buf.read_ptr(), buf.bytes_init()) } +} diff --git a/vendor/monoio/src/buf/msg.rs b/vendor/monoio/src/buf/msg.rs new file mode 100644 index 000000000..d95ad3c47 --- /dev/null +++ b/vendor/monoio/src/buf/msg.rs @@ -0,0 +1,124 @@ +use std::ops::{Deref, DerefMut}; + +#[cfg(unix)] +use libc::msghdr; +#[cfg(windows)] +use windows_sys::Win32::Networking::WinSock::WSAMSG; + +/// An `io_uring` compatible msg buffer. +/// +/// # Safety +/// See the safety note of the methods. +#[allow(clippy::unnecessary_safety_doc)] +pub unsafe trait MsgBuf: Unpin + 'static { + /// Returns a raw pointer to msghdr struct. + /// + /// # Safety + /// The implementation must ensure that, while the runtime owns the value, + /// the pointer returned by `stable_mut_ptr` **does not** change. + /// Also, the value pointed must be a valid msghdr struct. + #[cfg(unix)] + fn read_msghdr_ptr(&self) -> *const msghdr; + + /// Returns a raw pointer to WSAMSG struct. + #[cfg(windows)] + fn read_wsamsg_ptr(&self) -> *const WSAMSG; +} + +/// An `io_uring` compatible msg buffer. +/// +/// # Safety +/// See the safety note of the methods. +#[allow(clippy::unnecessary_safety_doc)] +pub unsafe trait MsgBufMut: Unpin + 'static { + /// Returns a raw pointer to msghdr struct. + /// + /// # Safety + /// The implementation must ensure that, while the runtime owns the value, + /// the pointer returned by `stable_mut_ptr` **does not** change. + /// Also, the value pointed must be a valid msghdr struct. + #[cfg(unix)] + fn write_msghdr_ptr(&mut self) -> *mut msghdr; + + /// Returns a raw pointer to WSAMSG struct. + #[cfg(windows)] + fn write_wsamsg_ptr(&mut self) -> *mut WSAMSG; +} + +#[allow(missing_docs)] +pub struct MsgMeta { + #[cfg(unix)] + pub(crate) data: msghdr, + #[cfg(windows)] + pub(crate) data: WSAMSG, +} + +unsafe impl MsgBuf for MsgMeta { + #[cfg(unix)] + fn read_msghdr_ptr(&self) -> *const msghdr { + &self.data + } + + #[cfg(windows)] + fn read_wsamsg_ptr(&self) -> *const WSAMSG { + &self.data + } +} + +unsafe impl MsgBufMut for MsgMeta { + #[cfg(unix)] + fn write_msghdr_ptr(&mut self) -> *mut msghdr { + &mut self.data + } + + #[cfg(windows)] + fn write_wsamsg_ptr(&mut self) -> *mut WSAMSG { + &mut self.data + } +} + +#[cfg(unix)] +impl From for MsgMeta { + fn from(data: msghdr) -> Self { + Self { data } + } +} + +#[cfg(unix)] +impl Deref for MsgMeta { + type Target = msghdr; + + fn deref(&self) -> &Self::Target { + &self.data + } +} + +#[cfg(unix)] +impl DerefMut for MsgMeta { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.data + } +} + +#[cfg(windows)] +impl From for MsgMeta { + fn from(data: WSAMSG) -> Self { + Self { data } + } +} + +#[cfg(windows)] +impl Deref for MsgMeta { + type Target = WSAMSG; + + fn deref(&self) -> &Self::Target { + &self.data + } +} + +#[cfg(windows)] +impl DerefMut for MsgMeta { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.data + } +} diff --git a/vendor/monoio/src/buf/raw_buf.rs b/vendor/monoio/src/buf/raw_buf.rs new file mode 100644 index 000000000..ad64bc41c --- /dev/null +++ b/vendor/monoio/src/buf/raw_buf.rs @@ -0,0 +1,196 @@ +use std::ptr::null; + +#[cfg(windows)] +use windows_sys::Win32::Networking::WinSock::WSABUF; + +use super::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut}; + +/// RawBuf is not a real buf. It only hold the pointer of the buffer. +/// Users must make sure the buffer behind the pointer is always valid. +/// Which means, user must: +/// 1. await the future with RawBuf Ready before drop the real buffer +/// 2. make sure the pointer and length is valid before the future Ready +pub struct RawBuf { + ptr: *const u8, + len: usize, +} + +impl RawBuf { + /// Create a empty RawBuf. + /// # Safety + /// do not use uninitialized RawBuf directly. + #[inline] + pub unsafe fn uninit() -> Self { + Self { + ptr: null(), + len: 0, + } + } + + /// Create a new RawBuf with given pointer and length. + /// # Safety + /// make sure the pointer and length is valid when RawBuf is used. + #[inline] + pub const unsafe fn new(ptr: *const u8, len: usize) -> Self { + Self { ptr, len } + } +} + +unsafe impl IoBuf for RawBuf { + #[inline] + fn read_ptr(&self) -> *const u8 { + self.ptr + } + + #[inline] + fn bytes_init(&self) -> usize { + self.len + } +} + +unsafe impl IoBufMut for RawBuf { + #[inline] + fn write_ptr(&mut self) -> *mut u8 { + self.ptr as *mut u8 + } + + #[inline] + fn bytes_total(&mut self) -> usize { + self.len + } + + #[inline] + unsafe fn set_init(&mut self, _pos: usize) {} +} + +impl RawBuf { + /// Create a new RawBuf with the first iovec part. + /// # Safety + /// make sure the pointer and length is valid when RawBuf is used. + #[inline] + pub unsafe fn new_from_iovec_mut(data: &mut T) -> Option { + #[cfg(unix)] + { + if data.write_iovec_len() == 0 { + return None; + } + let iovec = *data.write_iovec_ptr(); + Some(Self::new(iovec.iov_base as *const u8, iovec.iov_len)) + } + #[cfg(windows)] + { + if data.write_wsabuf_len() == 0 { + return None; + } + let wsabuf = *data.write_wsabuf_ptr(); + Some(Self::new(wsabuf.buf as *const u8, wsabuf.len as _)) + } + } + + /// Create a new RawBuf with the first iovec part. + /// # Safety + /// make sure the pointer and length is valid when RawBuf is used. + #[inline] + pub unsafe fn new_from_iovec(data: &T) -> Option { + #[cfg(unix)] + { + if data.read_iovec_len() == 0 { + return None; + } + let iovec = *data.read_iovec_ptr(); + Some(Self::new(iovec.iov_base as *const u8, iovec.iov_len)) + } + #[cfg(windows)] + { + if data.read_wsabuf_len() == 0 { + return None; + } + let wsabuf = *data.read_wsabuf_ptr(); + Some(Self::new(wsabuf.buf as *const u8, wsabuf.len as _)) + } + } +} + +/// RawBufVectored behaves like RawBuf. +/// And user must obey the following restrictions: +/// 1. await the future with RawBuf Ready before drop the real buffer +/// 2. make sure the pointer and length is valid before the future Ready +pub struct RawBufVectored { + #[cfg(unix)] + ptr: *const libc::iovec, + #[cfg(windows)] + ptr: *const WSABUF, + len: usize, +} + +impl RawBufVectored { + /// Create a new RawBuf with given pointer and length. + /// # Safety + /// make sure the pointer and length is valid when RawBuf is used. + #[cfg(unix)] + #[inline] + pub const unsafe fn new(ptr: *const libc::iovec, len: usize) -> Self { + Self { ptr, len } + } + + /// Create a new RawBuf with given pointer and length. + /// # Safety + /// make sure the pointer and length is valid when RawBuf is used. + #[cfg(windows)] + #[inline] + pub const unsafe fn new(ptr: *const WSABUF, len: usize) -> Self { + Self { ptr, len } + } +} + +unsafe impl IoVecBuf for RawBufVectored { + #[cfg(unix)] + #[inline] + fn read_iovec_ptr(&self) -> *const libc::iovec { + self.ptr + } + + #[cfg(unix)] + #[inline] + fn read_iovec_len(&self) -> usize { + self.len + } + + #[cfg(windows)] + #[inline] + fn read_wsabuf_ptr(&self) -> *const WSABUF { + self.ptr + } + + #[cfg(windows)] + #[inline] + fn read_wsabuf_len(&self) -> usize { + self.len + } +} + +unsafe impl IoVecBufMut for RawBufVectored { + #[cfg(unix)] + fn write_iovec_ptr(&mut self) -> *mut libc::iovec { + self.ptr as *mut libc::iovec + } + + #[cfg(unix)] + fn write_iovec_len(&mut self) -> usize { + self.len + } + + #[cfg(windows)] + #[inline] + fn write_wsabuf_ptr(&mut self) -> *mut WSABUF { + self.ptr as *mut WSABUF + } + + #[cfg(windows)] + #[inline] + fn write_wsabuf_len(&mut self) -> usize { + self.len + } + + unsafe fn set_init(&mut self, _pos: usize) {} +} diff --git a/vendor/monoio/src/buf/slice.rs b/vendor/monoio/src/buf/slice.rs new file mode 100644 index 000000000..05dfc4437 --- /dev/null +++ b/vendor/monoio/src/buf/slice.rs @@ -0,0 +1,374 @@ +use std::ops; + +use super::{IoVecBuf, IoVecBufMut}; +use crate::buf::{IoBuf, IoBufMut}; + +/// An owned view into a contiguous sequence of bytes. +/// SliceMut implements IoBuf and IoBufMut. +/// +/// This is similar to Rust slices (`&buf[..]`) but owns the underlying buffer. +/// This type is useful for performing io_uring read and write operations using +/// a subset of a buffer. +/// +/// Slices are created using [`IoBuf::slice`]. +/// +/// # Examples +/// +/// Creating a slice +/// +/// ``` +/// use monoio::buf::{IoBuf, IoBufMut}; +/// +/// let buf = b"hello world".to_vec(); +/// let slice = buf.slice_mut(..5); +/// +/// assert_eq!(&slice[..], b"hello"); +/// ``` +pub struct SliceMut { + buf: T, + begin: usize, + end: usize, +} + +impl SliceMut { + /// Create a SliceMut from a buffer and range. + #[inline] + pub fn new(mut buf: T, begin: usize, end: usize) -> Self { + assert!(end <= buf.bytes_total()); + assert!(begin <= buf.bytes_init()); + assert!(begin <= end); + Self { buf, begin, end } + } +} + +impl SliceMut { + /// Create a SliceMut from a buffer and range without boundary checking. + /// + /// # Safety + /// begin must be initialized, and end must be within the buffer capacity. + #[inline] + pub const unsafe fn new_unchecked(buf: T, begin: usize, end: usize) -> Self { + Self { buf, begin, end } + } + + /// Offset in the underlying buffer at which this slice starts. + /// + /// # Examples + /// + /// ``` + /// use monoio::buf::IoBuf; + /// + /// let buf = b"hello world".to_vec(); + /// let slice = buf.slice(1..5); + /// + /// assert_eq!(1, slice.begin()); + /// ``` + #[inline] + pub const fn begin(&self) -> usize { + self.begin + } + + /// Offset in the underlying buffer at which this slice ends. + /// + /// # Examples + /// + /// ``` + /// use monoio::buf::IoBuf; + /// + /// let buf = b"hello world".to_vec(); + /// let slice = buf.slice(1..5); + /// + /// assert_eq!(5, slice.end()); + /// ``` + #[inline] + pub const fn end(&self) -> usize { + self.end + } + + /// Gets a reference to the underlying buffer. + /// + /// This method escapes the slice's view. + /// + /// # Examples + /// + /// ``` + /// use monoio::buf::{IoBuf, IoBufMut}; + /// + /// let buf = b"hello world".to_vec(); + /// let slice = buf.slice_mut(..5); + /// + /// assert_eq!(slice.get_ref(), b"hello world"); + /// assert_eq!(&slice[..], b"hello"); + /// ``` + #[inline] + pub const fn get_ref(&self) -> &T { + &self.buf + } + + /// Gets a mutable reference to the underlying buffer. + /// + /// This method escapes the slice's view. + /// + /// # Examples + /// + /// ``` + /// use monoio::buf::{IoBuf, IoBufMut}; + /// + /// let buf = b"hello world".to_vec(); + /// let mut slice = buf.slice_mut(..5); + /// + /// slice.get_mut()[0] = b'b'; + /// + /// assert_eq!(slice.get_mut(), b"bello world"); + /// assert_eq!(&slice[..], b"bello"); + /// ``` + #[inline] + pub fn get_mut(&mut self) -> &mut T { + &mut self.buf + } + + /// Unwraps this `Slice`, returning the underlying buffer. + /// + /// # Examples + /// + /// ``` + /// use monoio::buf::IoBuf; + /// + /// let buf = b"hello world".to_vec(); + /// let slice = buf.slice(..5); + /// + /// let buf = slice.into_inner(); + /// assert_eq!(buf, b"hello world"); + /// ``` + #[inline] + pub fn into_inner(self) -> T { + self.buf + } +} + +impl ops::Deref for SliceMut { + type Target = [u8]; + + #[inline] + fn deref(&self) -> &[u8] { + let buf_bytes = super::deref(&self.buf); + let end = std::cmp::min(self.end, buf_bytes.len()); + &buf_bytes[self.begin..end] + } +} + +unsafe impl IoBuf for SliceMut { + #[inline] + fn read_ptr(&self) -> *const u8 { + super::deref(&self.buf)[self.begin..].as_ptr() + } + + #[inline] + fn bytes_init(&self) -> usize { + ops::Deref::deref(self).len() + } +} + +unsafe impl IoBufMut for SliceMut { + #[inline] + fn write_ptr(&mut self) -> *mut u8 { + unsafe { self.buf.write_ptr().add(self.begin) } + } + + #[inline] + fn bytes_total(&mut self) -> usize { + self.end - self.begin + } + + #[inline] + unsafe fn set_init(&mut self, n: usize) { + self.buf.set_init(self.begin + n); + } +} + +/// An owned view into a contiguous sequence of bytes. +/// Slice implements IoBuf. +pub struct Slice { + buf: T, + begin: usize, + end: usize, +} + +impl Slice { + /// Create a Slice from a buffer and range. + #[inline] + pub fn new(buf: T, begin: usize, end: usize) -> Self { + assert!(end <= buf.bytes_init()); + assert!(begin <= end); + Self { buf, begin, end } + } +} + +impl Slice { + /// Create a Slice from a buffer and range without boundary checking. + /// + /// # Safety + /// begin and end must be within the buffer initialized range. + #[inline] + pub const unsafe fn new_unchecked(buf: T, begin: usize, end: usize) -> Self { + Self { buf, begin, end } + } + + /// Offset in the underlying buffer at which this slice starts. + #[inline] + pub const fn begin(&self) -> usize { + self.begin + } + + /// Ofset in the underlying buffer at which this slice ends. + #[inline] + pub const fn end(&self) -> usize { + self.end + } + + /// Gets a reference to the underlying buffer. + #[inline] + pub const fn get_ref(&self) -> &T { + &self.buf + } + + /// Gets a mutable reference to the underlying buffer. + #[inline] + pub fn get_mut(&mut self) -> &mut T { + &mut self.buf + } + + /// Unwraps this `Slice`, returning the underlying buffer. + #[inline] + pub fn into_inner(self) -> T { + self.buf + } +} + +unsafe impl IoBuf for Slice { + #[inline] + fn read_ptr(&self) -> *const u8 { + unsafe { self.buf.read_ptr().add(self.begin) } + } + + #[inline] + fn bytes_init(&self) -> usize { + self.end - self.begin + } +} + +/// A wrapper to make IoVecBuf impl IoBuf. +pub struct IoVecWrapper { + // we must make sure raw contains at least one iovec. + raw: T, +} + +impl IoVecWrapper { + /// Create a new IoVecWrapper with something that impl IoVecBuf. + #[inline] + pub fn new(buf: T) -> Result { + #[cfg(unix)] + if buf.read_iovec_len() == 0 { + return Err(buf); + } + #[cfg(windows)] + if buf.read_wsabuf_len() == 0 { + return Err(buf); + } + Ok(Self { raw: buf }) + } + + /// Consume self and return raw iovec buf. + #[inline] + pub fn into_inner(self) -> T { + self.raw + } +} + +unsafe impl IoBuf for IoVecWrapper { + #[inline] + fn read_ptr(&self) -> *const u8 { + #[cfg(unix)] + { + let iovec = unsafe { *self.raw.read_iovec_ptr() }; + iovec.iov_base as *const u8 + } + #[cfg(windows)] + { + let wsabuf = unsafe { *self.raw.read_wsabuf_ptr() }; + wsabuf.buf as *const u8 + } + } + + #[inline] + fn bytes_init(&self) -> usize { + #[cfg(unix)] + { + let iovec = unsafe { *self.raw.read_iovec_ptr() }; + iovec.iov_len + } + #[cfg(windows)] + { + let wsabuf = unsafe { *self.raw.read_wsabuf_ptr() }; + wsabuf.len as _ + } + } +} + +/// A wrapper to make IoVecBufMut impl IoBufMut. +pub struct IoVecWrapperMut { + // we must make sure raw contains at least one iovec. + raw: T, +} + +impl IoVecWrapperMut { + /// Create a new IoVecWrapperMut with something that impl IoVecBufMut. + #[inline] + pub fn new(mut iovec_buf: T) -> Result { + #[cfg(unix)] + if iovec_buf.write_iovec_len() == 0 { + return Err(iovec_buf); + } + #[cfg(windows)] + if iovec_buf.write_wsabuf_len() == 0 { + return Err(iovec_buf); + } + Ok(Self { raw: iovec_buf }) + } + + /// Consume self and return raw iovec buf. + #[inline] + pub fn into_inner(self) -> T { + self.raw + } +} + +unsafe impl IoBufMut for IoVecWrapperMut { + fn write_ptr(&mut self) -> *mut u8 { + #[cfg(unix)] + { + let iovec = unsafe { *self.raw.write_iovec_ptr() }; + iovec.iov_base as *mut u8 + } + #[cfg(windows)] + { + let wsabuf = unsafe { *self.raw.write_wsabuf_ptr() }; + wsabuf.buf + } + } + + fn bytes_total(&mut self) -> usize { + #[cfg(unix)] + { + let iovec = unsafe { *self.raw.write_iovec_ptr() }; + iovec.iov_len + } + #[cfg(windows)] + { + let wsabuf = unsafe { *self.raw.write_wsabuf_ptr() }; + wsabuf.len as _ + } + } + + unsafe fn set_init(&mut self, _pos: usize) {} +} diff --git a/vendor/monoio/src/buf/vec_wrapper.rs b/vendor/monoio/src/buf/vec_wrapper.rs new file mode 100644 index 000000000..01feee869 --- /dev/null +++ b/vendor/monoio/src/buf/vec_wrapper.rs @@ -0,0 +1,291 @@ +#[cfg(windows)] +use {std::ops::Add, windows_sys::Win32::Networking::WinSock::WSABUF}; + +use super::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut}; + +pub(crate) struct IoVecMeta { + #[cfg(unix)] + data: Vec, + #[cfg(windows)] + data: Vec, + offset: usize, + len: usize, +} + +/// Read IoVecBuf meta data into a Vec. +pub(crate) fn read_vec_meta(buf: &T) -> IoVecMeta { + #[cfg(unix)] + { + let ptr = buf.read_iovec_ptr(); + let iovec_len = buf.read_iovec_len(); + + let mut data = Vec::with_capacity(iovec_len); + let mut len = 0; + for i in 0..iovec_len { + let iovec = unsafe { *ptr.add(i) }; + data.push(iovec); + len += iovec.iov_len; + } + IoVecMeta { + data, + offset: 0, + len, + } + } + #[cfg(windows)] + { + let ptr = buf.read_wsabuf_ptr(); + let wsabuf_len = buf.read_wsabuf_len(); + + let mut data = Vec::with_capacity(wsabuf_len); + let mut len = 0; + for i in 0..wsabuf_len { + let wsabuf = unsafe { *ptr.add(i) }; + data.push(wsabuf); + len += wsabuf.len; + } + let len = len as _; + IoVecMeta { + data, + offset: 0, + len, + } + } +} + +/// Read IoVecBufMut meta data into a Vec. +pub(crate) fn write_vec_meta(buf: &mut T) -> IoVecMeta { + #[cfg(unix)] + { + let ptr = buf.write_iovec_ptr(); + let iovec_len = buf.write_iovec_len(); + + let mut data = Vec::with_capacity(iovec_len); + let mut len = 0; + for i in 0..iovec_len { + let iovec = unsafe { *ptr.add(i) }; + data.push(iovec); + len += iovec.iov_len; + } + IoVecMeta { + data, + offset: 0, + len, + } + } + #[cfg(windows)] + { + let ptr = buf.write_wsabuf_ptr(); + let wsabuf_len = buf.write_wsabuf_len(); + + let mut data = Vec::with_capacity(wsabuf_len); + let mut len = 0; + for i in 0..wsabuf_len { + let wsabuf = unsafe { *ptr.add(i) }; + data.push(wsabuf); + len += wsabuf.len; + } + let len = len as _; + IoVecMeta { + data, + offset: 0, + len, + } + } +} + +impl IoVecMeta { + #[allow(unused_mut)] + pub(crate) fn consume(&mut self, mut amt: usize) { + #[cfg(unix)] + { + if amt == 0 { + return; + } + let mut offset = self.offset; + while let Some(iovec) = self.data.get_mut(offset) { + match iovec.iov_len.cmp(&amt) { + std::cmp::Ordering::Less => { + amt -= iovec.iov_len; + offset += 1; + continue; + } + std::cmp::Ordering::Equal => { + offset += 1; + self.offset = offset; + return; + } + std::cmp::Ordering::Greater => { + let _ = unsafe { iovec.iov_base.add(amt) }; + iovec.iov_len -= amt; + self.offset = offset; + return; + } + } + } + panic!("try to consume more than owned") + } + #[cfg(windows)] + { + let mut amt = amt as _; + if amt == 0 { + return; + } + let mut offset = self.offset; + while let Some(wsabuf) = self.data.get_mut(offset) { + match wsabuf.len.cmp(&amt) { + std::cmp::Ordering::Less => { + amt -= wsabuf.len; + offset += 1; + continue; + } + std::cmp::Ordering::Equal => { + offset += 1; + self.offset = offset; + return; + } + std::cmp::Ordering::Greater => { + _ = wsabuf.len.add(amt); + wsabuf.len -= amt; + self.offset = offset; + return; + } + } + } + panic!("try to consume more than owned") + } + } + + pub(crate) fn len(&self) -> usize { + self.len + } +} + +unsafe impl IoVecBuf for IoVecMeta { + #[cfg(unix)] + fn read_iovec_ptr(&self) -> *const libc::iovec { + unsafe { self.data.as_ptr().add(self.offset) } + } + #[cfg(unix)] + fn read_iovec_len(&self) -> usize { + self.data.len() + } + #[cfg(windows)] + fn read_wsabuf_ptr(&self) -> *const WSABUF { + unsafe { self.data.as_ptr().add(self.offset) } + } + #[cfg(windows)] + fn read_wsabuf_len(&self) -> usize { + self.data.len() + } +} + +unsafe impl IoVecBufMut for IoVecMeta { + #[cfg(unix)] + fn write_iovec_ptr(&mut self) -> *mut libc::iovec { + unsafe { self.data.as_mut_ptr().add(self.offset) } + } + + #[cfg(unix)] + fn write_iovec_len(&mut self) -> usize { + self.data.len() + } + + #[cfg(windows)] + fn write_wsabuf_ptr(&mut self) -> *mut WSABUF { + unsafe { self.data.as_mut_ptr().add(self.offset) } + } + + #[cfg(windows)] + fn write_wsabuf_len(&mut self) -> usize { + self.data.len() + } + + unsafe fn set_init(&mut self, pos: usize) { + self.consume(pos) + } +} + +impl<'t, T: IoBuf> From<&'t T> for IoVecMeta { + fn from(buf: &'t T) -> Self { + let ptr = buf.read_ptr() as *const _ as *mut _; + let len = buf.bytes_init() as _; + #[cfg(unix)] + let item = libc::iovec { + iov_base: ptr, + iov_len: len, + }; + #[cfg(windows)] + let item = WSABUF { buf: ptr, len }; + Self { + data: vec![item], + offset: 0, + len: 1, + } + } +} + +impl<'t, T: IoBufMut> From<&'t mut T> for IoVecMeta { + fn from(buf: &'t mut T) -> Self { + let ptr = buf.write_ptr() as *mut _; + let len = buf.bytes_total() as _; + #[cfg(unix)] + let item = libc::iovec { + iov_base: ptr, + iov_len: len, + }; + #[cfg(windows)] + let item = WSABUF { buf: ptr, len }; + Self { + data: vec![item], + offset: 0, + len: 1, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::buf::VecBuf; + + #[test] + fn test_read_vec_meta() { + let iovec = VecBuf::from(vec![vec![0; 10], vec![0; 20], vec![0; 30]]); + let meta = read_vec_meta(&iovec); + assert_eq!(meta.len(), 60); + assert_eq!(meta.data.len(), 3); + #[cfg(unix)] + { + assert_eq!(meta.data[0].iov_len, 10); + assert_eq!(meta.data[1].iov_len, 20); + assert_eq!(meta.data[2].iov_len, 30); + } + #[cfg(windows)] + { + assert_eq!(meta.data[0].len, 10); + assert_eq!(meta.data[1].len, 20); + assert_eq!(meta.data[2].len, 30); + } + } + + #[test] + fn test_write_vec_meta() { + let mut iovec = VecBuf::from(vec![vec![0; 10], vec![0; 20], vec![0; 30]]); + let meta = write_vec_meta(&mut iovec); + assert_eq!(meta.len(), 60); + assert_eq!(meta.data.len(), 3); + #[cfg(unix)] + { + assert_eq!(meta.data[0].iov_len, 10); + assert_eq!(meta.data[1].iov_len, 20); + assert_eq!(meta.data[2].iov_len, 30); + } + #[cfg(windows)] + { + assert_eq!(meta.data[0].len, 10); + assert_eq!(meta.data[1].len, 20); + assert_eq!(meta.data[2].len, 30); + } + } +} diff --git a/vendor/monoio/src/builder.rs b/vendor/monoio/src/builder.rs new file mode 100644 index 000000000..b6a7c9f0a --- /dev/null +++ b/vendor/monoio/src/builder.rs @@ -0,0 +1,368 @@ +use std::{io, marker::PhantomData}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use crate::driver::IoUringDriver; +#[cfg(feature = "legacy")] +use crate::driver::LegacyDriver; +#[cfg(any(feature = "legacy", feature = "iouring"))] +use crate::utils::thread_id::gen_id; +use crate::{ + driver::Driver, + time::{driver::TimeDriver, Clock}, + Runtime, +}; + +// ===== basic builder structure definition ===== + +/// Runtime builder +pub struct RuntimeBuilder { + // iouring entries + entries: Option, + + #[cfg(all(target_os = "linux", feature = "iouring"))] + urb: io_uring::Builder, + + // blocking handle + #[cfg(feature = "sync")] + blocking_handle: crate::blocking::BlockingHandle, + // driver mark + _mark: PhantomData, +} + +scoped_thread_local!(pub(crate) static BUILD_THREAD_ID: usize); + +impl Default for RuntimeBuilder { + /// Create a default runtime builder + #[must_use] + fn default() -> Self { + RuntimeBuilder::::new() + } +} + +impl RuntimeBuilder { + /// Create a default runtime builder + #[must_use] + pub fn new() -> Self { + Self { + entries: None, + + #[cfg(all(target_os = "linux", feature = "iouring"))] + urb: io_uring::IoUring::builder(), + + #[cfg(feature = "sync")] + blocking_handle: crate::blocking::BlockingStrategy::Panic.into(), + _mark: PhantomData, + } + } +} + +// ===== buildable trait and forward methods ===== + +/// Buildable trait. +pub trait Buildable: Sized { + /// Build the runtime. + fn build(this: RuntimeBuilder) -> io::Result>; +} + +#[allow(unused)] +macro_rules! direct_build { + ($ty: ty) => { + impl RuntimeBuilder<$ty> { + /// Build the runtime. + pub fn build(self) -> io::Result> { + Buildable::build(self) + } + } + }; +} + +#[cfg(all(target_os = "linux", feature = "iouring"))] +direct_build!(IoUringDriver); +#[cfg(all(target_os = "linux", feature = "iouring"))] +direct_build!(TimeDriver); +#[cfg(feature = "legacy")] +direct_build!(LegacyDriver); +#[cfg(feature = "legacy")] +direct_build!(TimeDriver); + +// ===== builder impl ===== + +#[cfg(feature = "legacy")] +impl Buildable for LegacyDriver { + fn build(this: RuntimeBuilder) -> io::Result> { + let thread_id = gen_id(); + #[cfg(feature = "sync")] + let blocking_handle = this.blocking_handle; + + BUILD_THREAD_ID.set(&thread_id, || { + let driver = match this.entries { + Some(entries) => LegacyDriver::new_with_entries(entries)?, + None => LegacyDriver::new()?, + }; + #[cfg(feature = "sync")] + let context = crate::runtime::Context::new(blocking_handle); + #[cfg(not(feature = "sync"))] + let context = crate::runtime::Context::new(); + Ok(Runtime::new(context, driver)) + }) + } +} + +#[cfg(all(target_os = "linux", feature = "iouring"))] +impl Buildable for IoUringDriver { + fn build(this: RuntimeBuilder) -> io::Result> { + let thread_id = gen_id(); + #[cfg(feature = "sync")] + let blocking_handle = this.blocking_handle; + + BUILD_THREAD_ID.set(&thread_id, || { + let driver = match this.entries { + Some(entries) => IoUringDriver::new_with_entries(&this.urb, entries)?, + None => IoUringDriver::new(&this.urb)?, + }; + #[cfg(feature = "sync")] + let context = crate::runtime::Context::new(blocking_handle); + #[cfg(not(feature = "sync"))] + let context = crate::runtime::Context::new(); + Ok(Runtime::new(context, driver)) + }) + } +} + +impl RuntimeBuilder { + const MIN_ENTRIES: u32 = 256; + + /// Set io_uring entries, min size is 256 and the default size is 1024. + #[must_use] + pub fn with_entries(mut self, entries: u32) -> Self { + // If entries is less than 256, it will be 256. + if entries < Self::MIN_ENTRIES { + self.entries = Some(Self::MIN_ENTRIES); + return self; + } + self.entries = Some(entries); + self + } + + /// Replaces the default [`io_uring::Builder`], which controls the settings for the + /// inner `io_uring` API. + /// + /// Refer to the [`io_uring::Builder`] documentation for all the supported methods. + + #[cfg(all(target_os = "linux", feature = "iouring"))] + #[must_use] + pub fn uring_builder(mut self, urb: io_uring::Builder) -> Self { + self.urb = urb; + self + } +} + +// ===== FusionDriver ===== + +/// Fake driver only for conditionally building. +#[cfg(any(all(target_os = "linux", feature = "iouring"), feature = "legacy"))] +pub struct FusionDriver; + +#[cfg(any(all(target_os = "linux", feature = "iouring"), feature = "legacy"))] +impl RuntimeBuilder { + /// Build the runtime. + #[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] + pub fn build(self) -> io::Result> { + if crate::utils::detect_uring() { + let builder = RuntimeBuilder:: { + entries: self.entries, + urb: self.urb, + #[cfg(feature = "sync")] + blocking_handle: self.blocking_handle, + _mark: PhantomData, + }; + info!("io_uring driver built"); + Ok(builder.build()?.into()) + } else { + let builder = RuntimeBuilder:: { + entries: self.entries, + urb: self.urb, + #[cfg(feature = "sync")] + blocking_handle: self.blocking_handle, + _mark: PhantomData, + }; + info!("legacy driver built"); + Ok(builder.build()?.into()) + } + } + + /// Build the runtime. + #[cfg(not(all(target_os = "linux", feature = "iouring")))] + pub fn build(self) -> io::Result> { + let builder = RuntimeBuilder:: { + entries: self.entries, + #[cfg(feature = "sync")] + blocking_handle: self.blocking_handle, + _mark: PhantomData, + }; + Ok(builder.build()?.into()) + } + + /// Build the runtime. + #[cfg(all(target_os = "linux", feature = "iouring", not(feature = "legacy")))] + pub fn build(self) -> io::Result> { + let builder = RuntimeBuilder:: { + entries: self.entries, + urb: self.urb, + #[cfg(feature = "sync")] + blocking_handle: self.blocking_handle, + _mark: PhantomData, + }; + Ok(builder.build()?.into()) + } +} + +#[cfg(any(all(target_os = "linux", feature = "iouring"), feature = "legacy"))] +impl RuntimeBuilder> { + /// Build the runtime. + #[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] + pub fn build( + self, + ) -> io::Result, TimeDriver>> { + if crate::utils::detect_uring() { + let builder = RuntimeBuilder::> { + entries: self.entries, + urb: self.urb, + #[cfg(feature = "sync")] + blocking_handle: self.blocking_handle, + _mark: PhantomData, + }; + info!("io_uring driver with timer built"); + Ok(builder.build()?.into()) + } else { + let builder = RuntimeBuilder::> { + entries: self.entries, + urb: self.urb, + #[cfg(feature = "sync")] + blocking_handle: self.blocking_handle, + _mark: PhantomData, + }; + info!("legacy driver with timer built"); + Ok(builder.build()?.into()) + } + } + + /// Build the runtime. + #[cfg(not(all(target_os = "linux", feature = "iouring")))] + pub fn build(self) -> io::Result>> { + let builder = RuntimeBuilder::> { + entries: self.entries, + #[cfg(feature = "sync")] + blocking_handle: self.blocking_handle, + _mark: PhantomData, + }; + Ok(builder.build()?.into()) + } + + /// Build the runtime. + #[cfg(all(target_os = "linux", feature = "iouring", not(feature = "legacy")))] + pub fn build(self) -> io::Result>> { + let builder = RuntimeBuilder::> { + entries: self.entries, + urb: self.urb, + #[cfg(feature = "sync")] + blocking_handle: self.blocking_handle, + _mark: PhantomData, + }; + Ok(builder.build()?.into()) + } +} + +// ===== enable_timer related ===== +mod time_wrap { + pub trait TimeWrapable {} +} + +#[cfg(all(target_os = "linux", feature = "iouring"))] +impl time_wrap::TimeWrapable for IoUringDriver {} +#[cfg(feature = "legacy")] +impl time_wrap::TimeWrapable for LegacyDriver {} +#[cfg(any(all(target_os = "linux", feature = "iouring"), feature = "legacy"))] +impl time_wrap::TimeWrapable for FusionDriver {} + +impl Buildable for TimeDriver +where + D: Buildable, +{ + /// Build the runtime + fn build(this: RuntimeBuilder) -> io::Result>> { + let Runtime { + driver, + mut context, + } = Buildable::build(RuntimeBuilder:: { + entries: this.entries, + #[cfg(all(target_os = "linux", feature = "iouring"))] + urb: this.urb, + #[cfg(feature = "sync")] + blocking_handle: this.blocking_handle, + _mark: PhantomData, + })?; + + let timer_driver = TimeDriver::new(driver, Clock::new()); + context.time_handle = Some(timer_driver.handle.clone()); + Ok(Runtime { + driver: timer_driver, + context, + }) + } +} + +impl RuntimeBuilder { + /// Enable all(currently only timer) + #[must_use] + pub fn enable_all(self) -> RuntimeBuilder> { + self.enable_timer() + } + + /// Enable timer + #[must_use] + pub fn enable_timer(self) -> RuntimeBuilder> { + let Self { + entries, + #[cfg(all(target_os = "linux", feature = "iouring"))] + urb, + #[cfg(feature = "sync")] + blocking_handle, + .. + } = self; + RuntimeBuilder { + entries, + #[cfg(all(target_os = "linux", feature = "iouring"))] + urb, + #[cfg(feature = "sync")] + blocking_handle, + _mark: PhantomData, + } + } +} + +impl RuntimeBuilder { + /// Attach thread pool, this will overwrite blocking strategy. + /// All `spawn_blocking` will be executed on given thread pool. + #[cfg(feature = "sync")] + #[must_use] + pub fn attach_thread_pool( + mut self, + tp: Box, + ) -> Self { + self.blocking_handle = crate::blocking::BlockingHandle::Attached(tp); + self + } + + /// Set blocking strategy, this will overwrite thread pool setting. + /// If `BlockingStrategy::Panic` is used, it will panic if `spawn_blocking` on this thread. + /// If `BlockingStrategy::ExecuteLocal` is used, it will execute with current thread, and may + /// cause tasks high latency. + /// Attaching a thread pool is recommended if `spawn_blocking` will be used. + #[cfg(feature = "sync")] + #[must_use] + pub fn with_blocking_strategy(mut self, strategy: crate::blocking::BlockingStrategy) -> Self { + self.blocking_handle = crate::blocking::BlockingHandle::Empty(strategy); + self + } +} diff --git a/vendor/monoio/src/driver/legacy/iocp/afd.rs b/vendor/monoio/src/driver/legacy/iocp/afd.rs new file mode 100644 index 000000000..05d730562 --- /dev/null +++ b/vendor/monoio/src/driver/legacy/iocp/afd.rs @@ -0,0 +1,201 @@ +use std::{ + ffi::c_void, + fs::File, + os::windows::prelude::{AsRawHandle, FromRawHandle, RawHandle}, + sync::atomic::{AtomicUsize, Ordering}, +}; + +use windows_sys::Win32::{ + Foundation::{ + RtlNtStatusToDosError, HANDLE, INVALID_HANDLE_VALUE, NTSTATUS, STATUS_NOT_FOUND, + STATUS_PENDING, STATUS_SUCCESS, UNICODE_STRING, + }, + Storage::FileSystem::{ + NtCreateFile, SetFileCompletionNotificationModes, FILE_OPEN, FILE_SHARE_READ, + FILE_SHARE_WRITE, SYNCHRONIZE, + }, + System::WindowsProgramming::{ + NtDeviceIoControlFile, FILE_SKIP_SET_EVENT_ON_HANDLE, IO_STATUS_BLOCK, IO_STATUS_BLOCK_0, + OBJECT_ATTRIBUTES, + }, +}; + +use super::CompletionPort; + +#[link(name = "ntdll")] +extern "system" { + /// See + /// + /// This is an undocumented API and as such not part of + /// from which `windows-sys` is generated, and also unlikely to be added, so + /// we manually declare it here + fn NtCancelIoFileEx( + FileHandle: HANDLE, + IoRequestToCancel: *mut IO_STATUS_BLOCK, + IoStatusBlock: *mut IO_STATUS_BLOCK, + ) -> NTSTATUS; +} + +static NEXT_TOKEN: AtomicUsize = AtomicUsize::new(0); + +macro_rules! s { + ($($id:expr)+) => { + &[$($id as u16),+] + } +} + +pub const POLL_RECEIVE: u32 = 0b0_0000_0001; +pub const POLL_RECEIVE_EXPEDITED: u32 = 0b0_0000_0010; +pub const POLL_SEND: u32 = 0b0_0000_0100; +pub const POLL_DISCONNECT: u32 = 0b0_0000_1000; +pub const POLL_ABORT: u32 = 0b0_0001_0000; +pub const POLL_LOCAL_CLOSE: u32 = 0b0_0010_0000; +// Not used as it indicated in each event where a connection is connected, not +// just the first time a connection is established. +// Also see https://github.com/piscisaureus/wepoll/commit/8b7b340610f88af3d83f40fb728e7b850b090ece. +pub const POLL_CONNECT: u32 = 0b0_0100_0000; +pub const POLL_ACCEPT: u32 = 0b0_1000_0000; +pub const POLL_CONNECT_FAIL: u32 = 0b1_0000_0000; + +pub const KNOWN_EVENTS: u32 = POLL_RECEIVE + | POLL_RECEIVE_EXPEDITED + | POLL_SEND + | POLL_DISCONNECT + | POLL_ABORT + | POLL_LOCAL_CLOSE + | POLL_ACCEPT + | POLL_CONNECT_FAIL; + +#[repr(C)] +#[derive(Debug)] +pub struct AfdPollHandleInfo { + pub handle: HANDLE, + pub events: u32, + pub status: NTSTATUS, +} + +#[repr(C)] +#[derive(Debug)] +pub struct AfdPollInfo { + pub timeout: i64, + pub number_of_handles: u32, + pub exclusive: u32, + pub handles: [AfdPollHandleInfo; 1], +} + +#[derive(Debug)] +pub struct Afd { + file: File, +} + +impl Afd { + pub fn new(cp: &CompletionPort) -> std::io::Result { + const AFD_NAME: &[u16] = s!['\\' 'D' 'e' 'v' 'i' 'c' 'e' '\\' 'A' 'f' 'd' '\\' 'I' 'o']; + let mut device_name = UNICODE_STRING { + Length: std::mem::size_of_val(AFD_NAME) as u16, + MaximumLength: std::mem::size_of_val(AFD_NAME) as u16, + Buffer: AFD_NAME.as_ptr() as *mut u16, + }; + let mut device_attributes = OBJECT_ATTRIBUTES { + Length: std::mem::size_of::() as u32, + RootDirectory: 0, + ObjectName: &mut device_name, + Attributes: 0, + SecurityDescriptor: std::ptr::null_mut(), + SecurityQualityOfService: std::ptr::null_mut(), + }; + let mut handle = INVALID_HANDLE_VALUE; + let mut iosb = unsafe { std::mem::zeroed::() }; + let result = unsafe { + NtCreateFile( + &mut handle, + SYNCHRONIZE, + &mut device_attributes, + &mut iosb, + std::ptr::null_mut(), + 0, + FILE_SHARE_READ | FILE_SHARE_WRITE, + FILE_OPEN, + 0, + std::ptr::null_mut(), + 0, + ) + }; + + if result != STATUS_SUCCESS { + let error = unsafe { RtlNtStatusToDosError(result) }; + return Err(std::io::Error::from_raw_os_error(error as i32)); + } + + let file = unsafe { File::from_raw_handle(handle as RawHandle) }; + // Increment by 2 to reserve space for other types of handles. + // Non-AFD types (currently only NamedPipe), use odd numbered + // tokens. This allows the selector to differentiate between them + // and dispatch events accordingly. + let token = NEXT_TOKEN.fetch_add(2, Ordering::Relaxed) + 2; + cp.add_handle(token, file.as_raw_handle() as HANDLE)?; + let result = unsafe { + SetFileCompletionNotificationModes( + handle, + FILE_SKIP_SET_EVENT_ON_HANDLE as u8, // This is just 2, so fits in u8 + ) + }; + + if result == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(Self { file }) + } + } + + pub unsafe fn poll( + &self, + info: &mut AfdPollInfo, + iosb: *mut IO_STATUS_BLOCK, + overlapped: *mut c_void, + ) -> std::io::Result { + const IOCTL_AFD_POLL: u32 = 0x00012024; + let info_ptr = info as *mut _ as *mut c_void; + (*iosb).Anonymous.Status = STATUS_PENDING; + + let result = NtDeviceIoControlFile( + self.file.as_raw_handle() as HANDLE, + 0, + None, + overlapped, + iosb, + IOCTL_AFD_POLL, + info_ptr, + std::mem::size_of::() as u32, + info_ptr, + std::mem::size_of::() as u32, + ); + + match result { + STATUS_SUCCESS => Ok(true), + STATUS_PENDING => Ok(false), + status => { + let error = RtlNtStatusToDosError(status); + Err(std::io::Error::from_raw_os_error(error as i32)) + } + } + } + + pub unsafe fn cancel(&self, iosb: *mut IO_STATUS_BLOCK) -> std::io::Result<()> { + if (*iosb).Anonymous.Status != STATUS_PENDING { + return Ok(()); + } + let mut cancel_iosb = IO_STATUS_BLOCK { + Anonymous: IO_STATUS_BLOCK_0 { Status: 0 }, + Information: 0, + }; + let status = NtCancelIoFileEx(self.file.as_raw_handle() as HANDLE, iosb, &mut cancel_iosb); + + if status == STATUS_SUCCESS || status == STATUS_NOT_FOUND { + Ok(()) + } else { + let error = RtlNtStatusToDosError(status); + Err(std::io::Error::from_raw_os_error(error as i32)) + } + } +} diff --git a/vendor/monoio/src/driver/legacy/iocp/core.rs b/vendor/monoio/src/driver/legacy/iocp/core.rs new file mode 100644 index 000000000..0c5cb4f18 --- /dev/null +++ b/vendor/monoio/src/driver/legacy/iocp/core.rs @@ -0,0 +1,123 @@ +use std::{ + os::windows::prelude::{AsRawHandle, FromRawHandle, IntoRawHandle, RawHandle}, + time::Duration, +}; + +use windows_sys::Win32::{ + Foundation::{CloseHandle, HANDLE, INVALID_HANDLE_VALUE}, + System::IO::{ + CreateIoCompletionPort, GetQueuedCompletionStatusEx, PostQueuedCompletionStatus, + OVERLAPPED_ENTRY, + }, +}; + +#[derive(Debug)] +pub struct CompletionPort { + handle: HANDLE, +} + +impl CompletionPort { + pub fn new(value: u32) -> std::io::Result { + let handle = unsafe { CreateIoCompletionPort(INVALID_HANDLE_VALUE, 0, 0, value) }; + + if handle == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(Self { handle }) + } + } + + pub fn add_handle(&self, token: usize, handle: HANDLE) -> std::io::Result<()> { + let result = unsafe { CreateIoCompletionPort(handle, self.handle, token, 0) }; + + if result == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } + } + + pub fn get_many<'a>( + &self, + entries: &'a mut [OVERLAPPED_ENTRY], + timeout: Option, + ) -> std::io::Result<&'a mut [OVERLAPPED_ENTRY]> { + let mut count = 0; + let result = unsafe { + GetQueuedCompletionStatusEx( + self.handle, + entries.as_mut_ptr(), + std::cmp::min(entries.len(), u32::MAX as usize) as u32, + &mut count, + duration_millis(timeout), + 0, + ) + }; + + if result == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(&mut entries[..count as usize]) + } + } + + pub fn post(&self, entry: OVERLAPPED_ENTRY) -> std::io::Result<()> { + let result = unsafe { + PostQueuedCompletionStatus( + self.handle, + entry.dwNumberOfBytesTransferred, + entry.lpCompletionKey, + entry.lpOverlapped, + ) + }; + + if result == 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } + } +} + +impl Drop for CompletionPort { + fn drop(&mut self) { + unsafe { CloseHandle(self.handle) }; + } +} + +impl AsRawHandle for CompletionPort { + fn as_raw_handle(&self) -> RawHandle { + self.handle as RawHandle + } +} + +impl FromRawHandle for CompletionPort { + unsafe fn from_raw_handle(handle: RawHandle) -> Self { + Self { + handle: handle as HANDLE, + } + } +} + +impl IntoRawHandle for CompletionPort { + fn into_raw_handle(self) -> RawHandle { + self.handle as RawHandle + } +} + +#[inline] +fn duration_millis(dur: Option) -> u32 { + if let Some(dur) = dur { + // `Duration::as_millis` truncates, so round up. This avoids + // turning sub-millisecond timeouts into a zero timeout, unless + // the caller explicitly requests that by specifying a zero + // timeout. + let dur_ms = dur + .checked_add(Duration::from_nanos(999_999)) + .unwrap_or(dur) + .as_millis(); + std::cmp::min(dur_ms, u32::MAX as u128) as u32 + } else { + u32::MAX + } +} diff --git a/vendor/monoio/src/driver/legacy/iocp/event.rs b/vendor/monoio/src/driver/legacy/iocp/event.rs new file mode 100644 index 000000000..0f962ff84 --- /dev/null +++ b/vendor/monoio/src/driver/legacy/iocp/event.rs @@ -0,0 +1,120 @@ +use mio::Token; +use windows_sys::Win32::System::IO::OVERLAPPED_ENTRY; + +use super::afd; + +#[derive(Clone)] +pub struct Event { + pub flags: u32, + pub data: u64, +} + +impl Event { + pub fn new(token: Token) -> Event { + Event { + flags: 0, + data: usize::from(token) as u64, + } + } + + pub fn token(&self) -> Token { + Token(self.data as usize) + } + + pub fn set_readable(&mut self) { + self.flags |= afd::POLL_RECEIVE + } + + pub fn set_writable(&mut self) { + self.flags |= afd::POLL_SEND; + } + + pub fn from_entry(status: &OVERLAPPED_ENTRY) -> Event { + Event { + flags: status.dwNumberOfBytesTransferred, + data: status.lpCompletionKey as u64, + } + } + + pub fn to_entry(&self) -> OVERLAPPED_ENTRY { + OVERLAPPED_ENTRY { + dwNumberOfBytesTransferred: self.flags, + lpCompletionKey: self.data as usize, + lpOverlapped: std::ptr::null_mut(), + Internal: 0, + } + } + + pub fn is_readable(&self) -> bool { + self.flags & READABLE_FLAGS != 0 + } + + pub fn is_writable(&self) -> bool { + self.flags & WRITABLE_FLAGS != 0 + } + + pub fn is_error(&self) -> bool { + self.flags & ERROR_FLAGS != 0 + } + + pub fn is_read_closed(&self) -> bool { + self.flags & READ_CLOSED_FLAGS != 0 + } + + pub fn is_write_closed(&self) -> bool { + self.flags & WRITE_CLOSED_FLAGS != 0 + } + + pub fn is_priority(&self) -> bool { + self.flags & afd::POLL_RECEIVE_EXPEDITED != 0 + } +} + +pub(crate) const READABLE_FLAGS: u32 = afd::POLL_RECEIVE + | afd::POLL_DISCONNECT + | afd::POLL_ACCEPT + | afd::POLL_ABORT + | afd::POLL_CONNECT_FAIL; +pub(crate) const WRITABLE_FLAGS: u32 = afd::POLL_SEND | afd::POLL_ABORT | afd::POLL_CONNECT_FAIL; +pub(crate) const ERROR_FLAGS: u32 = afd::POLL_CONNECT_FAIL; +pub(crate) const READ_CLOSED_FLAGS: u32 = + afd::POLL_DISCONNECT | afd::POLL_ABORT | afd::POLL_CONNECT_FAIL; +pub(crate) const WRITE_CLOSED_FLAGS: u32 = afd::POLL_ABORT | afd::POLL_CONNECT_FAIL; + +pub struct Events { + pub statuses: Box<[OVERLAPPED_ENTRY]>, + + pub events: Vec, +} + +impl Events { + pub fn with_capacity(cap: usize) -> Events { + Events { + statuses: unsafe { vec![std::mem::zeroed(); cap].into_boxed_slice() }, + events: Vec::with_capacity(cap), + } + } + + pub fn is_empty(&self) -> bool { + self.events.is_empty() + } + + pub fn capacity(&self) -> usize { + self.events.capacity() + } + + pub fn len(&self) -> usize { + self.events.len() + } + + pub fn get(&self, idx: usize) -> Option<&Event> { + self.events.get(idx) + } + + pub fn clear(&mut self) { + self.events.clear(); + for status in self.statuses.iter_mut() { + *status = unsafe { std::mem::zeroed() }; + } + } +} diff --git a/vendor/monoio/src/driver/legacy/iocp/mod.rs b/vendor/monoio/src/driver/legacy/iocp/mod.rs new file mode 100644 index 000000000..3aad2057e --- /dev/null +++ b/vendor/monoio/src/driver/legacy/iocp/mod.rs @@ -0,0 +1,312 @@ +mod afd; +mod core; +mod event; +mod state; +mod waker; + +pub use core::*; +use std::{ + collections::VecDeque, + os::windows::prelude::RawSocket, + pin::Pin, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, + }, + time::Duration, +}; + +pub use afd::*; +pub use event::*; +pub use state::*; +pub use waker::*; +use windows_sys::Win32::{ + Foundation::WAIT_TIMEOUT, + System::IO::{OVERLAPPED, OVERLAPPED_ENTRY}, +}; + +pub struct Poller { + is_polling: AtomicBool, + cp: Arc, + update_queue: Mutex>>>>, + afd: Mutex>>, +} + +impl Poller { + pub fn new() -> std::io::Result { + Ok(Self { + is_polling: AtomicBool::new(false), + cp: Arc::new(CompletionPort::new(0)?), + update_queue: Mutex::new(VecDeque::new()), + afd: Mutex::new(Vec::new()), + }) + } + + pub fn poll(&self, events: &mut Events, timeout: Option) -> std::io::Result<()> { + events.clear(); + + if timeout.is_none() { + loop { + let len = self.poll_inner(&mut events.statuses, &mut events.events, None)?; + if len == 0 { + continue; + } + break Ok(()); + } + } else { + self.poll_inner(&mut events.statuses, &mut events.events, timeout)?; + Ok(()) + } + } + + pub fn poll_inner( + &self, + entries: &mut [OVERLAPPED_ENTRY], + events: &mut Vec, + timeout: Option, + ) -> std::io::Result { + self.is_polling.swap(true, Ordering::AcqRel); + + unsafe { self.update_sockets_events() }?; + + let result = self.cp.get_many(entries, timeout); + + self.is_polling.store(false, Ordering::Relaxed); + + match result { + Ok(iocp_events) => Ok(unsafe { self.feed_events(events, iocp_events) }), + Err(ref e) if e.raw_os_error() == Some(WAIT_TIMEOUT as i32) => Ok(0), + Err(e) => Err(e), + } + } + + unsafe fn update_sockets_events(&self) -> std::io::Result<()> { + let mut queue = self.update_queue.lock().unwrap(); + for sock in queue.iter_mut() { + let mut sock_internal = sock.lock().unwrap(); + if !sock_internal.delete_pending { + sock_internal.update(sock)?; + } + } + + queue.retain(|sock| sock.lock().unwrap().error.is_some()); + + let mut afd = self.afd.lock().unwrap(); + afd.retain(|g| Arc::strong_count(g) > 1); + Ok(()) + } + + unsafe fn feed_events(&self, events: &mut Vec, entries: &[OVERLAPPED_ENTRY]) -> usize { + let mut n = 0; + let mut update_queue = self.update_queue.lock().unwrap(); + for entry in entries.iter() { + if entry.lpOverlapped.is_null() { + events.push(Event::from_entry(entry)); + n += 1; + continue; + } + + let sock_state = from_overlapped(entry.lpOverlapped); + let mut sock_guard = sock_state.lock().unwrap(); + if let Some(e) = sock_guard.feed_event() { + events.push(e); + n += 1; + } + + if !sock_guard.delete_pending { + update_queue.push_back(sock_state.clone()); + } + } + let mut afd = self.afd.lock().unwrap(); + afd.retain(|sock| Arc::strong_count(sock) > 1); + n + } + + pub fn register( + &self, + state: &mut SocketState, + token: mio::Token, + interests: mio::Interest, + ) -> std::io::Result<()> { + if state.inner.is_none() { + let flags = interests_to_afd_flags(interests); + + let inner = { + let sock = self._alloc_sock_for_rawsocket(state.socket)?; + let event = Event { + flags, + data: token.0 as u64, + }; + sock.lock().unwrap().set_event(event); + sock + }; + + self.queue_state(inner.clone()); + unsafe { self.update_sockets_events_if_polling()? }; + state.inner = Some(inner); + state.token = token; + state.interest = interests; + + Ok(()) + } else { + Err(std::io::ErrorKind::AlreadyExists.into()) + } + } + + pub fn reregister( + &self, + state: &mut SocketState, + token: mio::Token, + interests: mio::Interest, + ) -> std::io::Result<()> { + if let Some(inner) = state.inner.as_mut() { + { + let event = Event { + flags: interests_to_afd_flags(interests), + data: token.0 as u64, + }; + + inner.lock().unwrap().set_event(event); + } + + state.token = token; + state.interest = interests; + + self.queue_state(inner.clone()); + unsafe { self.update_sockets_events_if_polling() } + } else { + Err(std::io::ErrorKind::NotFound.into()) + } + } + + pub fn deregister(&mut self, state: &mut SocketState) -> std::io::Result<()> { + if let Some(inner) = state.inner.as_mut() { + { + let mut sock_state = inner.lock().unwrap(); + sock_state.mark_delete(); + } + state.inner = None; + Ok(()) + } else { + Err(std::io::ErrorKind::NotFound.into()) + } + } + + /// This function is called by register() and reregister() to start an + /// IOCTL_AFD_POLL operation corresponding to the registered events, but + /// only if necessary. + /// + /// Since it is not possible to modify or synchronously cancel an AFD_POLL + /// operation, and there can be only one active AFD_POLL operation per + /// (socket, completion port) pair at any time, it is expensive to change + /// a socket's event registration after it has been submitted to the kernel. + /// + /// Therefore, if no other threads are polling when interest in a socket + /// event is (re)registered, the socket is added to the 'update queue', but + /// the actual syscall to start the IOCTL_AFD_POLL operation is deferred + /// until just before the GetQueuedCompletionStatusEx() syscall is made. + /// + /// However, when another thread is already blocked on + /// GetQueuedCompletionStatusEx() we tell the kernel about the registered + /// socket event(s) immediately. + unsafe fn update_sockets_events_if_polling(&self) -> std::io::Result<()> { + if self.is_polling.load(Ordering::Acquire) { + self.update_sockets_events() + } else { + Ok(()) + } + } + + fn queue_state(&self, sock_state: Pin>>) { + let mut update_queue = self.update_queue.lock().unwrap(); + update_queue.push_back(sock_state); + } + + fn _alloc_sock_for_rawsocket( + &self, + raw_socket: RawSocket, + ) -> std::io::Result>>> { + const POLL_GROUP__MAX_GROUP_SIZE: usize = 32; + + let mut afd_group = self.afd.lock().unwrap(); + if afd_group.len() == 0 { + self._alloc_afd_group(&mut afd_group)?; + } else { + // + 1 reference in Vec + if Arc::strong_count(afd_group.last().unwrap()) > POLL_GROUP__MAX_GROUP_SIZE { + self._alloc_afd_group(&mut afd_group)?; + } + } + let afd = match afd_group.last() { + Some(arc) => arc.clone(), + None => unreachable!("Cannot acquire afd"), + }; + + Ok(Arc::pin(Mutex::new(SockState::new(raw_socket, afd)?))) + } + + fn _alloc_afd_group(&self, afd_group: &mut Vec>) -> std::io::Result<()> { + let afd = Afd::new(&self.cp)?; + let arc = Arc::new(afd); + afd_group.push(arc); + Ok(()) + } +} + +impl Drop for Poller { + fn drop(&mut self) { + loop { + let count: usize; + let mut statuses: [OVERLAPPED_ENTRY; 1024] = unsafe { std::mem::zeroed() }; + + let result = self + .cp + .get_many(&mut statuses, Some(std::time::Duration::from_millis(0))); + match result { + Ok(events) => { + count = events.iter().len(); + for event in events.iter() { + if event.lpOverlapped.is_null() { + } else { + // drain sock state to release memory of Arc reference + let _ = from_overlapped(event.lpOverlapped); + } + } + } + Err(_) => break, + } + + if count == 0 { + break; + } + } + + let mut afd_group = self.afd.lock().unwrap(); + afd_group.retain(|g| Arc::strong_count(g) > 1); + } +} + +pub fn from_overlapped(ptr: *mut OVERLAPPED) -> Pin>> { + let sock_ptr: *const Mutex = ptr as *const _; + unsafe { Pin::new_unchecked(Arc::from_raw(sock_ptr)) } +} + +pub fn into_overlapped(sock_state: Pin>>) -> *mut std::ffi::c_void { + let overlapped_ptr: *const Mutex = + unsafe { Arc::into_raw(Pin::into_inner_unchecked(sock_state)) }; + overlapped_ptr as *mut _ +} + +pub fn interests_to_afd_flags(interests: mio::Interest) -> u32 { + let mut flags = 0; + + if interests.is_readable() { + flags |= READABLE_FLAGS | READ_CLOSED_FLAGS | ERROR_FLAGS; + } + + if interests.is_writable() { + flags |= WRITABLE_FLAGS | WRITE_CLOSED_FLAGS | ERROR_FLAGS; + } + + flags +} diff --git a/vendor/monoio/src/driver/legacy/iocp/state.rs b/vendor/monoio/src/driver/legacy/iocp/state.rs new file mode 100644 index 000000000..a550eb6e0 --- /dev/null +++ b/vendor/monoio/src/driver/legacy/iocp/state.rs @@ -0,0 +1,291 @@ +use core::fmt::Debug; +use std::{ + marker::PhantomPinned, + os::windows::prelude::RawSocket, + pin::Pin, + sync::{Arc, Mutex}, +}; + +use windows_sys::Win32::{ + Foundation::{ERROR_INVALID_HANDLE, ERROR_IO_PENDING, HANDLE, STATUS_CANCELLED}, + Networking::WinSock::{ + WSAGetLastError, WSAIoctl, SIO_BASE_HANDLE, SIO_BSP_HANDLE, SIO_BSP_HANDLE_POLL, + SIO_BSP_HANDLE_SELECT, SOCKET_ERROR, + }, + System::WindowsProgramming::IO_STATUS_BLOCK, +}; + +use super::{afd, from_overlapped, into_overlapped, Afd, AfdPollInfo, Event}; + +#[derive(Debug, Copy, Clone, PartialEq, Eq)] +pub enum SockPollStatus { + Idle, + Pending, + Cancelled, +} + +#[derive(Debug)] +pub struct SocketState { + pub socket: RawSocket, + pub inner: Option>>>, + pub token: mio::Token, + pub interest: mio::Interest, +} + +impl SocketState { + pub fn new(socket: RawSocket) -> Self { + Self { + socket, + inner: None, + token: mio::Token(0), + interest: mio::Interest::READABLE, + } + } +} + +pub struct SockState { + pub iosb: IO_STATUS_BLOCK, + pub poll_info: AfdPollInfo, + pub afd: Arc, + + pub base_socket: RawSocket, + + pub user_evts: u32, + pub pending_evts: u32, + + pub user_data: u64, + + pub poll_status: SockPollStatus, + pub delete_pending: bool, + + pub error: Option, + + _pinned: PhantomPinned, +} + +impl SockState { + pub fn new(raw_socket: RawSocket, afd: Arc) -> std::io::Result { + Ok(SockState { + iosb: unsafe { std::mem::zeroed() }, + poll_info: unsafe { std::mem::zeroed() }, + afd, + base_socket: get_base_socket(raw_socket)?, + user_evts: 0, + pending_evts: 0, + user_data: 0, + poll_status: SockPollStatus::Idle, + delete_pending: false, + error: None, + _pinned: PhantomPinned, + }) + } + + pub fn update(&mut self, self_arc: &Pin>>) -> std::io::Result<()> { + assert!(!self.delete_pending); + + // make sure to reset previous error before a new update + self.error = None; + + if let SockPollStatus::Pending = self.poll_status { + if (self.user_evts & afd::KNOWN_EVENTS & !self.pending_evts) == 0 { + // All the events the user is interested in are already being monitored by + // the pending poll operation. It might spuriously complete because of an + // event that we're no longer interested in; when that happens we'll submit + // a new poll operation with the updated event mask. + } else { + // A poll operation is already pending, but it's not monitoring for all the + // events that the user is interested in. Therefore, cancel the pending + // poll operation; when we receive it's completion package, a new poll + // operation will be submitted with the correct event mask. + if let Err(e) = self.cancel() { + self.error = e.raw_os_error(); + return Err(e); + } + return Ok(()); + } + } else if let SockPollStatus::Cancelled = self.poll_status { + // The poll operation has already been cancelled, we're still waiting for + // it to return. For now, there's nothing that needs to be done. + } else if let SockPollStatus::Idle = self.poll_status { + // No poll operation is pending; start one. + self.poll_info.exclusive = 0; + self.poll_info.number_of_handles = 1; + self.poll_info.timeout = i64::MAX; + self.poll_info.handles[0].handle = self.base_socket as HANDLE; + self.poll_info.handles[0].status = 0; + self.poll_info.handles[0].events = self.user_evts | afd::POLL_LOCAL_CLOSE; + + // Increase the ref count as the memory will be used by the kernel. + let overlapped_ptr = into_overlapped(self_arc.clone()); + + let result = unsafe { + self.afd + .poll(&mut self.poll_info, &mut self.iosb, overlapped_ptr) + }; + if let Err(e) = result { + let code = e.raw_os_error().unwrap(); + if code == ERROR_IO_PENDING as i32 { + // Overlapped poll operation in progress; this is expected. + } else { + // Since the operation failed it means the kernel won't be + // using the memory any more. + drop(from_overlapped(overlapped_ptr as *mut _)); + if code == ERROR_INVALID_HANDLE as i32 { + // Socket closed; it'll be dropped. + self.mark_delete(); + return Ok(()); + } else { + self.error = e.raw_os_error(); + return Err(e); + } + } + } + + self.poll_status = SockPollStatus::Pending; + self.pending_evts = self.user_evts; + } else { + unreachable!("Invalid poll status during update") + } + + Ok(()) + } + + pub fn feed_event(&mut self) -> Option { + self.poll_status = SockPollStatus::Idle; + self.pending_evts = 0; + + let mut afd_events = 0; + // We use the status info in IO_STATUS_BLOCK to determine the socket poll status. It is + // unsafe to use a pointer of IO_STATUS_BLOCK. + unsafe { + if self.delete_pending { + return None; + } else if self.iosb.Anonymous.Status == STATUS_CANCELLED { + // The poll request was cancelled by CancelIoEx. + } else if self.iosb.Anonymous.Status < 0 { + // The overlapped request itself failed in an unexpected way. + afd_events = afd::POLL_CONNECT_FAIL; + } else if self.poll_info.number_of_handles < 1 { + // This poll operation succeeded but didn't report any socket events. + } else if self.poll_info.handles[0].events & afd::POLL_LOCAL_CLOSE != 0 { + // The poll operation reported that the socket was closed. + self.mark_delete(); + return None; + } else { + afd_events = self.poll_info.handles[0].events; + } + } + + afd_events &= self.user_evts; + + if afd_events == 0 { + return None; + } + + self.user_evts &= !afd_events; + + Some(Event { + data: self.user_data, + flags: afd_events, + }) + } + + pub fn mark_delete(&mut self) { + if !self.delete_pending { + if let SockPollStatus::Pending = self.poll_status { + drop(self.cancel()); + } + + self.delete_pending = true; + } + } + + pub fn set_event(&mut self, ev: Event) -> bool { + // afd::POLL_CONNECT_FAIL and afd::POLL_ABORT are always reported, even when not requested + // by the caller. + let events = ev.flags | afd::POLL_CONNECT_FAIL | afd::POLL_ABORT; + + self.user_evts = events; + self.user_data = ev.data; + + (events & !self.pending_evts) != 0 + } + + pub fn cancel(&mut self) -> std::io::Result<()> { + match self.poll_status { + SockPollStatus::Pending => {} + _ => unreachable!("Invalid poll status during cancel"), + }; + unsafe { + self.afd.cancel(&mut self.iosb)?; + } + self.poll_status = SockPollStatus::Cancelled; + self.pending_evts = 0; + Ok(()) + } +} + +impl Debug for SockState { + #[allow(unused_variables)] + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + unimplemented!() + } +} + +impl Drop for SockState { + fn drop(&mut self) { + self.mark_delete(); + } +} + +fn get_base_socket(raw_socket: RawSocket) -> std::io::Result { + let res = try_get_base_socket(raw_socket, SIO_BASE_HANDLE); + if let Ok(base_socket) = res { + return Ok(base_socket); + } + + // The `SIO_BASE_HANDLE` should not be intercepted by LSPs, therefore + // it should not fail as long as `raw_socket` is a valid socket. See + // https://docs.microsoft.com/en-us/windows/win32/winsock/winsock-ioctls. + // However, at least one known LSP deliberately breaks it, so we try + // some alternative IOCTLs, starting with the most appropriate one. + for &ioctl in &[SIO_BSP_HANDLE_SELECT, SIO_BSP_HANDLE_POLL, SIO_BSP_HANDLE] { + if let Ok(base_socket) = try_get_base_socket(raw_socket, ioctl) { + // Since we know now that we're dealing with an LSP (otherwise + // SIO_BASE_HANDLE would't have failed), only return any result + // when it is different from the original `raw_socket`. + if base_socket != raw_socket { + return Ok(base_socket); + } + } + } + + // If the alternative IOCTLs also failed, return the original error. + let os_error = res.unwrap_err(); + let err = std::io::Error::from_raw_os_error(os_error); + Err(err) +} + +fn try_get_base_socket(raw_socket: RawSocket, ioctl: u32) -> Result { + let mut base_socket: RawSocket = 0; + let mut bytes: u32 = 0; + let result = unsafe { + WSAIoctl( + raw_socket as usize, + ioctl, + std::ptr::null_mut(), + 0, + &mut base_socket as *mut _ as *mut std::ffi::c_void, + std::mem::size_of::() as u32, + &mut bytes, + std::ptr::null_mut(), + None, + ) + }; + + if result != SOCKET_ERROR { + Ok(base_socket) + } else { + Err(unsafe { WSAGetLastError() }) + } +} diff --git a/vendor/monoio/src/driver/legacy/iocp/waker.rs b/vendor/monoio/src/driver/legacy/iocp/waker.rs new file mode 100644 index 000000000..ab21813a1 --- /dev/null +++ b/vendor/monoio/src/driver/legacy/iocp/waker.rs @@ -0,0 +1,25 @@ +use std::{io, sync::Arc}; + +use super::{CompletionPort, Event, Poller}; + +#[derive(Debug)] +pub struct Waker { + token: mio::Token, + port: Arc, +} + +impl Waker { + #[allow(unreachable_code, unused_variables)] + pub fn new(poller: &Poller, token: mio::Token) -> io::Result { + Ok(Waker { + token, + port: poller.cp.clone(), + }) + } + + pub fn wake(&self) -> io::Result<()> { + let mut ev = Event::new(self.token); + ev.set_readable(); + self.port.post(ev.to_entry()) + } +} diff --git a/vendor/monoio/src/driver/legacy/mod.rs b/vendor/monoio/src/driver/legacy/mod.rs new file mode 100644 index 000000000..92c454876 --- /dev/null +++ b/vendor/monoio/src/driver/legacy/mod.rs @@ -0,0 +1,537 @@ +//! Monoio Legacy Driver. + +use std::{ + cell::UnsafeCell, + io, + rc::Rc, + task::{Context, Poll}, + time::Duration, +}; + +use super::{ + op::{CompletionMeta, Op, OpAble}, + ready::{self, Ready}, + scheduled_io::ScheduledIo, + Driver, Inner, CURRENT, +}; +use crate::utils::slab::Slab; + +#[allow(missing_docs, unreachable_pub, dead_code, unused_imports)] +#[cfg(windows)] +pub(super) mod iocp; + +#[cfg(feature = "sync")] +mod waker; +#[cfg(feature = "sync")] +pub(crate) use waker::UnparkHandle; + +pub(crate) struct LegacyInner { + pub(crate) io_dispatch: Slab, + #[cfg(unix)] + events: mio::Events, + #[cfg(unix)] + poll: mio::Poll, + #[cfg(windows)] + events: iocp::Events, + #[cfg(windows)] + poll: iocp::Poller, + + #[cfg(feature = "sync")] + shared_waker: std::sync::Arc, + + // Waker receiver + #[cfg(feature = "sync")] + waker_receiver: flume::Receiver, +} + +/// Driver with Poll-like syscall. +#[allow(unreachable_pub)] +pub struct LegacyDriver { + inner: Rc>, + + // Used for drop + #[cfg(feature = "sync")] + thread_id: usize, +} + +#[cfg(feature = "sync")] +const TOKEN_WAKEUP: mio::Token = mio::Token(1 << 31); + +// moon patch: programmatic spin-budget override (host CLI flag path). +// u64::MAX = unset -> fall back to the MOON_EPOLL_SPIN_US env. Must be set by +// the host before runtime threads first park. +pub(crate) static SPIN_BUDGET_US: std::sync::atomic::AtomicU64 = + std::sync::atomic::AtomicU64::new(u64::MAX); + +// moon patch: epoll spin budget (0 = disabled): programmatic override first, +// else MOON_EPOLL_SPIN_US env, read once. See inner_park for the poll-mode +// rationale. +fn moon_epoll_spin_budget_us() -> u64 { + let ovr = SPIN_BUDGET_US.load(std::sync::atomic::Ordering::Relaxed); + if ovr != u64::MAX { + return ovr; + } + static SPIN_US: std::sync::OnceLock = std::sync::OnceLock::new(); + *SPIN_US.get_or_init(|| { + std::env::var("MOON_EPOLL_SPIN_US") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(0) + }) +} + +// moon patch: per-thread spin-park hooks (skip-notify handshake with the +// host's cross-shard mesh). `advertise(true)` is called at spin entry and +// `advertise(false)` at every spin exit; the host publishes this so remote +// producers can elide their cross-thread wake (flume send + waker relay + +// eventfd write) while this thread is polling anyway. `probe()` is called +// each spin iteration AND once after `advertise(false)` (the Dekker final +// check — a producer that skipped its wake concurrently with spin exit is +// caught either by its own flag re-read or by this probe): it returns true +// when host-level work (SPSC ringbuf items) is pending, after delivering a +// thread-local wake to the host task so the executor runs it on return. +// Not Send: registered and invoked on this driver's thread only. +struct MoonSpinHooks { + advertise: Box, + probe: Box bool>, +} + +thread_local! { + static MOON_SPIN_HOOKS: std::cell::RefCell> = + const { std::cell::RefCell::new(None) }; +} + +pub(crate) fn set_spin_hooks(advertise: Box, probe: Box bool>) { + MOON_SPIN_HOOKS.with(|h| *h.borrow_mut() = Some(MoonSpinHooks { advertise, probe })); +} + +// Short borrows per call: the closures run host code (ringbuf checks, a +// local flume send) that must not observe a held RefCell borrow. +fn moon_spin_hooks_installed() -> bool { + MOON_SPIN_HOOKS.with(|h| h.borrow().is_some()) +} + +fn moon_spin_advertise(spinning: bool) { + MOON_SPIN_HOOKS.with(|h| { + if let Some(hooks) = &*h.borrow() { + (hooks.advertise)(spinning); + } + }); +} + +fn moon_spin_probe() -> bool { + MOON_SPIN_HOOKS.with(|h| { + h.borrow() + .as_ref() + .map_or(false, |hooks| (hooks.probe)()) + }) +} + +#[allow(dead_code)] +impl LegacyDriver { + const DEFAULT_ENTRIES: u32 = 1024; + + pub(crate) fn new() -> io::Result { + Self::new_with_entries(Self::DEFAULT_ENTRIES) + } + + pub(crate) fn new_with_entries(entries: u32) -> io::Result { + #[cfg(unix)] + let poll = mio::Poll::new()?; + #[cfg(windows)] + let poll = iocp::Poller::new()?; + + #[cfg(all(unix, feature = "sync"))] + let shared_waker = std::sync::Arc::new(waker::EventWaker::new(mio::Waker::new( + poll.registry(), + TOKEN_WAKEUP, + )?)); + #[cfg(all(windows, feature = "sync"))] + let shared_waker = std::sync::Arc::new(waker::EventWaker::new(iocp::Waker::new( + &poll, + TOKEN_WAKEUP, + )?)); + #[cfg(feature = "sync")] + let (waker_sender, waker_receiver) = flume::unbounded::(); + #[cfg(feature = "sync")] + let thread_id = crate::builder::BUILD_THREAD_ID.with(|id| *id); + + let inner = LegacyInner { + io_dispatch: Slab::new(), + #[cfg(unix)] + events: mio::Events::with_capacity(entries as usize), + #[cfg(unix)] + poll, + #[cfg(windows)] + events: iocp::Events::with_capacity(entries as usize), + #[cfg(windows)] + poll, + #[cfg(feature = "sync")] + shared_waker, + #[cfg(feature = "sync")] + waker_receiver, + }; + let driver = Self { + inner: Rc::new(UnsafeCell::new(inner)), + #[cfg(feature = "sync")] + thread_id, + }; + + // Register unpark handle + #[cfg(feature = "sync")] + { + let unpark = driver.unpark(); + super::thread::register_unpark_handle(thread_id, unpark.into()); + super::thread::register_waker_sender(thread_id, waker_sender); + } + + Ok(driver) + } + + fn inner_park(&self, mut timeout: Option) -> io::Result<()> { + let inner = unsafe { &mut *self.inner.get() }; + + #[allow(unused_mut)] + let mut need_wait = true; + #[cfg(feature = "sync")] + { + // Process foreign wakers + while let Ok(w) = inner.waker_receiver.try_recv() { + w.wake(); + need_wait = false; + } + + // Set status as not awake if we are going to sleep + if need_wait { + inner + .shared_waker + .awake + .store(false, std::sync::atomic::Ordering::Release); + } + + // Process foreign wakers left + while let Ok(w) = inner.waker_receiver.try_recv() { + w.wake(); + need_wait = false; + } + } + + if !need_wait { + timeout = Some(Duration::ZERO); + } + + // here we borrow 2 mut self, but its safe. + let events = unsafe { &mut (*self.inner.get()).events }; + + // ---- moon patch: readiness spin-poll before blocking (MOON_EPOLL_SPIN_US) ---- + // Poll-mode for request/response workloads: instead of sleeping in + // epoll_wait and paying the scheduler wake on every op, busy-loop + // zero-timeout polls (~0.3µs each; readiness is published by softirq + // with no wake machinery) for a bounded window, falling back to the + // stock blocking poll on miss. Foreign wakes surface during the spin + // too: the waker eventfd is registered in this same mio poll. Off + // unless MOON_EPOLL_SPIN_US is set; zero behavior change otherwise. + let mut polled = false; + #[cfg(unix)] + if need_wait { + let spin_us = moon_epoll_spin_budget_us(); + if spin_us > 0 { + let budget = match timeout { + Some(d) => d.min(Duration::from_micros(spin_us)), + None => Duration::from_micros(spin_us), + }; + let deadline = std::time::Instant::now() + budget; + // Skip-notify handshake (see MoonSpinHooks above): advertise + // the spin window so remote producers elide their wake; probe + // host ringbufs each iteration in their stead. + let hooks = moon_spin_hooks_installed(); + if hooks { + moon_spin_advertise(true); + } + let mut probe_hit = false; + loop { + match inner.poll.poll(events, Some(Duration::ZERO)) { + Ok(_) => {} + Err(ref e) if e.kind() == io::ErrorKind::Interrupted => {} + Err(e) => { + if hooks { + moon_spin_advertise(false); + } + return Err(e); + } + } + if !events.is_empty() { + polled = true; + break; + } + if hooks && moon_spin_probe() { + probe_hit = true; + break; + } + if std::time::Instant::now() >= deadline { + break; + } + for _ in 0..16 { + std::hint::spin_loop(); + } + } + if hooks { + // Retract BEFORE the final probe (SeqCst handshake lives + // in the host closures): a producer that skipped its wake + // during our exit is guaranteed visible to this probe, or + // it re-reads the cleared flag and sends normally. Runs on + // ALL exits — an fd-event exit can also have raced a + // skipped notify. + moon_spin_advertise(false); + if !probe_hit && moon_spin_probe() { + probe_hit = true; + } + } + if probe_hit { + // Host work is pending and a local wake was delivered — + // return to the executor instead of blocking. + polled = true; + } + } + } + // ---- end moon patch ---- + + if !polled { + match inner.poll.poll(events, timeout) { + Ok(_) => {} + Err(ref e) if e.kind() == io::ErrorKind::Interrupted => {} + Err(e) => return Err(e), + } + } + #[cfg(unix)] + let iter = events.iter(); + #[cfg(windows)] + let iter = events.events.iter(); + for event in iter { + let token = event.token(); + + #[cfg(feature = "sync")] + if token != TOKEN_WAKEUP { + inner.dispatch(token, Ready::from_mio(event)); + } + + #[cfg(not(feature = "sync"))] + inner.dispatch(token, Ready::from_mio(event)); + } + Ok(()) + } + + #[cfg(windows)] + pub(crate) fn register( + this: &Rc>, + state: &mut iocp::SocketState, + interest: mio::Interest, + ) -> io::Result { + let inner = unsafe { &mut *this.get() }; + let io = ScheduledIo::default(); + let token = inner.io_dispatch.insert(io); + + match inner.poll.register(state, mio::Token(token), interest) { + Ok(_) => Ok(token), + Err(e) => { + inner.io_dispatch.remove(token); + Err(e) + } + } + } + + #[cfg(windows)] + pub(crate) fn deregister( + this: &Rc>, + token: usize, + state: &mut iocp::SocketState, + ) -> io::Result<()> { + let inner = unsafe { &mut *this.get() }; + + // try to deregister fd first, on success we will remove it from slab. + match inner.poll.deregister(state) { + Ok(_) => { + inner.io_dispatch.remove(token); + Ok(()) + } + Err(e) => Err(e), + } + } + + #[cfg(unix)] + pub(crate) fn register( + this: &Rc>, + source: &mut impl mio::event::Source, + interest: mio::Interest, + ) -> io::Result { + let inner = unsafe { &mut *this.get() }; + let token = inner.io_dispatch.insert(ScheduledIo::new()); + + let registry = inner.poll.registry(); + match registry.register(source, mio::Token(token), interest) { + Ok(_) => Ok(token), + Err(e) => { + inner.io_dispatch.remove(token); + Err(e) + } + } + } + + #[cfg(unix)] + pub(crate) fn deregister( + this: &Rc>, + token: usize, + source: &mut impl mio::event::Source, + ) -> io::Result<()> { + let inner = unsafe { &mut *this.get() }; + + // try to deregister fd first, on success we will remove it from slab. + match inner.poll.registry().deregister(source) { + Ok(_) => { + inner.io_dispatch.remove(token); + Ok(()) + } + Err(e) => Err(e), + } + } +} + +impl LegacyInner { + fn dispatch(&mut self, token: mio::Token, ready: Ready) { + let mut sio = match self.io_dispatch.get(token.0) { + Some(io) => io, + None => { + return; + } + }; + let ref_mut = sio.as_mut(); + ref_mut.set_readiness(|curr| curr | ready); + ref_mut.wake(ready); + } + + pub(crate) fn poll_op( + this: &Rc>, + data: &mut T, + cx: &mut Context<'_>, + ) -> Poll { + let inner = unsafe { &mut *this.get() }; + let (direction, index) = match data.legacy_interest() { + Some(x) => x, + None => { + // if there is no index provided, it means the action does not rely on fd + // readiness. do syscall right now. + return Poll::Ready(CompletionMeta { + result: OpAble::legacy_call(data), + flags: 0, + }); + } + }; + + // wait io ready and do syscall + let mut scheduled_io = inner.io_dispatch.get(index).expect("scheduled_io lost"); + let ref_mut = scheduled_io.as_mut(); + + let readiness = ready!(ref_mut.poll_readiness(cx, direction)); + + // check if canceled + if readiness.is_canceled() { + // clear CANCELED part only + ref_mut.clear_readiness(readiness & Ready::CANCELED); + return Poll::Ready(CompletionMeta { + result: Err(io::Error::from_raw_os_error(125)), + flags: 0, + }); + } + + match OpAble::legacy_call(data) { + Ok(n) => Poll::Ready(CompletionMeta { + result: Ok(n), + flags: 0, + }), + Err(ref e) if e.kind() == io::ErrorKind::WouldBlock => { + ref_mut.clear_readiness(direction.mask()); + ref_mut.set_waker(cx, direction); + Poll::Pending + } + Err(e) => Poll::Ready(CompletionMeta { + result: Err(e), + flags: 0, + }), + } + } + + pub(crate) fn cancel_op( + this: &Rc>, + index: usize, + direction: ready::Direction, + ) { + let inner = unsafe { &mut *this.get() }; + let ready = match direction { + ready::Direction::Read => Ready::READ_CANCELED, + ready::Direction::Write => Ready::WRITE_CANCELED, + }; + inner.dispatch(mio::Token(index), ready); + } + + pub(crate) fn submit_with_data( + this: &Rc>, + data: T, + ) -> io::Result> + where + T: OpAble, + { + Ok(Op { + driver: Inner::Legacy(this.clone()), + // useless for legacy + index: 0, + data: Some(data), + }) + } + + #[cfg(feature = "sync")] + pub(crate) fn unpark(this: &Rc>) -> waker::UnparkHandle { + let inner = unsafe { &*this.get() }; + let weak = std::sync::Arc::downgrade(&inner.shared_waker); + waker::UnparkHandle(weak) + } +} + +impl Driver for LegacyDriver { + fn with(&self, f: impl FnOnce() -> R) -> R { + let inner = Inner::Legacy(self.inner.clone()); + CURRENT.set(&inner, f) + } + + fn submit(&self) -> io::Result<()> { + // wait with timeout = 0 + self.park_timeout(Duration::ZERO) + } + + fn park(&self) -> io::Result<()> { + self.inner_park(None) + } + + fn park_timeout(&self, duration: Duration) -> io::Result<()> { + self.inner_park(Some(duration)) + } + + #[cfg(feature = "sync")] + type Unpark = waker::UnparkHandle; + + #[cfg(feature = "sync")] + fn unpark(&self) -> Self::Unpark { + LegacyInner::unpark(&self.inner) + } +} + +impl Drop for LegacyDriver { + fn drop(&mut self) { + // Deregister thread id + #[cfg(feature = "sync")] + { + use crate::driver::thread::{unregister_unpark_handle, unregister_waker_sender}; + unregister_unpark_handle(self.thread_id); + unregister_waker_sender(self.thread_id); + } + } +} diff --git a/vendor/monoio/src/driver/legacy/waker.rs b/vendor/monoio/src/driver/legacy/waker.rs new file mode 100644 index 000000000..40290e968 --- /dev/null +++ b/vendor/monoio/src/driver/legacy/waker.rs @@ -0,0 +1,50 @@ +use crate::driver::unpark::Unpark; + +pub(crate) struct EventWaker { + // raw waker + #[cfg(windows)] + waker: super::iocp::Waker, + #[cfg(unix)] + waker: mio::Waker, + // Atomic awake status + pub(crate) awake: std::sync::atomic::AtomicBool, +} + +impl EventWaker { + #[cfg(unix)] + pub(crate) fn new(waker: mio::Waker) -> Self { + Self { + waker, + awake: std::sync::atomic::AtomicBool::new(true), + } + } + + #[cfg(windows)] + pub(crate) fn new(waker: super::iocp::Waker) -> Self { + Self { + waker, + awake: std::sync::atomic::AtomicBool::new(true), + } + } + + pub(crate) fn wake(&self) -> std::io::Result<()> { + // Skip wake if already awake + if self.awake.load(std::sync::atomic::Ordering::Acquire) { + return Ok(()); + } + self.waker.wake() + } +} + +#[derive(Clone)] +pub struct UnparkHandle(pub(crate) std::sync::Weak); + +impl Unpark for UnparkHandle { + fn unpark(&self) -> std::io::Result<()> { + if let Some(w) = self.0.upgrade() { + w.wake() + } else { + Ok(()) + } + } +} diff --git a/vendor/monoio/src/driver/mod.rs b/vendor/monoio/src/driver/mod.rs new file mode 100644 index 000000000..58de6f44e --- /dev/null +++ b/vendor/monoio/src/driver/mod.rs @@ -0,0 +1,282 @@ +/// Monoio Driver. +#[allow(dead_code)] +pub(crate) mod op; +#[cfg(all(feature = "poll-io", unix))] +pub(crate) mod poll; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +pub(crate) mod ready; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +pub(crate) mod scheduled_io; +#[allow(dead_code)] +pub(crate) mod shared_fd; +#[cfg(feature = "sync")] +pub(crate) mod thread; + +#[cfg(feature = "legacy")] +mod legacy; +#[cfg(all(target_os = "linux", feature = "iouring"))] +mod uring; + +mod util; + +use std::{ + io, + task::{Context, Poll}, + time::Duration, +}; + +#[allow(unreachable_pub)] +#[cfg(feature = "legacy")] +pub use self::legacy::LegacyDriver; +#[cfg(feature = "legacy")] +use self::legacy::LegacyInner; +use self::op::{CompletionMeta, Op, OpAble}; +#[cfg(all(target_os = "linux", feature = "iouring"))] +pub use self::uring::IoUringDriver; +#[cfg(all(target_os = "linux", feature = "iouring"))] +use self::uring::UringInner; + +// moon patch: programmatic legacy-driver spin-budget setter (see +// legacy/mod.rs). Re-exported at the crate root for the host application. +#[cfg(feature = "legacy")] +pub(crate) fn set_legacy_spin_budget_us(us: u64) { + legacy::SPIN_BUDGET_US.store(us, std::sync::atomic::Ordering::Relaxed); +} + +// moon patch: register per-thread spin-park hooks (skip-notify handshake +// with the host's cross-shard mesh). See legacy::MoonSpinHooks. +#[cfg(feature = "legacy")] +pub(crate) fn set_legacy_spin_hooks(advertise: Box, probe: Box bool>) { + legacy::set_spin_hooks(advertise, probe); +} + +/// Unpark a runtime of another thread. +pub(crate) mod unpark { + #[allow(unreachable_pub)] + pub trait Unpark: Sync + Send + 'static { + /// Unblocks a thread that is blocked by the associated `Park` handle. + /// + /// Calling `unpark` atomically makes available the unpark token, if it + /// is not already available. + /// + /// # Panics + /// + /// This function **should** not panic, but ultimately, panics are left + /// as an implementation detail. Refer to the documentation for + /// the specific `Unpark` implementation + fn unpark(&self) -> std::io::Result<()>; + } +} + +impl unpark::Unpark for Box { + fn unpark(&self) -> io::Result<()> { + (**self).unpark() + } +} + +impl unpark::Unpark for std::sync::Arc { + fn unpark(&self) -> io::Result<()> { + (**self).unpark() + } +} + +/// Core driver trait. +pub trait Driver { + /// Run with driver TLS. + fn with(&self, f: impl FnOnce() -> R) -> R; + /// Submit ops to kernel and process returned events. + fn submit(&self) -> io::Result<()>; + /// Wait infinitely and process returned events. + fn park(&self) -> io::Result<()>; + /// Wait with timeout and process returned events. + fn park_timeout(&self, duration: Duration) -> io::Result<()>; + + /// The struct to wake thread from another. + #[cfg(feature = "sync")] + type Unpark: unpark::Unpark; + + /// Get Unpark. + #[cfg(feature = "sync")] + fn unpark(&self) -> Self::Unpark; +} + +scoped_thread_local!(pub(crate) static CURRENT: Inner); + +#[derive(Clone)] +pub(crate) enum Inner { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Uring(std::rc::Rc>), + #[cfg(feature = "legacy")] + Legacy(std::rc::Rc>), +} + +impl Inner { + fn submit_with(&self, data: T) -> io::Result> { + match self { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Inner::Uring(this) => UringInner::submit_with_data(this, data), + #[cfg(feature = "legacy")] + Inner::Legacy(this) => LegacyInner::submit_with_data(this, data), + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + _ => { + util::feature_panic(); + } + } + } + + #[allow(unused)] + fn poll_op( + &self, + data: &mut T, + index: usize, + cx: &mut Context<'_>, + ) -> Poll { + match self { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Inner::Uring(this) => UringInner::poll_op(this, index, cx), + #[cfg(feature = "legacy")] + Inner::Legacy(this) => LegacyInner::poll_op::(this, data, cx), + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + _ => { + util::feature_panic(); + } + } + } + + #[cfg(feature = "poll-io")] + fn poll_legacy_op( + &self, + data: &mut T, + cx: &mut Context<'_>, + ) -> Poll { + match self { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Inner::Uring(this) => UringInner::poll_legacy_op(this, data, cx), + #[cfg(feature = "legacy")] + Inner::Legacy(this) => LegacyInner::poll_op::(this, data, cx), + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + _ => { + util::feature_panic(); + } + } + } + + #[allow(unused)] + fn drop_op(&self, index: usize, data: &mut Option) { + match self { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Inner::Uring(this) => UringInner::drop_op(this, index, data), + #[cfg(feature = "legacy")] + Inner::Legacy(_) => {} + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + _ => { + util::feature_panic(); + } + } + } + + #[allow(unused)] + pub(super) unsafe fn cancel_op(&self, op_canceller: &op::OpCanceller) { + match self { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Inner::Uring(this) => UringInner::cancel_op(this, op_canceller.index), + #[cfg(feature = "legacy")] + Inner::Legacy(this) => { + if let Some(direction) = op_canceller.direction { + LegacyInner::cancel_op(this, op_canceller.index, direction) + } + } + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + _ => { + util::feature_panic(); + } + } + } + + #[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] + fn is_legacy(&self) -> bool { + matches!(self, Inner::Legacy(..)) + } + + #[cfg(all(target_os = "linux", feature = "iouring", not(feature = "legacy")))] + fn is_legacy(&self) -> bool { + false + } + + #[allow(unused)] + #[cfg(not(all(target_os = "linux", feature = "iouring")))] + fn is_legacy(&self) -> bool { + true + } +} + +/// The unified UnparkHandle. +#[cfg(feature = "sync")] +#[derive(Clone)] +pub(crate) enum UnparkHandle { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Uring(self::uring::UnparkHandle), + #[cfg(feature = "legacy")] + Legacy(self::legacy::UnparkHandle), +} + +#[cfg(feature = "sync")] +impl unpark::Unpark for UnparkHandle { + fn unpark(&self) -> io::Result<()> { + match self { + #[cfg(all(target_os = "linux", feature = "iouring"))] + UnparkHandle::Uring(inner) => inner.unpark(), + #[cfg(feature = "legacy")] + UnparkHandle::Legacy(inner) => inner.unpark(), + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + _ => { + util::feature_panic(); + } + } + } +} + +#[cfg(all(feature = "sync", target_os = "linux", feature = "iouring"))] +impl From for UnparkHandle { + fn from(inner: self::uring::UnparkHandle) -> Self { + Self::Uring(inner) + } +} + +#[cfg(all(feature = "sync", feature = "legacy"))] +impl From for UnparkHandle { + fn from(inner: self::legacy::UnparkHandle) -> Self { + Self::Legacy(inner) + } +} + +#[cfg(feature = "sync")] +impl UnparkHandle { + #[allow(unused)] + pub(crate) fn current() -> Self { + CURRENT.with(|inner| match inner { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Inner::Uring(this) => UringInner::unpark(this).into(), + #[cfg(feature = "legacy")] + Inner::Legacy(this) => LegacyInner::unpark(this).into(), + }) + } +} diff --git a/vendor/monoio/src/driver/op.rs b/vendor/monoio/src/driver/op.rs new file mode 100644 index 000000000..d2daafff4 --- /dev/null +++ b/vendor/monoio/src/driver/op.rs @@ -0,0 +1,214 @@ +use std::{ + future::Future, + io, + pin::Pin, + task::{Context, Poll}, +}; + +use crate::driver; + +pub(crate) mod close; + +mod accept; +mod connect; +mod fsync; +mod open; +mod poll; +mod read; +mod recv; +mod send; +mod write; + +#[cfg(unix)] +mod statx; + +#[cfg(all(unix, feature = "mkdirat"))] +mod mkdir; + +#[cfg(all(unix, feature = "unlinkat"))] +mod unlink; + +#[cfg(all(unix, feature = "renameat"))] +mod rename; + +#[cfg(all(target_os = "linux", feature = "splice"))] +mod splice; + +/// In-flight operation +pub(crate) struct Op { + // Driver running the operation + pub(super) driver: driver::Inner, + + // Operation index in the slab(useless for legacy) + pub(super) index: usize, + + // Per-operation data + pub(super) data: Option, +} + +/// Operation completion. Returns stored state with the result of the operation. +#[derive(Debug)] +pub(crate) struct Completion { + pub(crate) data: T, + pub(crate) meta: CompletionMeta, +} + +/// Operation completion meta info. +#[derive(Debug)] +pub(crate) struct CompletionMeta { + pub(crate) result: io::Result, + #[allow(unused)] + pub(crate) flags: u32, +} + +pub(crate) trait OpAble { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry; + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_interest(&self) -> Option<(super::ready::Direction, usize)>; + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_call(&mut self) -> io::Result; +} + +/// If legacy is enabled and iouring is not, we can expose io interface in a poll-like way. +/// This can provide better compatibility for crates programmed in poll-like way. +#[allow(dead_code)] +#[cfg(any(feature = "legacy", feature = "poll-io"))] +pub(crate) trait PollLegacy { + #[cfg(feature = "legacy")] + fn poll_legacy(&mut self, cx: &mut std::task::Context<'_>) -> std::task::Poll; + #[cfg(feature = "poll-io")] + fn poll_io(&mut self, cx: &mut std::task::Context<'_>) -> std::task::Poll; +} + +#[cfg(any(feature = "legacy", feature = "poll-io"))] +impl PollLegacy for T +where + T: OpAble, +{ + #[cfg(feature = "legacy")] + #[inline] + fn poll_legacy(&mut self, _cx: &mut std::task::Context<'_>) -> std::task::Poll { + #[cfg(all(feature = "iouring", feature = "tokio-compat"))] + unsafe { + extern "C" { + #[link_name = "tokio-compat can only be enabled when legacy feature is enabled and \ + iouring is not"] + fn trigger() -> !; + } + trigger() + } + + #[cfg(not(all(feature = "iouring", feature = "tokio-compat")))] + driver::CURRENT.with(|this| this.poll_op(self, 0, _cx)) + } + + #[cfg(feature = "poll-io")] + #[inline] + fn poll_io(&mut self, cx: &mut std::task::Context<'_>) -> std::task::Poll { + driver::CURRENT.with(|this| this.poll_legacy_op(self, cx)) + } +} + +impl Op { + /// Submit an operation to uring. + /// + /// `state` is stored during the operation tracking any state submitted to + /// the kernel. + pub(super) fn submit_with(data: T) -> io::Result> + where + T: OpAble, + { + driver::CURRENT.with(|this| this.submit_with(data)) + } + + /// Try submitting an operation to uring + #[allow(unused)] + pub(super) fn try_submit_with(data: T) -> io::Result> + where + T: OpAble, + { + if driver::CURRENT.is_set() { + Op::submit_with(data) + } else { + Err(io::ErrorKind::Other.into()) + } + } + + pub(crate) fn op_canceller(&self) -> OpCanceller + where + T: OpAble, + { + #[cfg(feature = "legacy")] + if is_legacy() { + return if let Some((dir, id)) = self.data.as_ref().unwrap().legacy_interest() { + OpCanceller { + index: id, + direction: Some(dir), + } + } else { + OpCanceller { + index: 0, + direction: None, + } + }; + } + OpCanceller { + index: self.index, + #[cfg(feature = "legacy")] + direction: None, + } + } +} + +impl Future for Op +where + T: Unpin + OpAble + 'static, +{ + type Output = Completion; + + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + let me = &mut *self; + let data_mut = me.data.as_mut().expect("unexpected operation state"); + let meta = ready!(me.driver.poll_op::(data_mut, me.index, cx)); + + me.index = usize::MAX; + let data = me.data.take().expect("unexpected operation state"); + Poll::Ready(Completion { data, meta }) + } +} + +impl Drop for Op { + fn drop(&mut self) { + self.driver.drop_op(self.index, &mut self.data); + } +} + +/// Check if current driver is legacy. +#[allow(unused)] +#[cfg(not(target_os = "linux"))] +#[inline] +pub const fn is_legacy() -> bool { + true +} + +/// Check if current driver is legacy. +#[cfg(target_os = "linux")] +#[inline] +pub fn is_legacy() -> bool { + super::CURRENT.with(|inner| inner.is_legacy()) +} + +#[derive(Debug, Eq, PartialEq, Clone, Hash)] +pub(crate) struct OpCanceller { + pub(super) index: usize, + #[cfg(feature = "legacy")] + pub(super) direction: Option, +} + +impl OpCanceller { + pub(crate) unsafe fn cancel(&self) { + super::CURRENT.with(|inner| inner.cancel_op(self)) + } +} diff --git a/vendor/monoio/src/driver/op/accept.rs b/vendor/monoio/src/driver/op/accept.rs new file mode 100644 index 000000000..e683ed96f --- /dev/null +++ b/vendor/monoio/src/driver/op/accept.rs @@ -0,0 +1,131 @@ +#[cfg(all(unix, any(feature = "legacy", feature = "poll-io")))] +use std::os::unix::prelude::AsRawFd; +use std::{ + io, + mem::{size_of, MaybeUninit}, +}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(windows)] +use { + crate::syscall, + std::os::windows::prelude::AsRawSocket, + windows_sys::Win32::Networking::WinSock::{ + accept, socklen_t, INVALID_SOCKET, SOCKADDR_STORAGE, + }, +}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; +#[cfg(all(unix, any(feature = "legacy", feature = "poll-io")))] +use crate::syscall_u32; + +/// Accept +pub(crate) struct Accept { + pub(crate) fd: SharedFd, + #[cfg(unix)] + pub(crate) addr: Box<(MaybeUninit, libc::socklen_t)>, + #[cfg(windows)] + pub(crate) addr: Box<(MaybeUninit, socklen_t)>, +} + +impl Op { + /// Accept a connection + pub(crate) fn accept(fd: &SharedFd) -> io::Result { + #[cfg(unix)] + let addr = Box::new(( + MaybeUninit::uninit(), + size_of::() as libc::socklen_t, + )); + + #[cfg(windows)] + let addr = Box::new(( + MaybeUninit::uninit(), + size_of::() as socklen_t, + )); + + Op::submit_with(Accept { + fd: fd.clone(), + addr, + }) + } +} + +impl OpAble for Accept { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::Accept::new( + types::Fd(self.fd.raw_fd()), + self.addr.0.as_mut_ptr() as *mut _, + &mut self.addr.1, + ) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd.registered_index().map(|idx| (Direction::Read, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_socket(); + let addr = self.addr.0.as_mut_ptr() as *mut _; + let len = &mut self.addr.1; + + syscall!(accept(fd as _, addr, len), PartialEq::eq, INVALID_SOCKET) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_fd(); + let addr = self.addr.0.as_mut_ptr() as *mut _; + let len = &mut self.addr.1; + // Here I use copied some code from mio because I don't want the conversion. + + // On platforms that support it we can use `accept4(2)` to set `NONBLOCK` + // and `CLOEXEC` in the call to accept the connection. + #[cfg(any( + // Android x86's seccomp profile forbids calls to `accept4(2)` + // See https://github.com/tokio-rs/mio/issues/1445 for details + all( + not(target_arch="x86"), + target_os = "android" + ), + target_os = "dragonfly", + target_os = "freebsd", + target_os = "illumos", + target_os = "linux", + target_os = "netbsd", + target_os = "openbsd" + ))] + return syscall_u32!(accept4( + fd, + addr, + len, + libc::SOCK_CLOEXEC | libc::SOCK_NONBLOCK, + )); + + // But not all platforms have the `accept4(2)` call. Luckily BSD (derived) + // OSes inherit the non-blocking flag from the listener, so we just have to + // set `CLOEXEC`. + #[cfg(any( + all(target_arch = "x86", target_os = "android"), + target_os = "ios", + target_os = "macos", + target_os = "redox" + ))] + return { + let stream_fd = syscall_u32!(accept(fd, addr, len))? as i32; + syscall_u32!(fcntl(stream_fd, libc::F_SETFD, libc::FD_CLOEXEC)) + .and_then(|_| syscall_u32!(fcntl(stream_fd, libc::F_SETFL, libc::O_NONBLOCK))) + .inspect_err(|_| { + let _ = syscall_u32!(close(stream_fd)); + })?; + Ok(stream_fd as _) + }; + } +} diff --git a/vendor/monoio/src/driver/op/close.rs b/vendor/monoio/src/driver/op/close.rs new file mode 100644 index 000000000..0ac7a7317 --- /dev/null +++ b/vendor/monoio/src/driver/op/close.rs @@ -0,0 +1,55 @@ +use std::io; +#[cfg(unix)] +use std::os::unix::io::RawFd; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(windows)] +use { + crate::syscall, std::os::windows::io::RawSocket, + windows_sys::Win32::Networking::WinSock::closesocket, +}; + +use super::{Op, OpAble}; + +pub(crate) struct Close { + #[cfg(unix)] + fd: RawFd, + #[cfg(windows)] + fd: RawSocket, +} + +impl Op { + #[allow(unused)] + #[cfg(unix)] + pub(crate) fn close(fd: RawFd) -> io::Result> { + Op::try_submit_with(Close { fd }) + } + + #[cfg(windows)] + pub(crate) fn close(fd: RawSocket) -> io::Result> { + Op::try_submit_with(Close { fd }) + } +} + +impl OpAble for Close { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::Close::new(types::Fd(self.fd)).build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(crate::driver::ready::Direction, usize)> { + None + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_call(&mut self) -> io::Result { + #[cfg(unix)] + return crate::syscall_u32!(close(self.fd)); + + #[cfg(windows)] + return syscall!(closesocket(self.fd as _), PartialEq::ne, 0); + } +} diff --git a/vendor/monoio/src/driver/op/connect.rs b/vendor/monoio/src/driver/op/connect.rs new file mode 100644 index 000000000..1fa923e98 --- /dev/null +++ b/vendor/monoio/src/driver/op/connect.rs @@ -0,0 +1,302 @@ +use std::{io, net::SocketAddr}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(windows)] +use windows_sys::Win32::Networking::WinSock::{ + connect, socklen_t, AF_INET, AF_INET6, IN6_ADDR, IN6_ADDR_0, IN_ADDR, IN_ADDR_0, SOCKADDR_IN, + SOCKADDR_IN6, SOCKADDR_IN6_0, SOCKET_ERROR, +}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; + +pub(crate) struct Connect { + pub(crate) fd: SharedFd, + socket_addr: Box, + #[cfg(windows)] + socket_addr_len: socklen_t, + #[cfg(unix)] + socket_addr_len: libc::socklen_t, + #[cfg(any(target_os = "ios", target_os = "macos"))] + tfo: bool, +} + +impl Op { + /// Submit a request to connect. + pub(crate) fn connect( + socket: SharedFd, + addr: SocketAddr, + _tfo: bool, + ) -> io::Result> { + let (raw_addr, raw_addr_length) = socket_addr(&addr); + Op::submit_with(Connect { + fd: socket, + socket_addr: Box::new(raw_addr), + socket_addr_len: raw_addr_length, + #[cfg(any(target_os = "ios", target_os = "macos"))] + tfo: _tfo, + }) + } +} + +impl OpAble for Connect { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::Connect::new( + types::Fd(self.fd.raw_fd()), + self.socket_addr.as_ptr(), + self.socket_addr_len, + ) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + None + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_call(&mut self) -> io::Result { + // For ios/macos, if tfo is enabled, we will + // call connectx here. + // For linux/android, we have already set socket + // via set_tcp_fastopen_connect. + #[cfg(any(target_os = "ios", target_os = "macos"))] + if self.tfo { + let mut endpoints: libc::sa_endpoints_t = unsafe { std::mem::zeroed() }; + endpoints.sae_dstaddr = self.socket_addr.as_ptr(); + endpoints.sae_dstaddrlen = self.socket_addr_len; + + return match crate::syscall_u32!(connectx( + self.fd.raw_fd(), + &endpoints as *const _, + libc::SAE_ASSOCID_ANY, + libc::CONNECT_DATA_IDEMPOTENT | libc::CONNECT_RESUME_ON_READ_WRITE, + std::ptr::null(), + 0, + std::ptr::null_mut(), + std::ptr::null_mut(), + )) { + Err(err) if err.raw_os_error() != Some(libc::EINPROGRESS) => Err(err), + _ => Ok(self.fd.raw_fd() as u32), + }; + } + + #[cfg(unix)] + match crate::syscall_u32!(connect( + self.fd.raw_fd(), + self.socket_addr.as_ptr(), + self.socket_addr_len, + )) { + Err(err) if err.raw_os_error() != Some(libc::EINPROGRESS) => Err(err), + _ => Ok(self.fd.raw_fd() as u32), + } + + #[cfg(windows)] + { + let res = unsafe { + connect( + self.fd.raw_socket() as _, + self.socket_addr.as_ptr().cast(), + self.socket_addr_len, + ) + }; + if res == SOCKET_ERROR { + let err = io::Error::last_os_error(); + if err.kind() != io::ErrorKind::WouldBlock { + return Err(err); + } + } + #[allow(clippy::unnecessary_cast)] + Ok(self.fd.raw_socket() as u32) + } + } +} + +#[cfg(unix)] +pub(crate) struct ConnectUnix { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + pub(crate) fd: SharedFd, + socket_addr: Box<(libc::sockaddr_un, libc::socklen_t)>, +} + +#[cfg(unix)] +impl Op { + /// Submit a request to connect. + pub(crate) fn connect_unix( + socket: SharedFd, + socket_addr: libc::sockaddr_un, + socket_len: libc::socklen_t, + ) -> io::Result> { + Op::submit_with(ConnectUnix { + fd: socket, + socket_addr: Box::new((socket_addr, socket_len)), + }) + } +} + +#[cfg(unix)] +impl OpAble for ConnectUnix { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::Connect::new( + types::Fd(self.fd.raw_fd()), + &self.socket_addr.0 as *const _ as *const _, + self.socket_addr.1, + ) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + None + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_call(&mut self) -> io::Result { + match crate::syscall_u32!(connect( + self.fd.raw_fd(), + &self.socket_addr.0 as *const _ as *const _, + self.socket_addr.1 + )) { + Err(err) if err.raw_os_error() != Some(libc::EINPROGRESS) => Err(err), + _ => Ok(self.fd.raw_fd() as u32), + } + } +} + +/// A type with the same memory layout as `libc::sockaddr`. Used in converting Rust level +/// SocketAddr* types into their system representation. The benefit of this specific +/// type over using `libc::sockaddr_storage` is that this type is exactly as large as it +/// needs to be and not a lot larger. And it can be initialized cleaner from Rust. +// Copied from mio. +#[repr(C)] +pub(crate) union SocketAddrCRepr { + #[cfg(unix)] + v4: libc::sockaddr_in, + #[cfg(unix)] + v6: libc::sockaddr_in6, + #[cfg(windows)] + v4: SOCKADDR_IN, + #[cfg(windows)] + v6: SOCKADDR_IN6, +} + +impl SocketAddrCRepr { + pub(crate) fn as_ptr(&self) -> *const libc::sockaddr { + self as *const _ as *const libc::sockaddr + } +} + +#[cfg(windows)] +pub(crate) fn socket_addr(addr: &SocketAddr) -> (SocketAddrCRepr, i32) { + match addr { + SocketAddr::V4(ref addr) => { + // `s_addr` is stored as BE on all machine and the array is in BE order. + // So the native endian conversion method is used so that it's never swapped. + let sin_addr = unsafe { + let mut s_un = std::mem::zeroed::(); + s_un.S_addr = u32::from_ne_bytes(addr.ip().octets()); + IN_ADDR { S_un: s_un } + }; + + let sockaddr_in = SOCKADDR_IN { + sin_family: AF_INET, // 1 + sin_port: addr.port().to_be(), + sin_addr, + sin_zero: [0; 8], + }; + + let sockaddr = SocketAddrCRepr { v4: sockaddr_in }; + (sockaddr, std::mem::size_of::() as i32) + } + SocketAddr::V6(ref addr) => { + let sin6_addr = unsafe { + let mut u = std::mem::zeroed::(); + u.Byte = addr.ip().octets(); + IN6_ADDR { u } + }; + let u = unsafe { + let mut u = std::mem::zeroed::(); + u.sin6_scope_id = addr.scope_id(); + u + }; + + let sockaddr_in6 = SOCKADDR_IN6 { + sin6_family: AF_INET6, // 23 + sin6_port: addr.port().to_be(), + sin6_addr, + sin6_flowinfo: addr.flowinfo(), + Anonymous: u, + }; + + let sockaddr = SocketAddrCRepr { v6: sockaddr_in6 }; + (sockaddr, std::mem::size_of::() as i32) + } + } +} + +#[cfg(unix)] +/// Converts a Rust `SocketAddr` into the system representation. +pub(crate) fn socket_addr(addr: &SocketAddr) -> (SocketAddrCRepr, libc::socklen_t) { + match addr { + SocketAddr::V4(ref addr) => { + // `s_addr` is stored as BE on all machine and the array is in BE order. + // So the native endian conversion method is used so that it's never swapped. + let sin_addr = libc::in_addr { + s_addr: u32::from_ne_bytes(addr.ip().octets()), + }; + + let sockaddr_in = libc::sockaddr_in { + sin_family: libc::AF_INET as libc::sa_family_t, + sin_port: addr.port().to_be(), + sin_addr, + sin_zero: [0; 8], + #[cfg(any( + target_os = "dragonfly", + target_os = "freebsd", + target_os = "ios", + target_os = "macos", + target_os = "netbsd", + target_os = "openbsd" + ))] + sin_len: 0, + }; + + let sockaddr = SocketAddrCRepr { v4: sockaddr_in }; + let socklen = std::mem::size_of::() as libc::socklen_t; + (sockaddr, socklen) + } + SocketAddr::V6(ref addr) => { + let sockaddr_in6 = libc::sockaddr_in6 { + sin6_family: libc::AF_INET6 as libc::sa_family_t, + sin6_port: addr.port().to_be(), + sin6_addr: libc::in6_addr { + s6_addr: addr.ip().octets(), + }, + sin6_flowinfo: addr.flowinfo(), + sin6_scope_id: addr.scope_id(), + #[cfg(any( + target_os = "dragonfly", + target_os = "freebsd", + target_os = "ios", + target_os = "macos", + target_os = "netbsd", + target_os = "openbsd" + ))] + sin6_len: 0, + #[cfg(target_os = "illumos")] + __sin6_src_id: 0, + }; + + let sockaddr = SocketAddrCRepr { v6: sockaddr_in6 }; + let socklen = std::mem::size_of::() as libc::socklen_t; + (sockaddr, socklen) + } + } +} diff --git a/vendor/monoio/src/driver/op/fsync.rs b/vendor/monoio/src/driver/op/fsync.rs new file mode 100644 index 000000000..0dfd1707c --- /dev/null +++ b/vendor/monoio/src/driver/op/fsync.rs @@ -0,0 +1,78 @@ +use std::io; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(windows)] +use { + crate::syscall, std::os::windows::prelude::AsRawHandle, + windows_sys::Win32::Storage::FileSystem::FlushFileBuffers, +}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; +#[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] +use crate::syscall_u32; + +pub(crate) struct Fsync { + #[allow(unused)] + fd: SharedFd, + #[cfg(target_os = "linux")] + data_sync: bool, +} + +impl Op { + pub(crate) fn fsync(fd: &SharedFd) -> io::Result> { + Op::submit_with(Fsync { + fd: fd.clone(), + #[cfg(target_os = "linux")] + data_sync: false, + }) + } + + pub(crate) fn datasync(fd: &SharedFd) -> io::Result> { + Op::submit_with(Fsync { + fd: fd.clone(), + #[cfg(target_os = "linux")] + data_sync: true, + }) + } +} + +impl OpAble for Fsync { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + let mut opc = opcode::Fsync::new(types::Fd(self.fd.raw_fd())); + if self.data_sync { + opc = opc.flags(types::FsyncFlags::DATASYNC) + } + opc.build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + None + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + syscall!( + FlushFileBuffers(self.fd.as_raw_handle() as _), + PartialEq::eq, + 0 + ) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + #[cfg(target_os = "linux")] + if self.data_sync { + syscall_u32!(fdatasync(self.fd.raw_fd())) + } else { + syscall_u32!(fsync(self.fd.raw_fd())) + } + #[cfg(not(target_os = "linux"))] + syscall_u32!(fsync(self.fd.raw_fd())) + } +} diff --git a/vendor/monoio/src/driver/op/mkdir.rs b/vendor/monoio/src/driver/op/mkdir.rs new file mode 100644 index 000000000..be418c511 --- /dev/null +++ b/vendor/monoio/src/driver/op/mkdir.rs @@ -0,0 +1,47 @@ +use std::{ffi::CString, path::Path}; + +use libc::mode_t; + +use super::{Op, OpAble}; +use crate::driver::util::cstr; + +pub(crate) struct MkDir { + path: CString, + mode: mode_t, +} + +impl Op { + pub(crate) fn mkdir>(path: P, mode: mode_t) -> std::io::Result> { + let path = cstr(path.as_ref())?; + Op::submit_with(MkDir { path, mode }) + } +} + +impl OpAble for MkDir { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + use io_uring::{opcode, types}; + + opcode::MkDirAt::new(types::Fd(libc::AT_FDCWD), self.path.as_ptr()) + .mode(self.mode) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(crate::driver::ready::Direction, usize)> { + None + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> std::io::Result { + use crate::syscall_u32; + + syscall_u32!(mkdirat(libc::AT_FDCWD, self.path.as_ptr(), self.mode)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + unimplemented!() + } +} diff --git a/vendor/monoio/src/driver/op/open.rs b/vendor/monoio/src/driver/op/open.rs new file mode 100644 index 000000000..1cb81dbdf --- /dev/null +++ b/vendor/monoio/src/driver/op/open.rs @@ -0,0 +1,96 @@ +use std::{ffi::CString, io, path::Path}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(windows)] +use windows_sys::Win32::{Foundation::INVALID_HANDLE_VALUE, Storage::FileSystem::CreateFileW}; + +use super::{Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; +#[cfg(windows)] +use crate::syscall; +#[cfg(all(unix, any(feature = "legacy", feature = "poll-io")))] +use crate::syscall_u32; +use crate::{driver::util::cstr, fs::OpenOptions}; + +/// Open a file +pub(crate) struct Open { + pub(crate) path: CString, + #[cfg(unix)] + flags: i32, + #[cfg(unix)] + mode: libc::mode_t, + #[cfg(windows)] + opts: OpenOptions, +} + +impl Op { + #[cfg(unix)] + /// Submit a request to open a file. + pub(crate) fn open>(path: P, options: &OpenOptions) -> io::Result> { + // Here the path will be copied, so its safe. + let path = cstr(path.as_ref())?; + let flags = libc::O_CLOEXEC + | options.access_mode()? + | options.creation_mode()? + | (options.custom_flags & !libc::O_ACCMODE); + let mode = options.mode; + + Op::submit_with(Open { path, flags, mode }) + } + + #[cfg(windows)] + /// Submit a request to open a file. + pub(crate) fn open>(path: P, options: &OpenOptions) -> io::Result> { + // Here the path will be copied, so its safe. + let path = cstr(path.as_ref())?; + + Op::submit_with(Open { + path, + opts: options.clone(), + }) + } +} + +impl OpAble for Open { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::OpenAt::new(types::Fd(libc::AT_FDCWD), self.path.as_c_str().as_ptr()) + .flags(self.flags) + .mode(self.mode) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + None + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), not(windows)))] + fn legacy_call(&mut self) -> io::Result { + syscall_u32!(open( + self.path.as_c_str().as_ptr(), + self.flags, + self.mode as libc::c_int + )) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + syscall!( + CreateFileW( + self.path.as_c_str().as_ptr().cast(), + self.opts.access_mode()?, + self.opts.share_mode, + self.opts.security_attributes, + self.opts.creation_mode()?, + self.opts.get_flags_and_attributes(), + 0, + ), + PartialEq::eq, + INVALID_HANDLE_VALUE + ) + } +} diff --git a/vendor/monoio/src/driver/op/poll.rs b/vendor/monoio/src/driver/op/poll.rs new file mode 100644 index 000000000..56aaae0ef --- /dev/null +++ b/vendor/monoio/src/driver/op/poll.rs @@ -0,0 +1,130 @@ +use std::io; +#[cfg(windows)] +use std::{ + io::{Error, ErrorKind}, + os::windows::prelude::AsRawSocket, +}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(windows)] +use windows_sys::Win32::Networking::WinSock::{ + WSAGetLastError, WSAPoll, POLLIN, POLLOUT, SOCKET_ERROR, WSAPOLLFD, +}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; + +pub(crate) struct PollAdd { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + // true: read; false: write + is_read: bool, + #[cfg(any(feature = "legacy", feature = "poll-io"))] + relaxed: bool, +} + +impl Op { + pub(crate) fn poll_read(fd: &SharedFd, _relaxed: bool) -> io::Result> { + Op::submit_with(PollAdd { + fd: fd.clone(), + is_read: true, + #[cfg(any(feature = "legacy", feature = "poll-io"))] + relaxed: _relaxed, + }) + } + + pub(crate) fn poll_write(fd: &SharedFd, _relaxed: bool) -> io::Result> { + Op::submit_with(PollAdd { + fd: fd.clone(), + is_read: false, + #[cfg(any(feature = "legacy", feature = "poll-io"))] + relaxed: _relaxed, + }) + } + + pub(crate) async fn wait(self) -> io::Result<()> { + let complete = self.await; + complete.meta.result.map(|_| ()) + } +} + +impl OpAble for PollAdd { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::PollAdd::new( + types::Fd(self.fd.raw_fd()), + if self.is_read { + libc::POLLIN as _ + } else { + libc::POLLOUT as _ + }, + ) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd.registered_index().map(|idx| { + ( + if self.is_read { + Direction::Read + } else { + Direction::Write + }, + idx, + ) + }) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), not(windows)))] + fn legacy_call(&mut self) -> io::Result { + if !self.relaxed { + use std::{io::ErrorKind, os::fd::AsRawFd}; + + let mut pollfd = libc::pollfd { + fd: self.fd.as_raw_fd(), + events: if self.is_read { + libc::POLLIN as _ + } else { + libc::POLLOUT as _ + }, + revents: 0, + }; + let ret = crate::syscall_u32!(poll(&mut pollfd as *mut _, 1, 0))?; + if ret == 0 { + return Err(ErrorKind::WouldBlock.into()); + } + } + Ok(0) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + if !self.relaxed { + let mut pollfd = WSAPOLLFD { + fd: self.fd.as_raw_socket() as _, + events: if self.is_read { + POLLIN as _ + } else { + POLLOUT as _ + }, + revents: 0, + }; + let ret = unsafe { WSAPoll(&mut pollfd as *mut _, 1, 0) }; + match ret { + 0 => return Err(ErrorKind::WouldBlock.into()), + SOCKET_ERROR => { + let error = unsafe { WSAGetLastError() }; + return Err(Error::from_raw_os_error(error)); + } + _ => (), + } + } + Ok(0) + } +} diff --git a/vendor/monoio/src/driver/op/read.rs b/vendor/monoio/src/driver/op/read.rs new file mode 100644 index 000000000..eff9d7274 --- /dev/null +++ b/vendor/monoio/src/driver/op/read.rs @@ -0,0 +1,214 @@ +use std::io; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(all(unix, any(feature = "legacy", feature = "poll-io")))] +use {crate::syscall_u32, std::os::unix::prelude::AsRawFd}; +#[cfg(all(windows, any(feature = "legacy", feature = "poll-io")))] +use { + std::ffi::c_void, + windows_sys::Win32::{ + Foundation::TRUE, + Networking::WinSock::{WSAGetLastError, WSARecv, WSAESHUTDOWN}, + Storage::FileSystem::{ReadFile, SetFilePointer, FILE_CURRENT, INVALID_SET_FILE_POINTER}, + }, +}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; +use crate::{ + buf::{IoBufMut, IoVecBufMut}, + BufResult, +}; + +pub(crate) struct Read { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + offset: u64, + + /// Reference to the in-flight buffer. + pub(crate) buf: T, +} + +impl Op> { + pub(crate) fn read_at(fd: &SharedFd, buf: T, offset: u64) -> io::Result>> { + Op::submit_with(Read { + fd: fd.clone(), + offset, + buf, + }) + } + + pub(crate) async fn read(self) -> BufResult { + let complete = self.await; + + // Convert the operation result to `usize` + let res = complete.meta.result.map(|v| v as usize); + // Recover the buffer + let mut buf = complete.data.buf; + + // If the operation was successful, advance the initialized cursor. + if let Ok(n) = res { + // Safety: the kernel wrote `n` bytes to the buffer. + unsafe { + buf.set_init(n); + } + } + + (res, buf) + } +} + +impl OpAble for Read { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::Read::new( + types::Fd(self.fd.raw_fd()), + self.buf.write_ptr(), + self.buf.bytes_total() as _, + ) + .offset(self.offset) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd.registered_index().map(|idx| (Direction::Read, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_fd(); + let seek_offset = libc::off_t::try_from(self.offset) + .map_err(|_| io::Error::new(io::ErrorKind::Other, "offset too big"))?; + #[cfg(not(target_os = "macos"))] + return syscall_u32!(pread64( + fd, + self.buf.write_ptr() as _, + self.buf.bytes_total(), + seek_offset as _ + )); + + #[cfg(target_os = "macos")] + return syscall_u32!(pread( + fd, + self.buf.write_ptr() as _, + self.buf.bytes_total(), + seek_offset + )); + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.raw_handle() as _; + let seek_offset = libc::off_t::try_from(self.offset) + .map_err(|_| io::Error::new(io::ErrorKind::Other, "offset too big"))?; + let mut bytes_read = 0; + let ret = unsafe { + // see https://learn.microsoft.com/zh-cn/windows/win32/api/fileapi/nf-fileapi-setfilepointer + if seek_offset != 0 { + let r = SetFilePointer(fd, seek_offset, std::ptr::null_mut(), FILE_CURRENT); + if INVALID_SET_FILE_POINTER == r { + return Err(io::Error::last_os_error()); + } + } + // see https://learn.microsoft.com/zh-cn/windows/win32/api/fileapi/nf-fileapi-readfile + ReadFile( + fd, + self.buf.write_ptr().cast::(), + self.buf.bytes_total() as u32, + &mut bytes_read, + std::ptr::null_mut(), + ) + }; + if TRUE == ret { + Ok(bytes_read) + } else { + Err(io::Error::last_os_error()) + } + } +} + +pub(crate) struct ReadVec { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + + /// Reference to the in-flight buffer. + pub(crate) buf_vec: T, +} + +impl Op> { + pub(crate) fn readv(fd: SharedFd, buf_vec: T) -> io::Result { + Op::submit_with(ReadVec { fd, buf_vec }) + } + + pub(crate) async fn read(self) -> BufResult { + let complete = self.await; + let res = complete.meta.result.map(|v| v as _); + let mut buf_vec = complete.data.buf_vec; + + if let Ok(n) = res { + // Safety: the kernel wrote `n` bytes to the buffer. + unsafe { buf_vec.set_init(n) }; + } + (res, buf_vec) + } +} + +impl OpAble for ReadVec { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + let ptr = self.buf_vec.write_iovec_ptr() as _; + let len = self.buf_vec.write_iovec_len() as _; + opcode::Readv::new(types::Fd(self.fd.raw_fd()), ptr, len).build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd.registered_index().map(|idx| (Direction::Read, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + syscall_u32!(readv( + self.fd.raw_fd(), + self.buf_vec.write_iovec_ptr(), + self.buf_vec.write_iovec_len().min(i32::MAX as usize) as _ + )) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let mut nread = 0; + let mut flags = 0; + let ret = unsafe { + WSARecv( + self.fd.raw_socket() as _, + self.buf_vec.write_wsabuf_ptr(), + self.buf_vec.write_wsabuf_len().min(u32::MAX as usize) as _, + &mut nread, + &mut flags, + std::ptr::null_mut(), + None, + ) + }; + match ret { + 0 => Ok(nread), + _ => { + let error = unsafe { WSAGetLastError() }; + if error == WSAESHUTDOWN { + Ok(0) + } else { + Err(io::Error::from_raw_os_error(error)) + } + } + } + } +} diff --git a/vendor/monoio/src/driver/op/recv.rs b/vendor/monoio/src/driver/op/recv.rs new file mode 100644 index 000000000..a995695d0 --- /dev/null +++ b/vendor/monoio/src/driver/op/recv.rs @@ -0,0 +1,361 @@ +use std::{ + io, + mem::{transmute, MaybeUninit}, + net::{Ipv4Addr, Ipv6Addr, SocketAddr, SocketAddrV4, SocketAddrV6}, +}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(unix)] +use { + crate::net::unix::SocketAddr as UnixSocketAddr, + libc::{sockaddr_in, sockaddr_in6, sockaddr_storage, socklen_t, AF_INET, AF_INET6}, +}; +#[cfg(all(windows, any(feature = "legacy", feature = "poll-io")))] +use { + crate::syscall, + std::os::windows::io::AsRawSocket, + windows_sys::Win32::Networking::WinSock::recv, + windows_sys::{ + core::GUID, + Win32::{ + Networking::WinSock::{ + WSAGetLastError, WSAIoctl, AF_INET, AF_INET6, LPFN_WSARECVMSG, + LPWSAOVERLAPPED_COMPLETION_ROUTINE, SIO_GET_EXTENSION_FUNCTION_POINTER, SOCKADDR, + SOCKADDR_IN as sockaddr_in, SOCKADDR_IN6 as sockaddr_in6, + SOCKADDR_STORAGE as sockaddr_storage, SOCKET, SOCKET_ERROR, WSAID_WSARECVMSG, + WSAMSG, + }, + System::IO::OVERLAPPED, + }, + }, +}; +#[cfg(all(unix, any(feature = "legacy", feature = "poll-io")))] +use {crate::syscall_u32, std::os::unix::prelude::AsRawFd}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; +use crate::{ + buf::{IoBufMut, IoVecBufMut, IoVecMeta, MsgMeta}, + BufResult, +}; + +pub(crate) struct Recv { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + + /// Reference to the in-flight buffer. + pub(crate) buf: T, +} + +impl Op> { + pub(crate) fn recv(fd: SharedFd, buf: T) -> io::Result { + Op::submit_with(Recv { fd, buf }) + } + + #[allow(unused)] + pub(crate) fn recv_raw(fd: &SharedFd, buf: T) -> Recv { + Recv { + fd: fd.clone(), + buf, + } + } + + pub(crate) async fn read(self) -> BufResult { + let complete = self.await; + let res = complete.meta.result.map(|v| v as _); + let mut buf = complete.data.buf; + + if let Ok(n) = res { + // Safety: the kernel wrote `n` bytes to the buffer. + unsafe { + buf.set_init(n); + } + } + (res, buf) + } +} + +impl OpAble for Recv { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::Recv::new( + types::Fd(self.fd.raw_fd()), + self.buf.write_ptr(), + self.buf.bytes_total() as _, + ) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd.registered_index().map(|idx| (Direction::Read, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_fd(); + syscall_u32!(recv( + fd, + self.buf.write_ptr() as _, + self.buf.bytes_total().min(u32::MAX as usize), + 0 + )) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_socket(); + syscall!( + recv( + fd as _, + self.buf.write_ptr(), + self.buf.bytes_total().min(i32::MAX as usize) as _, + 0 + ), + PartialOrd::lt, + 0 + ) + } +} + +pub(crate) struct RecvMsg { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + + /// Reference to the in-flight buffer. + pub(crate) buf: T, + /// For multiple message recv in the future + pub(crate) info: Box<(MaybeUninit, IoVecMeta, MsgMeta)>, +} + +impl Op> { + pub(crate) fn recv_msg(fd: SharedFd, mut buf: T) -> io::Result { + let mut info: Box<(MaybeUninit, IoVecMeta, MsgMeta)> = + Box::new((MaybeUninit::uninit(), IoVecMeta::from(&mut buf), unsafe { + std::mem::zeroed() + })); + + #[cfg(unix)] + { + info.2.msg_iov = info.1.write_iovec_ptr(); + info.2.msg_iovlen = info.1.write_iovec_len() as _; + info.2.msg_name = &mut info.0 as *mut _ as *mut libc::c_void; + info.2.msg_namelen = std::mem::size_of::() as socklen_t; + } + #[cfg(windows)] + { + info.2.lpBuffers = info.1.write_wsabuf_ptr(); + info.2.dwBufferCount = info.1.write_wsabuf_len() as _; + info.2.name = &mut info.0 as *mut _ as *mut SOCKADDR; + info.2.namelen = std::mem::size_of::() as _; + } + + Op::submit_with(RecvMsg { fd, buf, info }) + } + + pub(crate) async fn wait(self) -> BufResult<(usize, SocketAddr), T> { + let complete = self.await; + let res = complete.meta.result.map(|v| v as _); + let mut buf = complete.data.buf; + + let res = res.map(|n| { + let storage = unsafe { complete.data.info.0.assume_init() }; + + let addr = unsafe { + match storage.ss_family as _ { + AF_INET => { + // Safety: if the ss_family field is AF_INET then storage must be a + // sockaddr_in. + let addr: &sockaddr_in = transmute(&storage); + #[cfg(unix)] + let ip = Ipv4Addr::from(addr.sin_addr.s_addr.to_ne_bytes()); + #[cfg(windows)] + let ip = Ipv4Addr::from(addr.sin_addr.S_un.S_addr.to_ne_bytes()); + let port = u16::from_be(addr.sin_port); + SocketAddr::V4(SocketAddrV4::new(ip, port)) + } + AF_INET6 => { + // Safety: if the ss_family field is AF_INET6 then storage must be a + // sockaddr_in6. + let addr: &sockaddr_in6 = transmute(&storage); + #[cfg(unix)] + let ip = Ipv6Addr::from(addr.sin6_addr.s6_addr); + #[cfg(windows)] + let ip = Ipv6Addr::from(addr.sin6_addr.u.Byte); + let port = u16::from_be(addr.sin6_port); + #[cfg(unix)] + let scope_id = addr.sin6_scope_id; + #[cfg(windows)] + let scope_id = addr.Anonymous.sin6_scope_id; + SocketAddr::V6(SocketAddrV6::new(ip, port, addr.sin6_flowinfo, scope_id)) + } + _ => { + unreachable!() + } + } + }; + + // Safety: the kernel wrote `n` bytes to the buffer. + unsafe { buf.set_init(n) }; + + (n, addr) + }); + (res, buf) + } +} + +/// see https://github.com/microsoft/windows-rs/issues/2530 +#[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] +static WSA_RECV_MSG: std::sync::OnceLock< + unsafe extern "system" fn( + SOCKET, + *mut WSAMSG, + *mut u32, + *mut OVERLAPPED, + LPWSAOVERLAPPED_COMPLETION_ROUTINE, + ) -> i32, +> = std::sync::OnceLock::new(); + +impl OpAble for RecvMsg { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::RecvMsg::new(types::Fd(self.fd.raw_fd()), &mut *self.info.2).build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd.registered_index().map(|idx| (Direction::Read, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_fd(); + syscall_u32!(recvmsg(fd, &mut *self.info.2, 0)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_socket() as _; + let func_ptr = WSA_RECV_MSG.get_or_init(|| unsafe { + let mut wsa_recv_msg: LPFN_WSARECVMSG = None; + let mut dw_bytes = 0; + let r = WSAIoctl( + fd, + SIO_GET_EXTENSION_FUNCTION_POINTER, + &WSAID_WSARECVMSG as *const _ as *const std::ffi::c_void, + std::mem::size_of:: as usize as u32, + &mut wsa_recv_msg as *mut _ as *mut std::ffi::c_void, + std::mem::size_of::() as _, + &mut dw_bytes, + std::ptr::null_mut(), + None, + ); + if r == SOCKET_ERROR || wsa_recv_msg.is_none() { + panic!( + "init WSARecvMsg failed with {}", + io::Error::from_raw_os_error(WSAGetLastError()) + ) + } else { + assert_eq!(dw_bytes, std::mem::size_of::() as _); + wsa_recv_msg.unwrap() + } + }); + let mut recved = 0; + let r = unsafe { + (func_ptr)( + fd, + &mut *self.info.2, + &mut recved, + std::ptr::null_mut(), + None, + ) + }; + if r == SOCKET_ERROR { + unsafe { Err(io::Error::from_raw_os_error(WSAGetLastError())) } + } else { + Ok(recved) + } + } +} + +#[cfg(unix)] +pub(crate) struct RecvMsgUnix { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + + /// Reference to the in-flight buffer. + pub(crate) buf: T, + /// For multiple message recv in the future + pub(crate) info: Box<(MaybeUninit, IoVecMeta, libc::msghdr)>, +} + +#[cfg(unix)] +impl Op> { + pub(crate) fn recv_msg_unix(fd: SharedFd, mut buf: T) -> io::Result { + let mut info: Box<(MaybeUninit, IoVecMeta, libc::msghdr)> = + Box::new((MaybeUninit::uninit(), IoVecMeta::from(&mut buf), unsafe { + std::mem::zeroed() + })); + + info.2.msg_iov = info.1.write_iovec_ptr(); + info.2.msg_iovlen = info.1.write_iovec_len() as _; + info.2.msg_name = &mut info.0 as *mut _ as *mut libc::c_void; + info.2.msg_namelen = std::mem::size_of::() as socklen_t; + + Op::submit_with(RecvMsgUnix { fd, buf, info }) + } + + pub(crate) async fn wait(self) -> BufResult<(usize, UnixSocketAddr), T> { + let complete = self.await; + let res = complete.meta.result.map(|v| v as _); + let mut buf = complete.data.buf; + + let res = res.map(|n| { + let storage = unsafe { complete.data.info.0.assume_init() }; + let name_len = complete.data.info.2.msg_namelen; + + let addr = unsafe { + let addr: &libc::sockaddr_un = transmute(&storage); + UnixSocketAddr::from_parts(*addr, name_len) + }; + + // Safety: the kernel wrote `n` bytes to the buffer. + unsafe { + buf.set_init(n); + } + + (n, addr) + }); + (res, buf) + } +} + +#[cfg(unix)] +impl OpAble for RecvMsgUnix { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::RecvMsg::new(types::Fd(self.fd.raw_fd()), &mut self.info.2 as *mut _).build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd.registered_index().map(|idx| (Direction::Read, idx)) + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_fd(); + syscall_u32!(recvmsg(fd, &mut self.info.2 as *mut _, 0)) + } +} diff --git a/vendor/monoio/src/driver/op/rename.rs b/vendor/monoio/src/driver/op/rename.rs new file mode 100644 index 000000000..df9719a02 --- /dev/null +++ b/vendor/monoio/src/driver/op/rename.rs @@ -0,0 +1,55 @@ +use std::{ffi::CString, path::Path}; + +use super::{Op, OpAble}; +use crate::driver::util::cstr; + +pub(crate) struct Rename { + from: CString, + to: CString, +} + +impl Op { + pub(crate) fn rename(from: &Path, to: &Path) -> std::io::Result { + let from = cstr(from)?; + let to = cstr(to)?; + + Op::submit_with(Rename { from, to }) + } +} + +impl OpAble for Rename { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + use io_uring::{opcode::RenameAt, types}; + use libc::AT_FDCWD; + + RenameAt::new( + types::Fd(AT_FDCWD), + self.from.as_ptr(), + types::Fd(AT_FDCWD), + self.to.as_ptr(), + ) + .build() + } + + fn legacy_interest(&self) -> Option<(crate::driver::ready::Direction, usize)> { + None + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> std::io::Result { + use crate::syscall_u32; + + syscall_u32!(renameat( + libc::AT_FDCWD, + self.from.as_ptr(), + libc::AT_FDCWD, + self.to.as_ptr() + )) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + unimplemented!() + } +} diff --git a/vendor/monoio/src/driver/op/send.rs b/vendor/monoio/src/driver/op/send.rs new file mode 100644 index 000000000..3c5f99bcf --- /dev/null +++ b/vendor/monoio/src/driver/op/send.rs @@ -0,0 +1,321 @@ +use std::{io, net::SocketAddr}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +use socket2::SockAddr; +#[cfg(all(windows, any(feature = "legacy", feature = "poll-io")))] +use { + crate::syscall, + std::os::windows::io::AsRawSocket, + windows_sys::Win32::Networking::WinSock::{send, WSASendMsg, SOCKET_ERROR}, +}; +#[cfg(all(unix, any(feature = "legacy", feature = "poll-io")))] +use {crate::syscall_u32, std::os::unix::prelude::AsRawFd}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; +#[cfg(unix)] +use crate::net::unix::SocketAddr as UnixSocketAddr; +use crate::{ + buf::{IoBuf, IoVecBufMut, IoVecMeta, MsgMeta}, + BufResult, +}; + +pub(crate) struct Send { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + + pub(crate) buf: T, +} + +impl Op> { + pub(crate) fn send(fd: SharedFd, buf: T) -> io::Result { + Op::submit_with(Send { fd, buf }) + } + + #[allow(unused)] + pub(crate) fn send_raw(fd: &SharedFd, buf: T) -> Send { + Send { + fd: fd.clone(), + buf, + } + } + + pub(crate) async fn write(self) -> BufResult { + let complete = self.await; + (complete.meta.result.map(|v| v as _), complete.data.buf) + } +} + +impl OpAble for Send { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + #[allow(deprecated)] + #[cfg(feature = "zero-copy")] + fn zero_copy_flag_guard(buf: &T) -> libc::c_int { + // TODO: use libc const after supported. + const MSG_ZEROCOPY: libc::c_int = 0x4000000; + // According to Linux's documentation, zero copy introduces extra overhead and + // is only considered effective for at writes over around 10 KB. + // see also: https://www.kernel.org/doc/html/v4.16/networking/msg_zerocopy.html + const MSG_ZEROCOPY_THRESHOLD: usize = 10 * 1024 * 1024; + if buf.bytes_init() >= MSG_ZEROCOPY_THRESHOLD { + libc::MSG_NOSIGNAL as libc::c_int | MSG_ZEROCOPY + } else { + libc::MSG_NOSIGNAL as libc::c_int + } + } + + #[cfg(feature = "zero-copy")] + let flags = zero_copy_flag_guard(&self.buf); + #[cfg(not(feature = "zero-copy"))] + #[allow(deprecated)] + let flags = libc::MSG_NOSIGNAL as libc::c_int; + + opcode::Send::new( + types::Fd(self.fd.raw_fd()), + self.buf.read_ptr(), + self.buf.bytes_init() as _, + ) + .flags(flags) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd + .registered_index() + .map(|idx| (Direction::Write, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_fd(); + #[cfg(target_os = "linux")] + #[allow(deprecated)] + let flags = libc::MSG_NOSIGNAL as _; + #[cfg(not(target_os = "linux"))] + let flags = 0; + + syscall_u32!(send( + fd, + self.buf.read_ptr() as _, + self.buf.bytes_init(), + flags + )) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_socket(); + syscall!( + send(fd as _, self.buf.read_ptr(), self.buf.bytes_init() as _, 0), + PartialOrd::lt, + 0 + ) + } +} + +pub(crate) struct SendMsg { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + + /// Reference to the in-flight buffer. + pub(crate) buf: T, + /// For multiple message send in the future + pub(crate) info: Box<(Option, IoVecMeta, MsgMeta)>, +} + +impl Op> { + pub(crate) fn send_msg( + fd: SharedFd, + buf: T, + socket_addr: Option, + ) -> io::Result { + let mut info: Box<(Option, IoVecMeta, MsgMeta)> = Box::new(( + socket_addr.map(Into::into), + IoVecMeta::from(&buf), + unsafe { std::mem::zeroed() }, + )); + + #[cfg(unix)] + { + info.2.msg_iov = info.1.write_iovec_ptr(); + info.2.msg_iovlen = info.1.write_iovec_len() as _; + match info.0.as_ref() { + Some(socket_addr) => { + info.2.msg_name = socket_addr.as_ptr() as *mut libc::c_void; + info.2.msg_namelen = socket_addr.len(); + } + None => { + info.2.msg_name = std::ptr::null_mut(); + info.2.msg_namelen = 0; + } + } + } + #[cfg(windows)] + { + info.2.lpBuffers = info.1.write_wsabuf_ptr(); + info.2.dwBufferCount = info.1.write_wsabuf_len() as _; + match info.0.as_ref() { + Some(socket_addr) => { + info.2.name = socket_addr.as_ptr() as *mut _; + info.2.namelen = socket_addr.len(); + } + None => { + info.2.name = std::ptr::null_mut(); + info.2.namelen = 0; + } + } + } + + Op::submit_with(SendMsg { fd, buf, info }) + } + + pub(crate) async fn wait(self) -> BufResult { + let complete = self.await; + let res = complete.meta.result.map(|v| v as _); + let buf = complete.data.buf; + (res, buf) + } +} + +impl OpAble for SendMsg { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + #[allow(deprecated)] + const FLAGS: u32 = libc::MSG_NOSIGNAL as u32; + opcode::SendMsg::new(types::Fd(self.fd.raw_fd()), &*self.info.2) + .flags(FLAGS) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd + .registered_index() + .map(|idx| (Direction::Write, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + #[cfg(target_os = "linux")] + #[allow(deprecated)] + const FLAGS: libc::c_int = libc::MSG_NOSIGNAL as libc::c_int; + #[cfg(not(target_os = "linux"))] + const FLAGS: libc::c_int = 0; + let fd = self.fd.as_raw_fd(); + syscall_u32!(sendmsg(fd, &*self.info.2, FLAGS)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_socket(); + let mut nsent = 0; + let ret = unsafe { + WSASendMsg( + fd as _, + &*self.info.2, + 0, + &mut nsent, + std::ptr::null_mut(), + None, + ) + }; + if ret == SOCKET_ERROR { + Err(io::Error::last_os_error()) + } else { + Ok(nsent) + } + } +} + +#[cfg(unix)] +pub(crate) struct SendMsgUnix { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + + /// Reference to the in-flight buffer. + pub(crate) buf: T, + /// For multiple message send in the future + pub(crate) info: Box<(Option, IoVecMeta, libc::msghdr)>, +} + +#[cfg(unix)] +impl Op> { + pub(crate) fn send_msg_unix( + fd: SharedFd, + buf: T, + socket_addr: Option, + ) -> io::Result { + let mut info: Box<(Option, IoVecMeta, libc::msghdr)> = Box::new(( + socket_addr.map(Into::into), + IoVecMeta::from(&buf), + unsafe { std::mem::zeroed() }, + )); + + info.2.msg_iov = info.1.write_iovec_ptr(); + info.2.msg_iovlen = info.1.write_iovec_len() as _; + + match info.0.as_ref() { + Some(socket_addr) => { + info.2.msg_name = socket_addr.as_ptr() as *mut libc::c_void; + info.2.msg_namelen = socket_addr.len(); + } + None => { + info.2.msg_name = std::ptr::null_mut(); + info.2.msg_namelen = 0; + } + } + + Op::submit_with(SendMsgUnix { fd, buf, info }) + } + + pub(crate) async fn wait(self) -> BufResult { + let complete = self.await; + let res = complete.meta.result.map(|v| v as _); + let buf = complete.data.buf; + (res, buf) + } +} + +#[cfg(unix)] +impl OpAble for SendMsgUnix { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + #[allow(deprecated)] + const FLAGS: u32 = libc::MSG_NOSIGNAL as u32; + opcode::SendMsg::new(types::Fd(self.fd.raw_fd()), &mut self.info.2 as *mut _) + .flags(FLAGS) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd + .registered_index() + .map(|idx| (Direction::Write, idx)) + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_call(&mut self) -> io::Result { + #[cfg(target_os = "linux")] + #[allow(deprecated)] + const FLAGS: libc::c_int = libc::MSG_NOSIGNAL as libc::c_int; + #[cfg(not(target_os = "linux"))] + const FLAGS: libc::c_int = 0; + let fd = self.fd.as_raw_fd(); + syscall_u32!(sendmsg(fd, &mut self.info.2 as *mut _, FLAGS)) + } +} diff --git a/vendor/monoio/src/driver/op/splice.rs b/vendor/monoio/src/driver/op/splice.rs new file mode 100644 index 000000000..21940576d --- /dev/null +++ b/vendor/monoio/src/driver/op/splice.rs @@ -0,0 +1,106 @@ +//! This module works only on linux. + +use std::io; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(all(unix, feature = "legacy"))] +use { + crate::{driver::ready::Direction, syscall_u32}, + std::os::unix::prelude::AsRawFd, +}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; + +// Currently our Splice does not support setting offset. +pub(crate) struct Splice { + fd_in: SharedFd, + fd_out: SharedFd, + len: u32, + direction: SpliceDirection, +} +enum SpliceDirection { + FromPipe, + ToPipe, +} + +impl Op { + pub(crate) fn splice_to_pipe( + fd_in: &SharedFd, + fd_out: &SharedFd, + len: u32, + ) -> io::Result> { + Op::submit_with(Splice { + fd_in: fd_in.clone(), + fd_out: fd_out.clone(), + len, + direction: SpliceDirection::ToPipe, + }) + } + + pub(crate) fn splice_from_pipe( + fd_in: &SharedFd, + fd_out: &SharedFd, + len: u32, + ) -> io::Result> { + Op::submit_with(Splice { + fd_in: fd_in.clone(), + fd_out: fd_out.clone(), + len, + direction: SpliceDirection::FromPipe, + }) + } + + pub(crate) async fn splice(self) -> io::Result { + let complete = self.await; + complete.meta.result + } +} + +impl OpAble for Splice { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + const FLAG: u32 = libc::SPLICE_F_MOVE; + opcode::Splice::new( + types::Fd(self.fd_in.raw_fd()), + -1, + types::Fd(self.fd_out.raw_fd()), + -1, + self.len, + ) + .flags(FLAG) + .build() + } + + #[cfg(all(unix, feature = "legacy"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + match self.direction { + SpliceDirection::FromPipe => self + .fd_out + .registered_index() + .map(|idx| (Direction::Write, idx)), + SpliceDirection::ToPipe => self + .fd_in + .registered_index() + .map(|idx| (Direction::Read, idx)), + } + } + + #[cfg(all(unix, feature = "legacy"))] + fn legacy_call(&mut self) -> io::Result { + const FLAG: u32 = libc::SPLICE_F_MOVE | libc::SPLICE_F_NONBLOCK; + let fd_in = self.fd_in.as_raw_fd(); + let fd_out = self.fd_out.as_raw_fd(); + let off_in = std::ptr::null_mut::(); + let off_out = std::ptr::null_mut::(); + syscall_u32!(splice( + fd_in, + off_in, + fd_out, + off_out, + self.len as usize, + FLAG + )) + } +} diff --git a/vendor/monoio/src/driver/op/statx.rs b/vendor/monoio/src/driver/op/statx.rs new file mode 100644 index 000000000..97a365fcb --- /dev/null +++ b/vendor/monoio/src/driver/op/statx.rs @@ -0,0 +1,212 @@ +use std::{ffi::CString, mem::MaybeUninit, path::Path}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(target_os = "linux")] +use libc::statx; + +use super::{Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; +use crate::driver::{shared_fd::SharedFd, util::cstr}; + +#[derive(Debug)] +pub(crate) struct Statx { + inner: T, + #[cfg(target_os = "linux")] + flags: i32, + #[cfg(target_os = "linux")] + statx_buf: Box>, + #[cfg(target_os = "macos")] + stat_buf: Box>, + #[cfg(target_os = "macos")] + follow_symlinks: bool, +} + +type FdStatx = Statx; + +impl Op { + /// submit a statx operation + #[cfg(target_os = "linux")] + pub(crate) fn statx_using_fd(fd: &SharedFd, flags: i32) -> std::io::Result { + Op::submit_with(Statx { + inner: fd.clone(), + flags, + statx_buf: Box::new(MaybeUninit::uninit()), + }) + } + + #[cfg(target_os = "linux")] + pub(crate) async fn statx_result(self) -> std::io::Result { + let complete = self.await; + complete.meta.result?; + + Ok(unsafe { MaybeUninit::assume_init(*complete.data.statx_buf) }) + } + + #[cfg(target_os = "macos")] + pub(crate) fn statx_using_fd(fd: &SharedFd, follow_symlinks: bool) -> std::io::Result { + Op::submit_with(Statx { + inner: fd.clone(), + follow_symlinks, + stat_buf: Box::new(MaybeUninit::uninit()), + }) + } + + #[cfg(target_os = "macos")] + pub(crate) async fn statx_result(self) -> std::io::Result { + let complete = self.await; + complete.meta.result?; + + Ok(unsafe { MaybeUninit::assume_init(*complete.data.stat_buf) }) + } +} + +impl OpAble for FdStatx { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + use std::os::fd::AsRawFd; + + let statxbuf = self.statx_buf.as_mut_ptr() as *mut _; + + opcode::Statx::new(types::Fd(self.inner.as_raw_fd()), c"".as_ptr(), statxbuf) + .flags(libc::AT_EMPTY_PATH | libc::AT_STATX_SYNC_AS_STAT) + .mask(libc::STATX_ALL) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_interest(&self) -> Option<(crate::driver::ready::Direction, usize)> { + self.inner + .registered_index() + .map(|idx| (Direction::Read, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), target_os = "linux"))] + fn legacy_call(&mut self) -> std::io::Result { + use std::os::fd::AsRawFd; + + use crate::syscall_u32; + + syscall_u32!(statx( + self.inner.as_raw_fd(), + c"".as_ptr(), + libc::AT_EMPTY_PATH, + libc::STATX_ALL, + self.statx_buf.as_mut_ptr() as *mut _ + )) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> std::io::Result { + unimplemented!() + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), target_os = "macos"))] + fn legacy_call(&mut self) -> std::io::Result { + use std::os::fd::AsRawFd; + + use crate::syscall_u32; + + syscall_u32!(fstat( + self.inner.as_raw_fd(), + self.stat_buf.as_mut_ptr() as *mut _ + )) + } +} + +type PathStatx = Statx; + +impl Op { + /// submit a statx operation + #[cfg(target_os = "linux")] + pub(crate) fn statx_using_path>(path: P, flags: i32) -> std::io::Result { + let path = cstr(path.as_ref())?; + Op::submit_with(Statx { + inner: path, + flags, + statx_buf: Box::new(MaybeUninit::uninit()), + }) + } + + #[cfg(target_os = "linux")] + pub(crate) async fn statx_result(self) -> std::io::Result { + let complete = self.await; + complete.meta.result?; + + Ok(unsafe { MaybeUninit::assume_init(*complete.data.statx_buf) }) + } + + #[cfg(target_os = "macos")] + pub(crate) fn statx_using_path>( + path: P, + follow_symlinks: bool, + ) -> std::io::Result { + let path = cstr(path.as_ref())?; + Op::submit_with(Statx { + inner: path, + follow_symlinks, + stat_buf: Box::new(MaybeUninit::uninit()), + }) + } + + #[cfg(target_os = "macos")] + pub(crate) async fn statx_result(self) -> std::io::Result { + let complete = self.await; + complete.meta.result?; + + Ok(unsafe { MaybeUninit::assume_init(*complete.data.stat_buf) }) + } +} + +impl OpAble for PathStatx { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + let statxbuf = self.statx_buf.as_mut_ptr() as *mut _; + + opcode::Statx::new(types::Fd(libc::AT_FDCWD), self.inner.as_ptr(), statxbuf) + .flags(self.flags) + .mask(libc::STATX_ALL) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_interest(&self) -> Option<(crate::driver::ready::Direction, usize)> { + None + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), target_os = "linux"))] + fn legacy_call(&mut self) -> std::io::Result { + use crate::syscall_u32; + + syscall_u32!(statx( + libc::AT_FDCWD, + self.inner.as_ptr(), + self.flags, + libc::STATX_ALL, + self.statx_buf.as_mut_ptr() as *mut _ + )) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> std::io::Result { + unimplemented!() + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), target_os = "macos"))] + fn legacy_call(&mut self) -> std::io::Result { + use crate::syscall_u32; + + if self.follow_symlinks { + syscall_u32!(stat( + self.inner.as_ptr(), + self.stat_buf.as_mut_ptr() as *mut _ + )) + } else { + syscall_u32!(lstat( + self.inner.as_ptr(), + self.stat_buf.as_mut_ptr() as *mut _ + )) + } + } +} diff --git a/vendor/monoio/src/driver/op/unlink.rs b/vendor/monoio/src/driver/op/unlink.rs new file mode 100644 index 000000000..3871d29c7 --- /dev/null +++ b/vendor/monoio/src/driver/op/unlink.rs @@ -0,0 +1,57 @@ +use std::{ffi::CString, io, path::Path}; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, squeue::Entry, types::Fd}; +#[cfg(all(target_os = "linux", feature = "iouring"))] +use libc::{AT_FDCWD, AT_REMOVEDIR}; + +use super::{Op, OpAble}; +use crate::driver::util::cstr; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::{driver::ready::Direction, syscall_u32}; + +pub(crate) struct Unlink { + path: CString, + remove_dir: bool, +} + +impl Op { + pub(crate) fn unlink>(path: P) -> io::Result> { + let path = cstr(path.as_ref())?; + Op::submit_with(Unlink { + path, + remove_dir: false, + }) + } + + pub(crate) fn rmdir>(path: P) -> io::Result> { + let path = cstr(path.as_ref())?; + Op::submit_with(Unlink { + path, + remove_dir: true, + }) + } +} + +impl OpAble for Unlink { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> Entry { + opcode::UnlinkAt::new(Fd(AT_FDCWD), self.path.as_c_str().as_ptr()) + .flags(if self.remove_dir { AT_REMOVEDIR } else { 0 }) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + None + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + fn legacy_call(&mut self) -> io::Result { + if self.remove_dir { + syscall_u32!(rmdir(self.path.as_c_str().as_ptr())) + } else { + syscall_u32!(unlink(self.path.as_c_str().as_ptr())) + } + } +} diff --git a/vendor/monoio/src/driver/op/write.rs b/vendor/monoio/src/driver/op/write.rs new file mode 100644 index 000000000..f32295ceb --- /dev/null +++ b/vendor/monoio/src/driver/op/write.rs @@ -0,0 +1,190 @@ +use std::io; +#[cfg(all(unix, any(feature = "legacy", feature = "poll-io")))] +use std::os::unix::prelude::AsRawFd; + +#[cfg(all(target_os = "linux", feature = "iouring"))] +use io_uring::{opcode, types}; +#[cfg(all(windows, any(feature = "legacy", feature = "poll-io")))] +use windows_sys::Win32::{ + Foundation::TRUE, + Networking::WinSock::WSASend, + Storage::FileSystem::{SetFilePointer, WriteFile, FILE_CURRENT, INVALID_SET_FILE_POINTER}, +}; + +use super::{super::shared_fd::SharedFd, Op, OpAble}; +#[cfg(any(feature = "legacy", feature = "poll-io"))] +use crate::driver::ready::Direction; +use crate::{ + buf::{IoBuf, IoVecBuf}, + syscall_u32, BufResult, +}; + +pub(crate) struct Write { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + offset: u64, + + pub(crate) buf: T, +} + +impl Op> { + pub(crate) fn write_at(fd: &SharedFd, buf: T, offset: u64) -> io::Result>> { + Op::submit_with(Write { + fd: fd.clone(), + offset, + buf, + }) + } + + pub(crate) async fn write(self) -> BufResult { + let complete = self.await; + (complete.meta.result.map(|v| v as _), complete.data.buf) + } +} + +impl OpAble for Write { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + opcode::Write::new( + types::Fd(self.fd.raw_fd()), + self.buf.read_ptr(), + self.buf.bytes_init() as _, + ) + .offset(self.offset) + .build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd + .registered_index() + .map(|idx| (Direction::Write, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.as_raw_fd(); + let seek_offset = libc::off_t::try_from(self.offset) + .map_err(|_| io::Error::new(io::ErrorKind::Other, "offset too big"))?; + #[cfg(not(target_os = "macos"))] + return syscall_u32!(pwrite64( + fd, + self.buf.read_ptr() as _, + self.buf.bytes_init(), + seek_offset as _ + )); + + #[cfg(target_os = "macos")] + return syscall_u32!(pwrite( + fd, + self.buf.read_ptr() as _, + self.buf.bytes_init(), + seek_offset + )); + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let fd = self.fd.raw_handle() as _; + let seek_offset = libc::off_t::try_from(self.offset) + .map_err(|_| io::Error::new(io::ErrorKind::Other, "offset too big"))?; + let mut bytes_write = 0; + let ret = unsafe { + // see https://learn.microsoft.com/zh-cn/windows/win32/api/fileapi/nf-fileapi-setfilepointer + if seek_offset != 0 { + let r = SetFilePointer(fd, seek_offset, std::ptr::null_mut(), FILE_CURRENT); + if INVALID_SET_FILE_POINTER == r { + return Err(io::Error::last_os_error()); + } + } + // see https://learn.microsoft.com/zh-cn/windows/win32/api/fileapi/nf-fileapi-writefile + WriteFile( + fd, + self.buf.read_ptr(), + self.buf.bytes_init() as u32, + &mut bytes_write, + std::ptr::null_mut(), + ) + }; + if TRUE == ret { + Ok(bytes_write) + } else { + Err(io::Error::last_os_error()) + } + } +} + +pub(crate) struct WriteVec { + /// Holds a strong ref to the FD, preventing the file from being closed + /// while the operation is in-flight. + #[allow(unused)] + fd: SharedFd, + + pub(crate) buf_vec: T, +} + +impl Op> { + pub(crate) fn writev(fd: &SharedFd, buf_vec: T) -> io::Result { + Op::submit_with(WriteVec { + fd: fd.clone(), + buf_vec, + }) + } + + #[allow(unused)] + pub(crate) fn writev_raw(fd: &SharedFd, buf_vec: T) -> WriteVec { + WriteVec { + fd: fd.clone(), + buf_vec, + } + } + + pub(crate) async fn write(self) -> BufResult { + let complete = self.await; + (complete.meta.result.map(|v| v as _), complete.data.buf_vec) + } +} + +impl OpAble for WriteVec { + #[cfg(all(target_os = "linux", feature = "iouring"))] + fn uring_op(&mut self) -> io_uring::squeue::Entry { + let ptr = self.buf_vec.read_iovec_ptr() as *const _; + let len = self.buf_vec.read_iovec_len() as _; + opcode::Writev::new(types::Fd(self.fd.raw_fd()), ptr, len).build() + } + + #[cfg(any(feature = "legacy", feature = "poll-io"))] + #[inline] + fn legacy_interest(&self) -> Option<(Direction, usize)> { + self.fd + .registered_index() + .map(|idx| (Direction::Write, idx)) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), unix))] + fn legacy_call(&mut self) -> io::Result { + syscall_u32!(writev( + self.fd.raw_fd(), + self.buf_vec.read_iovec_ptr(), + self.buf_vec.read_iovec_len().min(i32::MAX as usize) as _ + )) + } + + #[cfg(all(any(feature = "legacy", feature = "poll-io"), windows))] + fn legacy_call(&mut self) -> io::Result { + let mut bytes_sent = 0; + syscall_u32!(WSASend( + self.fd.raw_socket() as _, + self.buf_vec.read_wsabuf_ptr(), + self.buf_vec.read_wsabuf_len() as _, + &mut bytes_sent, + 0, + std::ptr::null_mut(), + None, + )) + .map(|_| bytes_sent) + } +} diff --git a/vendor/monoio/src/driver/poll.rs b/vendor/monoio/src/driver/poll.rs new file mode 100644 index 000000000..e0ae0b41f --- /dev/null +++ b/vendor/monoio/src/driver/poll.rs @@ -0,0 +1,109 @@ +use std::{io, task::Context, time::Duration}; + +use super::{ready::Direction, scheduled_io::ScheduledIo}; +use crate::{driver::op::CompletionMeta, utils::slab::Slab}; + +/// Poller with io dispatch. +// TODO: replace legacy impl with this Poll. +pub(crate) struct Poll { + pub(crate) io_dispatch: Slab, + poll: mio::Poll, + events: mio::Events, +} + +impl Poll { + #[inline] + pub(crate) fn with_capacity(capacity: usize) -> io::Result { + Ok(Self { + io_dispatch: Slab::new(), + poll: mio::Poll::new()?, + events: mio::Events::with_capacity(capacity), + }) + } + + #[inline] + pub(crate) fn tick(&mut self, timeout: Option) -> io::Result<()> { + match self.poll.poll(&mut self.events, timeout) { + Ok(_) => {} + Err(ref e) if e.kind() == io::ErrorKind::Interrupted => {} + Err(e) => return Err(e), + } + for event in self.events.iter() { + let token = event.token(); + + if let Some(mut sio) = self.io_dispatch.get(token.0) { + let ref_mut = sio.as_mut(); + let ready = super::ready::Ready::from_mio(event); + ref_mut.set_readiness(|curr| curr | ready); + ref_mut.wake(ready); + } + } + Ok(()) + } + + pub(crate) fn register( + &mut self, + source: &mut impl mio::event::Source, + interest: mio::Interest, + ) -> io::Result { + let token = self.io_dispatch.insert(ScheduledIo::new()); + let registry = self.poll.registry(); + match registry.register(source, mio::Token(token), interest) { + Ok(_) => Ok(token), + Err(e) => { + self.io_dispatch.remove(token); + Err(e) + } + } + } + + pub(crate) fn deregister( + &mut self, + source: &mut impl mio::event::Source, + token: usize, + ) -> io::Result<()> { + match self.poll.registry().deregister(source) { + Ok(_) => { + self.io_dispatch.remove(token); + Ok(()) + } + Err(e) => Err(e), + } + } + + #[inline] + pub(crate) fn poll_syscall( + &mut self, + cx: &mut Context<'_>, + token: usize, + direction: Direction, + syscall: impl FnOnce() -> io::Result, + ) -> std::task::Poll { + let mut scheduled_io = self.io_dispatch.get(token).expect("scheduled_io lost"); + let ref_mut = scheduled_io.as_mut(); + ready!(ref_mut.poll_readiness(cx, direction)); + match syscall() { + Ok(n) => std::task::Poll::Ready(CompletionMeta { + result: Ok(n), + flags: 0, + }), + Err(ref e) if e.kind() == io::ErrorKind::WouldBlock => { + ref_mut.clear_readiness(direction.mask()); + ref_mut.set_waker(cx, direction); + std::task::Poll::Pending + } + Err(e) => std::task::Poll::Ready(CompletionMeta { + result: Err(e), + flags: 0, + }), + } + } +} + +#[cfg(unix)] +impl std::os::fd::AsRawFd for Poll { + #[inline] + fn as_raw_fd(&self) -> std::os::fd::RawFd { + self.poll.as_raw_fd() + } +} diff --git a/vendor/monoio/src/driver/pool.rs b/vendor/monoio/src/driver/pool.rs new file mode 100644 index 000000000..5fe25cee5 --- /dev/null +++ b/vendor/monoio/src/driver/pool.rs @@ -0,0 +1,78 @@ +use crate::driver; + +use io_uring::{opcode, IoUring}; +use std::io; +use std::mem::ManuallyDrop; + +/// Buffer pool shared with kernel +pub(crate) struct Pool { + mem: *mut u8, + num: usize, + size: usize, +} + +pub(crate) struct ProvidedBuf { + buf: ManuallyDrop>, + driver: driver::Handle, +} + +impl Pool { + pub(super) fn new(num: usize, size: usize) -> Pool { + let total = num * size; + let mut mem = ManuallyDrop::new(Vec::::with_capacity(total)); + + assert_eq!(mem.capacity(), total); + + Pool { + mem: mem.as_mut_ptr(), + num, + size, + } + } + + pub(super) fn provide_buffers(&self, uring: &mut IoUring) -> io::Result<()> { + let op = opcode::ProvideBuffers::new(self.mem, self.size as _, self.num as _, 0, 0) + .build() + .user_data(0); + + // Scoped to ensure `sq` drops before trying to submit + { + let mut sq = uring.submission(); + + if unsafe { sq.push(&op) }.is_err() { + unimplemented!("when is this hit?"); + } + } + + uring.submit_and_wait(1)?; + + let mut cq = uring.completion(); + for cqe in &mut cq { + assert_eq!(cqe.user_data(), 0); + } + + Ok(()) + } +} + +impl ProvidedBuf {} + +impl Drop for ProvidedBuf { + fn drop(&mut self) { + let mut driver = self.driver.borrow_mut(); + let pool = &driver.pool; + + let ptr = self.buf.as_mut_ptr(); + let bid = (ptr as usize - pool.mem as usize) / pool.size; + + let op = opcode::ProvideBuffers::new(ptr, pool.size as _, 1, 0, bid as _) + .build() + .user_data(u64::MAX); + + let mut sq = driver.uring.submission(); + + if unsafe { sq.push(&op) }.is_err() { + unimplemented!(); + } + } +} diff --git a/vendor/monoio/src/driver/ready.rs b/vendor/monoio/src/driver/ready.rs new file mode 100644 index 000000000..df25b8a3b --- /dev/null +++ b/vendor/monoio/src/driver/ready.rs @@ -0,0 +1,248 @@ +//! Copied from tokio. +//! Ready and Interest. + +use std::{fmt, ops}; + +const READABLE: u8 = 0b0_01; +const WRITABLE: u8 = 0b0_10; +const READ_CLOSED: u8 = 0b0_0100; +const WRITE_CLOSED: u8 = 0b0_1000; +const READ_CANCELED: u8 = 0b01_0000; +const WRITE_CANCELED: u8 = 0b10_0000; + +/// Describes the readiness state of an I/O resources. +/// +/// `Ready` tracks which operation an I/O resource is ready to perform. +#[cfg_attr(docsrs, doc(cfg(feature = "net")))] +#[derive(Clone, Copy, PartialEq, PartialOrd, Eq)] +pub(crate) struct Ready(u8); + +impl Ready { + /// Returns the empty `Ready` set. + pub(crate) const EMPTY: Ready = Ready(0); + + /// Returns a `Ready` representing readable readiness. + pub(crate) const READABLE: Ready = Ready(READABLE); + + /// Returns a `Ready` representing writable readiness. + pub(crate) const WRITABLE: Ready = Ready(WRITABLE); + + /// Returns a `Ready` representing read closed readiness. + pub(crate) const READ_CLOSED: Ready = Ready(READ_CLOSED); + + /// Returns a `Ready` representing write closed readiness. + pub(crate) const WRITE_CLOSED: Ready = Ready(WRITE_CLOSED); + + /// Returns a `Ready` representing read canceled readiness. + pub(crate) const READ_CANCELED: Ready = Ready(READ_CANCELED); + + /// Returns a `Ready` representing write canceled readiness. + pub(crate) const WRITE_CANCELED: Ready = Ready(WRITE_CANCELED); + + /// Returns a `Ready` representing read or write canceled readiness. + pub(crate) const CANCELED: Ready = Ready(READ_CANCELED | WRITE_CANCELED); + + pub(crate) const READ_ALL: Ready = Ready(READABLE | READ_CLOSED | READ_CANCELED); + pub(crate) const WRITE_ALL: Ready = Ready(WRITABLE | WRITE_CLOSED | WRITE_CANCELED); + + #[cfg(windows)] + pub(crate) fn from_mio(event: &super::legacy::iocp::Event) -> Ready { + let mut ready = Ready::EMPTY; + + if event.is_readable() { + ready |= Ready::READABLE; + } + + if event.is_writable() { + ready |= Ready::WRITABLE; + } + + if event.is_read_closed() { + ready |= Ready::READ_CLOSED; + } + + if event.is_write_closed() { + ready |= Ready::WRITE_CLOSED; + } + + ready + } + + #[cfg(unix)] + // Must remain crate-private to avoid adding a public dependency on Mio. + pub(crate) fn from_mio(event: &mio::event::Event) -> Ready { + let mut ready = Ready::EMPTY; + + #[cfg(all(target_os = "freebsd", feature = "legacy"))] + { + if event.is_aio() { + ready |= Ready::READABLE; + } + + if event.is_lio() { + ready |= Ready::READABLE; + } + } + + if event.is_readable() { + ready |= Ready::READABLE; + } + + if event.is_writable() { + ready |= Ready::WRITABLE; + } + + if event.is_read_closed() { + ready |= Ready::READ_CLOSED; + } + + if event.is_write_closed() { + ready |= Ready::WRITE_CLOSED; + } + + ready + } + + /// Returns true if `Ready` is the empty set. + pub(crate) fn is_empty(self) -> bool { + self == Ready::EMPTY + } + + /// Returns `true` if the value includes `readable`. + pub(crate) fn is_readable(self) -> bool { + !(self & Ready::READ_ALL).is_empty() + } + + /// Returns `true` if the value includes writable `readiness`. + pub(crate) fn is_writable(self) -> bool { + !(self & Ready::WRITE_ALL).is_empty() + } + + /// Returns `true` if the value includes read-closed `readiness`. + pub(crate) fn is_read_closed(self) -> bool { + self.contains(Ready::READ_CLOSED) + } + + /// Returns `true` if the value includes write-closed `readiness`. + pub(crate) fn is_write_closed(self) -> bool { + self.contains(Ready::WRITE_CLOSED) + } + + #[allow(dead_code)] + pub(crate) fn is_canceled(self) -> bool { + !(self & Ready::CANCELED).is_empty() + } + + /// Returns true if `self` is a superset of `other`. + /// + /// `other` may represent more than one readiness operations, in which case + /// the function only returns true if `self` contains all readiness + /// specified in `other`. + pub(crate) fn contains>(self, other: T) -> bool { + let other = other.into(); + (self & other) == other + } +} + +/// Readiness event interest. +/// +/// Specifies the readiness events the caller is interested in when awaiting on +/// I/O resource readiness states. +#[derive(Clone, Copy, Eq, PartialEq)] +// TODO: Do we need to remove it? +#[allow(dead_code)] +pub(crate) struct Interest(mio::Interest); + +impl Interest { + /// Add together two `Interest` values. + /// + /// This function works from a `const` context. + pub(crate) const fn add(self, other: Interest) -> Interest { + Interest(self.0.add(other.0)) + } +} + +impl ops::BitOr for Interest { + type Output = Self; + + #[inline] + fn bitor(self, other: Self) -> Self { + self.add(other) + } +} + +impl ops::BitOrAssign for Interest { + #[inline] + fn bitor_assign(&mut self, other: Self) { + self.0 = (*self | other).0; + } +} + +impl fmt::Debug for Interest { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + self.0.fmt(fmt) + } +} + +impl ops::BitOr for Ready { + type Output = Ready; + + #[inline] + fn bitor(self, other: Ready) -> Ready { + Ready(self.0 | other.0) + } +} + +impl ops::BitOrAssign for Ready { + #[inline] + fn bitor_assign(&mut self, other: Ready) { + self.0 |= other.0; + } +} + +impl ops::BitAnd for Ready { + type Output = Ready; + + #[inline] + fn bitand(self, other: Ready) -> Ready { + Ready(self.0 & other.0) + } +} + +impl ops::Sub for Ready { + type Output = Ready; + + #[inline] + fn sub(self, other: Ready) -> Ready { + Ready(self.0 & !other.0) + } +} + +impl fmt::Debug for Ready { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt.debug_struct("Ready") + .field("is_readable", &self.is_readable()) + .field("is_writable", &self.is_writable()) + .field("is_read_closed", &self.is_read_closed()) + .field("is_write_closed", &self.is_write_closed()) + .finish() + } +} + +#[derive(Debug, Eq, PartialEq, Clone, Copy, Hash)] +pub(crate) enum Direction { + Read, + Write, +} + +impl Direction { + pub(crate) fn mask(self) -> Ready { + match self { + Direction::Read => Ready::READABLE | Ready::READ_CLOSED | Ready::READ_CANCELED, + Direction::Write => Ready::WRITABLE | Ready::WRITE_CLOSED | Ready::WRITE_CANCELED, + } + } +} + +#[allow(dead_code)] +pub(crate) const RW_INTERESTS: mio::Interest = mio::Interest::READABLE.add(mio::Interest::WRITABLE); diff --git a/vendor/monoio/src/driver/scheduled_io.rs b/vendor/monoio/src/driver/scheduled_io.rs new file mode 100644 index 000000000..d164a1d36 --- /dev/null +++ b/vendor/monoio/src/driver/scheduled_io.rs @@ -0,0 +1,92 @@ +use std::task::{Context, Poll, Waker}; + +use super::ready::{Direction, Ready}; + +pub(crate) struct ScheduledIo { + readiness: Ready, + + /// Waker used for AsyncRead. + reader: Option, + /// Waker used for AsyncWrite. + writer: Option, +} + +impl Default for ScheduledIo { + #[inline] + fn default() -> Self { + Self::new() + } +} + +impl ScheduledIo { + pub(crate) const fn new() -> Self { + Self { + readiness: Ready::EMPTY, + reader: None, + writer: None, + } + } + + #[allow(unused)] + #[inline] + pub(crate) fn set_writable(&mut self) { + self.readiness |= Ready::WRITABLE; + } + + #[inline] + pub(crate) fn set_readiness(&mut self, f: impl Fn(Ready) -> Ready) { + self.readiness = f(self.readiness); + } + + #[inline] + pub(crate) fn wake(&mut self, ready: Ready) { + if ready.is_readable() { + if let Some(waker) = self.reader.take() { + waker.wake(); + } + } + if ready.is_writable() { + if let Some(waker) = self.writer.take() { + waker.wake(); + } + } + } + + #[inline] + pub(crate) fn clear_readiness(&mut self, ready: Ready) { + self.readiness = self.readiness - ready; + } + + #[allow(clippy::needless_pass_by_ref_mut)] + #[inline] + pub(crate) fn poll_readiness( + &mut self, + cx: &mut Context<'_>, + direction: Direction, + ) -> Poll { + let ready = direction.mask() & self.readiness; + if !ready.is_empty() { + return Poll::Ready(ready); + } + self.set_waker(cx, direction); + Poll::Pending + } + + #[inline] + pub(crate) fn set_waker(&mut self, cx: &mut Context<'_>, direction: Direction) { + let slot = match direction { + Direction::Read => &mut self.reader, + Direction::Write => &mut self.writer, + }; + match slot { + Some(existing) => { + if !existing.will_wake(cx.waker()) { + existing.clone_from(cx.waker()); + } + } + None => { + *slot = Some(cx.waker().clone()); + } + } + } +} diff --git a/vendor/monoio/src/driver/shared_fd.rs b/vendor/monoio/src/driver/shared_fd.rs new file mode 100644 index 000000000..b9f370fb1 --- /dev/null +++ b/vendor/monoio/src/driver/shared_fd.rs @@ -0,0 +1,600 @@ +#[cfg(unix)] +use std::os::unix::io::{AsRawFd, FromRawFd, RawFd}; +#[cfg(windows)] +use std::os::windows::io::{ + AsRawHandle, AsRawSocket, FromRawSocket, OwnedSocket, RawHandle, RawSocket, +}; +use std::{cell::UnsafeCell, io, rc::Rc}; + +#[cfg(windows)] +use super::legacy::iocp::SocketState as RawFd; +use super::CURRENT; + +// Tracks in-flight operations on a file descriptor. Ensures all in-flight +// operations complete before submitting the close. +#[derive(Clone, Debug)] +pub(crate) struct SharedFd { + inner: Rc, +} + +struct Inner { + // Open file descriptor + #[cfg(any(unix, windows))] + fd: RawFd, + + // Waker to notify when the close operation completes. + state: UnsafeCell, +} + +enum State { + #[cfg(all(target_os = "linux", feature = "iouring"))] + Uring(UringState), + #[cfg(feature = "legacy")] + Legacy(Option), +} + +#[cfg(feature = "poll-io")] +impl State { + #[cfg(all(target_os = "linux", feature = "iouring"))] + #[allow(unreachable_patterns)] + pub(crate) fn cvt_uring_poll(&mut self, fd: RawFd) -> io::Result<()> { + let state = match self { + State::Uring(state) => state, + _ => return Ok(()), + }; + // TODO: only Init state can convert? + if matches!(state, UringState::Init) { + let mut source = mio::unix::SourceFd(&fd); + crate::syscall!(fcntl(fd, libc::F_SETFL, libc::O_NONBLOCK))?; + let reg = CURRENT + .with(|inner| match inner { + #[cfg(all(target_os = "linux", feature = "iouring"))] + crate::driver::Inner::Uring(r) => super::IoUringDriver::register_poll_io( + r, + &mut source, + super::ready::RW_INTERESTS, + ), + #[cfg(feature = "legacy")] + crate::driver::Inner::Legacy(_) => panic!("unexpected legacy runtime"), + }) + .inspect_err(|_| { + let _ = crate::syscall!(fcntl(fd, libc::F_SETFL, 0)); + })?; + *state = UringState::Legacy(Some(reg)); + } else { + return Err(io::Error::new( + io::ErrorKind::Other, + "not clear uring state", + )); + } + Ok(()) + } + + #[cfg(not(all(target_os = "linux", feature = "iouring")))] + #[inline] + pub(crate) fn cvt_uring_poll(&mut self, _fd: RawFd) -> io::Result<()> { + Ok(()) + } + + #[cfg(all(target_os = "linux", feature = "iouring"))] + pub(crate) fn cvt_comp(&mut self, fd: RawFd) -> io::Result<()> { + let inner = match self { + Self::Uring(UringState::Legacy(inner)) => inner, + _ => return Ok(()), + }; + let Some(token) = inner else { + return Err(io::Error::new(io::ErrorKind::Other, "empty token")); + }; + let mut source = mio::unix::SourceFd(&fd); + crate::syscall!(fcntl(fd, libc::F_SETFL, 0))?; + CURRENT + .with(|inner| match inner { + #[cfg(all(target_os = "linux", feature = "iouring"))] + crate::driver::Inner::Uring(r) => { + super::IoUringDriver::deregister_poll_io(r, &mut source, *token) + } + #[cfg(feature = "legacy")] + crate::driver::Inner::Legacy(_) => panic!("unexpected legacy runtime"), + }) + .inspect_err(|_| { + let _ = crate::syscall!(fcntl(fd, libc::F_SETFL, libc::O_NONBLOCK)); + })?; + *self = State::Uring(UringState::Init); + Ok(()) + } + + #[cfg(not(all(target_os = "linux", feature = "iouring")))] + #[inline] + pub(crate) fn cvt_comp(&mut self, _fd: RawFd) -> io::Result<()> { + Ok(()) + } +} + +impl std::fmt::Debug for Inner { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Inner").field("fd", &self.fd).finish() + } +} + +#[cfg(all(target_os = "linux", feature = "iouring"))] +enum UringState { + /// Initial state + Init, + + /// Waiting for all in-flight operation to complete. + Waiting(Option), + + /// The FD is closing + Closing(super::op::Op), + + /// The FD is fully closed + Closed, + + /// Poller + #[cfg(feature = "poll-io")] + Legacy(Option), +} + +#[cfg(unix)] +impl AsRawFd for SharedFd { + fn as_raw_fd(&self) -> RawFd { + self.raw_fd() + } +} + +#[cfg(windows)] +impl AsRawSocket for SharedFd { + fn as_raw_socket(&self) -> RawSocket { + self.raw_socket() + } +} + +#[cfg(windows)] +impl AsRawHandle for SharedFd { + fn as_raw_handle(&self) -> RawHandle { + self.raw_handle() + } +} + +impl SharedFd { + #[cfg(unix)] + #[allow(unreachable_code, unused)] + pub(crate) fn new(fd: RawFd) -> io::Result { + enum Reg { + Uring, + #[cfg(feature = "poll-io")] + UringLegacy(io::Result), + Legacy(io::Result), + } + + #[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] + let state = match CURRENT.with(|inner| match inner { + super::Inner::Uring(inner) => match FORCE_LEGACY { + false => Reg::Uring, + true => { + #[cfg(feature = "poll-io")] + { + let mut source = mio::unix::SourceFd(&fd); + Reg::UringLegacy(super::IoUringDriver::register_poll_io( + inner, + &mut source, + super::ready::RW_INTERESTS, + )) + } + #[cfg(not(feature = "poll-io"))] + Reg::Uring + } + }, + super::Inner::Legacy(inner) => { + let mut source = mio::unix::SourceFd(&fd); + Reg::Legacy(super::legacy::LegacyDriver::register( + inner, + &mut source, + super::ready::RW_INTERESTS, + )) + } + }) { + Reg::Uring => State::Uring(UringState::Init), + #[cfg(feature = "poll-io")] + Reg::UringLegacy(idx) => State::Uring(UringState::Legacy(Some(idx?))), + Reg::Legacy(idx) => State::Legacy(Some(idx?)), + }; + + #[cfg(all(not(feature = "legacy"), target_os = "linux", feature = "iouring"))] + let state = State::Uring(UringState::Init); + + #[cfg(all( + unix, + feature = "legacy", + not(all(target_os = "linux", feature = "iouring")) + ))] + let state = { + let reg = CURRENT.with(|inner| match inner { + super::Inner::Legacy(inner) => { + let mut source = mio::unix::SourceFd(&fd); + super::legacy::LegacyDriver::register( + inner, + &mut source, + super::ready::RW_INTERESTS, + ) + } + }); + + State::Legacy(Some(reg?)) + }; + + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + #[allow(unused)] + let state = super::util::feature_panic(); + + #[allow(unreachable_code)] + Ok(SharedFd { + inner: Rc::new(Inner { + fd, + state: UnsafeCell::new(state), + }), + }) + } + + #[cfg(windows)] + pub(crate) fn new(fd: RawSocket) -> io::Result { + const RW_INTERESTS: mio::Interest = mio::Interest::READABLE.add(mio::Interest::WRITABLE); + + let mut fd = RawFd::new(fd); + + let state = { + let reg = CURRENT.with(|inner| match inner { + super::Inner::Legacy(inner) => { + super::legacy::LegacyDriver::register(inner, &mut fd, RW_INTERESTS) + } + }); + + State::Legacy(Some(reg?)) + }; + + #[allow(unreachable_code)] + Ok(SharedFd { + inner: Rc::new(Inner { + fd, + state: UnsafeCell::new(state), + }), + }) + } + + #[cfg(unix)] + #[allow(unreachable_code, unused)] + pub(crate) fn new_without_register(fd: RawFd) -> SharedFd { + let state = CURRENT.with(|inner| match inner { + #[cfg(all(target_os = "linux", feature = "iouring"))] + super::Inner::Uring(_) => State::Uring(UringState::Init), + #[cfg(feature = "legacy")] + super::Inner::Legacy(_) => State::Legacy(None), + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + _ => { + super::util::feature_panic(); + } + }); + + SharedFd { + inner: Rc::new(Inner { + fd, + state: UnsafeCell::new(state), + }), + } + } + + #[cfg(windows)] + #[allow(unreachable_code, unused)] + pub(crate) fn new_without_register(fd: RawSocket) -> SharedFd { + let state = CURRENT.with(|inner| match inner { + super::Inner::Legacy(_) => State::Legacy(None), + }); + + SharedFd { + inner: Rc::new(Inner { + fd: RawFd::new(fd), + state: UnsafeCell::new(state), + }), + } + } + + #[cfg(unix)] + /// Returns the RawFd + pub(crate) fn raw_fd(&self) -> RawFd { + self.inner.fd + } + + #[cfg(windows)] + /// Returns the RawSocket + pub(crate) fn raw_socket(&self) -> RawSocket { + self.inner.fd.socket + } + + #[cfg(windows)] + pub(crate) fn raw_handle(&self) -> RawHandle { + self.inner.fd.socket as _ + } + + #[cfg(unix)] + /// Try unwrap Rc, then deregister if registered and return rawfd. + /// Note: this action will consume self and return rawfd without closing it. + pub(crate) fn try_unwrap(self) -> Result { + use std::mem::{ManuallyDrop, MaybeUninit}; + + let fd = self.inner.fd; + match Rc::try_unwrap(self.inner) { + Ok(inner) => { + // Only drop Inner's state, skip its drop impl. + let mut inner_skip_drop = ManuallyDrop::new(inner); + #[allow(invalid_value)] + #[allow(clippy::uninit_assumed_init)] + let mut state = unsafe { MaybeUninit::uninit().assume_init() }; + std::mem::swap(&mut inner_skip_drop.state, &mut state); + + #[cfg(feature = "legacy")] + let state = unsafe { &*state.get() }; + + #[cfg(feature = "legacy")] + #[allow(irrefutable_let_patterns)] + if let State::Legacy(idx) = state { + if CURRENT.is_set() { + CURRENT.with(|inner| { + match inner { + #[cfg(all(target_os = "linux", feature = "iouring"))] + super::Inner::Uring(_) => { + unreachable!("try_unwrap legacy fd with uring runtime") + } + super::Inner::Legacy(inner) => { + // deregister it from driver(Poll and slab) and close fd + if let Some(idx) = idx { + let mut source = mio::unix::SourceFd(&fd); + let _ = super::legacy::LegacyDriver::deregister( + inner, + *idx, + &mut source, + ); + } + } + } + }) + } + } + Ok(fd) + } + Err(inner) => Err(Self { inner }), + } + } + + #[cfg(windows)] + /// Try unwrap Rc, then deregister if registered and return rawfd. + /// Note: this action will consume self and return rawfd without closing it. + pub(crate) fn try_unwrap(self) -> Result { + match Rc::try_unwrap(self.inner) { + Ok(_inner) => { + let mut fd = _inner.fd; + let state = unsafe { &*_inner.state.get() }; + + #[allow(irrefutable_let_patterns)] + if let State::Legacy(idx) = state { + if CURRENT.is_set() { + CURRENT.with(|inner| { + match inner { + super::Inner::Legacy(inner) => { + // deregister it from driver(Poll and slab) and close fd + if let Some(idx) = idx { + let _ = super::legacy::LegacyDriver::deregister( + inner, *idx, &mut fd, + ); + } + } + } + }) + } + } + Ok(fd.socket) + } + Err(inner) => Err(Self { inner }), + } + } + + #[allow(unused)] + pub(crate) fn registered_index(&self) -> Option { + let state = unsafe { &*self.inner.state.get() }; + match state { + #[cfg(all(target_os = "linux", feature = "iouring", feature = "poll-io"))] + State::Uring(UringState::Legacy(s)) => *s, + #[cfg(all(target_os = "linux", feature = "iouring"))] + State::Uring(_) => None, + #[cfg(feature = "legacy")] + State::Legacy(s) => *s, + #[cfg(all( + not(feature = "legacy"), + not(all(target_os = "linux", feature = "iouring")) + ))] + _ => { + super::util::feature_panic(); + } + } + } + + /// An FD cannot be closed until all in-flight operation have completed. + /// This prevents bugs where in-flight reads could operate on the incorrect + /// file descriptor. + pub(crate) async fn close(self) { + // Here we only submit close op for uring mode. + // Fd will be closed when Inner drops for legacy mode. + #[cfg(all(target_os = "linux", feature = "iouring"))] + { + let fd = self.inner.fd; + let mut this = self; + #[allow(irrefutable_let_patterns)] + if let State::Uring(uring_state) = unsafe { &mut *this.inner.state.get() } { + if Rc::get_mut(&mut this.inner).is_some() { + *uring_state = match super::op::Op::close(fd) { + Ok(op) => UringState::Closing(op), + Err(_) => { + let _ = unsafe { std::fs::File::from_raw_fd(fd) }; + return; + } + }; + } + this.inner.closed().await; + } + } + } + + #[cfg(feature = "poll-io")] + #[inline] + pub(crate) fn cvt_poll(&mut self) -> io::Result<()> { + let state = unsafe { &mut *self.inner.state.get() }; + #[cfg(unix)] + let r = state.cvt_uring_poll(self.inner.fd); + #[cfg(windows)] + let r = Ok(()); + r + } + + #[cfg(feature = "poll-io")] + #[inline] + pub(crate) fn cvt_comp(&mut self) -> io::Result<()> { + let state = unsafe { &mut *self.inner.state.get() }; + #[cfg(unix)] + let r = state.cvt_comp(self.inner.fd); + #[cfg(windows)] + let r = Ok(()); + r + } +} + +#[cfg(all(target_os = "linux", feature = "iouring"))] +impl Inner { + /// Completes when the FD has been closed. + /// Should only be called for uring mode. + async fn closed(&self) { + use std::task::Poll; + + crate::macros::support::poll_fn(|cx| { + let state = unsafe { &mut *self.state.get() }; + + #[allow(irrefutable_let_patterns)] + if let State::Uring(uring_state) = state { + use std::{future::Future, pin::Pin}; + + return match uring_state { + UringState::Init => { + *uring_state = UringState::Waiting(Some(cx.waker().clone())); + Poll::Pending + } + UringState::Waiting(Some(waker)) => { + if !waker.will_wake(cx.waker()) { + waker.clone_from(cx.waker()); + } + + Poll::Pending + } + UringState::Waiting(None) => { + *uring_state = UringState::Waiting(Some(cx.waker().clone())); + Poll::Pending + } + UringState::Closing(op) => { + // Nothing to do if the close operation failed. + let _ = ready!(Pin::new(op).poll(cx)); + *uring_state = UringState::Closed; + Poll::Ready(()) + } + UringState::Closed => Poll::Ready(()), + #[cfg(feature = "poll-io")] + UringState::Legacy(_) => Poll::Ready(()), + }; + } + Poll::Ready(()) + }) + .await; + } +} + +#[cfg(unix)] +impl Drop for Inner { + fn drop(&mut self) { + let fd = self.fd; + let state = unsafe { &mut *self.state.get() }; + #[allow(unreachable_patterns)] + match state { + #[cfg(all(target_os = "linux", feature = "iouring"))] + State::Uring(UringState::Init) | State::Uring(UringState::Waiting(..)) => { + if super::op::Op::close(fd).is_err() { + let _ = unsafe { std::fs::File::from_raw_fd(fd) }; + }; + } + #[cfg(feature = "legacy")] + State::Legacy(idx) => drop_legacy(fd, *idx), + #[cfg(all(target_os = "linux", feature = "iouring", feature = "poll-io"))] + State::Uring(UringState::Legacy(idx)) => drop_uring_legacy(fd, *idx), + _ => {} + } + } +} + +#[allow(unused_mut)] +#[cfg(feature = "legacy")] +fn drop_legacy(mut fd: RawFd, idx: Option) { + if CURRENT.is_set() { + CURRENT.with(|inner| { + #[cfg(any(all(target_os = "linux", feature = "iouring"), feature = "legacy"))] + match inner { + #[cfg(all(target_os = "linux", feature = "iouring"))] + super::Inner::Uring(_) => { + unreachable!("close legacy fd with uring runtime") + } + super::Inner::Legacy(inner) => { + // deregister it from driver(Poll and slab) and close fd + #[cfg(not(windows))] + if let Some(idx) = idx { + let mut source = mio::unix::SourceFd(&fd); + let _ = super::legacy::LegacyDriver::deregister(inner, idx, &mut source); + } + #[cfg(windows)] + if let Some(idx) = idx { + let _ = super::legacy::LegacyDriver::deregister(inner, idx, &mut fd); + } + } + } + }) + } + #[cfg(all(unix, feature = "legacy"))] + let _ = unsafe { std::fs::File::from_raw_fd(fd) }; + #[cfg(all(windows, feature = "legacy"))] + let _ = unsafe { OwnedSocket::from_raw_socket(fd.socket) }; +} + +#[cfg(feature = "poll-io")] +fn drop_uring_legacy(fd: RawFd, idx: Option) { + if CURRENT.is_set() { + CURRENT.with(|inner| { + match inner { + #[cfg(feature = "legacy")] + super::Inner::Legacy(_) => { + unreachable!("close uring fd with legacy runtime") + } + #[cfg(all(target_os = "linux", feature = "iouring"))] + super::Inner::Uring(inner) => { + // deregister it from driver(Poll and slab) and close fd + if let Some(idx) = idx { + let mut source = mio::unix::SourceFd(&fd); + let _ = super::IoUringDriver::deregister_poll_io(inner, &mut source, idx); + } + } + } + }) + } + #[cfg(unix)] + let _ = unsafe { std::fs::File::from_raw_fd(fd) }; + #[cfg(windows)] + let _ = unsafe { OwnedSocket::from_raw_socket(fd.socket) }; +} diff --git a/vendor/monoio/src/driver/thread.rs b/vendor/monoio/src/driver/thread.rs new file mode 100644 index 000000000..e85b25067 --- /dev/null +++ b/vendor/monoio/src/driver/thread.rs @@ -0,0 +1,48 @@ +#[cfg(feature = "unstable")] +use std::sync::LazyLock; +use std::{sync::Mutex, task::Waker}; + +use flume::Sender; +use fxhash::FxHashMap; +#[cfg(not(feature = "unstable"))] +use once_cell::sync::Lazy as LazyLock; + +use crate::driver::UnparkHandle; + +static UNPARK: LazyLock>> = + LazyLock::new(|| Mutex::new(FxHashMap::default())); + +// Global waker sender map +static WAKER_SENDER: LazyLock>>> = + LazyLock::new(|| Mutex::new(FxHashMap::default())); + +macro_rules! lock { + ($x: ident) => { + $x.lock() + .expect("Unable to lock global map, which is unexpected") + }; +} + +pub(crate) fn register_unpark_handle(id: usize, unpark: UnparkHandle) { + lock!(UNPARK).insert(id, unpark); +} + +pub(crate) fn unregister_unpark_handle(id: usize) { + lock!(UNPARK).remove(&id); +} + +pub(crate) fn get_unpark_handle(id: usize) -> Option { + lock!(UNPARK).get(&id).cloned() +} + +pub(crate) fn register_waker_sender(id: usize, sender: Sender) { + lock!(WAKER_SENDER).insert(id, sender); +} + +pub(crate) fn unregister_waker_sender(id: usize) { + lock!(WAKER_SENDER).remove(&id); +} + +pub(crate) fn get_waker_sender(id: usize) -> Option> { + lock!(WAKER_SENDER).get(&id).cloned() +} diff --git a/vendor/monoio/src/driver/uring/lifecycle.rs b/vendor/monoio/src/driver/uring/lifecycle.rs new file mode 100644 index 000000000..3c7551c11 --- /dev/null +++ b/vendor/monoio/src/driver/uring/lifecycle.rs @@ -0,0 +1,93 @@ +//! Uring state lifecycle. +//! Partly borrow from tokio-uring. + +use std::{ + io, + task::{Context, Poll, Waker}, +}; + +use crate::{driver::op::CompletionMeta, utils::slab::Ref}; + +pub(crate) enum Lifecycle { + /// The operation has been submitted to uring and is currently in-flight + Submitted, + + /// The submitter is waiting for the completion of the operation + Waiting(Waker), + + /// The submitter no longer has interest in the operation result. The state + /// must be passed to the driver and held until the operation completes. + #[allow(dead_code)] + Ignored(Box), + + /// The operation has completed. + Completed(io::Result, u32), +} + +impl<'a> Ref<'a, Lifecycle> { + pub(crate) fn complete(mut self, result: io::Result, flags: u32) { + let ref_mut = &mut *self; + match ref_mut { + Lifecycle::Submitted => { + *ref_mut = Lifecycle::Completed(result, flags); + } + Lifecycle::Waiting(_) => { + let old = std::mem::replace(ref_mut, Lifecycle::Completed(result, flags)); + match old { + Lifecycle::Waiting(waker) => { + waker.wake(); + } + _ => unsafe { std::hint::unreachable_unchecked() }, + } + } + Lifecycle::Ignored(..) => { + self.remove(); + } + Lifecycle::Completed(..) => unsafe { std::hint::unreachable_unchecked() }, + } + } + + #[allow(clippy::needless_pass_by_ref_mut)] + pub(crate) fn poll_op(mut self, cx: &mut Context<'_>) -> Poll { + let ref_mut = &mut *self; + match ref_mut { + Lifecycle::Submitted => { + *ref_mut = Lifecycle::Waiting(cx.waker().clone()); + return Poll::Pending; + } + Lifecycle::Waiting(waker) => { + if !waker.will_wake(cx.waker()) { + *ref_mut = Lifecycle::Waiting(cx.waker().clone()); + } + return Poll::Pending; + } + _ => {} + } + + match self.remove() { + Lifecycle::Completed(result, flags) => Poll::Ready(CompletionMeta { result, flags }), + _ => unsafe { std::hint::unreachable_unchecked() }, + } + } + + // return if the op must has been finished + pub(crate) fn drop_op(mut self, data: &mut Option) -> bool { + let ref_mut = &mut *self; + match ref_mut { + Lifecycle::Submitted | Lifecycle::Waiting(_) => { + if let Some(data) = data.take() { + *ref_mut = Lifecycle::Ignored(Box::new(data)); + } else { + *ref_mut = Lifecycle::Ignored(Box::new(())); // () is a ZST, so it does not + // allocate + }; + return false; + } + Lifecycle::Completed(..) => { + self.remove(); + } + Lifecycle::Ignored(..) => unsafe { std::hint::unreachable_unchecked() }, + } + true + } +} diff --git a/vendor/monoio/src/driver/uring/mod.rs b/vendor/monoio/src/driver/uring/mod.rs new file mode 100644 index 000000000..5af2cfbde --- /dev/null +++ b/vendor/monoio/src/driver/uring/mod.rs @@ -0,0 +1,677 @@ +//! Monoio Uring Driver. + +use std::{ + cell::UnsafeCell, + io, + mem::ManuallyDrop, + os::unix::prelude::{AsRawFd, RawFd}, + rc::Rc, + task::{Context, Poll}, + time::Duration, +}; + +use io_uring::{cqueue, opcode, types::Timespec, IoUring}; +use lifecycle::Lifecycle; + +use super::{ + op::{CompletionMeta, Op, OpAble}, + // ready::Ready, + // scheduled_io::ScheduledIo, + util::timespec, + Driver, + Inner, + CURRENT, +}; +use crate::utils::slab::Slab; + +mod lifecycle; +#[cfg(feature = "sync")] +mod waker; +#[cfg(feature = "sync")] +pub(crate) use waker::UnparkHandle; + +#[allow(unused)] +pub(crate) const CANCEL_USERDATA: u64 = u64::MAX; +pub(crate) const TIMEOUT_USERDATA: u64 = u64::MAX - 1; +#[allow(unused)] +pub(crate) const EVENTFD_USERDATA: u64 = u64::MAX - 2; +#[cfg(feature = "poll-io")] +pub(crate) const POLLER_USERDATA: u64 = u64::MAX - 3; + +pub(crate) const MIN_REVERSED_USERDATA: u64 = u64::MAX - 3; + +// moon patch: CQ spin budget from MOON_URING_SPIN_US (0 = disabled), read once. +// See inner_park below for the poll-mode rationale. +fn moon_spin_budget_us() -> u64 { + static SPIN_US: std::sync::OnceLock = std::sync::OnceLock::new(); + *SPIN_US.get_or_init(|| { + std::env::var("MOON_URING_SPIN_US") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(0) + }) +} + +/// Driver with uring. +pub struct IoUringDriver { + inner: Rc>, + + // Used as timeout buffer + timespec: *mut Timespec, + + // Used as read eventfd buffer + #[cfg(feature = "sync")] + eventfd_read_dst: *mut u8, + + // Used for drop + #[cfg(feature = "sync")] + thread_id: usize, +} + +pub(crate) struct UringInner { + /// In-flight operations + ops: Ops, + + #[cfg(feature = "poll-io")] + poll: super::poll::Poll, + #[cfg(feature = "poll-io")] + poller_installed: bool, + + /// IoUring bindings + uring: ManuallyDrop, + + /// Shared waker + #[cfg(feature = "sync")] + shared_waker: std::sync::Arc, + + // Mark if eventfd is in the ring + #[cfg(feature = "sync")] + eventfd_installed: bool, + + // Waker receiver + #[cfg(feature = "sync")] + waker_receiver: flume::Receiver, + + // Uring support ext_arg + ext_arg: bool, +} + +// When dropping the driver, all in-flight operations must have completed. This +// type wraps the slab and ensures that, on drop, the slab is empty. +struct Ops { + slab: Slab, +} + +impl IoUringDriver { + const DEFAULT_ENTRIES: u32 = 1024; + + pub(crate) fn new(b: &io_uring::Builder) -> io::Result { + Self::new_with_entries(b, Self::DEFAULT_ENTRIES) + } + + #[cfg(not(feature = "sync"))] + pub(crate) fn new_with_entries( + urb: &io_uring::Builder, + entries: u32, + ) -> io::Result { + let uring = ManuallyDrop::new(urb.build(entries)?); + + let inner = Rc::new(UnsafeCell::new(UringInner { + #[cfg(feature = "poll-io")] + poll: super::poll::Poll::with_capacity(entries as usize)?, + #[cfg(feature = "poll-io")] + poller_installed: false, + ops: Ops::new(), + ext_arg: uring.params().is_feature_ext_arg(), + uring, + })); + + Ok(IoUringDriver { + inner, + timespec: Box::leak(Box::new(Timespec::new())) as *mut Timespec, + }) + } + + #[cfg(feature = "sync")] + pub(crate) fn new_with_entries( + urb: &io_uring::Builder, + entries: u32, + ) -> io::Result { + let uring = ManuallyDrop::new(urb.build(entries)?); + + // Create eventfd and register it to the ring. + let waker = { + let fd = crate::syscall!(eventfd(0, libc::EFD_CLOEXEC))?; + unsafe { + use std::os::unix::io::FromRawFd; + std::fs::File::from_raw_fd(fd) + } + }; + + let (waker_sender, waker_receiver) = flume::unbounded::(); + + let inner = Rc::new(UnsafeCell::new(UringInner { + #[cfg(feature = "poll-io")] + poller_installed: false, + #[cfg(feature = "poll-io")] + poll: super::poll::Poll::with_capacity(entries as usize)?, + ops: Ops::new(), + ext_arg: uring.params().is_feature_ext_arg(), + uring, + shared_waker: std::sync::Arc::new(waker::EventWaker::new(waker)), + eventfd_installed: false, + waker_receiver, + })); + + let thread_id = crate::builder::BUILD_THREAD_ID.with(|id| *id); + let driver = IoUringDriver { + inner, + timespec: Box::leak(Box::new(Timespec::new())) as *mut Timespec, + eventfd_read_dst: Box::leak(Box::new([0_u8; 8])) as *mut u8, + thread_id, + }; + + // Register unpark handle + super::thread::register_unpark_handle(thread_id, driver.unpark().into()); + super::thread::register_waker_sender(thread_id, waker_sender); + Ok(driver) + } + + #[allow(unused)] + fn num_operations(&self) -> usize { + let inner = self.inner.get(); + unsafe { (*inner).ops.slab.len() } + } + + // Flush to make enough space + fn flush_space(inner: &mut UringInner, need: usize) -> io::Result<()> { + let sq = inner.uring.submission(); + debug_assert!(sq.capacity() >= need); + if sq.len() + need > sq.capacity() { + drop(sq); + inner.submit()?; + } + Ok(()) + } + + #[cfg(feature = "sync")] + fn install_eventfd(&self, inner: &mut UringInner, fd: RawFd) { + let entry = opcode::Read::new(io_uring::types::Fd(fd), self.eventfd_read_dst, 8) + .build() + .user_data(EVENTFD_USERDATA); + + let mut sq = inner.uring.submission(); + let _ = unsafe { sq.push(&entry) }; + inner.eventfd_installed = true; + } + + #[cfg(feature = "poll-io")] + fn install_poller(&self, inner: &mut UringInner, fd: RawFd) { + let entry = opcode::PollAdd::new(io_uring::types::Fd(fd), libc::POLLIN as _) + .build() + .user_data(POLLER_USERDATA); + + let mut sq = inner.uring.submission(); + let _ = unsafe { sq.push(&entry) }; + inner.poller_installed = true; + } + + fn install_timeout(&self, inner: &mut UringInner, duration: Duration) { + let timespec = timespec(duration); + unsafe { + std::ptr::replace(self.timespec, timespec); + } + let entry = opcode::Timeout::new(self.timespec as *const Timespec) + .build() + .user_data(TIMEOUT_USERDATA); + + let mut sq = inner.uring.submission(); + let _ = unsafe { sq.push(&entry) }; + } + + fn inner_park(&self, timeout: Option) -> io::Result<()> { + let inner = unsafe { &mut *self.inner.get() }; + + #[allow(unused_mut)] + let mut need_wait = true; + + #[cfg(feature = "sync")] + { + // Process foreign wakers + while let Ok(w) = inner.waker_receiver.try_recv() { + w.wake(); + need_wait = false; + } + + // Set status as not awake if we are going to sleep + if need_wait { + inner + .shared_waker + .awake + .store(false, std::sync::atomic::Ordering::Release); + } + + // Process foreign wakers left + while let Ok(w) = inner.waker_receiver.try_recv() { + w.wake(); + need_wait = false; + } + } + + if need_wait { + // ---- moon patch: CQ spin-poll before blocking (MOON_URING_SPIN_US) ---- + // Poll-mode reaping for request/response workloads: submit pending + // SQEs immediately (one enter), then busy-poll the completion queue + // (shared-memory head/tail check, zero syscalls) for a bounded + // window before falling back to the stock blocking wait. When a + // completion lands inside the window the thread never sleeps, so + // the scheduler wakeup disappears from the per-op path. Foreign + // (cross-thread) wakes during the window are delayed at most by the + // window: the eventfd is already written, so the stock fallback's + // submit_and_wait returns immediately. Off unless MOON_URING_SPIN_US + // is set; 0 behavior change otherwise. + let spin_us = moon_spin_budget_us(); + if spin_us > 0 { + let budget = match timeout { + Some(d) => d.min(Duration::from_micros(spin_us)), + None => Duration::from_micros(spin_us), + }; + inner.uring.submit()?; + let deadline = std::time::Instant::now() + budget; + let mut hit = false; + loop { + let mut cq = inner.uring.completion(); + cq.sync(); + if !cq.is_empty() { + hit = true; + break; + } + if std::time::Instant::now() >= deadline { + break; + } + for _ in 0..32 { + std::hint::spin_loop(); + } + } + if hit { + #[cfg(feature = "sync")] + inner + .shared_waker + .awake + .store(true, std::sync::atomic::Ordering::Release); + inner.tick()?; + return Ok(()); + } + } + // ---- end moon patch ---- + + // Install timeout and eventfd for unpark if sync is enabled + + // 1. alloc spaces + let mut space = 0; + #[cfg(feature = "sync")] + if !inner.eventfd_installed { + space += 1; + } + #[cfg(feature = "poll-io")] + if !inner.poller_installed { + space += 1; + } + if timeout.is_some() { + space += 1; + } + if space != 0 { + Self::flush_space(inner, space)?; + } + + // 2.1 install poller + #[cfg(feature = "poll-io")] + if !inner.poller_installed { + self.install_poller(inner, inner.poll.as_raw_fd()); + } + + // 2.2 install eventfd and timeout + #[cfg(feature = "sync")] + if !inner.eventfd_installed { + self.install_eventfd(inner, inner.shared_waker.as_raw_fd()); + } + + // 2.3 install timeout and submit_and_wait with timeout + if let Some(duration) = timeout { + match inner.ext_arg { + // Submit and Wait with timeout in an TimeoutOp way. + // Better compatibility(5.4+). + false => { + self.install_timeout(inner, duration); + inner.uring.submit_and_wait(1)?; + } + // Submit and Wait with enter args. + // Better performance(5.11+). + true => { + let timespec = timespec(duration); + let args = io_uring::types::SubmitArgs::new().timespec(×pec); + if let Err(e) = inner.uring.submitter().submit_with_args(1, &args) { + if e.raw_os_error() != Some(libc::ETIME) { + return Err(e); + } + } + } + } + } else { + // Submit and Wait without timeout + inner.uring.submit_and_wait(1)?; + } + } else { + // Submit only + inner.uring.submit()?; + } + + // Set status as awake + #[cfg(feature = "sync")] + inner + .shared_waker + .awake + .store(true, std::sync::atomic::Ordering::Release); + + // Process CQ + inner.tick()?; + + Ok(()) + } + + #[cfg(feature = "poll-io")] + #[inline] + pub(crate) fn register_poll_io( + this: &Rc>, + source: &mut impl mio::event::Source, + interest: mio::Interest, + ) -> io::Result { + let inner = unsafe { &mut *this.get() }; + inner.poll.register(source, interest) + } + + #[cfg(feature = "poll-io")] + #[inline] + pub(crate) fn deregister_poll_io( + this: &Rc>, + source: &mut impl mio::event::Source, + token: usize, + ) -> io::Result<()> { + let inner = unsafe { &mut *this.get() }; + inner.poll.deregister(source, token) + } +} + +impl Driver for IoUringDriver { + /// Enter the driver context. This enables using uring types. + fn with(&self, f: impl FnOnce() -> R) -> R { + // TODO(ihciah): remove clone + let inner = Inner::Uring(self.inner.clone()); + CURRENT.set(&inner, f) + } + + fn submit(&self) -> io::Result<()> { + let inner = unsafe { &mut *self.inner.get() }; + inner.submit()?; + inner.tick()?; + Ok(()) + } + + fn park(&self) -> io::Result<()> { + self.inner_park(None) + } + + fn park_timeout(&self, duration: Duration) -> io::Result<()> { + self.inner_park(Some(duration)) + } + + #[cfg(feature = "sync")] + type Unpark = waker::UnparkHandle; + + #[cfg(feature = "sync")] + fn unpark(&self) -> Self::Unpark { + UringInner::unpark(&self.inner) + } +} + +impl UringInner { + fn tick(&mut self) -> io::Result<()> { + let cq = self.uring.completion(); + + for cqe in cq { + let index = cqe.user_data(); + match index { + #[cfg(feature = "sync")] + EVENTFD_USERDATA => self.eventfd_installed = false, + #[cfg(feature = "poll-io")] + POLLER_USERDATA => { + self.poller_installed = false; + self.poll.tick(Some(Duration::ZERO))?; + } + _ if index >= MIN_REVERSED_USERDATA => (), + _ => self.ops.complete(index as _, resultify(&cqe), cqe.flags()), + } + } + Ok(()) + } + + fn submit(&mut self) -> io::Result<()> { + loop { + match self.uring.submit() { + #[cfg(feature = "unstable")] + Err(ref e) + if matches!(e.kind(), io::ErrorKind::Other | io::ErrorKind::ResourceBusy) => + { + self.tick()?; + } + #[cfg(not(feature = "unstable"))] + Err(ref e) + if matches!(e.raw_os_error(), Some(libc::EAGAIN) | Some(libc::EBUSY)) => + { + // This error is constructed with io::Error::last_os_error(): + // https://github.com/tokio-rs/io-uring/blob/01c83bbce965d4aaf93ebfaa08c3aa8b7b0f5335/src/sys/mod.rs#L32 + // So we can use https://doc.rust-lang.org/nightly/std/io/struct.Error.html#method.raw_os_error + // to get the raw error code. + self.tick()?; + } + e => return e.map(|_| ()), + } + } + } + + fn new_op(data: T, inner: &mut UringInner, driver: Inner) -> Op { + Op { + driver, + index: inner.ops.insert(), + data: Some(data), + } + } + + pub(crate) fn submit_with_data( + this: &Rc>, + data: T, + ) -> io::Result> + where + T: OpAble, + { + let inner = unsafe { &mut *this.get() }; + // If the submission queue is full, flush it to the kernel + if inner.uring.submission().is_full() { + inner.submit()?; + } + + // Create the operation + let mut op = Self::new_op(data, inner, Inner::Uring(this.clone())); + + // Configure the SQE + let data_mut = unsafe { op.data.as_mut().unwrap_unchecked() }; + let sqe = OpAble::uring_op(data_mut).user_data(op.index as _); + + { + let mut sq = inner.uring.submission(); + + // Push the new operation + if unsafe { sq.push(&sqe).is_err() } { + unimplemented!("when is this hit?"); + } + } + + // Submit the new operation. At this point, the operation has been + // pushed onto the queue and the tail pointer has been updated, so + // the submission entry is visible to the kernel. If there is an + // error here (probably EAGAIN), we still return the operation. A + // future `io_uring_enter` will fully submit the event. + + // CHIHAI: We are not going to do syscall now. If we are waiting + // for IO, we will submit on `park`. + // let _ = inner.submit(); + Ok(op) + } + + pub(crate) fn poll_op( + this: &Rc>, + index: usize, + cx: &mut Context<'_>, + ) -> Poll { + let inner = unsafe { &mut *this.get() }; + let lifecycle = unsafe { inner.ops.slab.get(index).unwrap_unchecked() }; + lifecycle.poll_op(cx) + } + + #[cfg(feature = "poll-io")] + pub(crate) fn poll_legacy_op( + this: &Rc>, + data: &mut T, + cx: &mut Context<'_>, + ) -> Poll { + let inner = unsafe { &mut *this.get() }; + let (direction, index) = match data.legacy_interest() { + Some(x) => x, + None => { + // if there is no index provided, it means the action does not rely on fd + // readiness. do syscall right now. + return Poll::Ready(CompletionMeta { + result: OpAble::legacy_call(data), + flags: 0, + }); + } + }; + + // wait io ready and do syscall + inner + .poll + .poll_syscall(cx, index, direction, || OpAble::legacy_call(data)) + } + + pub(crate) fn drop_op( + this: &Rc>, + index: usize, + data: &mut Option, + ) { + let inner = unsafe { &mut *this.get() }; + if index == usize::MAX { + // already finished + return; + } + if let Some(lifecycle) = inner.ops.slab.get(index) { + let _must_finished = lifecycle.drop_op(data); + #[cfg(feature = "async-cancel")] + if !_must_finished { + unsafe { + let cancel = opcode::AsyncCancel::new(index as u64) + .build() + .user_data(u64::MAX); + + // Try push cancel, if failed, will submit and re-push. + if inner.uring.submission().push(&cancel).is_err() { + let _ = inner.submit(); + let _ = inner.uring.submission().push(&cancel); + } + } + } + } + } + + pub(crate) unsafe fn cancel_op(this: &Rc>, index: usize) { + let inner = &mut *this.get(); + let cancel = opcode::AsyncCancel::new(index as u64) + .build() + .user_data(u64::MAX); + if inner.uring.submission().push(&cancel).is_err() { + let _ = inner.submit(); + let _ = inner.uring.submission().push(&cancel); + } + } + + #[cfg(feature = "sync")] + pub(crate) fn unpark(this: &Rc>) -> waker::UnparkHandle { + let inner = unsafe { &*this.get() }; + let weak = std::sync::Arc::downgrade(&inner.shared_waker); + waker::UnparkHandle(weak) + } +} + +impl AsRawFd for IoUringDriver { + fn as_raw_fd(&self) -> RawFd { + unsafe { (*self.inner.get()).uring.as_raw_fd() } + } +} + +impl Drop for IoUringDriver { + fn drop(&mut self) { + trace!("MONOIO DEBUG[IoUringDriver]: drop"); + + // Dealloc leaked memory + unsafe { std::ptr::drop_in_place(self.timespec) }; + + #[cfg(feature = "sync")] + unsafe { + std::ptr::drop_in_place(self.eventfd_read_dst) + }; + + // Deregister thread id + #[cfg(feature = "sync")] + { + use crate::driver::thread::{unregister_unpark_handle, unregister_waker_sender}; + unregister_unpark_handle(self.thread_id); + unregister_waker_sender(self.thread_id); + } + } +} + +impl Drop for UringInner { + fn drop(&mut self) { + // no need to wait for completion, as the kernel will clean up the ring asynchronically. + let _ = self.uring.submitter().submit(); + unsafe { + ManuallyDrop::drop(&mut self.uring); + } + } +} + +impl Ops { + const fn new() -> Self { + Ops { slab: Slab::new() } + } + + // Insert a new operation + pub(crate) fn insert(&mut self) -> usize { + self.slab.insert(Lifecycle::Submitted) + } + + fn complete(&mut self, index: usize, result: io::Result, flags: u32) { + let lifecycle = unsafe { self.slab.get(index).unwrap_unchecked() }; + lifecycle.complete(result, flags); + } +} + +#[inline] +fn resultify(cqe: &cqueue::Entry) -> io::Result { + let res = cqe.result(); + + if res >= 0 { + Ok(res as u32) + } else { + Err(io::Error::from_raw_os_error(-res)) + } +} diff --git a/vendor/monoio/src/driver/uring/waker.rs b/vendor/monoio/src/driver/uring/waker.rs new file mode 100644 index 000000000..83c56b16a --- /dev/null +++ b/vendor/monoio/src/driver/uring/waker.rs @@ -0,0 +1,57 @@ +//! Custom thread waker based on eventfd. + +use std::os::unix::prelude::{AsRawFd, RawFd}; + +use crate::driver::unpark::Unpark; + +pub(crate) struct EventWaker { + // RawFd + raw: RawFd, + // File hold the ownership of fd, only useful when drop + _file: std::fs::File, + // Atomic awake status + pub(crate) awake: std::sync::atomic::AtomicBool, +} + +impl EventWaker { + pub(crate) fn new(file: std::fs::File) -> Self { + Self { + raw: file.as_raw_fd(), + _file: file, + awake: std::sync::atomic::AtomicBool::new(true), + } + } + + pub(crate) fn wake(&self) -> std::io::Result<()> { + // Skip wake if already awake + if self.awake.load(std::sync::atomic::Ordering::Acquire) { + return Ok(()); + } + // Write data into EventFd to wake the executor. + let buf = 0x1u64.to_ne_bytes(); + unsafe { + // SAFETY: Writing number to eventfd is thread safe. + libc::write(self.raw, buf.as_ptr().cast(), buf.len()); + Ok(()) + } + } +} + +impl AsRawFd for EventWaker { + fn as_raw_fd(&self) -> RawFd { + self.raw + } +} + +#[derive(Clone)] +pub struct UnparkHandle(pub(crate) std::sync::Weak); + +impl Unpark for UnparkHandle { + fn unpark(&self) -> std::io::Result<()> { + if let Some(w) = self.0.upgrade() { + w.wake() + } else { + Ok(()) + } + } +} diff --git a/vendor/monoio/src/driver/util.rs b/vendor/monoio/src/driver/util.rs new file mode 100644 index 000000000..20f16fc50 --- /dev/null +++ b/vendor/monoio/src/driver/util.rs @@ -0,0 +1,80 @@ +use std::{ffi::CString, io, path::Path}; + +#[allow(unused_variables)] +pub(super) fn cstr(p: &Path) -> io::Result { + #[cfg(unix)] + { + use std::os::unix::ffi::OsStrExt; + Ok(CString::new(p.as_os_str().as_bytes())?) + } + #[cfg(windows)] + if let Some(s) = p.as_os_str().to_str() { + Ok(CString::new(s)?) + } else { + Err(io::Error::new( + io::ErrorKind::InvalidInput, + "invalid utf-8: corrupt contents", + )) + } +} + +// Convert Duration to Timespec +// It's strange that io_uring does not impl From for Timespec. +#[cfg(all(target_os = "linux", feature = "iouring"))] +pub(super) fn timespec(duration: std::time::Duration) -> io_uring::types::Timespec { + io_uring::types::Timespec::new() + .sec(duration.as_secs()) + .nsec(duration.subsec_nanos()) +} + +/// Do syscall and return Result +#[cfg(unix)] +#[macro_export] +macro_rules! syscall { + ($fn: ident ( $($arg: expr),* $(,)* ) ) => {{ + let res = unsafe { libc::$fn($($arg, )*) }; + if res == -1 { + Err(std::io::Error::last_os_error()) + } else { + Ok(res) + } + }}; +} + +/// Do syscall and return Result +#[cfg(windows)] +#[macro_export] +macro_rules! syscall { + ($fn: ident ( $($arg: expr),* $(,)* ), $err_test: path, $err_value: expr) => {{ + let res = unsafe { $fn($($arg, )*) }; + if $err_test(&res, &$err_value) { + Err(std::io::Error::last_os_error()) + } else { + Ok(res.try_into().unwrap()) + } + }}; +} + +/// Do syscall and return Result +#[macro_export] +macro_rules! syscall_u32 { + ($fn: ident ( $($arg: expr),* $(,)* ) ) => {{ + #[cfg(windows)] + let res = unsafe { $fn($($arg, )*) }; + #[cfg(unix)] + let res = unsafe { libc::$fn($($arg, )*) }; + if res < 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(res as u32) + } + }}; +} + +#[cfg(all( + not(all(target_os = "linux", feature = "iouring")), + not(feature = "legacy") +))] +pub(crate) fn feature_panic() -> ! { + panic!("one of iouring and legacy features must be enabled"); +} diff --git a/vendor/monoio/src/fs/create_dir.rs b/vendor/monoio/src/fs/create_dir.rs new file mode 100644 index 000000000..9fc59494c --- /dev/null +++ b/vendor/monoio/src/fs/create_dir.rs @@ -0,0 +1,62 @@ +use std::{io, path::Path}; + +use super::DirBuilder; + +/// Create a new directory at the target path +/// +/// # Note +/// +/// - This function require the provided path's parent are all existing. +/// - To create a directory and all its missing parents at the same time, use the +/// [`create_dir_all`] function. +/// - Currently this function is supported on unix, windows is unimplement. +/// +/// # Errors +/// +/// This function will return an error in the following situations, but is not +/// limited to just these cases: +/// +/// * User lacks permissions to create directory at `path`. +/// * A parent of the given path doesn't exist. (To create a directory and all its missing parents +/// at the same time, use the [`create_dir_all`] function.) +/// * `path` already exists. +/// +/// # Examples +/// +/// ```no_run +/// use monoio::fs; +/// +/// #[monoio::main] +/// async fn main() -> std::io::Result<()> { +/// fs::create_dir("/some/dir").await?; +/// Ok(()) +/// } +/// ``` +pub async fn create_dir>(path: P) -> io::Result<()> { + DirBuilder::new().create(path).await +} + +/// Recursively create a directory and all of its missing components +/// +/// # Note +/// +/// - Currently this function is supported on unix, windows is unimplement. +/// +/// # Errors +/// +/// Same with [`create_dir`] +/// +/// # Examples +/// +/// ```no_run +/// use monoio::fs; +/// +/// #[monoio::main] +/// async fn main() -> std::io::Result<()> { +/// fs::create_dir_all("/some/dir").await?; +/// Ok(()) +/// } +/// ``` +pub async fn create_dir_all>(path: P) -> io::Result<()> { + DirBuilder::new().recursive(true).create(path).await +} diff --git a/vendor/monoio/src/fs/dir_builder/mod.rs b/vendor/monoio/src/fs/dir_builder/mod.rs new file mode 100644 index 000000000..b9efd11bb --- /dev/null +++ b/vendor/monoio/src/fs/dir_builder/mod.rs @@ -0,0 +1,152 @@ +mod unix; + +use std::{io, os::unix::fs::DirBuilderExt, path::Path}; + +#[cfg(unix)] +use unix as sys; + +/// A builder used to create directories in various manners. +/// +/// This builder also supports platform-specific options. +pub struct DirBuilder { + recursive: bool, + inner: sys::BuilderInner, +} + +impl DirBuilder { + /// Creates a new set of options with default mode/security settings for all + /// platforms and also non-recursive. + /// + /// This an async version of [`std::fs::DirBuilder::new`] + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::DirBuilder; + /// + /// let builder = DirBuilder::new(); + /// ``` + pub fn new() -> Self { + Self { + recursive: false, + inner: sys::BuilderInner::new(), + } + } + + /// Indicates that directories should be created recursively, creating all + /// parent directories. Parents that do not exist are created with the same + /// security and permissions settings. + /// + /// This option defaults to `false`. + /// + /// This an async version of [`std::fs::DirBuilder::recursive`] + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::DirBuilder; + /// + /// let mut builder = DirBuilder::new(); + /// builder.recursive(true); + /// ``` + pub fn recursive(&mut self, recursive: bool) -> &mut Self { + self.recursive = recursive; + self + } + + /// Creates the specified directory with the options configured in this + /// builder. + /// + /// It is considered an error if the directory already exists unless + /// recursive mode is enabled. + /// + /// This is async version of [`std::fs::DirBuilder::create`] and use io-uring + /// in support platform. + /// + /// # Errors + /// + /// An error will be returned under the following circumstances: + /// + /// * Path already points to an existing file. + /// * Path already points to an existing directory and the mode is non-recursive. + /// * The calling process doesn't have permissions to create the directory or its missing + /// parents. + /// * Other I/O error occurred. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::DirBuilder; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// DirBuilder::new() + /// .recursive(true) + /// .create("/some/dir") + /// .await?; + /// + /// Ok(()) + /// } + /// ``` + pub async fn create>(&self, path: P) -> io::Result<()> { + if self.recursive { + self.create_dir_all(path.as_ref()).await + } else { + self.inner.mkdir(path.as_ref()).await + } + } + + async fn create_dir_all(&self, path: &Path) -> io::Result<()> { + if path == Path::new("") { + return Ok(()); + } + + let mut inexist_path = path; + let mut need_create = vec![]; + + while match self.inner.mkdir(inexist_path).await { + Ok(()) => false, + Err(ref e) if e.kind() == io::ErrorKind::NotFound => true, + Err(_) if is_dir(inexist_path).await => false, + Err(e) => return Err(e), + } { + match inexist_path.parent() { + Some(p) => { + need_create.push(inexist_path); + inexist_path = p; + } + None => { + return Err(io::Error::new( + io::ErrorKind::Other, + "failed to create whole tree", + )) + } + } + } + + for p in need_create.into_iter().rev() { + self.inner.mkdir(p).await?; + } + + Ok(()) + } +} + +impl Default for DirBuilder { + fn default() -> Self { + Self::new() + } +} + +impl DirBuilderExt for DirBuilder { + fn mode(&mut self, mode: u32) -> &mut Self { + self.inner.set_mode(mode); + self + } +} + +// currently, will use the std version of metadata, will change to use the io-uring version +// when the statx is merge +async fn is_dir(path: &Path) -> bool { + std::fs::metadata(path).is_ok_and(|metadata| metadata.is_dir()) +} diff --git a/vendor/monoio/src/fs/dir_builder/unix.rs b/vendor/monoio/src/fs/dir_builder/unix.rs new file mode 100644 index 000000000..ec5745a3f --- /dev/null +++ b/vendor/monoio/src/fs/dir_builder/unix.rs @@ -0,0 +1,23 @@ +use std::path::Path; + +use libc::mode_t; + +use crate::driver::op::Op; + +pub(super) struct BuilderInner { + mode: libc::mode_t, +} + +impl BuilderInner { + pub(super) fn new() -> Self { + Self { mode: 0o777 } + } + + pub(super) async fn mkdir(&self, path: &Path) -> std::io::Result<()> { + Op::mkdir(path, self.mode)?.await.meta.result.map(|_| ()) + } + + pub(super) fn set_mode(&mut self, mode: u32) { + self.mode = mode as mode_t; + } +} diff --git a/vendor/monoio/src/fs/file.rs b/vendor/monoio/src/fs/file.rs new file mode 100644 index 000000000..70b1ab3f8 --- /dev/null +++ b/vendor/monoio/src/fs/file.rs @@ -0,0 +1,543 @@ +#[cfg(windows)] +use std::os::windows::io::{AsRawHandle, RawHandle}; +#[cfg(unix)] +use std::{ + fs::File as StdFile, + os::{ + fd::IntoRawFd, + unix::io::{AsRawFd, RawFd}, + }, +}; +use std::{io, path::Path}; + +#[cfg(unix)] +use super::{metadata::FileAttr, Metadata}; +use crate::{ + buf::{IoBuf, IoBufMut}, + driver::{op::Op, shared_fd::SharedFd}, + fs::OpenOptions, +}; + +/// A reference to an open file on the filesystem. +/// +/// An instance of a `File` can be read and/or written depending on what options +/// it was opened with. The `File` type provides **positional** read and write +/// operations. The file does not maintain an internal cursor. The caller is +/// required to specify an offset when issuing an operation. +/// +/// While files are automatically closed when they go out of scope, the +/// operation happens asynchronously in the background. It is recommended to +/// call the `close()` function in order to guarantee that the file successfully +/// closed before exiting the scope. Closing a file does not guarantee writes +/// have persisted to disk. Use [`sync_all`] to ensure all writes have reached +/// the filesystem. +/// +/// [`sync_all`]: File::sync_all +/// +/// # Examples +/// +/// Creates a new file and write data to it: +/// +/// ```no_run +/// use monoio::fs::File; +/// +/// #[monoio::main] +/// async fn main() -> Result<(), Box> { +/// // Open a file +/// let file = File::create("hello.txt").await?; +/// +/// // Write some data +/// let (res, buf) = file.write_at(&b"hello world"[..], 0).await; +/// let n = res?; +/// +/// println!("wrote {} bytes", n); +/// +/// // Sync data to the file system. +/// file.sync_all().await?; +/// +/// // Close the file +/// file.close().await?; +/// +/// Ok(()) +/// } +/// ``` +#[derive(Debug)] +pub struct File { + /// Open file descriptor + fd: SharedFd, +} + +impl File { + /// Attempts to open a file in read-only mode. + /// + /// See the [`OpenOptions::open`] method for more details. + /// + /// # Errors + /// + /// This function will return an error if `path` does not already exist. + /// Other errors may also be returned according to [`OpenOptions::open`]. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let f = File::open("foo.txt").await?; + /// + /// // Close the file + /// f.close().await?; + /// Ok(()) + /// } + /// ``` + pub async fn open(path: impl AsRef) -> io::Result { + OpenOptions::new().read(true).open(path).await + } + + /// Opens a file in write-only mode. + /// + /// This function will create a file if it does not exist, + /// and will truncate it if it does. + /// + /// See the [`OpenOptions::open`] function for more details. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let f = File::create("foo.txt").await?; + /// + /// // Close the file + /// f.close().await?; + /// Ok(()) + /// } + /// ``` + pub async fn create(path: impl AsRef) -> io::Result { + OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .open(path) + .await + } + + pub(crate) fn from_shared_fd(fd: SharedFd) -> File { + File { fd } + } + + /// Converts a [`std::fs::File`] to a [`monoio::fs::File`](File). + /// + /// # Examples + /// + /// ```no_run + /// // This line could block. It is not recommended to do this on the monoio + /// // runtime. + /// let std_file = std::fs::File::open("foo.txt").unwrap(); + /// let file = monoio::fs::File::from_std(std_file); + /// ``` + #[cfg(unix)] + pub fn from_std(std: StdFile) -> io::Result { + Ok(File { + fd: SharedFd::new_without_register(std.into_raw_fd()), + }) + } + + /// Read some bytes at the specified offset from the file into the specified + /// buffer, returning how many bytes were read. + /// + /// # Return + /// + /// The method returns the operation result and the same buffer value passed + /// as an argument. + /// + /// If the method returns [`Ok(n)`], then the read was successful. A nonzero + /// `n` value indicates that the buffer has been filled with `n` bytes of + /// data from the file. If `n` is `0`, then one of the following happened: + /// + /// 1. The specified offset is the end of the file. + /// 2. The buffer specified was 0 bytes in length. + /// + /// It is not an error if the returned value `n` is smaller than the buffer + /// size, even when the file contains enough data to fill the buffer. + /// + /// # Errors + /// + /// If this function encounters any form of I/O or other error, an error + /// variant will be returned. The buffer is returned on error. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let f = File::open("foo.txt").await?; + /// let buffer = vec![0; 10]; + /// + /// // Read up to 10 bytes + /// let (res, buffer) = f.read_at(buffer, 0).await; + /// let n = res?; + /// + /// println!("The bytes: {:?}", &buffer[..n]); + /// + /// // Close the file + /// f.close().await?; + /// Ok(()) + /// } + /// ``` + pub async fn read_at(&self, buf: T, pos: u64) -> crate::BufResult { + // Submit the read operation + let op = Op::read_at(&self.fd, buf, pos).unwrap(); + op.read().await + } + + /// Read the exact number of bytes required to fill `buf` at the specified + /// offset from the file. + /// + /// This function reads as many as bytes as necessary to completely fill the + /// specified buffer `buf`. + /// + /// # Return + /// + /// The method returns the operation result and the same buffer value passed + /// as an argument. + /// + /// If the method returns [`Ok(())`], then the read was successful. + /// + /// # Errors + /// + /// If this function encounters an error of the kind + /// [`ErrorKind::Interrupted`] then the error is ignored and the + /// operation will continue. + /// + /// If this function encounters an "end of file" before completely filling + /// the buffer, it returns an error of the kind + /// [`ErrorKind::UnexpectedEof`]. The buffer is returned on error. + /// + /// If this function encounters any form of I/O or other error, an error + /// variant will be returned. The buffer is returned on error. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let f = File::open("foo.txt").await?; + /// let buffer = vec![0; 10]; + /// + /// // Read up to 10 bytes + /// let (res, buffer) = f.read_exact_at(buffer, 0).await; + /// res?; + /// + /// println!("The bytes: {:?}", buffer); + /// + /// // Close the file + /// f.close().await?; + /// Ok(()) + /// } + /// ``` + /// + /// [`ErrorKind::Interrupted`]: std::io::ErrorKind::Interrupted + /// [`ErrorKind::UnexpectedEof`]: std::io::ErrorKind::UnexpectedEof + pub async fn read_exact_at( + &self, + mut buf: T, + pos: u64, + ) -> crate::BufResult<(), T> { + let len = buf.bytes_total(); + let mut read = 0; + while read < len { + let slice = unsafe { buf.slice_mut_unchecked(read..len) }; + let (res, slice) = self.read_at(slice, pos + read as u64).await; + buf = slice.into_inner(); + match res { + Ok(0) => { + return ( + Err(io::Error::new( + io::ErrorKind::UnexpectedEof, + "failed to fill whole buffer", + )), + buf, + ) + } + Ok(n) => { + read += n; + } + Err(ref e) if e.kind() == io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + }; + } + + (Ok(()), buf) + } + + /// Write a buffer into this file at the specified offset, returning how + /// many bytes were written. + /// + /// This function will attempt to write the entire contents of `buf`, but + /// the entire write may not succeed, or the write may also generate an + /// error. The bytes will be written starting at the specified offset. + /// + /// # Return + /// + /// The method returns the operation result and the same buffer value passed + /// in as an argument. A return value of `0` typically means that the + /// underlying file is no longer able to accept bytes and will likely not be + /// able to in the future as well, or that the buffer provided is empty. + /// + /// # Errors + /// + /// Each call to `write` may generate an I/O error indicating that the + /// operation could not be completed. If an error is returned then no bytes + /// in the buffer were written to this writer. + /// + /// It is **not** considered an error if the entire buffer could not be + /// written to this writer. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = File::create("foo.txt").await?; + /// + /// // Writes some prefix of the byte string, not necessarily all of it. + /// let (res, _) = file.write_at(&b"some bytes"[..], 0).await; + /// let n = res?; + /// + /// println!("wrote {} bytes", n); + /// + /// // Close the file + /// file.close().await?; + /// Ok(()) + /// } + /// ``` + /// + /// [`Ok(n)`]: Ok + pub async fn write_at(&self, buf: T, pos: u64) -> crate::BufResult { + let op = Op::write_at(&self.fd, buf, pos).unwrap(); + op.write().await + } + + /// Attempts to write an entire buffer into this file at the specified + /// offset. + /// + /// This method will continuously call [`write_at`] until there is no more + /// data to be written or an error of non-[`ErrorKind::Interrupted`] + /// kind is returned. This method will not return until the entire + /// buffer has been successfully written or such an error occurs. + /// + /// If the buffer contains no data, this will never call [`write_at`]. + /// + /// # Return + /// + /// The method returns the operation result and the same buffer value passed + /// in as an argument. + /// + /// # Errors + /// + /// This function will return the first error of + /// non-[`ErrorKind::Interrupted`] kind that [`write_at`] returns. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = File::create("foo.txt").await?; + /// + /// // Writes some prefix of the byte string, not necessarily all of it. + /// let (res, _) = file.write_all_at(&b"some bytes"[..], 0).await; + /// res?; + /// + /// println!("wrote all bytes"); + /// + /// // Close the file + /// file.close().await?; + /// Ok(()) + /// } + /// ``` + /// + /// [`write_at`]: File::write_at + /// [`ErrorKind::Interrupted`]: std::io::ErrorKind::Interrupted + pub async fn write_all_at(&self, mut buf: T, pos: u64) -> crate::BufResult<(), T> { + let len = buf.bytes_init(); + let mut written = 0; + while written < len { + let slice = unsafe { buf.slice_unchecked(written..len) }; + let (res, slice) = self.write_at(slice, pos + written as u64).await; + buf = slice.into_inner(); + match res { + Ok(0) => { + return ( + Err(io::Error::new( + io::ErrorKind::WriteZero, + "failed to write whole buffer", + )), + buf, + ) + } + Ok(n) => written += n, + Err(ref e) if e.kind() == io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + }; + } + + (Ok(()), buf) + } + + /// Attempts to sync all OS-internal metadata to disk. + /// + /// This function will attempt to ensure that all in-memory data reaches the + /// filesystem before completing. + /// + /// This can be used to handle errors that would otherwise only be caught + /// when the `File` is closed. Dropping a file will ignore errors in + /// synchronizing this in-memory data. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let f = File::create("foo.txt").await?; + /// let (res, buf) = f.write_at(&b"Hello, world!"[..], 0).await; + /// let n = res?; + /// + /// f.sync_all().await?; + /// + /// // Close the file + /// f.close().await?; + /// Ok(()) + /// } + /// ``` + pub async fn sync_all(&self) -> io::Result<()> { + let op = Op::fsync(&self.fd).unwrap(); + let completion = op.await; + + completion.meta.result?; + Ok(()) + } + + /// Attempts to sync file data to disk. + /// + /// This method is similar to [`sync_all`], except that it may not + /// synchronize file metadata to the filesystem. + /// + /// This is intended for use cases that must synchronize content, but don't + /// need the metadata on disk. The goal of this method is to reduce disk + /// operations. + /// + /// Note that some platforms may simply implement this in terms of + /// [`sync_all`]. + /// + /// [`sync_all`]: File::sync_all + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let f = File::create("foo.txt").await?; + /// let (res, buf) = f.write_at(&b"Hello, world!"[..], 0).await; + /// let n = res?; + /// + /// f.sync_data().await?; + /// + /// // Close the file + /// f.close().await?; + /// Ok(()) + /// } + /// ``` + pub async fn sync_data(&self) -> io::Result<()> { + let op = Op::datasync(&self.fd).unwrap(); + let completion = op.await; + + completion.meta.result?; + Ok(()) + } + + /// Closes the file. + /// + /// The method completes once the close operation has completed, + /// guaranteeing that resources associated with the file have been released. + /// + /// If `close` is not called before dropping the file, the file is closed in + /// the background, but there is no guarantee as to **when** the close + /// operation will complete. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// // Open the file + /// let f = File::open("foo.txt").await?; + /// // Close the file + /// f.close().await?; + /// + /// Ok(()) + /// } + /// ``` + pub async fn close(self) -> io::Result<()> { + self.fd.close().await; + Ok(()) + } + + /// Queries metadata about the underlying file. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::File; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let mut f = File::open("foo.txt").await?; + /// let metadata = f.metadata().await?; + /// Ok(()) + /// } + /// ``` + #[cfg(unix)] + pub async fn metadata(&self) -> io::Result { + #[cfg(target_os = "linux")] + let flags = libc::AT_STATX_SYNC_AS_STAT | libc::AT_EMPTY_PATH; + #[cfg(target_os = "linux")] + let op = Op::statx_using_fd(&self.fd, flags)?; + #[cfg(target_os = "macos")] + let op = Op::statx_using_fd(&self.fd, true)?; + + op.statx_result().await.map(FileAttr::from).map(Metadata) + } +} + +#[cfg(unix)] +impl AsRawFd for File { + fn as_raw_fd(&self) -> RawFd { + self.fd.raw_fd() + } +} + +#[cfg(windows)] +impl AsRawHandle for File { + fn as_raw_handle(&self) -> RawHandle { + self.fd.raw_handle() + } +} diff --git a/vendor/monoio/src/fs/file_type.rs b/vendor/monoio/src/fs/file_type.rs new file mode 100644 index 000000000..a5a8884cf --- /dev/null +++ b/vendor/monoio/src/fs/file_type.rs @@ -0,0 +1,63 @@ +use std::{fmt::Debug, os::unix::fs::FileTypeExt}; + +use libc::mode_t; + +/// A structure representing a type of file with accessors for each file type. +#[derive(PartialEq, Eq, Clone, Copy, Hash)] +pub struct FileType { + pub(crate) mode: mode_t, +} + +#[cfg(unix)] +impl FileType { + /// Returns `true` if this file type is a directory. + pub fn is_dir(&self) -> bool { + self.is(libc::S_IFDIR) + } + + /// Returns `true` if this file type is a regular file. + pub fn is_file(&self) -> bool { + self.is(libc::S_IFREG) + } + + /// Returns `true` if this file type is a symbolic link. + pub fn is_symlink(&self) -> bool { + self.is(libc::S_IFLNK) + } + + pub(crate) fn is(&self, mode: mode_t) -> bool { + self.masked() == mode + } + + fn masked(&self) -> mode_t { + self.mode & libc::S_IFMT + } +} + +impl FileTypeExt for FileType { + fn is_block_device(&self) -> bool { + self.is(libc::S_IFBLK) + } + + fn is_char_device(&self) -> bool { + self.is(libc::S_IFCHR) + } + + fn is_fifo(&self) -> bool { + self.is(libc::S_IFIFO) + } + + fn is_socket(&self) -> bool { + self.is(libc::S_IFSOCK) + } +} + +impl Debug for FileType { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("FileType") + .field("is_file", &self.is_file()) + .field("is_dir", &self.is_dir()) + .field("is_symlink", &self.is_symlink()) + .finish_non_exhaustive() + } +} diff --git a/vendor/monoio/src/fs/metadata/mod.rs b/vendor/monoio/src/fs/metadata/mod.rs new file mode 100644 index 000000000..55359bf0a --- /dev/null +++ b/vendor/monoio/src/fs/metadata/mod.rs @@ -0,0 +1,532 @@ +mod unix; +mod windows; + +use std::{os::unix::fs::MetadataExt, path::Path, time::SystemTime}; + +use super::{file_type::FileType, permissions::Permissions}; +use crate::driver::op::Op; + +/// Given a path, query the file system to get information about a file, +/// directory, etc. +/// +/// This function will traverse symbolic links to query information about the +/// destination file. +/// +/// # Platform-specific behavior +/// +/// current implementation is only for Linux. +/// +/// # Errors +/// +/// This function will return an error in the following situations, but is not +/// limited to just these cases: +/// +/// * The user lacks permissions to perform `metadata` call on `path`. +/// * execute(search) permission is required on all of the directories in path that lead to the +/// file. +/// * `path` does not exist. +/// +/// # Examples +/// +/// ```rust,no_run +/// use monoio::fs; +/// +/// #[monoio::main] +/// async fn main() -> std::io::Result<()> { +/// let attr = fs::metadata("/some/file/path.txt").await?; +/// // inspect attr ... +/// Ok(()) +/// } +/// ``` +pub async fn metadata>(path: P) -> std::io::Result { + #[cfg(target_os = "linux")] + let flags = libc::AT_STATX_SYNC_AS_STAT; + + #[cfg(target_os = "linux")] + let op = Op::statx_using_path(path, flags)?; + + #[cfg(target_os = "macos")] + let op = Op::statx_using_path(path, true)?; + + op.statx_result().await.map(FileAttr::from).map(Metadata) +} + +/// Query the metadata about a file without following symlinks. +/// +/// # Platform-specific behavior +/// +/// This function currently corresponds to the `lstat` function on linux +/// +/// # Errors +/// +/// This function will return an error in the following situations, but is not +/// limited to just these cases: +/// +/// * The user lacks permissions to perform `metadata` call on `path`. +/// * execute(search) permission is required on all of the directories in path that lead to the +/// file. +/// * `path` does not exist. +/// +/// # Examples +/// ```rust,no_run +/// use monoio::fs; +/// +/// #[monoio::main] +/// async fn main() -> std::io::Result<()> { +/// let attr = fs::symlink_metadata("/some/file/path.txt").await?; +/// // inspect attr ... +/// Ok(()) +/// } +/// ``` +pub async fn symlink_metadata>(path: P) -> std::io::Result { + #[cfg(target_os = "linux")] + let flags = libc::AT_STATX_SYNC_AS_STAT | libc::AT_SYMLINK_NOFOLLOW; + + #[cfg(target_os = "linux")] + let op = Op::statx_using_path(path, flags)?; + + #[cfg(target_os = "macos")] + let op = Op::statx_using_path(path, false)?; + + op.statx_result().await.map(FileAttr::from).map(Metadata) +} + +#[cfg(unix)] +pub(crate) use unix::FileAttr; + +/// Metadata information about a file. +/// +/// This structure is returned from the [`metadata`] or +/// [`symlink_metadata`] function or method and represents known +/// metadata about a file such as its permissions, size, modification +/// times, etc. +#[cfg(unix)] +pub struct Metadata(pub(crate) FileAttr); + +impl Metadata { + /// Returns `true` if this metadata is for a directory. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("path/to/dir").await?; + /// + /// println!("{:?}", metadata.is_dir()); + /// Ok(()) + /// } + /// ``` + pub fn is_dir(&self) -> bool { + self.0.stat.st_mode & libc::S_IFMT == libc::S_IFDIR + } + + /// Returns `true` if this metadata is for a regular file. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("foo.txt").await?; + /// + /// println!("{:?}", metadata.is_file()); + /// Ok(()) + /// } + /// ``` + pub fn is_file(&self) -> bool { + self.0.stat.st_mode & libc::S_IFMT == libc::S_IFREG + } + + /// Returns `true` if this metadata is for a symbolic link. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("foo.txt").await?; + /// + /// println!("{:?}", metadata.is_symlink()); + /// Ok(()) + /// } + /// ``` + pub fn is_symlink(&self) -> bool { + self.0.stat.st_mode & libc::S_IFMT == libc::S_IFLNK + } + + /// Returns the size of the file, in bytes, this metadata is for. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("foo.txt").await?; + /// + /// println!("{:?}", metadata.len()); + /// Ok(()) + /// } + /// ``` + #[allow(clippy::len_without_is_empty)] + pub fn len(&self) -> u64 { + self.0.size() + } + + /// Returns the last modification time listed in this metadata. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("foo.txt").await?; + /// + /// println!("{:?}", metadata.modified()); + /// Ok(()) + /// } + pub fn modified(&self) -> std::io::Result { + let mtime = self.0.stat.st_mtime; + let mtime_nsec = self.0.stat.st_mtime_nsec as u32; + + Ok(SystemTime::UNIX_EPOCH + std::time::Duration::new(mtime as u64, mtime_nsec)) + } + + /// Returns the last access time listed in this metadata. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("foo.txt").await?; + /// + /// println!("{:?}", metadata.accessed()); + /// Ok(()) + /// } + /// ``` + pub fn accessed(&self) -> std::io::Result { + let atime = self.0.stat.st_atime; + let atime_nsec = self.0.stat.st_atime_nsec as u32; + + Ok(SystemTime::UNIX_EPOCH + std::time::Duration::new(atime as u64, atime_nsec)) + } + + /// Returns the creation time listed in this metadata. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("foo.txt").await?; + /// + /// println!("{:?}", metadata.created()); + /// Ok(()) + /// } + /// ``` + #[cfg(target_os = "linux")] + pub fn created(&self) -> std::io::Result { + if let Some(extra) = self.0.statx_extra_fields.as_ref() { + return if extra.stx_mask & libc::STATX_BTIME != 0 { + let btime = extra.stx_btime.tv_sec; + let btime_nsec = extra.stx_btime.tv_nsec; + + Ok(SystemTime::UNIX_EPOCH + std::time::Duration::new(btime as u64, btime_nsec)) + } else { + Err(std::io::Error::new( + std::io::ErrorKind::Other, + "Creation time is not available", + )) + }; + } + + Err(std::io::Error::new( + std::io::ErrorKind::Other, + "Creation time is not available", + )) + } + + /// Returns the permissions of the file this metadata is for. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("foo.txt").await?; + /// + /// println!("{:?}", metadata.permissions()); + /// Ok(()) + /// } + /// ``` + #[cfg(unix)] + pub fn permissions(&self) -> Permissions { + use super::permissions::Permissions; + + Permissions(self.0.perm()) + } + + /// Returns the file type for this metadata. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs; + /// + /// #[monoio::main] + /// async fn main() -> std::io::Result<()> { + /// let metadata = fs::metadata("foo.txt").await?; + /// + /// println!("{:?}", metadata.file_type()); + /// Ok(()) + /// } + /// ``` + #[cfg(unix)] + pub fn file_type(&self) -> FileType { + self.0.file_type() + } +} + +impl std::fmt::Debug for Metadata { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + let mut debug = f.debug_struct("Metadata"); + // debug.field("file_type", &self.file_type()); + debug.field("permissions", &self.permissions()); + debug.field("len", &self.len()); + if let Ok(modified) = self.modified() { + debug.field("modified", &modified); + } + if let Ok(accessed) = self.accessed() { + debug.field("accessed", &accessed); + } + #[cfg(target_os = "linux")] + if let Ok(created) = self.created() { + debug.field("created", &created); + } + debug.finish_non_exhaustive() + } +} + +#[cfg(all(target_os = "linux", not(target_pointer_width = "32")))] +impl MetadataExt for Metadata { + fn dev(&self) -> u64 { + self.0.stat.st_dev + } + + fn ino(&self) -> u64 { + self.0.stat.st_ino + } + + fn mode(&self) -> u32 { + self.0.stat.st_mode + } + + #[cfg(any(target_arch = "aarch64", target_arch = "riscv64"))] + fn nlink(&self) -> u64 { + self.0.stat.st_nlink.into() + } + + #[cfg(not(any(target_arch = "aarch64", target_arch = "riscv64")))] + fn nlink(&self) -> u64 { + self.0.stat.st_nlink + } + + fn uid(&self) -> u32 { + self.0.stat.st_uid + } + + fn gid(&self) -> u32 { + self.0.stat.st_gid + } + + fn rdev(&self) -> u64 { + self.0.stat.st_rdev + } + + fn size(&self) -> u64 { + self.0.stat.st_size as u64 + } + + fn atime(&self) -> i64 { + self.0.stat.st_atime + } + + fn atime_nsec(&self) -> i64 { + self.0.stat.st_atime_nsec + } + + fn mtime(&self) -> i64 { + self.0.stat.st_mtime + } + + fn mtime_nsec(&self) -> i64 { + self.0.stat.st_mtime_nsec + } + + fn ctime(&self) -> i64 { + self.0.stat.st_ctime + } + + fn ctime_nsec(&self) -> i64 { + self.0.stat.st_ctime_nsec + } + + fn blksize(&self) -> u64 { + self.0.stat.st_blksize as u64 + } + + fn blocks(&self) -> u64 { + self.0.stat.st_blocks as u64 + } +} + +#[cfg(all(target_os = "macos", not(target_pointer_width = "32")))] +impl MetadataExt for Metadata { + fn dev(&self) -> u64 { + self.0.stat.st_dev as u64 + } + + fn ino(&self) -> u64 { + self.0.stat.st_ino + } + + fn mode(&self) -> u32 { + self.0.stat.st_mode as u32 + } + + fn nlink(&self) -> u64 { + self.0.stat.st_nlink.into() + } + + fn uid(&self) -> u32 { + self.0.stat.st_uid + } + + fn gid(&self) -> u32 { + self.0.stat.st_gid + } + + fn rdev(&self) -> u64 { + self.0.stat.st_rdev as u64 + } + + fn size(&self) -> u64 { + self.0.stat.st_size as u64 + } + + fn atime(&self) -> i64 { + self.0.stat.st_atime + } + + fn atime_nsec(&self) -> i64 { + self.0.stat.st_atime_nsec + } + + fn mtime(&self) -> i64 { + self.0.stat.st_mtime + } + + fn mtime_nsec(&self) -> i64 { + self.0.stat.st_mtime_nsec + } + + fn ctime(&self) -> i64 { + self.0.stat.st_ctime + } + + fn ctime_nsec(&self) -> i64 { + self.0.stat.st_ctime_nsec + } + + fn blksize(&self) -> u64 { + self.0.stat.st_blksize as u64 + } + + fn blocks(&self) -> u64 { + self.0.stat.st_blocks as u64 + } +} + +#[cfg(all(unix, target_pointer_width = "32"))] +impl MetadataExt for Metadata { + fn dev(&self) -> u64 { + self.0.stat.st_dev.into() + } + + fn ino(&self) -> u64 { + self.0.stat.st_ino.into() + } + + fn mode(&self) -> u32 { + self.0.stat.st_mode + } + + fn nlink(&self) -> u64 { + self.0.stat.st_nlink.into() + } + + fn uid(&self) -> u32 { + self.0.stat.st_uid + } + + fn gid(&self) -> u32 { + self.0.stat.st_gid + } + + fn rdev(&self) -> u64 { + self.0.stat.st_rdev.into() + } + + fn size(&self) -> u64 { + self.0.stat.st_size as u64 + } + + fn atime(&self) -> i64 { + self.0.stat.st_atime.into() + } + + fn atime_nsec(&self) -> i64 { + self.0.stat.st_atime_nsec.into() + } + + fn mtime(&self) -> i64 { + self.0.stat.st_mtime.into() + } + + fn mtime_nsec(&self) -> i64 { + self.0.stat.st_mtime_nsec.into() + } + + fn ctime(&self) -> i64 { + self.0.stat.st_ctime.into() + } + + fn ctime_nsec(&self) -> i64 { + self.0.stat.st_ctime_nsec.into() + } + + fn blksize(&self) -> u64 { + self.0.stat.st_blksize as u64 + } + + fn blocks(&self) -> u64 { + self.0.stat.st_blocks as u64 + } +} diff --git a/vendor/monoio/src/fs/metadata/unix.rs b/vendor/monoio/src/fs/metadata/unix.rs new file mode 100644 index 000000000..352c8f783 --- /dev/null +++ b/vendor/monoio/src/fs/metadata/unix.rs @@ -0,0 +1,82 @@ +use libc::mode_t; + +use crate::fs::{file_type::FileType, permissions::FilePermissions}; + +pub(crate) struct FileAttr { + #[cfg(target_os = "linux")] + pub(crate) stat: libc::stat64, + #[cfg(target_os = "macos")] + pub(crate) stat: libc::stat, + #[cfg(target_os = "linux")] + pub(crate) statx_extra_fields: Option, +} + +#[cfg(unix)] +impl FileAttr { + pub(crate) fn size(&self) -> u64 { + self.stat.st_size as u64 + } + + pub(crate) fn perm(&self) -> FilePermissions { + FilePermissions { + mode: (self.stat.st_mode as mode_t), + } + } + + pub(crate) fn file_type(&self) -> FileType { + FileType { + mode: self.stat.st_mode as mode_t, + } + } +} + +/// Extra fields that are available in `statx` struct. +#[cfg(target_os = "linux")] +pub(crate) struct StatxExtraFields { + pub(crate) stx_mask: u32, + pub(crate) stx_btime: libc::statx_timestamp, +} + +/// Convert a `statx` struct to not platform-specific `FileAttr`. +/// Current implementation is only for Linux. +#[cfg(target_os = "linux")] +impl From for FileAttr { + fn from(buf: libc::statx) -> Self { + let mut stat: libc::stat64 = unsafe { std::mem::zeroed() }; + + stat.st_dev = libc::makedev(buf.stx_dev_major, buf.stx_dev_minor) as _; + stat.st_ino = buf.stx_ino as libc::ino64_t; + stat.st_nlink = buf.stx_nlink as libc::nlink_t; + stat.st_mode = buf.stx_mode as libc::mode_t; + stat.st_uid = buf.stx_uid as libc::uid_t; + stat.st_gid = buf.stx_gid as libc::gid_t; + stat.st_rdev = libc::makedev(buf.stx_rdev_major, buf.stx_rdev_minor) as _; + stat.st_size = buf.stx_size as libc::off64_t; + stat.st_blksize = buf.stx_blksize as libc::blksize_t; + stat.st_blocks = buf.stx_blocks as libc::blkcnt64_t; + stat.st_atime = buf.stx_atime.tv_sec as libc::time_t; + // `i64` on gnu-x86_64-x32, `c_ulong` otherwise. + stat.st_atime_nsec = buf.stx_atime.tv_nsec as _; + stat.st_mtime = buf.stx_mtime.tv_sec as libc::time_t; + stat.st_mtime_nsec = buf.stx_mtime.tv_nsec as _; + stat.st_ctime = buf.stx_ctime.tv_sec as libc::time_t; + stat.st_ctime_nsec = buf.stx_ctime.tv_nsec as _; + + let extra = StatxExtraFields { + stx_mask: buf.stx_mask, + stx_btime: buf.stx_btime, + }; + + Self { + stat, + statx_extra_fields: Some(extra), + } + } +} + +#[cfg(target_os = "macos")] +impl From for FileAttr { + fn from(stat: libc::stat) -> Self { + Self { stat } + } +} diff --git a/vendor/monoio/src/fs/metadata/windows.rs b/vendor/monoio/src/fs/metadata/windows.rs new file mode 100644 index 000000000..8b1378917 --- /dev/null +++ b/vendor/monoio/src/fs/metadata/windows.rs @@ -0,0 +1 @@ + diff --git a/vendor/monoio/src/fs/mod.rs b/vendor/monoio/src/fs/mod.rs new file mode 100644 index 000000000..665bc2dcd --- /dev/null +++ b/vendor/monoio/src/fs/mod.rs @@ -0,0 +1,168 @@ +//! Filesystem manipulation operations. + +mod file; +use std::{io, path::Path}; + +pub use file::File; + +#[cfg(all(unix, feature = "mkdirat"))] +mod dir_builder; +#[cfg(all(unix, feature = "mkdirat"))] +pub use dir_builder::DirBuilder; + +#[cfg(all(unix, feature = "mkdirat"))] +mod create_dir; +#[cfg(all(unix, feature = "mkdirat"))] +pub use create_dir::*; + +mod open_options; +pub use open_options::OpenOptions; + +#[cfg(unix)] +mod metadata; +#[cfg(unix)] +pub use metadata::{metadata, symlink_metadata, Metadata}; + +#[cfg(unix)] +mod file_type; +#[cfg(target_os = "linux")] +pub use file_type::FileType; + +#[cfg(unix)] +mod permissions; +#[cfg(target_os = "linux")] +pub use permissions::Permissions; + +use crate::buf::IoBuf; +#[cfg(all(unix, feature = "unlinkat"))] +use crate::driver::op::Op; + +/// Read the entire contents of a file into a bytes vector. +#[cfg(unix)] +pub async fn read>(path: P) -> io::Result> { + use std::os::fd::{AsRawFd, FromRawFd, IntoRawFd}; + + use crate::buf::IoBufMut; + + let file = File::open(path).await?; + let sys_file = unsafe { std::fs::File::from_raw_fd(file.as_raw_fd()) }; + let size = sys_file.metadata()?.len() as usize; + let _ = sys_file.into_raw_fd(); + + let (res, buf) = file + .read_exact_at(Vec::with_capacity(size).slice_mut(0..size), 0) + .await; + res?; + Ok(buf.into_inner()) +} + +/// Write a buffer as the entire contents of a file. +pub async fn write, C: IoBuf>(path: P, contents: C) -> (io::Result<()>, C) { + let file = match File::create(path).await { + Ok(f) => f, + Err(e) => return (Err(e), contents), + }; + file.write_all_at(contents, 0).await +} + +/// Removes a file from the filesystem. +/// +/// Note that there is no +/// guarantee that the file is immediately deleted (e.g., depending on +/// platform, other open file descriptors may prevent immediate removal). +/// +/// # Platform-specific behavior +/// +/// This function is currently only implemented for Unix. +/// +/// # Errors +/// +/// This function will return an error in the following situations, but is not +/// limited to just these cases: +/// +/// * `path` points to a directory. +/// * The file doesn't exist. +/// * The user lacks permissions to remove the file. +/// +/// # Examples +/// +/// ```no_run +/// use monoio::fs::File; +/// +/// #[monoio::main] +/// async fn main() -> std::io::Result<()> { +/// fs::remove_file("a.txt").await?; +/// Ok(()) +/// } +/// ``` +#[cfg(all(unix, feature = "unlinkat"))] +pub async fn remove_file>(path: P) -> io::Result<()> { + Op::unlink(path)?.await.meta.result?; + Ok(()) +} + +/// Removes an empty directory. +/// +/// # Platform-specific behavior +/// +/// This function is currently only implemented for Unix. +/// +/// # Errors +/// +/// This function will return an error in the following situations, but is not +/// limited to just these cases: +/// +/// * `path` doesn't exist. +/// * `path` isn't a directory. +/// * The user lacks permissions to remove the directory at the provided `path`. +/// * The directory isn't empty. +/// +/// # Examples +/// +/// ```no_run +/// use monoio::fs::File; +/// +/// #[monoio::main] +/// async fn main() -> std::io::Result<()> { +/// fs::remove_dir("/some/dir").await?; +/// Ok(()) +/// } +/// ``` +#[cfg(all(unix, feature = "unlinkat"))] +pub async fn remove_dir>(path: P) -> io::Result<()> { + Op::rmdir(path)?.await.meta.result?; + Ok(()) +} + +/// Rename a file or directory to a new name, replacing the original file if +/// `to` already exists. +/// +/// This will not work if the new name is on a different mount point. +/// +/// This is async version of [std::fs::rename]. +/// +/// # Errors +/// +/// This function will return an error in the following situations, but is not +/// limited to just these cases: +/// +/// * `from` does not exist. +/// * The user lacks permissions to view contents. +/// * `from` and `to` are on separate filesystems. +/// +/// # Examples +/// +/// ```no_run +/// use monoio::fs; +/// +/// #[monoio::main] +/// async fn main() -> std::io::Result<()> { +/// fs::rename("a.txt", "b.txt")?; // Rename a.txt to b.txt +/// Ok(()) +/// } +/// ``` +#[cfg(all(unix, feature = "renameat"))] +pub async fn rename, Q: AsRef>(from: P, to: Q) -> io::Result<()> { + Op::rename(from.as_ref(), to.as_ref())?.await.meta.result?; + Ok(()) +} diff --git a/vendor/monoio/src/fs/open_options.rs b/vendor/monoio/src/fs/open_options.rs new file mode 100644 index 000000000..73c3d737a --- /dev/null +++ b/vendor/monoio/src/fs/open_options.rs @@ -0,0 +1,460 @@ +#[cfg(unix)] +use std::os::unix::prelude::OpenOptionsExt; +use std::{io, path::Path}; + +#[cfg(windows)] +use windows_sys::Win32::{ + Foundation::{ERROR_INVALID_PARAMETER, GENERIC_READ, GENERIC_WRITE}, + Security::SECURITY_ATTRIBUTES, + Storage::FileSystem::{ + CREATE_ALWAYS, CREATE_NEW, FILE_FLAG_OPEN_REPARSE_POINT, FILE_GENERIC_WRITE, + FILE_SHARE_DELETE, FILE_SHARE_READ, FILE_SHARE_WRITE, FILE_WRITE_DATA, OPEN_ALWAYS, + OPEN_EXISTING, TRUNCATE_EXISTING, + }, +}; + +use crate::{ + driver::{op::Op, shared_fd::SharedFd}, + fs::File, +}; + +/// Options and flags which can be used to configure how a file is opened. +/// +/// This builder exposes the ability to configure how a [`File`] is opened and +/// what operations are permitted on the open file. The [`File::open`] and +/// [`File::create`] methods are aliases for commonly used options using this +/// builder. +/// +/// Generally speaking, when using `OpenOptions`, you'll first call +/// [`OpenOptions::new`], then chain calls to methods to set each option, then +/// call [`OpenOptions::open`], passing the path of the file you're trying to +/// open. This will give you a [`io::Result`] with a [`File`] inside that you +/// can further operate on. +/// +/// # Examples +/// +/// Opening a file to read: +/// +/// ```no_run +/// use monoio::fs::OpenOptions; +/// +/// #[monoio::main] +/// async fn main() -> Result<(), Box> { +/// let file = OpenOptions::new().read(true).open("foo.txt").await?; +/// Ok(()) +/// } +/// ``` +/// +/// Opening a file for both reading and writing, as well as creating it if it +/// doesn't exist: +/// +/// ```no_run +/// use monoio::fs::OpenOptions; +/// +/// #[monoio::main] +/// async fn main() -> Result<(), Box> { +/// let file = OpenOptions::new() +/// .read(true) +/// .write(true) +/// .create(true) +/// .open("foo.txt") +/// .await?; +/// Ok(()) +/// } +/// ``` +#[derive(Debug, Clone)] +pub struct OpenOptions { + read: bool, + write: bool, + append: bool, + truncate: bool, + create: bool, + create_new: bool, + #[cfg(unix)] + pub(crate) mode: libc::mode_t, + #[cfg(unix)] + pub(crate) custom_flags: libc::c_int, + #[cfg(windows)] + pub(crate) custom_flags: u32, + #[cfg(windows)] + pub(crate) access_mode: Option, + #[cfg(windows)] + pub(crate) attributes: u32, + #[cfg(windows)] + pub(crate) share_mode: u32, + #[cfg(windows)] + pub(crate) security_qos_flags: u32, + #[cfg(windows)] + pub(crate) security_attributes: *mut SECURITY_ATTRIBUTES, +} + +impl OpenOptions { + /// Creates a blank new set of options ready for configuration. + /// + /// All options are initially set to `false`. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::OpenOptions; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = OpenOptions::new().read(true).open("foo.txt").await?; + /// Ok(()) + /// } + /// ``` + #[allow(clippy::new_without_default)] + pub fn new() -> OpenOptions { + OpenOptions { + // generic + read: false, + write: false, + append: false, + truncate: false, + create: false, + create_new: false, + #[cfg(unix)] + mode: 0o666, + #[cfg(unix)] + custom_flags: 0, + #[cfg(windows)] + custom_flags: 0, + #[cfg(windows)] + access_mode: None, + #[cfg(windows)] + share_mode: FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, + #[cfg(windows)] + attributes: 0, + #[cfg(windows)] + security_qos_flags: 0, + #[cfg(windows)] + security_attributes: std::ptr::null_mut(), + } + } + + /// Sets the option for read access. + /// + /// This option, when true, will indicate that the file should be + /// `read`-able if opened. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::OpenOptions; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = OpenOptions::new().read(true).open("foo.txt").await?; + /// Ok(()) + /// } + /// ``` + pub fn read(&mut self, read: bool) -> &mut OpenOptions { + self.read = read; + self + } + + /// Sets the option for write access. + /// + /// This option, when true, will indicate that the file should be + /// `write`-able if opened. + /// + /// If the file already exists, any write calls on it will overwrite its + /// contents, without truncating it. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::OpenOptions; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = OpenOptions::new().write(true).open("foo.txt").await?; + /// Ok(()) + /// } + /// ``` + pub fn write(&mut self, write: bool) -> &mut OpenOptions { + self.write = write; + self + } + + /// Sets the option for the append mode. + /// + /// This option, when true, means that writes will append to a file instead + /// of overwriting previous contents. Note that setting + /// `.write(true).append(true)` has the same effect as setting only + /// `.append(true)`. + /// + /// For most filesystems, the operating system guarantees that all writes + /// are atomic: no writes get mangled because another process writes at the + /// same time. + /// + /// ## Note + /// + /// This function doesn't create the file if it doesn't exist. Use the + /// [`OpenOptions::create`] method to do so. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::OpenOptions; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = OpenOptions::new().append(true).open("foo.txt").await?; + /// Ok(()) + /// } + /// ``` + pub fn append(&mut self, append: bool) -> &mut OpenOptions { + self.append = append; + self + } + + /// Sets the option for truncating a previous file. + /// + /// If a file is successfully opened with this option set it will truncate + /// the file to 0 length if it already exists. + /// + /// The file must be opened with write access for truncate to work. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::OpenOptions; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = OpenOptions::new() + /// .write(true) + /// .truncate(true) + /// .open("foo.txt") + /// .await?; + /// Ok(()) + /// } + /// ``` + pub fn truncate(&mut self, truncate: bool) -> &mut OpenOptions { + self.truncate = truncate; + self + } + + /// Sets the option to create a new file, or open it if it already exists. + /// + /// In order for the file to be created, [`OpenOptions::write`] or + /// [`OpenOptions::append`] access must be used. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::OpenOptions; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = OpenOptions::new() + /// .write(true) + /// .create(true) + /// .open("foo.txt") + /// .await?; + /// Ok(()) + /// } + /// ``` + pub fn create(&mut self, create: bool) -> &mut OpenOptions { + self.create = create; + self + } + + /// Sets the option to create a new file, failing if it already exists. + /// + /// No file is allowed to exist at the target location, also no (dangling) + /// symlink. In this way, if the call succeeds, the file returned is + /// guaranteed to be new. + /// + /// This option is useful because it is atomic. Otherwise between checking + /// whether a file exists and creating a new one, the file may have been + /// created by another process (a TOCTOU race condition / attack). + /// + /// If `.create_new(true)` is set, [`.create()`] and [`.truncate()`] are + /// ignored. + /// + /// The file must be opened with write or append access in order to create + /// a new file. + /// + /// [`.create()`]: OpenOptions::create + /// [`.truncate()`]: OpenOptions::truncate + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::OpenOptions; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = OpenOptions::new() + /// .write(true) + /// .create_new(true) + /// .open("foo.txt") + /// .await?; + /// Ok(()) + /// } + /// ``` + pub fn create_new(&mut self, create_new: bool) -> &mut OpenOptions { + self.create_new = create_new; + self + } + + /// Opens a file at `path` with the options specified by `self`. + /// + /// # Errors + /// + /// This function will return an error under a number of different + /// circumstances. Some of these error conditions are listed here, together + /// with their [`io::ErrorKind`]. The mapping to [`io::ErrorKind`]s is not + /// part of the compatibility contract of the function, especially the + /// [`Other`] kind might change to more specific kinds in the future. + /// + /// * [`NotFound`]: The specified file does not exist and neither `create` or `create_new` is + /// set. + /// * [`NotFound`]: One of the directory components of the file path does not exist. + /// * [`PermissionDenied`]: The user lacks permission to get the specified access rights for the + /// file. + /// * [`PermissionDenied`]: The user lacks permission to open one of the directory components of + /// the specified path. + /// * [`AlreadyExists`]: `create_new` was specified and the file already exists. + /// * [`InvalidInput`]: Invalid combinations of open options (truncate without write access, no + /// access mode set, etc.). + /// * [`Other`]: One of the directory components of the specified file path was not, in fact, a + /// directory. + /// * [`Other`]: Filesystem-level errors: full disk, write permission requested on a read-only + /// file system, exceeded disk quota, too many open files, too long filename, too many + /// symbolic links in the specified path (Unix-like systems only), etc. + /// + /// # Examples + /// + /// ```no_run + /// use monoio::fs::OpenOptions; + /// + /// #[monoio::main] + /// async fn main() -> Result<(), Box> { + /// let file = OpenOptions::new().read(true).open("foo.txt").await?; + /// Ok(()) + /// } + /// ``` + /// + /// [`AlreadyExists`]: io::ErrorKind::AlreadyExists + /// [`InvalidInput`]: io::ErrorKind::InvalidInput + /// [`NotFound`]: io::ErrorKind::NotFound + /// [`Other`]: io::ErrorKind::Other + /// [`PermissionDenied`]: io::ErrorKind::PermissionDenied + pub async fn open(&self, path: impl AsRef) -> io::Result { + let op = Op::open(path.as_ref(), self)?; + + // Await the completion of the event + let completion = op.await; + + // The file is open + Ok(File::from_shared_fd(SharedFd::new_without_register( + completion.meta.result? as _, + ))) + } + + #[cfg(unix)] + pub(crate) fn access_mode(&self) -> io::Result { + match (self.read, self.write, self.append) { + (true, false, false) => Ok(libc::O_RDONLY), + (false, true, false) => Ok(libc::O_WRONLY), + (true, true, false) => Ok(libc::O_RDWR), + (false, _, true) => Ok(libc::O_WRONLY | libc::O_APPEND), + (true, _, true) => Ok(libc::O_RDWR | libc::O_APPEND), + (false, false, false) => Err(io::Error::from_raw_os_error(libc::EINVAL)), + } + } + + #[cfg(windows)] + pub(crate) fn access_mode(&self) -> io::Result { + match (self.read, self.write, self.append, self.access_mode) { + (.., Some(mode)) => Ok(mode), + (true, false, false, None) => Ok(GENERIC_READ), + (false, true, false, None) => Ok(GENERIC_WRITE), + (true, true, false, None) => Ok(GENERIC_READ | GENERIC_WRITE), + (false, _, true, None) => Ok(FILE_GENERIC_WRITE & !FILE_WRITE_DATA), + (true, _, true, None) => Ok(GENERIC_READ | (FILE_GENERIC_WRITE & !FILE_WRITE_DATA)), + (false, false, false, None) => { + Err(io::Error::from_raw_os_error(ERROR_INVALID_PARAMETER as _)) + } + } + } + + #[cfg(unix)] + pub(crate) fn creation_mode(&self) -> io::Result { + match (self.write, self.append) { + (true, false) => {} + (false, false) => { + if self.truncate || self.create || self.create_new { + return Err(io::Error::from_raw_os_error(libc::EINVAL)); + } + } + (_, true) => { + if self.truncate && !self.create_new { + return Err(io::Error::from_raw_os_error(libc::EINVAL)); + } + } + } + + Ok(match (self.create, self.truncate, self.create_new) { + (false, false, false) => 0, + (true, false, false) => libc::O_CREAT, + (false, true, false) => libc::O_TRUNC, + (true, true, false) => libc::O_CREAT | libc::O_TRUNC, + (_, _, true) => libc::O_CREAT | libc::O_EXCL, + }) + } + + #[cfg(windows)] + pub(crate) fn creation_mode(&self) -> io::Result { + match (self.write, self.append) { + (true, false) => {} + (false, false) => { + if self.truncate || self.create || self.create_new { + return Err(io::Error::from_raw_os_error(ERROR_INVALID_PARAMETER as _)); + } + } + (_, true) => { + if self.truncate && !self.create_new { + return Err(io::Error::from_raw_os_error(ERROR_INVALID_PARAMETER as _)); + } + } + } + + Ok(match (self.create, self.truncate, self.create_new) { + (false, false, false) => OPEN_EXISTING, + (true, false, false) => OPEN_ALWAYS, + (false, true, false) => TRUNCATE_EXISTING, + (true, true, false) => CREATE_ALWAYS, + (_, _, true) => CREATE_NEW, + }) + } + + #[cfg(windows)] + pub(crate) fn get_flags_and_attributes(&self) -> u32 { + self.custom_flags + | self.attributes + | self.security_qos_flags + | if self.create_new { + FILE_FLAG_OPEN_REPARSE_POINT as _ + } else { + 0 + } + } +} + +#[cfg(unix)] +impl OpenOptionsExt for OpenOptions { + fn mode(&mut self, mode: u32) -> &mut Self { + self.mode = mode as libc::mode_t; + self + } + + fn custom_flags(&mut self, flags: i32) -> &mut Self { + self.custom_flags = flags as libc::c_int; + self + } +} diff --git a/vendor/monoio/src/fs/permissions.rs b/vendor/monoio/src/fs/permissions.rs new file mode 100644 index 000000000..5b69ea5fd --- /dev/null +++ b/vendor/monoio/src/fs/permissions.rs @@ -0,0 +1,86 @@ +use std::{fmt::Debug, os::unix::fs::PermissionsExt}; + +#[cfg(unix)] +use libc::mode_t; + +#[cfg(unix)] +#[derive(Clone, PartialEq, Eq)] +pub(crate) struct FilePermissions { + pub(crate) mode: mode_t, +} + +impl FilePermissions { + fn readonly(&self) -> bool { + self.mode & 0o222 == 0 + } + + #[cfg(target_os = "linux")] + fn mode(&self) -> u32 { + self.mode + } + + fn set_readonly(&mut self, readonly: bool) { + if readonly { + self.mode &= !0o222; + } else { + self.mode |= 0o222; + } + } + + #[cfg(not(target_os = "linux"))] + fn mode(&self) -> u32 { + unimplemented!() + } +} + +/// Representation of the various permissions on a file. +#[cfg(unix)] +pub struct Permissions(pub(crate) FilePermissions); + +impl Permissions { + /// Returns `true` if these permissions describe a readonly (unwritable) file. + pub fn readonly(&self) -> bool { + self.0.readonly() + } + + /// Set the readonly flag for this set of permissions. + /// + /// This will not change the file's permissions, only the in-memory representation. + /// Same with the `std::fs`, if you want to change the file's permissions, you should use + /// `monoio::fs::set_permissions`(currently not support) or `std::fs::set_permissions`. + #[allow(unused)] + pub fn set_readonly(&mut self, readonly: bool) { + self.0.set_readonly(readonly) + } +} + +impl PermissionsExt for Permissions { + /// Returns the underlying raw `mode_t` bits that are used by the OS. + fn mode(&self) -> u32 { + self.0.mode() + } + + /// Set the mode bits for this set of permissions. + /// + /// This will not change the file's permissions, only the in-memory representation. + /// Same with the `std::fs`, if you want to change the file's permissions, you should use + /// `monoio::fs::set_permissions`(currently not support) or `std::fs::set_permissions`. + fn set_mode(&mut self, mode: u32) { + *self = Self::from_mode(mode); + } + + /// Create a new instance of `Permissions` from the given mode bits. + fn from_mode(mode: u32) -> Self { + Self(FilePermissions { + mode: mode as mode_t, + }) + } +} + +impl Debug for Permissions { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Permissions") + .field("readonly", &self.readonly()) + .finish_non_exhaustive() + } +} diff --git a/vendor/monoio/src/io/as_fd.rs b/vendor/monoio/src/io/as_fd.rs new file mode 100644 index 000000000..745a2e5a4 --- /dev/null +++ b/vendor/monoio/src/io/as_fd.rs @@ -0,0 +1,30 @@ +//! We impl AsReadFd and AsWriteFd for some structs. + +use crate::driver::shared_fd::SharedFd; + +/// Get a readable shared fd from self. +pub trait AsReadFd { + /// Get fd. + fn as_reader_fd(&mut self) -> &SharedFdWrapper; +} + +/// Get a writable shared fd from self. +pub trait AsWriteFd { + /// Get fd. + fn as_writer_fd(&mut self) -> &SharedFdWrapper; +} + +/// A wrapper of SharedFd to solve pub problem. +#[repr(transparent)] +pub struct SharedFdWrapper(SharedFd); + +impl SharedFdWrapper { + #[allow(unused)] + pub(crate) fn as_ref(&self) -> &SharedFd { + &self.0 + } + + pub(crate) fn new(inner: &SharedFd) -> &Self { + unsafe { std::mem::transmute(inner) } + } +} diff --git a/vendor/monoio/src/io/async_buf_read.rs b/vendor/monoio/src/io/async_buf_read.rs new file mode 100644 index 000000000..149544df3 --- /dev/null +++ b/vendor/monoio/src/io/async_buf_read.rs @@ -0,0 +1,11 @@ +use std::future::Future; + +use crate::io::AsyncReadRent; + +/// AsyncBufRead: async read with buffered content +pub trait AsyncBufRead: AsyncReadRent { + /// Try read data and get a reference to the internal buffer + fn fill_buf(&mut self) -> impl Future>; + /// Mark how much data is read + fn consume(&mut self, amt: usize); +} diff --git a/vendor/monoio/src/io/async_buf_read_ext.rs b/vendor/monoio/src/io/async_buf_read_ext.rs new file mode 100644 index 000000000..1236d697b --- /dev/null +++ b/vendor/monoio/src/io/async_buf_read_ext.rs @@ -0,0 +1,121 @@ +use std::{ + future::Future, + io::{Error, ErrorKind, Result}, + str::from_utf8, +}; + +use memchr::memchr; + +use crate::io::AsyncBufRead; + +struct Guard<'a> { + buf: &'a mut Vec, + len: usize, +} + +impl<'a> Drop for Guard<'a> { + fn drop(&mut self) { + unsafe { + self.buf.set_len(self.len); + } + } +} + +async fn read_until(r: &mut A, delim: u8, buf: &mut Vec) -> Result +where + A: AsyncBufRead + ?Sized, +{ + let mut read = 0; + loop { + let (done, used) = { + let available = match r.fill_buf().await { + Ok(n) => n, + Err(ref e) if e.kind() == ErrorKind::Interrupted => continue, + Err(e) => return Err(e), + }; + + match memchr(delim, available) { + Some(i) => { + buf.extend_from_slice(&available[..=i]); + (true, i + 1) + } + None => { + buf.extend_from_slice(available); + (false, available.len()) + } + } + }; + r.consume(used); + read += used; + if done || used == 0 { + return Ok(read); + } + } +} + +/// AsyncBufReadExt +pub trait AsyncBufReadExt { + /// This function will read bytes from the underlying stream until the delimiter or EOF is + /// found. Once found, all bytes up to, and including, the delimiter (if found) will be appended + /// to buf. + /// + /// If successful, this function will return the total number of bytes read. + /// + /// # Errors + /// This function will ignore all instances of ErrorKind::Interrupted and will otherwise return + /// any errors returned by fill_buf. + fn read_until<'a>( + &'a mut self, + byte: u8, + buf: &'a mut Vec, + ) -> impl Future>; + + /// This function will read bytes from the underlying stream until the newline delimiter (the + /// 0xA byte) or EOF is found. Once found, all bytes up to, and including, the delimiter (if + /// found) will be appended to buf. + /// + /// If successful, this function will return the total number of bytes read. + /// + /// If this function returns Ok(0), the stream has reached EOF. + /// + /// # Errors + /// This function has the same error semantics as read_until and will also return an error if + /// the read bytes are not valid UTF-8. If an I/O error is encountered then buf may contain some + /// bytes already read in the event that all data read so far was valid UTF-8. + fn read_line<'a>(&'a mut self, buf: &'a mut String) -> impl Future>; +} + +impl AsyncBufReadExt for A +where + A: AsyncBufRead + ?Sized, +{ + fn read_until<'a>( + &'a mut self, + byte: u8, + buf: &'a mut Vec, + ) -> impl Future> { + read_until(self, byte, buf) + } + + async fn read_line<'a>(&'a mut self, buf: &'a mut String) -> Result { + unsafe { + let mut g = Guard { + len: buf.len(), + buf: buf.as_mut_vec(), + }; + + let ret = read_until(self, b'\n', g.buf).await; + if from_utf8(&g.buf[g.len..]).is_err() { + ret.and_then(|_| { + Err(Error::new( + ErrorKind::InvalidData, + "stream did not contain valid UTF-8", + )) + }) + } else { + g.len = g.buf.len(); + ret + } + } + } +} diff --git a/vendor/monoio/src/io/async_read_rent.rs b/vendor/monoio/src/io/async_read_rent.rs new file mode 100644 index 000000000..e61fd6280 --- /dev/null +++ b/vendor/monoio/src/io/async_read_rent.rs @@ -0,0 +1,72 @@ +use std::future::Future; + +use crate::{ + buf::{IoBufMut, IoVecBufMut, RawBuf}, + BufResult, +}; + +/// AsyncReadRent: async read with a ownership of a buffer +pub trait AsyncReadRent { + /// Same as read(2) + fn read(&mut self, buf: T) -> impl Future>; + /// Same as readv(2) + fn readv(&mut self, buf: T) -> impl Future>; +} + +/// AsyncReadRentAt: async read with a ownership of a buffer and a position +pub trait AsyncReadRentAt { + /// Same as pread(2) + fn read_at( + &mut self, + buf: T, + pos: usize, + ) -> impl Future>; +} + +impl AsyncReadRent for &mut A { + #[inline] + fn read(&mut self, buf: T) -> impl Future> { + (**self).read(buf) + } + + #[inline] + fn readv(&mut self, buf: T) -> impl Future> { + (**self).readv(buf) + } +} + +impl AsyncReadRent for &[u8] { + fn read(&mut self, mut buf: T) -> impl Future> { + let amt = std::cmp::min(self.len(), buf.bytes_total()); + let (a, b) = self.split_at(amt); + unsafe { + buf.write_ptr().copy_from_nonoverlapping(a.as_ptr(), amt); + buf.set_init(amt); + } + *self = b; + async move { (Ok(amt), buf) } + } + + fn readv(&mut self, mut buf: T) -> impl Future> { + // # Safety + // We do it in pure sync way. + let n = match unsafe { RawBuf::new_from_iovec_mut(&mut buf) } { + Some(mut raw_buf) => { + // copy from read to avoid await + let amt = std::cmp::min(self.len(), raw_buf.bytes_total()); + let (a, b) = self.split_at(amt); + unsafe { + raw_buf + .write_ptr() + .copy_from_nonoverlapping(a.as_ptr(), amt); + raw_buf.set_init(amt); + } + *self = b; + amt + } + None => 0, + }; + unsafe { buf.set_init(n) }; + async move { (Ok(n), buf) } + } +} diff --git a/vendor/monoio/src/io/async_read_rent_ext.rs b/vendor/monoio/src/io/async_read_rent_ext.rs new file mode 100644 index 000000000..de60dd4a6 --- /dev/null +++ b/vendor/monoio/src/io/async_read_rent_ext.rs @@ -0,0 +1,169 @@ +use std::future::Future; + +use super::AsyncReadRent; +use crate::{ + buf::{IoBufMut, IoVecBufMut, SliceMut}, + BufResult, +}; + +macro_rules! reader_trait { + ($future: ident, $n_ty: ty, $f: ident) => { + /// Read number in async way + fn $f(&mut self) -> impl Future>; + }; +} + +macro_rules! reader_be_impl { + ($future: ident, $n_ty: ty, $f: ident) => { + async fn $f(&mut self) -> std::io::Result<$n_ty> { + let (res, buf) = self + .read_exact(std::boxed::Box::new([0; std::mem::size_of::<$n_ty>()])) + .await; + res?; + use crate::utils::box_into_inner::IntoInner; + Ok(<$n_ty>::from_be_bytes(Box::consume(buf))) + } + }; +} + +macro_rules! reader_le_impl { + ($future: ident, $n_ty: ty, $f: ident) => { + async fn $f(&mut self) -> std::io::Result<$n_ty> { + let (res, buf) = self + .read_exact(std::boxed::Box::new([0; std::mem::size_of::<$n_ty>()])) + .await; + res?; + use crate::utils::box_into_inner::IntoInner; + Ok(<$n_ty>::from_le_bytes(Box::consume(buf))) + } + }; +} + +/// AsyncReadRentExt +pub trait AsyncReadRentExt { + /// Read until buf capacity is fulfilled + fn read_exact( + &mut self, + buf: T, + ) -> impl Future>; + + /// Readv until buf capacity is fulfilled + fn read_vectored_exact( + &mut self, + buf: T, + ) -> impl Future>; + + reader_trait!(ReadU8Future, u8, read_u8); + reader_trait!(ReadU16Future, u16, read_u16); + reader_trait!(ReadU32Future, u32, read_u32); + reader_trait!(ReadU64Future, u64, read_u64); + reader_trait!(ReadU128Future, u16, read_u128); + reader_trait!(ReadI8Future, i8, read_i8); + reader_trait!(ReadI16Future, i16, read_i16); + reader_trait!(ReadI32Future, i32, read_i32); + reader_trait!(ReadI64Future, i64, read_i64); + reader_trait!(ReadI128Future, i128, read_i128); + reader_trait!(ReadF32Future, f32, read_f32); + reader_trait!(ReadF64Future, f64, read_f64); + + reader_trait!(ReadU8LEFuture, u8, read_u8_le); + reader_trait!(ReadU16LEFuture, u16, read_u16_le); + reader_trait!(ReadU32LEFuture, u32, read_u32_le); + reader_trait!(ReadU64LEFuture, u64, read_u64_le); + reader_trait!(ReadU128LEFuture, u128, read_u128_le); + reader_trait!(ReadI8LEFuture, i8, read_i8_le); + reader_trait!(ReadI16LEFuture, i16, read_i16_le); + reader_trait!(ReadI32LEFuture, i32, read_i32_le); + reader_trait!(ReadI64LEFuture, i64, read_i64_le); + reader_trait!(ReadI128LEFuture, i128, read_i128_le); + reader_trait!(ReadF32LEFuture, f32, read_f32_le); + reader_trait!(ReadF64LEFuture, f64, read_f64_le); +} + +impl AsyncReadRentExt for A +where + A: AsyncReadRent + ?Sized, +{ + async fn read_exact(&mut self, mut buf: T) -> BufResult { + let len = buf.bytes_total(); + let mut read = 0; + while read < len { + let buf_slice = unsafe { SliceMut::new_unchecked(buf, read, len) }; + let (result, buf_slice) = self.read(buf_slice).await; + buf = buf_slice.into_inner(); + match result { + Ok(0) => { + return ( + Err(std::io::Error::new( + std::io::ErrorKind::UnexpectedEof, + "failed to fill whole buffer", + )), + buf, + ) + } + Ok(n) => { + read += n; + unsafe { buf.set_init(read) }; + } + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + } + } + (Ok(read), buf) + } + + async fn read_vectored_exact( + &mut self, + mut buf: T, + ) -> BufResult { + let mut meta = crate::buf::write_vec_meta(&mut buf); + let len = meta.len(); + let mut read = 0; + + while read < len { + let (res, meta_) = self.readv(meta).await; + meta = meta_; + match res { + Ok(0) => { + return ( + Err(std::io::Error::new( + std::io::ErrorKind::UnexpectedEof, + "failed to fill whole buffer", + )), + buf, + ) + } + Ok(n) => read += n, + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + } + } + (Ok(read), buf) + } + + reader_be_impl!(ReadU8Future, u8, read_u8); + reader_be_impl!(ReadU16Future, u16, read_u16); + reader_be_impl!(ReadU32Future, u32, read_u32); + reader_be_impl!(ReadU64Future, u64, read_u64); + reader_be_impl!(ReadU128Future, u16, read_u128); + reader_be_impl!(ReadI8Future, i8, read_i8); + reader_be_impl!(ReadI16Future, i16, read_i16); + reader_be_impl!(ReadI32Future, i32, read_i32); + reader_be_impl!(ReadI64Future, i64, read_i64); + reader_be_impl!(ReadI128Future, i128, read_i128); + reader_be_impl!(ReadF32Future, f32, read_f32); + reader_be_impl!(ReadF64Future, f64, read_f64); + + reader_le_impl!(ReadU8LEFuture, u8, read_u8_le); + reader_le_impl!(ReadU16LEFuture, u16, read_u16_le); + reader_le_impl!(ReadU32LEFuture, u32, read_u32_le); + reader_le_impl!(ReadU64LEFuture, u64, read_u64_le); + reader_le_impl!(ReadU128LEFuture, u128, read_u128_le); + reader_le_impl!(ReadI8LEFuture, i8, read_i8_le); + reader_le_impl!(ReadI16LEFuture, i16, read_i16_le); + reader_le_impl!(ReadI32LEFuture, i32, read_i32_le); + reader_le_impl!(ReadI64LEFuture, i64, read_i64_le); + reader_le_impl!(ReadI128LEFuture, i128, read_i128_le); + reader_be_impl!(ReadF32LEFuture, f32, read_f32_le); + reader_be_impl!(ReadF64LEFuture, f64, read_f64_le); +} diff --git a/vendor/monoio/src/io/async_rent_cancelable.rs b/vendor/monoio/src/io/async_rent_cancelable.rs new file mode 100644 index 000000000..1dd447a23 --- /dev/null +++ b/vendor/monoio/src/io/async_rent_cancelable.rs @@ -0,0 +1,100 @@ +use std::future::Future; + +use super::{AsyncReadRent, AsyncWriteRent, CancelHandle}; +use crate::{ + buf::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut}, + BufResult, +}; + +/// CancelableAsyncReadRent: async read with a ownership of a buffer and ability to cancel io. +pub trait CancelableAsyncReadRent: AsyncReadRent { + /// Same as read(2) + fn cancelable_read( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future>; + /// Same as readv(2) + fn cancelable_readv( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future>; +} + +impl CancelableAsyncReadRent for &mut A { + #[inline] + fn cancelable_read( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future> { + (**self).cancelable_read(buf, c) + } + + #[inline] + fn cancelable_readv( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future> { + (**self).cancelable_readv(buf, c) + } +} + +/// CancelableAsyncWriteRent: async write with a ownership of a buffer and ability to cancel io. +pub trait CancelableAsyncWriteRent: AsyncWriteRent { + /// Same as write(2) + fn cancelable_write( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future>; + + /// Same as writev(2) + fn cancelable_writev( + &mut self, + buf_vec: T, + c: CancelHandle, + ) -> impl Future>; + + /// Flush buffered data if needed + fn cancelable_flush(&mut self, c: CancelHandle) -> impl Future>; + + /// Same as shutdown + fn cancelable_shutdown(&mut self, c: CancelHandle) + -> impl Future>; +} + +impl CancelableAsyncWriteRent for &mut A { + #[inline] + fn cancelable_write( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future> { + (**self).cancelable_write(buf, c) + } + + #[inline] + fn cancelable_writev( + &mut self, + buf_vec: T, + c: CancelHandle, + ) -> impl Future> { + (**self).cancelable_writev(buf_vec, c) + } + + #[inline] + fn cancelable_flush(&mut self, c: CancelHandle) -> impl Future> { + (**self).cancelable_flush(c) + } + + #[inline] + fn cancelable_shutdown( + &mut self, + c: CancelHandle, + ) -> impl Future> { + (**self).cancelable_shutdown(c) + } +} diff --git a/vendor/monoio/src/io/async_rent_cancelable_ext.rs b/vendor/monoio/src/io/async_rent_cancelable_ext.rs new file mode 100644 index 000000000..067070d7d --- /dev/null +++ b/vendor/monoio/src/io/async_rent_cancelable_ext.rs @@ -0,0 +1,260 @@ +use std::future::Future; + +use super::{CancelHandle, CancelableAsyncReadRent, CancelableAsyncWriteRent}; +use crate::{ + buf::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut, Slice, SliceMut}, + BufResult, +}; + +macro_rules! reader_trait { + ($future: ident, $n_ty: ty, $f: ident) => { + /// Read number in async way + fn $f(&mut self, c: CancelHandle) -> impl Future>; + }; +} + +macro_rules! reader_be_impl { + ($future: ident, $n_ty: ty, $f: ident) => { + async fn $f(&mut self, c: CancelHandle) -> std::io::Result<$n_ty> { + let (res, buf) = self + .cancelable_read_exact(std::boxed::Box::new([0; std::mem::size_of::<$n_ty>()]), c) + .await; + res?; + use crate::utils::box_into_inner::IntoInner; + Ok(<$n_ty>::from_be_bytes(Box::consume(buf))) + } + }; +} + +macro_rules! reader_le_impl { + ($future: ident, $n_ty: ty, $f: ident) => { + async fn $f(&mut self, c: CancelHandle) -> std::io::Result<$n_ty> { + let (res, buf) = self + .cancelable_read_exact(std::boxed::Box::new([0; std::mem::size_of::<$n_ty>()]), c) + .await; + res?; + use crate::utils::box_into_inner::IntoInner; + Ok(<$n_ty>::from_le_bytes(Box::consume(buf))) + } + }; +} + +/// CancelableAsyncReadRentExt +pub trait CancelableAsyncReadRentExt { + /// Read until buf capacity is fulfilled + fn cancelable_read_exact( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future>; + + /// Readv until buf capacity is fulfilled + fn cancelable_read_vectored_exact( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future>; + + reader_trait!(ReadU8Future, u8, cancelable_read_u8); + reader_trait!(ReadU16Future, u16, cancelable_read_u16); + reader_trait!(ReadU32Future, u32, cancelable_read_u32); + reader_trait!(ReadU64Future, u64, cancelable_read_u64); + reader_trait!(ReadU128Future, u128, cancelable_read_u128); + reader_trait!(ReadI8Future, i8, cancelable_read_i8); + reader_trait!(ReadI16Future, i16, cancelable_read_i16); + reader_trait!(ReadI32Future, i32, cancelable_read_i32); + reader_trait!(ReadI64Future, i64, cancelable_read_i64); + reader_trait!(ReadI128Future, i128, cancelable_read_i128); + reader_trait!(ReadF32Future, f32, cancelable_read_f32); + reader_trait!(ReadF64Future, f64, cancelable_read_f64); + + reader_trait!(ReadU8LEFuture, u8, cancelable_read_u8_le); + reader_trait!(ReadU16LEFuture, u16, cancelable_read_u16_le); + reader_trait!(ReadU32LEFuture, u32, cancelable_read_u32_le); + reader_trait!(ReadU64LEFuture, u64, cancelable_read_u64_le); + reader_trait!(ReadU128LEFuture, u128, cancelable_read_u128_le); + reader_trait!(ReadI8LEFuture, i8, cancelable_read_i8_le); + reader_trait!(ReadI16LEFuture, i16, cancelable_read_i16_le); + reader_trait!(ReadI32LEFuture, i32, cancelable_read_i32_le); + reader_trait!(ReadI64LEFuture, i64, cancelable_read_i64_le); + reader_trait!(ReadI128LEFuture, i128, cancelable_read_i128_le); + reader_trait!(ReadF32LEFuture, f32, cancelable_read_f32_le); + reader_trait!(ReadF64LEFuture, f64, cancelable_read_f64_le); +} + +impl CancelableAsyncReadRentExt for A +where + A: CancelableAsyncReadRent + ?Sized, +{ + async fn cancelable_read_exact( + &mut self, + mut buf: T, + c: CancelHandle, + ) -> BufResult { + let len = buf.bytes_total(); + let mut read = 0; + while read < len { + let buf_slice = unsafe { SliceMut::new_unchecked(buf, read, len) }; + let (result, buf_slice) = self.cancelable_read(buf_slice, c.clone()).await; + buf = buf_slice.into_inner(); + match result { + Ok(0) => { + return ( + Err(std::io::Error::new( + std::io::ErrorKind::UnexpectedEof, + "failed to fill whole buffer", + )), + buf, + ) + } + Ok(n) => { + read += n; + unsafe { buf.set_init(read) }; + } + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + } + } + (Ok(read), buf) + } + + async fn cancelable_read_vectored_exact( + &mut self, + mut buf: T, + c: CancelHandle, + ) -> BufResult { + let mut meta = crate::buf::write_vec_meta(&mut buf); + let len = meta.len(); + let mut read = 0; + + while read < len { + let (res, meta_) = self.cancelable_readv(meta, c.clone()).await; + meta = meta_; + match res { + Ok(0) => { + return ( + Err(std::io::Error::new( + std::io::ErrorKind::UnexpectedEof, + "failed to fill whole buffer", + )), + buf, + ) + } + Ok(n) => read += n, + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + } + } + (Ok(read), buf) + } + + reader_be_impl!(ReadU8Future, u8, cancelable_read_u8); + reader_be_impl!(ReadU16Future, u16, cancelable_read_u16); + reader_be_impl!(ReadU32Future, u32, cancelable_read_u32); + reader_be_impl!(ReadU64Future, u64, cancelable_read_u64); + reader_be_impl!(ReadU128Future, u128, cancelable_read_u128); + reader_be_impl!(ReadI8Future, i8, cancelable_read_i8); + reader_be_impl!(ReadI16Future, i16, cancelable_read_i16); + reader_be_impl!(ReadI32Future, i32, cancelable_read_i32); + reader_be_impl!(ReadI64Future, i64, cancelable_read_i64); + reader_be_impl!(ReadI128Future, i128, cancelable_read_i128); + reader_be_impl!(ReadF32Future, f32, cancelable_read_f32); + reader_be_impl!(ReadF64Future, f64, cancelable_read_f64); + + reader_le_impl!(ReadU8LEFuture, u8, cancelable_read_u8_le); + reader_le_impl!(ReadU16LEFuture, u16, cancelable_read_u16_le); + reader_le_impl!(ReadU32LEFuture, u32, cancelable_read_u32_le); + reader_le_impl!(ReadU64LEFuture, u64, cancelable_read_u64_le); + reader_le_impl!(ReadU128LEFuture, u128, cancelable_read_u128_le); + reader_le_impl!(ReadI8LEFuture, i8, cancelable_read_i8_le); + reader_le_impl!(ReadI16LEFuture, i16, cancelable_read_i16_le); + reader_le_impl!(ReadI32LEFuture, i32, cancelable_read_i32_le); + reader_le_impl!(ReadI64LEFuture, i64, cancelable_read_i64_le); + reader_le_impl!(ReadI128LEFuture, i128, cancelable_read_i128_le); + reader_be_impl!(ReadF32LEFuture, f32, cancelable_read_f32_le); + reader_be_impl!(ReadF64LEFuture, f64, cancelable_read_f64_le); +} + +/// CancelableAsyncWriteRentExt +pub trait CancelableAsyncWriteRentExt { + /// Write all + fn write_all( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future>; + + /// Write all + fn write_vectored_all( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future>; +} + +impl CancelableAsyncWriteRentExt for A +where + A: CancelableAsyncWriteRent + ?Sized, +{ + async fn write_all( + &mut self, + mut buf: T, + c: CancelHandle, + ) -> BufResult { + let len = buf.bytes_init(); + let mut written = 0; + while written < len { + let buf_slice = unsafe { Slice::new_unchecked(buf, written, len) }; + let (result, buf_slice) = self.cancelable_write(buf_slice, c.clone()).await; + buf = buf_slice.into_inner(); + match result { + Ok(0) => { + return ( + Err(std::io::Error::new( + std::io::ErrorKind::WriteZero, + "failed to write whole buffer", + )), + buf, + ) + } + Ok(n) => written += n, + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + } + } + (Ok(written), buf) + } + + async fn write_vectored_all( + &mut self, + buf: T, + c: CancelHandle, + ) -> BufResult { + let mut meta = crate::buf::read_vec_meta(&buf); + let len = meta.len(); + let mut written = 0; + + while written < len { + let (res, meta_) = self.cancelable_writev(meta, c.clone()).await; + meta = meta_; + match res { + Ok(0) => { + return ( + Err(std::io::Error::new( + std::io::ErrorKind::WriteZero, + "failed to write whole buffer", + )), + buf, + ) + } + Ok(n) => { + written += n; + meta.consume(n); + } + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + } + } + (Ok(written), buf) + } +} diff --git a/vendor/monoio/src/io/async_write_rent.rs b/vendor/monoio/src/io/async_write_rent.rs new file mode 100644 index 000000000..3c8c70bb5 --- /dev/null +++ b/vendor/monoio/src/io/async_write_rent.rs @@ -0,0 +1,53 @@ +use std::future::Future; + +use crate::{ + buf::{IoBuf, IoBufMut, IoVecBuf}, + BufResult, +}; + +/// AsyncWriteRent: async write with a ownership of a buffer +pub trait AsyncWriteRent { + /// Same as write(2) + fn write(&mut self, buf: T) -> impl Future>; + + /// Same as writev(2) + fn writev(&mut self, buf_vec: T) -> impl Future>; + + /// Flush buffered data if needed + fn flush(&mut self) -> impl Future>; + + /// Same as shutdown + fn shutdown(&mut self) -> impl Future>; +} + +/// AsyncWriteRentAt: async write with a ownership of a buffer and a position +pub trait AsyncWriteRentAt { + /// Write buf at given offset + fn write_at( + &self, + buf: T, + pos: usize, + ) -> impl Future>; +} + +impl AsyncWriteRent for &mut A { + #[inline] + fn write(&mut self, buf: T) -> impl Future> { + (**self).write(buf) + } + + #[inline] + fn writev(&mut self, buf_vec: T) -> impl Future> { + (**self).writev(buf_vec) + } + + #[inline] + fn flush(&mut self) -> impl Future> { + (**self).flush() + } + + #[inline] + fn shutdown(&mut self) -> impl Future> { + (**self).shutdown() + } +} diff --git a/vendor/monoio/src/io/async_write_rent_ext.rs b/vendor/monoio/src/io/async_write_rent_ext.rs new file mode 100644 index 000000000..7c2704463 --- /dev/null +++ b/vendor/monoio/src/io/async_write_rent_ext.rs @@ -0,0 +1,81 @@ +use std::future::Future; + +use crate::{ + buf::{IoBuf, IoVecBuf, Slice}, + io::AsyncWriteRent, + BufResult, +}; + +/// AsyncWriteRentExt +pub trait AsyncWriteRentExt { + /// Write all + fn write_all( + &mut self, + buf: T, + ) -> impl Future>; + + /// Write vectored all + fn write_vectored_all( + &mut self, + buf: T, + ) -> impl Future>; +} + +impl AsyncWriteRentExt for A +where + A: AsyncWriteRent + ?Sized, +{ + async fn write_all(&mut self, mut buf: T) -> BufResult { + let len = buf.bytes_init(); + let mut written = 0; + while written < len { + let buf_slice = unsafe { Slice::new_unchecked(buf, written, len) }; + let (result, buf_slice) = self.write(buf_slice).await; + buf = buf_slice.into_inner(); + match result { + Ok(0) => { + return ( + Err(std::io::Error::new( + std::io::ErrorKind::WriteZero, + "failed to write whole buffer", + )), + buf, + ) + } + Ok(n) => written += n, + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + } + } + (Ok(written), buf) + } + + async fn write_vectored_all(&mut self, buf: T) -> BufResult { + let mut meta = crate::buf::read_vec_meta(&buf); + let len = meta.len(); + let mut written = 0; + + while written < len { + let (res, meta_) = self.writev(meta).await; + meta = meta_; + match res { + Ok(0) => { + return ( + Err(std::io::Error::new( + std::io::ErrorKind::WriteZero, + "failed to write whole buffer", + )), + buf, + ) + } + Ok(n) => { + written += n; + meta.consume(n); + } + Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => {} + Err(e) => return (Err(e), buf), + } + } + (Ok(written), buf) + } +} diff --git a/vendor/monoio/src/io/mod.rs b/vendor/monoio/src/io/mod.rs new file mode 100644 index 000000000..7330c94bc --- /dev/null +++ b/vendor/monoio/src/io/mod.rs @@ -0,0 +1,69 @@ +//! IO traits + +mod async_buf_read; +mod async_buf_read_ext; +mod async_read_rent; +mod async_read_rent_ext; +mod async_rent_cancelable; +mod async_rent_cancelable_ext; +mod async_write_rent; +mod async_write_rent_ext; + +pub mod sink; +pub mod stream; + +pub mod as_fd; +#[cfg(all(target_os = "linux", feature = "splice"))] +pub mod splice; + +pub use async_buf_read::AsyncBufRead; +pub use async_buf_read_ext::AsyncBufReadExt; +pub use async_read_rent::{AsyncReadRent, AsyncReadRentAt}; +pub use async_read_rent_ext::AsyncReadRentExt; +pub use async_rent_cancelable::{CancelableAsyncReadRent, CancelableAsyncWriteRent}; +pub use async_rent_cancelable_ext::{CancelableAsyncReadRentExt, CancelableAsyncWriteRentExt}; +pub use async_write_rent::{AsyncWriteRent, AsyncWriteRentAt}; +pub use async_write_rent_ext::AsyncWriteRentExt; + +mod util; + +#[cfg(feature = "poll-io")] +pub use tokio::io as poll_io; +pub(crate) use util::operation_canceled; +#[cfg(all(target_os = "linux", feature = "splice"))] +pub use util::zero_copy; +pub use util::{ + copy, BufReader, BufWriter, CancelHandle, Canceller, OwnedReadHalf, OwnedWriteHalf, + PrefixedReadIo, Split, Splitable, +}; +#[cfg(feature = "poll-io")] +/// Convert a completion-based io to a poll-based io. +pub trait IntoPollIo: Sized { + /// The poll-based io type. + type PollIo; + + /// Convert a completion-based io to a poll-based io(able to get comp_io back). + fn try_into_poll_io(self) -> Result; + + /// Convert a completion-based io to a poll-based io. + #[inline] + fn into_poll_io(self) -> std::io::Result { + self.try_into_poll_io().map_err(|(e, _)| e) + } +} + +#[cfg(feature = "poll-io")] +/// Convert a poll-based io to a completion-based io. +pub trait IntoCompIo: Sized { + /// The completion-based io type. + type CompIo; + + /// Convert a poll-based io to a completion-based io(able to get poll_io back). + fn try_into_comp_io(self) -> Result; + + /// Convert a poll-based io to a completion-based io. + #[inline] + fn into_comp_io(self) -> std::io::Result { + self.try_into_comp_io().map_err(|(e, _)| e) + } +} diff --git a/vendor/monoio/src/io/sink/mod.rs b/vendor/monoio/src/io/sink/mod.rs new file mode 100644 index 000000000..73439847a --- /dev/null +++ b/vendor/monoio/src/io/sink/mod.rs @@ -0,0 +1,38 @@ +//! Sink trait in GAT style. +mod sink_ext; + +use std::future::Future; + +pub use sink_ext::SinkExt; + +/// A `Sink` is a value into which other values can be sent in pure async/await. +#[must_use = "sinks do nothing unless polled"] +pub trait Sink { + /// The type of value produced by the sink when an error occurs. + type Error; + + /// Send item. + fn send(&mut self, item: Item) -> impl Future>; + + /// Flush any remaining output from this sink. + fn flush(&mut self) -> impl Future>; + + /// Flush any remaining output and close this sink, if necessary. + fn close(&mut self) -> impl Future>; +} + +impl> Sink for &mut S { + type Error = S::Error; + + fn send(&mut self, item: T) -> impl Future> { + (**self).send(item) + } + + fn flush(&mut self) -> impl Future> { + (**self).flush() + } + + fn close(&mut self) -> impl Future> { + (**self).close() + } +} diff --git a/vendor/monoio/src/io/sink/sink_ext.rs b/vendor/monoio/src/io/sink/sink_ext.rs new file mode 100644 index 000000000..271218fc0 --- /dev/null +++ b/vendor/monoio/src/io/sink/sink_ext.rs @@ -0,0 +1,19 @@ +use std::future::Future; + +use super::Sink; + +/// Sink extensions. +pub trait SinkExt: Sink { + /// Send and flush. + fn send_and_flush(&mut self, item: T) -> impl Future>; +} + +impl SinkExt for A +where + A: Sink, +{ + async fn send_and_flush(&mut self, item: T) -> Result<(), Self::Error> { + Sink::::send(self, item).await?; + Sink::::flush(self).await + } +} diff --git a/vendor/monoio/src/io/splice.rs b/vendor/monoio/src/io/splice.rs new file mode 100644 index 000000000..1b8687963 --- /dev/null +++ b/vendor/monoio/src/io/splice.rs @@ -0,0 +1,52 @@ +//! Splice related trait and default impl. + +use std::future::Future; + +use super::as_fd::{AsReadFd, AsWriteFd}; +use crate::{driver::op::Op, net::Pipe}; + +/// Splice data from self to pipe. +pub trait SpliceSource { + /// Splice data from self to pipe. + fn splice_to_pipe<'a>( + &'a mut self, + pipe: &'a mut Pipe, + len: u32, + ) -> impl Future>; +} + +/// Splice data from self from pipe. +pub trait SpliceDestination { + /// Splice data from self from pipe. + fn splice_from_pipe<'a>( + &'a mut self, + pipe: &'a mut Pipe, + len: u32, + ) -> impl Future>; +} + +impl SpliceSource for T { + #[inline] + async fn splice_to_pipe<'a>( + &'a mut self, + pipe: &'a mut Pipe, + len: u32, + ) -> std::io::Result { + Op::splice_to_pipe(self.as_reader_fd().as_ref(), &pipe.fd, len)? + .splice() + .await + } +} + +impl SpliceDestination for T { + #[inline] + async fn splice_from_pipe<'a>( + &'a mut self, + pipe: &'a mut Pipe, + len: u32, + ) -> std::io::Result { + Op::splice_from_pipe(&pipe.fd, self.as_writer_fd().as_ref(), len)? + .splice() + .await + } +} diff --git a/vendor/monoio/src/io/stream/iter.rs b/vendor/monoio/src/io/stream/iter.rs new file mode 100644 index 000000000..0bd98227d --- /dev/null +++ b/vendor/monoio/src/io/stream/iter.rs @@ -0,0 +1,34 @@ +use super::{assert_stream, Stream}; + +/// Stream for the [`iter`] function. +#[derive(Debug, Clone)] +#[must_use = "streams do nothing unless polled"] +pub struct Iter { + iter: I, +} + +/// Converts an `Iterator` into a `Stream` which is always ready +/// to yield the next value. +pub fn iter(i: I) -> Iter +where + I: IntoIterator, +{ + assert_stream::(Iter { + iter: i.into_iter(), + }) +} + +impl Stream for Iter +where + I: Iterator, +{ + type Item = I::Item; + + async fn next(&mut self) -> Option { + self.iter.next() + } + + fn size_hint(&self) -> (usize, Option) { + self.iter.size_hint() + } +} diff --git a/vendor/monoio/src/io/stream/mod.rs b/vendor/monoio/src/io/stream/mod.rs new file mode 100644 index 000000000..3a70d3e2c --- /dev/null +++ b/vendor/monoio/src/io/stream/mod.rs @@ -0,0 +1,70 @@ +//! Stream trait in GAT style. + +mod iter; +mod stream_ext; + +use std::future::Future; + +pub use iter::{iter, Iter}; +pub use stream_ext::StreamExt; + +/// A stream of values produced asynchronously in pure async/await. +#[must_use = "streams do nothing unless polled"] +pub trait Stream { + /// Values yielded by the stream. + type Item; + + /// Attempt to pull out the next value of this stream, registering the + /// current task for wakeup if the value is not yet available, and returning + /// `None` if the stream is exhausted. + fn next(&mut self) -> impl Future>; + + /// Returns the bounds on the remaining length of the stream. + /// + /// Specifically, `size_hint()` returns a tuple where the first element + /// is the lower bound, and the second element is the upper bound. + /// + /// The second half of the tuple that is returned is an + /// [`Option`]`<`[`usize`]`>`. A [`None`] here means that either there + /// is no known upper bound, or the upper bound is larger than + /// [`usize`]. + /// + /// # Implementation notes + /// + /// It is not enforced that a stream implementation yields the declared + /// number of elements. A buggy stream may yield less than the lower bound + /// or more than the upper bound of elements. + /// + /// `size_hint()` is primarily intended to be used for optimizations such as + /// reserving space for the elements of the stream, but must not be + /// trusted to e.g., omit bounds checks in unsafe code. An incorrect + /// implementation of `size_hint()` should not lead to memory safety + /// violations. + /// + /// That said, the implementation should provide a correct estimation, + /// because otherwise it would be a violation of the trait's protocol. + /// + /// The default implementation returns `(0, `[`None`]`)` which is correct + /// for any stream. + #[inline] + fn size_hint(&self) -> (usize, Option) { + (0, None) + } +} + +impl Stream for &mut S { + type Item = S::Item; + + fn next(&mut self) -> impl Future> { + (**self).next() + } +} + +// Just a helper function to ensure the streams we're returning all have the +// right implementations. +pub(crate) fn assert_stream(stream: S) -> S +where + S: Stream, +{ + stream +} diff --git a/vendor/monoio/src/io/stream/stream_ext.rs b/vendor/monoio/src/io/stream/stream_ext.rs new file mode 100644 index 000000000..207bfe1fe --- /dev/null +++ b/vendor/monoio/src/io/stream/stream_ext.rs @@ -0,0 +1,96 @@ +use std::future::Future; + +use super::{assert_stream, Stream}; + +/// Stream extensions. +pub trait StreamExt: Stream { + /// Maps a stream to a stream of its items. + fn map(self, f: F) -> Map + where + F: FnMut(Self::Item) -> T, + Self: Sized, + { + assert_stream::(Map::new(self, f)) + } + + /// Computes from this stream's items new items of a different type using + /// an asynchronous closure. + fn then(self, f: F) -> Then + where + F: FnMut(Self::Item) -> Fut, + Fut: Future, + Self: Sized, + { + assert_stream::(Then::new(self, f)) + } + + /// Runs this stream to completion, executing the provided asynchronous + /// closure for each element on the stream. + fn for_each(mut self, mut f: F) -> impl Future + where + F: FnMut(Self::Item) -> Fut, + Fut: Future, + Self: Sized, + { + async move { + while let Some(item) = self.next().await { + (f)(item).await; + } + } + } +} + +impl StreamExt for T where T: Stream {} + +#[must_use = "streams do nothing unless polled"] +pub struct Map { + stream: St, + f: F, +} + +impl Map { + pub(crate) fn new(stream: St, f: F) -> Self { + Self { stream, f } + } +} + +impl Stream for Map +where + St: Stream, + F: FnMut(St::Item) -> Item, +{ + type Item = Item; + + async fn next(&mut self) -> Option { + self.stream.next().await.map(&mut self.f) + } +} + +#[must_use = "streams do nothing unless polled"] +pub struct Then { + stream: St, + f: F, +} + +impl Then +where + St: Stream, +{ + pub(super) fn new(stream: St, f: F) -> Self { + Self { stream, f } + } +} + +impl Stream for Then +where + St: Stream, + F: FnMut(St::Item) -> Fut, + Fut: Future, +{ + type Item = Fut::Output; + + async fn next(&mut self) -> Option { + let item = self.stream.next().await?; + Some((self.f)(item).await) + } +} diff --git a/vendor/monoio/src/io/util/buf_reader.rs b/vendor/monoio/src/io/util/buf_reader.rs new file mode 100644 index 000000000..174ed41d3 --- /dev/null +++ b/vendor/monoio/src/io/util/buf_reader.rs @@ -0,0 +1,176 @@ +use std::future::Future; + +use crate::{ + buf::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut, IoVecWrapperMut}, + io::{AsyncBufRead, AsyncReadRent, AsyncWriteRent}, + BufResult, +}; + +/// BufReader is a struct with a buffer. BufReader implements AsyncBufRead +/// and AsyncReadRent, and if the inner io implements AsyncWriteRent, it +/// will delegate the implementation. +pub struct BufReader { + inner: R, + buf: Option>, + pos: usize, + cap: usize, +} + +const DEFAULT_BUF_SIZE: usize = 8 * 1024; + +impl BufReader { + /// Create BufReader with default buffer size + #[inline] + pub fn new(inner: R) -> Self { + Self::with_capacity(DEFAULT_BUF_SIZE, inner) + } + + /// Create BufReader with given buffer size + #[inline] + pub fn with_capacity(capacity: usize, inner: R) -> Self { + let buffer = vec![0; capacity]; + Self { + inner, + buf: Some(buffer.into_boxed_slice()), + pos: 0, + cap: 0, + } + } + + /// Gets a reference to the underlying reader. + /// + /// It is inadvisable to directly read from the underlying reader. + #[inline] + pub const fn get_ref(&self) -> &R { + &self.inner + } + + /// Gets a mutable reference to the underlying reader. + #[inline] + pub fn get_mut(&mut self) -> &mut R { + &mut self.inner + } + + /// Consumes this `BufReader`, returning the underlying reader. + /// + /// Note that any leftover data in the internal buffer is lost. + #[inline] + pub fn into_inner(self) -> R { + self.inner + } + + /// Returns a reference to the internally buffered data. + /// + /// Unlike `fill_buf`, this will not attempt to fill the buffer if it is + /// empty. + #[inline] + pub fn buffer(&self) -> &[u8] { + &self.buf.as_ref().expect("unable to take buffer")[self.pos..self.cap] + } + + /// Invalidates all data in the internal buffer. + #[inline] + fn discard_buffer(&mut self) { + self.pos = 0; + self.cap = 0; + } +} + +impl AsyncReadRent for BufReader { + async fn read(&mut self, mut buf: T) -> BufResult { + // If we don't have any buffered data and we're doing a massive read + // (larger than our internal buffer), bypass our internal buffer + // entirely. + let owned_buf = self.buf.as_ref().unwrap(); + if self.pos == self.cap && buf.bytes_total() >= owned_buf.len() { + self.discard_buffer(); + return self.inner.read(buf).await; + } + + let rem = match self.fill_buf().await { + Ok(slice) => slice, + Err(e) => { + return (Err(e), buf); + } + }; + let amt = std::cmp::min(rem.len(), buf.bytes_total()); + unsafe { + buf.write_ptr().copy_from_nonoverlapping(rem.as_ptr(), amt); + buf.set_init(amt); + } + self.consume(amt); + (Ok(amt), buf) + } + + async fn readv(&mut self, mut buf: T) -> BufResult { + let slice = match IoVecWrapperMut::new(buf) { + Ok(slice) => slice, + Err(buf) => return (Ok(0), buf), + }; + + let (result, slice) = self.read(slice).await; + buf = slice.into_inner(); + if let Ok(n) = result { + unsafe { buf.set_init(n) }; + } + (result, buf) + } +} + +impl AsyncBufRead for BufReader { + async fn fill_buf(&mut self) -> std::io::Result<&[u8]> { + if self.pos == self.cap { + // there's no buffered data + let buf = self + .buf + .take() + .expect("no buffer available, generated future must be awaited"); + let (res, buf_) = self.inner.read(buf).await; + self.buf = Some(buf_); + match res { + Ok(n) => { + self.pos = 0; + self.cap = n; + return Ok(unsafe { + // We just put the buf into Option, so it must be Some. + &(self.buf.as_ref().unwrap_unchecked().as_ref())[self.pos..self.cap] + }); + } + Err(e) => { + return Err(e); + } + } + } + Ok(&(self + .buf + .as_ref() + .expect("no buffer available, generated future must be awaited") + .as_ref())[self.pos..self.cap]) + } + + fn consume(&mut self, amt: usize) { + self.pos = self.cap.min(self.pos + amt); + } +} + +impl AsyncWriteRent for BufReader { + #[inline] + fn write(&mut self, buf: T) -> impl Future> { + self.inner.write(buf) + } + + #[inline] + fn writev(&mut self, buf_vec: T) -> impl Future> { + self.inner.writev(buf_vec) + } + + #[inline] + fn flush(&mut self) -> impl Future> { + self.inner.flush() + } + + #[inline] + fn shutdown(&mut self) -> impl Future> { + self.inner.shutdown() + } +} diff --git a/vendor/monoio/src/io/util/buf_writer.rs b/vendor/monoio/src/io/util/buf_writer.rs new file mode 100644 index 000000000..92a524130 --- /dev/null +++ b/vendor/monoio/src/io/util/buf_writer.rs @@ -0,0 +1,176 @@ +use std::{future::Future, io}; + +use crate::{ + buf::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut, IoVecWrapper, Slice}, + io::{AsyncBufRead, AsyncReadRent, AsyncWriteRent, AsyncWriteRentExt}, + BufResult, +}; + +/// BufWriter is a struct with a buffer. BufWriter implements AsyncWriteRent, +/// and if the inner io implements AsyncReadRent, it will delegate the +/// implementation. +pub struct BufWriter { + inner: W, + buf: Option>, + pos: usize, + cap: usize, +} + +const DEFAULT_BUF_SIZE: usize = 8 * 1024; + +impl BufWriter { + /// Create BufWriter with default buffer size + #[inline] + pub fn new(inner: W) -> Self { + Self::with_capacity(DEFAULT_BUF_SIZE, inner) + } + + /// Create BufWriter with given buffer size + #[inline] + pub fn with_capacity(capacity: usize, inner: W) -> Self { + let buffer = vec![0; capacity]; + Self { + inner, + buf: Some(buffer.into_boxed_slice()), + pos: 0, + cap: 0, + } + } + + /// Gets a reference to the underlying writer. + #[inline] + pub fn get_ref(&self) -> &W { + &self.inner + } + + /// Gets a mutable reference to the underlying writer. + #[inline] + pub fn get_mut(&mut self) -> &mut W { + &mut self.inner + } + + /// Consumes this `BufWriter`, returning the underlying writer. + /// + /// Note that any leftover data in the internal buffer is lost. + #[inline] + pub fn into_inner(self) -> W { + self.inner + } + + /// Returns a reference to the internally buffered data. + #[inline] + pub fn buffer(&self) -> &[u8] { + &self.buf.as_ref().expect("unable to take buffer")[self.pos..self.cap] + } + + /// Invalidates all data in the internal buffer. + #[inline] + fn discard_buffer(&mut self) { + self.pos = 0; + self.cap = 0; + } +} + +impl BufWriter { + async fn flush_buf(&mut self) -> io::Result<()> { + if self.pos != self.cap { + // there is some data left inside internal buf + let buf = self + .buf + .take() + .expect("no buffer available, generated future must be awaited"); + // move buf to slice and write_all + let slice = Slice::new(buf, self.pos, self.cap); + let (ret, slice) = self.inner.write_all(slice).await; + // move it back and return + self.buf = Some(slice.into_inner()); + ret?; + self.discard_buffer(); + } + Ok(()) + } +} + +impl AsyncWriteRent for BufWriter { + async fn write(&mut self, buf: T) -> BufResult { + let owned_buf = self.buf.as_ref().unwrap(); + let owned_len = owned_buf.len(); + let amt = buf.bytes_init(); + + if self.pos + amt > owned_len { + // Buf can not be copied directly into OwnedBuf, + // we must flush OwnedBuf first. + match self.flush_buf().await { + Ok(_) => (), + Err(e) => { + return (Err(e), buf); + } + } + } + + // Now there are two situations here: + // 1. OwnedBuf has data, and self.pos + amt <= owned_len, + // which means the data can be copied into OwnedBuf. + // 2. OwnedBuf is empty. If we can copy buf into OwnedBuf, + // we will copy it, otherwise we will send it directly(in + // this situation, the OwnedBuf must be already empty). + if amt > owned_len { + self.inner.write(buf).await + } else { + unsafe { + let owned_buf = self.buf.as_mut().unwrap(); + owned_buf + .as_mut_ptr() + .add(self.cap) + .copy_from_nonoverlapping(buf.read_ptr(), amt); + } + self.cap += amt; + (Ok(amt), buf) + } + } + + // TODO: implement it as real io_vec + async fn writev(&mut self, buf: T) -> BufResult { + let slice = match IoVecWrapper::new(buf) { + Ok(slice) => slice, + Err(buf) => return (Ok(0), buf), + }; + + let (result, slice) = self.write(slice).await; + (result, slice.into_inner()) + } + + async fn flush(&mut self) -> std::io::Result<()> { + self.flush_buf().await?; + self.inner.flush().await + } + + async fn shutdown(&mut self) -> std::io::Result<()> { + self.flush_buf().await?; + self.inner.shutdown().await + } +} + +impl AsyncReadRent for BufWriter { + #[inline] + fn read(&mut self, buf: T) -> impl Future> { + self.inner.read(buf) + } + + #[inline] + fn readv(&mut self, buf: T) -> impl Future> { + self.inner.readv(buf) + } +} + +impl AsyncBufRead for BufWriter { + #[inline] + fn fill_buf(&mut self) -> impl Future> { + self.inner.fill_buf() + } + + #[inline] + fn consume(&mut self, amt: usize) { + self.inner.consume(amt) + } +} diff --git a/vendor/monoio/src/io/util/cancel.rs b/vendor/monoio/src/io/util/cancel.rs new file mode 100644 index 000000000..60bafc3fb --- /dev/null +++ b/vendor/monoio/src/io/util/cancel.rs @@ -0,0 +1,93 @@ +use std::{cell::RefCell, collections::HashSet, rc::Rc}; + +use crate::driver::op::OpCanceller; + +/// CancelHandle is used to pass to io actions with CancelableAsyncReadRent. +/// Create a CancelHandle with Canceller::handle. +#[derive(Clone)] +pub struct CancelHandle { + shared: Rc>, +} + +/// Canceller is a user-hold struct to cancel io operations. +/// A canceller can associate with multiple io operations. +#[derive(Default)] +pub struct Canceller { + shared: Rc>, +} + +pub(crate) struct AssociateGuard { + op_canceller: OpCanceller, + shared: Rc>, +} + +#[derive(Default)] +struct Shared { + canceled: bool, + slot_ref: HashSet, +} + +impl Canceller { + /// Create a new Canceller. + #[inline] + pub fn new() -> Self { + Default::default() + } + + /// Cancel all related operations. + pub fn cancel(self) -> Self { + let mut slot = HashSet::new(); + { + let mut shared = self.shared.borrow_mut(); + shared.canceled = true; + std::mem::swap(&mut slot, &mut shared.slot_ref); + } + + for op_canceller in slot.iter() { + unsafe { op_canceller.cancel() }; + } + slot.clear(); + Canceller { + shared: Rc::new(RefCell::new(Shared { + canceled: false, + slot_ref: slot, + })), + } + } + + /// Create a CancelHandle which can be used to pass to io operation. + #[inline] + pub fn handle(&self) -> CancelHandle { + CancelHandle { + shared: self.shared.clone(), + } + } +} + +impl CancelHandle { + pub(crate) fn canceled(&self) -> bool { + self.shared.borrow().canceled + } + + pub(crate) fn associate_op(self, op_canceller: OpCanceller) -> AssociateGuard { + { + let mut shared = self.shared.borrow_mut(); + shared.slot_ref.insert(op_canceller.clone()); + } + AssociateGuard { + op_canceller, + shared: self.shared, + } + } +} + +impl Drop for AssociateGuard { + fn drop(&mut self) { + let mut shared = self.shared.borrow_mut(); + shared.slot_ref.remove(&self.op_canceller); + } +} + +pub(crate) fn operation_canceled() -> std::io::Error { + std::io::Error::from_raw_os_error(125) +} diff --git a/vendor/monoio/src/io/util/copy.rs b/vendor/monoio/src/io/util/copy.rs new file mode 100644 index 000000000..40800ee85 --- /dev/null +++ b/vendor/monoio/src/io/util/copy.rs @@ -0,0 +1,98 @@ +#![allow(unused)] + +use std::io; + +use crate::io::{AsyncReadRent, AsyncWriteRent, AsyncWriteRentExt}; +#[cfg(unix)] +use crate::net::unix::new_pipe; + +const BUF_SIZE: usize = 4 * 1024; + +/// Copy data from reader to writer. +pub async fn copy<'a, R, W>(reader: &'a mut R, writer: &'a mut W) -> io::Result +where + R: AsyncReadRent + ?Sized, + W: AsyncWriteRent + ?Sized, +{ + let mut buf: Vec = Vec::with_capacity(BUF_SIZE); + let mut transferred: u64 = 0; + + 'r: loop { + let (read_res, mut buf_read) = reader.read(buf).await; + match read_res { + Ok(0) => { + // read closed + break; + } + Err(ref e) if e.kind() == io::ErrorKind::Interrupted => { + // retry + buf = buf_read; + continue; + } + Err(e) => { + // should return error + return Err(e); + } + Ok(_) => { + // go write data + } + } + + 'w: loop { + let (write_res, buf_) = writer.write_all(buf_read).await; + match write_res { + Ok(0) => { + // write closed + return Err(io::Error::new( + io::ErrorKind::WriteZero, + "write zero byte into writer", + )); + } + Err(ref e) if e.kind() == io::ErrorKind::Interrupted => { + // retry + buf_read = buf_; + continue 'w; + } + Err(e) => { + // should return error + return Err(e); + } + Ok(n) => { + // go read data + transferred += n as u64; + buf = buf_; + break; + } + } + } + } + + Ok(transferred) +} + +/// Copy with splice. +#[cfg(all(target_os = "linux", feature = "splice"))] +pub async fn zero_copy( + reader: &mut SRC, + writer: &mut DST, +) -> io::Result { + use crate::{ + driver::op::Op, + io::splice::{SpliceDestination, SpliceSource}, + }; + + let (mut pr, mut pw) = new_pipe()?; + let mut transferred: u64 = 0; + loop { + let mut to_write = reader.splice_to_pipe(&mut pw, BUF_SIZE as u32).await?; + if to_write == 0 { + break; + } + transferred += to_write as u64; + while to_write > 0 { + let written = writer.splice_from_pipe(&mut pr, to_write).await?; + to_write -= written; + } + } + Ok(transferred) +} diff --git a/vendor/monoio/src/io/util/mod.rs b/vendor/monoio/src/io/util/mod.rs new file mode 100644 index 000000000..45c3cc3a7 --- /dev/null +++ b/vendor/monoio/src/io/util/mod.rs @@ -0,0 +1,18 @@ +//! IO utils + +mod buf_reader; +mod buf_writer; +mod cancel; +mod copy; +mod prefixed_io; +mod split; + +pub use buf_reader::BufReader; +pub use buf_writer::BufWriter; +pub(crate) use cancel::operation_canceled; +pub use cancel::{CancelHandle, Canceller}; +pub use copy::copy; +#[cfg(all(target_os = "linux", feature = "splice"))] +pub use copy::zero_copy; +pub use prefixed_io::PrefixedReadIo; +pub use split::{OwnedReadHalf, OwnedWriteHalf, Split, Splitable}; diff --git a/vendor/monoio/src/io/util/prefixed_io.rs b/vendor/monoio/src/io/util/prefixed_io.rs new file mode 100644 index 000000000..6c6d37308 --- /dev/null +++ b/vendor/monoio/src/io/util/prefixed_io.rs @@ -0,0 +1,214 @@ +use std::future::Future; + +use super::{split::Split, CancelHandle}; +use crate::{ + buf::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut, IoVecWrapperMut}, + io::{AsyncReadRent, AsyncWriteRent, CancelableAsyncReadRent, CancelableAsyncWriteRent}, + BufResult, +}; + +/// PrefixedReadIO facilitates the addition of a prefix to an IO stream, +/// enabling stream rewinding and peeking capabilities. +/// Subsequent reads will preserve access to the original stream contents. +/// ``` +/// # use monoio::io::PrefixedReadIo; +/// # use monoio::io::{AsyncReadRent, AsyncWriteRent, AsyncReadRentExt}; +/// +/// async fn demo(mut stream: T) +/// where +/// T: AsyncReadRent + AsyncWriteRent, +/// { +/// // let stream = b"hello world"; +/// let buf = vec![0 as u8; 6]; +/// let (_, buf) = stream.read_exact(buf).await; +/// assert_eq!(buf, b"hello "); +/// +/// let prefix_buf = std::io::Cursor::new(buf); +/// let mut pio = PrefixedReadIo::new(stream, prefix_buf); +/// +/// let buf = vec![0 as u8; 11]; +/// let (_, buf) = pio.read_exact(buf).await; +/// assert_eq!(buf, b"hello world"); +/// } +/// ``` +pub struct PrefixedReadIo { + io: I, + prefix: P, + + prefix_finished: bool, +} + +impl PrefixedReadIo { + /// Create a PrefixedIo with given io and read prefix. + pub const fn new(io: I, prefix: P) -> Self { + Self { + io, + prefix, + prefix_finished: false, + } + } + + /// If the prefix has read to eof + pub const fn prefix_finished(&self) -> bool { + self.prefix_finished + } + + /// Into inner + #[inline] + pub fn into_inner(self) -> I { + self.io + } +} + +impl AsyncReadRent for PrefixedReadIo { + async fn read(&mut self, mut buf: T) -> BufResult { + if buf.bytes_total() == 0 { + return (Ok(0), buf); + } + if !self.prefix_finished { + let slice = unsafe { + &mut *std::ptr::slice_from_raw_parts_mut(buf.write_ptr(), buf.bytes_total()) + }; + match self.prefix.read(slice) { + Ok(0) => { + // prefix finished + self.prefix_finished = true; + } + Ok(n) => { + unsafe { buf.set_init(n) }; + return (Ok(n), buf); + } + Err(e) => { + return (Err(e), buf); + } + } + } + // prefix eof now, read io directly + self.io.read(buf).await + } + + async fn readv(&mut self, mut buf: T) -> BufResult { + let slice = match IoVecWrapperMut::new(buf) { + Ok(slice) => slice, + Err(buf) => return (Ok(0), buf), + }; + + let (result, slice) = self.read(slice).await; + buf = slice.into_inner(); + if let Ok(n) = result { + unsafe { buf.set_init(n) }; + } + (result, buf) + } +} + +impl CancelableAsyncReadRent + for PrefixedReadIo +{ + async fn cancelable_read( + &mut self, + mut buf: T, + c: CancelHandle, + ) -> crate::BufResult { + if buf.bytes_total() == 0 { + return (Ok(0), buf); + } + if !self.prefix_finished { + let slice = unsafe { + &mut *std::ptr::slice_from_raw_parts_mut(buf.write_ptr(), buf.bytes_total()) + }; + match self.prefix.read(slice) { + Ok(0) => { + // prefix finished + self.prefix_finished = true; + } + Ok(n) => { + unsafe { buf.set_init(n) }; + return (Ok(n), buf); + } + Err(e) => { + return (Err(e), buf); + } + } + } + // prefix eof now, read io directly + self.io.cancelable_read(buf, c).await + } + + async fn cancelable_readv( + &mut self, + mut buf: T, + c: CancelHandle, + ) -> crate::BufResult { + let slice = match IoVecWrapperMut::new(buf) { + Ok(slice) => slice, + Err(buf) => return (Ok(0), buf), + }; + + let (result, slice) = self.cancelable_read(slice, c).await; + buf = slice.into_inner(); + if let Ok(n) = result { + unsafe { buf.set_init(n) }; + } + (result, buf) + } +} + +impl AsyncWriteRent for PrefixedReadIo { + #[inline] + fn write(&mut self, buf: T) -> impl Future> { + self.io.write(buf) + } + + #[inline] + fn writev(&mut self, buf_vec: T) -> impl Future> { + self.io.writev(buf_vec) + } + + #[inline] + fn flush(&mut self) -> impl Future> { + self.io.flush() + } + + #[inline] + fn shutdown(&mut self) -> impl Future> { + self.io.shutdown() + } +} + +impl CancelableAsyncWriteRent for PrefixedReadIo { + #[inline] + fn cancelable_write( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future> { + self.io.cancelable_write(buf, c) + } + + #[inline] + fn cancelable_writev( + &mut self, + buf_vec: T, + c: CancelHandle, + ) -> impl Future> { + self.io.cancelable_writev(buf_vec, c) + } + + #[inline] + fn cancelable_flush(&mut self, c: CancelHandle) -> impl Future> { + self.io.cancelable_flush(c) + } + + #[inline] + fn cancelable_shutdown( + &mut self, + c: CancelHandle, + ) -> impl Future> { + self.io.cancelable_shutdown(c) + } +} + +/// implement unsafe Split for PrefixedReadIo, it's `safe` +/// because read/write are independent, we can safely split them into two I/O parts. +unsafe impl Split for PrefixedReadIo where I: Split {} diff --git a/vendor/monoio/src/io/util/split.rs b/vendor/monoio/src/io/util/split.rs new file mode 100644 index 000000000..a6364a3eb --- /dev/null +++ b/vendor/monoio/src/io/util/split.rs @@ -0,0 +1,253 @@ +use std::{ + cell::UnsafeCell, + error::Error, + fmt::{self, Debug}, + future::Future, + rc::Rc, +}; + +use super::CancelHandle; +use crate::{ + io::{AsyncReadRent, AsyncWriteRent, CancelableAsyncReadRent, CancelableAsyncWriteRent}, + BufResult, +}; + +/// Owned Read Half Part +#[derive(Debug)] +pub struct OwnedReadHalf(pub Rc>); +/// Owned Write Half Part +#[derive(Debug)] +#[repr(transparent)] +pub struct OwnedWriteHalf(pub Rc>) +where + T: AsyncWriteRent; + +/// This is a dummy unsafe trait to inform monoio, +/// the object with has this `Split` trait can be safely split +/// to read/write object in both form of `Owned` or `Borrowed`. +/// +/// # Safety +/// +/// monoio cannot guarantee whether the custom object can be +/// safely split to divided objects. Users should ensure the read +/// operations are indenpendence from the write ones, the methods +/// from `AsyncReadRent` and `AsyncWriteRent` can execute concurrently. +pub unsafe trait Split {} + +/// Inner split trait +pub trait Splitable { + /// Owned Read Split + type OwnedRead; + /// Owned Write Split + type OwnedWrite; + + /// Split into owned parts + fn into_split(self) -> (Self::OwnedRead, Self::OwnedWrite); +} + +impl Splitable for T +where + T: Split + AsyncWriteRent, +{ + type OwnedRead = OwnedReadHalf; + type OwnedWrite = OwnedWriteHalf; + + #[inline] + fn into_split(self) -> (Self::OwnedRead, Self::OwnedWrite) { + let shared = Rc::new(UnsafeCell::new(self)); + (OwnedReadHalf(shared.clone()), OwnedWriteHalf(shared)) + } +} + +impl AsyncReadRent for OwnedReadHalf +where + Inner: AsyncReadRent, +{ + #[inline] + fn read( + &mut self, + buf: T, + ) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.read(buf) + } + + #[inline] + fn readv( + &mut self, + buf: T, + ) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.readv(buf) + } +} + +impl CancelableAsyncReadRent for OwnedReadHalf +where + Inner: CancelableAsyncReadRent, +{ + #[inline] + fn cancelable_read( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.cancelable_read(buf, c) + } + + #[inline] + fn cancelable_readv( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.cancelable_readv(buf, c) + } +} + +impl AsyncWriteRent for OwnedWriteHalf +where + Inner: AsyncWriteRent, +{ + #[inline] + fn write(&mut self, buf: T) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.write(buf) + } + + #[inline] + fn writev( + &mut self, + buf_vec: T, + ) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.writev(buf_vec) + } + + #[inline] + fn flush(&mut self) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.flush() + } + + #[inline] + fn shutdown(&mut self) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.shutdown() + } +} + +impl CancelableAsyncWriteRent for OwnedWriteHalf +where + Inner: CancelableAsyncWriteRent, +{ + #[inline] + fn cancelable_write( + &mut self, + buf: T, + c: CancelHandle, + ) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.cancelable_write(buf, c) + } + + #[inline] + fn cancelable_writev( + &mut self, + buf_vec: T, + c: CancelHandle, + ) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.cancelable_writev(buf_vec, c) + } + + #[inline] + fn cancelable_flush(&mut self, c: CancelHandle) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.cancelable_flush(c) + } + + #[inline] + fn cancelable_shutdown( + &mut self, + c: CancelHandle, + ) -> impl Future> { + let stream = unsafe { &mut *self.0.get() }; + stream.cancelable_shutdown(c) + } +} + +impl OwnedReadHalf +where + T: AsyncWriteRent, +{ + /// reunite write half + #[inline] + pub fn reunite(self, other: OwnedWriteHalf) -> Result> { + reunite(self, other) + } +} + +impl OwnedWriteHalf +where + T: AsyncWriteRent, +{ + /// reunite read half + #[inline] + pub fn reunite(self, other: OwnedReadHalf) -> Result> { + reunite(other, self) + } +} + +impl Drop for OwnedWriteHalf +where + T: AsyncWriteRent, +{ + #[inline] + fn drop(&mut self) { + let write = unsafe { &mut *self.0.get() }; + // Notes:: shutdown is an async function but rust currently does not support async drop + // this drop will only execute sync part of `shutdown` function. + #[allow(unused_must_use)] + { + write.shutdown(); + } + } +} + +pub(crate) fn reunite( + read: OwnedReadHalf, + write: OwnedWriteHalf, +) -> Result> { + if Rc::ptr_eq(&read.0, &write.0) { + // we cannot execute drop for OwnedWriteHalf. + unsafe { + let _inner: Rc> = std::mem::transmute(write); + } + // This unwrap cannot fail as the api does not allow creating more than two + // Arcs, and we just dropped the other half. + Ok(Rc::try_unwrap(read.0) + .expect("try_unwrap failed in reunite") + .into_inner()) + } else { + Err(ReuniteError(read, write)) + } +} + +/// Error indicating that two halves were not from the same socket, and thus +/// could not be reunited. +#[derive(Debug)] +pub struct ReuniteError(pub OwnedReadHalf, pub OwnedWriteHalf); + +impl fmt::Display for ReuniteError +where + T: AsyncWriteRent, +{ + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "tried to reunite halves") + } +} + +impl Error for ReuniteError where T: AsyncWriteRent + Debug {} diff --git a/vendor/monoio/src/lib.rs b/vendor/monoio/src/lib.rs new file mode 100644 index 000000000..1f7b680fc --- /dev/null +++ b/vendor/monoio/src/lib.rs @@ -0,0 +1,147 @@ +#![doc = include_str!("../README.md")] +#![cfg_attr(docsrs, feature(doc_cfg))] +#![warn(missing_docs, unreachable_pub)] +#![allow(stable_features)] +#![allow(clippy::macro_metavars_in_unsafe)] +#![cfg_attr(feature = "unstable", feature(io_error_more))] +#![cfg_attr(feature = "unstable", feature(lazy_cell))] +#![cfg_attr(feature = "unstable", feature(stmt_expr_attributes))] +#![cfg_attr(feature = "unstable", feature(thread_local))] + +#[macro_use] +pub mod macros; +#[cfg(feature = "macros")] +#[doc(hidden)] +pub use monoio_macros::select_priv_declare_output_enum; +#[macro_use] +mod driver; +pub(crate) mod builder; +#[allow(dead_code)] +pub(crate) mod runtime; +mod scheduler; +pub mod time; + +extern crate alloc; + +#[cfg(feature = "sync")] +pub mod blocking; + +pub mod buf; +pub mod fs; +pub mod io; +pub mod net; +pub mod task; +pub mod utils; + +use std::future::Future; + +#[cfg(feature = "sync")] +pub use blocking::spawn_blocking; +pub use builder::{Buildable, RuntimeBuilder}; +pub use driver::Driver; +#[cfg(all(target_os = "linux", feature = "iouring"))] +pub use driver::IoUringDriver; +#[cfg(feature = "legacy")] +pub use driver::LegacyDriver; +#[cfg(feature = "macros")] +pub use monoio_macros::{main, test, test_all}; +pub use runtime::{spawn, Runtime}; +#[cfg(any(all(target_os = "linux", feature = "iouring"), feature = "legacy"))] +pub use {builder::FusionDriver, runtime::FusionRuntime}; + +/// moon patch: set the legacy-driver readiness spin budget in microseconds +/// before the runtime starts (poll-mode park; see driver/legacy). Equivalent +/// to the MOON_EPOLL_SPIN_US env var; the programmatic value wins. 0 disables +/// spinning. +#[cfg(feature = "legacy")] +pub fn set_legacy_spin_budget_us(us: u64) { + driver::set_legacy_spin_budget_us(us) +} + +/// moon patch: register per-thread spin-park hooks for the legacy driver's +/// poll-mode park (skip-notify handshake). `advertise(spinning)` is invoked +/// at spin entry/exit; `probe()` each spin iteration plus once after the +/// exit advertise (Dekker final check) — it must report (and locally wake +/// for) pending host work such as SPSC ringbuf items. Must be called on the +/// runtime's own thread before it first parks; closures are thread-local and +/// need not be Send. No effect unless a spin budget is active. +#[cfg(feature = "legacy")] +pub fn set_legacy_spin_hooks(advertise: Box, probe: Box bool>) { + driver::set_legacy_spin_hooks(advertise, probe) +} + +/// Start a monoio runtime. +/// +/// # Examples +/// +/// Basic usage +/// +/// ```no_run +/// fn main() -> Result<(), Box> { +/// #[cfg(not(all(target_os = "linux", feature = "iouring")))] +/// let r = monoio::start::(async { +/// // Open a file +/// let file = monoio::fs::File::open("hello.txt").await?; +/// +/// let buf = vec![0; 4096]; +/// // Read some data, the buffer is passed by ownership and +/// // submitted to the kernel. When the operation completes, +/// // we get the buffer back. +/// let (res, buf) = file.read_at(buf, 0).await; +/// let n = res?; +/// +/// // Display the contents +/// println!("{:?}", &buf[..n]); +/// +/// Ok(()) +/// }); +/// #[cfg(all(target_os = "linux", feature = "iouring"))] +/// let r = Ok(()); +/// r +/// } +/// ``` +pub fn start(future: F) -> F::Output +where + F: Future, + F::Output: 'static, + D: Buildable + Driver, +{ + let mut rt = builder::Buildable::build(builder::RuntimeBuilder::::new()) + .expect("Unable to build runtime."); + rt.block_on(future) +} + +/// A specialized `Result` type for `io-uring` operations with buffers. +/// +/// This type is used as a return value for asynchronous `io-uring` methods that +/// require passing ownership of a buffer to the runtime. When the operation +/// completes, the buffer is returned whether or not the operation completed +/// successfully. +/// +/// # Examples +/// +/// ```no_run +/// fn main() -> Result<(), Box> { +/// #[cfg(not(all(target_os = "linux", feature = "iouring")))] +/// let r = monoio::start::(async { +/// // Open a file +/// let file = monoio::fs::File::open("hello.txt").await?; +/// +/// let buf = vec![0; 4096]; +/// // Read some data, the buffer is passed by ownership and +/// // submitted to the kernel. When the operation completes, +/// // we get the buffer back. +/// let (res, buf) = file.read_at(buf, 0).await; +/// let n = res?; +/// +/// // Display the contents +/// println!("{:?}", &buf[..n]); +/// +/// Ok(()) +/// }); +/// #[cfg(all(target_os = "linux", feature = "iouring"))] +/// let r = Ok(()); +/// r +/// } +/// ``` +pub type BufResult = (std::io::Result, B); diff --git a/vendor/monoio/src/macros/debug.rs b/vendor/monoio/src/macros/debug.rs new file mode 100644 index 000000000..ee62ef05f --- /dev/null +++ b/vendor/monoio/src/macros/debug.rs @@ -0,0 +1,21 @@ +#[cfg(all(debug_assertions, feature = "debug"))] +macro_rules! trace { + ($( $args:expr ),*) => { tracing::trace!( $( $args ),* ); } +} + +#[cfg(not(all(debug_assertions, feature = "debug")))] +macro_rules! trace { + ($( $args:expr ),*) => {}; +} + +#[allow(unused_macros)] +#[cfg(all(debug_assertions, feature = "debug"))] +macro_rules! info { + ($( $args:expr ),*) => { tracing::info!( $( $args ),* ); } +} + +#[allow(unused_macros)] +#[cfg(not(all(debug_assertions, feature = "debug")))] +macro_rules! info { + ($( $args:expr ),*) => {}; +} diff --git a/vendor/monoio/src/macros/join.rs b/vendor/monoio/src/macros/join.rs new file mode 100644 index 000000000..6133a03e4 --- /dev/null +++ b/vendor/monoio/src/macros/join.rs @@ -0,0 +1,117 @@ +/// Wait on multiple concurrent branches, returning when **all** branches +/// complete. +/// +/// The `join!` macro must be used inside of async functions, closures, and +/// blocks. +/// +/// The `join!` macro takes a list of async expressions and evaluates them +/// concurrently on the same task. Each async expression evaluates to a future +/// and the futures from each expression are multiplexed on the current task. +/// +/// When working with async expressions returning `Result`, `join!` will wait +/// for **all** branches complete regardless if any complete with `Err`. Use +/// [`try_join!`] to return early when `Err` is encountered. +/// +/// [`try_join!`]: macro@try_join +/// +/// # Notes +/// +/// The supplied futures are stored inline and does not require allocating a +/// `Vec`. +/// +/// ### Runtime characteristics +/// +/// By running all async expressions on the current task, the expressions are +/// able to run **concurrently** but not in **parallel**. This means all +/// expressions are run on the same thread and if one branch blocks the thread, +/// all other expressions will be unable to continue. If parallelism is +/// required, spawn each async expression using [`monoio::spawn`] and pass the +/// join handle to `join!`. +/// +/// [`monoio::spawn`]: crate::spawn +/// +/// # Examples +/// +/// Basic join with two branches +/// +/// ``` +/// async fn do_stuff_async() { +/// // async work +/// } +/// +/// async fn more_async_work() { +/// // more here +/// } +/// +/// #[monoio::main] +/// async fn main() { +/// let (first, second) = monoio::join!(do_stuff_async(), more_async_work()); +/// +/// // do something with the values +/// } +/// ``` +#[macro_export] +#[cfg_attr(docsrs, doc(cfg(feature = "macros")))] +macro_rules! join { + (@ { + // One `_` for each branch in the `join!` macro. This is not used once + // normalization is complete. + ( $($count:tt)* ) + + // Normalized join! branches + $( ( $($skip:tt)* ) $e:expr, )* + + }) => {{ + use $crate::macros::support::{maybe_done, poll_fn, Future, Pin}; + use $crate::macros::support::Poll::{Ready, Pending}; + + // Safety: nothing must be moved out of `futures`. This is to satisfy + // the requirement of `Pin::new_unchecked` called below. + let mut futures = ( $( maybe_done($e), )* ); + + poll_fn(move |cx| { + let mut is_pending = false; + + $( + // Extract the future for this branch from the tuple. + let ( $($skip,)* fut, .. ) = &mut futures; + + // Safety: future is stored on the stack above + // and never moved. + let fut = unsafe { Pin::new_unchecked(fut) }; + + // Try polling + if fut.poll(cx).is_pending() { + is_pending = true; + } + )* + + if is_pending { + Pending + } else { + Ready(($({ + // Extract the future for this branch from the tuple. + let ( $($skip,)* fut, .. ) = &mut futures; + + // Safety: future is stored on the stack above + // and never moved. + let fut = unsafe { Pin::new_unchecked(fut) }; + + fut.take_output().expect("expected completed future") + },)*)) + } + }).await + }}; + + // ===== Normalize ===== + + (@ { ( $($s:tt)* ) $($t:tt)* } $e:expr, $($r:tt)* ) => { + $crate::join!(@{ ($($s)* _) $($t)* ($($s)*) $e, } $($r)*) + }; + + // ===== Entry point ===== + + ( $($e:expr),* $(,)?) => { + $crate::join!(@{ () } $($e,)*) + }; +} diff --git a/vendor/monoio/src/macros/mod.rs b/vendor/monoio/src/macros/mod.rs new file mode 100644 index 000000000..bd9f1738b --- /dev/null +++ b/vendor/monoio/src/macros/mod.rs @@ -0,0 +1,26 @@ +//! Useful macros. + +#[macro_use] +pub mod scoped_tls; + +#[macro_use] +mod pin; + +#[macro_use] +mod ready; + +#[macro_use] +mod select; + +#[macro_use] +mod join; + +#[macro_use] +mod try_join; + +// Includes re-exports needed to implement macros +#[doc(hidden)] +pub mod support; + +#[macro_use] +mod debug; diff --git a/vendor/monoio/src/macros/pin.rs b/vendor/monoio/src/macros/pin.rs new file mode 100644 index 000000000..1637d35c6 --- /dev/null +++ b/vendor/monoio/src/macros/pin.rs @@ -0,0 +1,112 @@ +/// Pins a value on the stack. +/// +/// Calls to `async fn` return anonymous [`Future`] values that are `!Unpin`. +/// These values must be pinned before they can be polled. Calling `.await` will +/// handle this, but consumes the future. If it is required to call `.await` on +/// a `&mut _` reference, the caller is responsible for pinning the future. +/// +/// Pinning may be done by allocating with [`Box::pin`] or by using the stack +/// with the `pin!` macro. +/// +/// The following will **fail to compile**: +/// +/// ```compile_fail +/// async fn my_async_fn() { +/// // async logic here +/// } +/// +/// #[monoio::main] +/// async fn main() { +/// let mut future = my_async_fn(); +/// (&mut future).await; +/// } +/// ``` +/// +/// To make this work requires pinning: +/// +/// ``` +/// use monoio::pin; +/// +/// async fn my_async_fn() { +/// // async logic here +/// } +/// +/// #[monoio::main] +/// async fn main() { +/// let future = my_async_fn(); +/// pin!(future); +/// +/// (&mut future).await; +/// } +/// ``` +/// +/// Pinning is useful when using `select!` and stream operators that require `T: +/// Stream + Unpin`. +/// +/// [`Future`]: trait@std::future::Future +/// [`Box::pin`]: std::boxed::Box::pin +/// +/// # Usage +/// +/// The `pin!` macro takes **identifiers** as arguments. It does **not** work +/// with expressions. +/// +/// The following does not compile as an expression is passed to `pin!`. +/// +/// ```compile_fail +/// async fn my_async_fn() { +/// // async logic here +/// } +/// +/// #[monoio::main] +/// async fn main() { +/// let mut future = pin!(my_async_fn()); +/// (&mut future).await; +/// } +/// ``` +/// +/// Because assigning to a variable followed by pinning is common, there is also +/// a variant of the macro that supports doing both in one go. +/// +/// ``` +/// use monoio::{pin, select}; +/// +/// async fn my_async_fn() { +/// // async logic here +/// } +/// +/// #[monoio::main] +/// async fn main() { +/// pin! { +/// let future1 = my_async_fn(); +/// let future2 = my_async_fn(); +/// } +/// +/// select! { +/// _ = &mut future1 => {} +/// _ = &mut future2 => {} +/// } +/// } +/// ``` +#[deprecated(note = "use std::pin::pin instead")] +#[macro_export] +macro_rules! pin { + ($($x:ident),*) => { $( + // Move the value to ensure that it is owned + let mut $x = $x; + // Shadow the original binding so that it can't be directly accessed + // ever again. + #[allow(unused_mut)] + let mut $x = unsafe { + $crate::macros::support::Pin::new_unchecked(&mut $x) + }; + )* }; + ($( + let $x:ident = $init:expr; + )*) => { + $( + let $x = $init; + $crate::pin!($x); + )* + }; +} diff --git a/vendor/monoio/src/macros/ready.rs b/vendor/monoio/src/macros/ready.rs new file mode 100644 index 000000000..1f48623b8 --- /dev/null +++ b/vendor/monoio/src/macros/ready.rs @@ -0,0 +1,8 @@ +macro_rules! ready { + ($e:expr $(,)?) => { + match $e { + std::task::Poll::Ready(t) => t, + std::task::Poll::Pending => return std::task::Poll::Pending, + } + }; +} diff --git a/vendor/monoio/src/macros/scoped_tls.rs b/vendor/monoio/src/macros/scoped_tls.rs new file mode 100644 index 000000000..25980628f --- /dev/null +++ b/vendor/monoio/src/macros/scoped_tls.rs @@ -0,0 +1,129 @@ +// Copyright 2014-2015 The Rust Project Developers. See the COPYRIGHT +// file at the top-level directory of this distribution and at +// http://rust-lang.org/COPYRIGHT. +// +// Licensed under the Apache License, Version 2.0 or the MIT license +// , at your +// option. This file may not be copied, modified, or distributed +// except according to those terms. + +//! Scoped thread-local storage +//! +//! This module provides the ability to generate *scoped* thread-local +//! variables. In this sense, scoped indicates that thread local storage +//! actually stores a reference to a value, and this reference is only placed +//! in storage for a scoped amount of time. +//! +//! There are no restrictions on what types can be placed into a scoped +//! variable, but all scoped variables are initialized to the equivalent of +//! null. Scoped thread local storage is useful when a value is present for a known +//! period of time and it is not required to relinquish ownership of the +//! contents. + +#![deny(missing_docs, warnings)] + +use std::{cell::Cell, marker, thread::LocalKey}; + +/// The macro. See the module level documentation for the description and examples. +#[macro_export] +macro_rules! scoped_thread_local { + ($(#[$attrs:meta])* $vis:vis static $name:ident: $ty:ty) => ( + $(#[$attrs])* + $vis static $name: $crate::macros::scoped_tls::ScopedKey<$ty> = $crate::macros::scoped_tls::ScopedKey { + inner: { + thread_local!(static FOO: ::std::cell::Cell<*const ()> = { + ::std::cell::Cell::new(::std::ptr::null()) + }); + &FOO + }, + _marker: ::std::marker::PhantomData, + }; + ) +} + +/// Type representing a thread local storage key corresponding to a reference +/// to the type parameter `T`. +/// +/// Keys are statically allocated and can contain a reference to an instance of +/// type `T` scoped to a particular lifetime. Keys provides two methods, `set` +/// and `with`, both of which currently use closures to control the scope of +/// their contents. +pub struct ScopedKey { + #[doc(hidden)] + pub inner: &'static LocalKey>, + #[doc(hidden)] + pub _marker: marker::PhantomData, +} + +unsafe impl Sync for ScopedKey {} + +impl ScopedKey { + /// Inserts a value into this scoped thread local storage slot for a + /// duration of a closure. + /// + /// While `cb` is running, the value `t` will be returned by `get` unless + /// this function is called recursively inside of `cb`. + /// + /// Upon return, this function will restore the previous value, if any + /// was available. + pub fn set(&'static self, t: &T, f: F) -> R + where + F: FnOnce() -> R, + { + struct Reset { + key: &'static LocalKey>, + val: *const (), + } + impl Drop for Reset { + fn drop(&mut self) { + self.key.with(|c| c.set(self.val)); + } + } + let prev = self.inner.with(|c| { + let prev = c.get(); + c.set(t as *const T as *const ()); + prev + }); + let _reset = Reset { + key: self.inner, + val: prev, + }; + f() + } + + /// Gets a value out of this scoped variable. + /// + /// This function takes a closure which receives the value of this + /// variable. + pub fn with(&'static self, f: F) -> R + where + F: FnOnce(&T) -> R, + { + let val = self.inner.with(|c| c.get()); + assert!( + !val.is_null(), + "cannot access a scoped thread local variable without calling `set` first" + ); + unsafe { f(&*(val as *const T)) } + } + + /// Gets a value out of this scoped variable. + pub fn try_with(&'static self, f: F) -> R + where + F: FnOnce(Option<&T>) -> R, + { + let val = self.inner.with(|c| c.get()); + if val.is_null() { + f(None) + } else { + unsafe { f(Some(&*(val as *const T))) } + } + } + + /// Test whether this TLS key has been `set` for the current thread. + #[inline] + pub fn is_set(&'static self) -> bool { + self.inner.with(|c| !c.get().is_null()) + } +} diff --git a/vendor/monoio/src/macros/select.rs b/vendor/monoio/src/macros/select.rs new file mode 100644 index 000000000..ad6029046 --- /dev/null +++ b/vendor/monoio/src/macros/select.rs @@ -0,0 +1,841 @@ +/// Wait on multiple concurrent branches, returning when the **first** branch +/// completes, cancelling the remaining branches. +/// +/// The `select!` macro must be used inside of async functions, closures, and +/// blocks. +/// +/// The `select!` macro accepts one or more branches with the following pattern: +/// +/// ```text +/// = (, if )? => , +/// ``` +/// +/// Additionally, the `select!` macro may include a single, optional `else` +/// branch, which evaluates if none of the other branches match their patterns: +/// +/// ```text +/// else => +/// ``` +/// +/// The macro aggregates all `` expressions and runs them +/// concurrently on the **current** task. Once the **first** expression +/// completes with a value that matches its ``, the `select!` macro +/// returns the result of evaluating the completed branch's `` +/// expression. +/// +/// Additionally, each branch may include an optional `if` precondition. If the +/// precondition returns `false`, then the branch is disabled. The provided +/// `` is still evaluated but the resulting future is never +/// polled. This capability is useful when using `select!` within a loop. +/// +/// The complete lifecycle of a `select!` expression is as follows: +/// +/// 1. Evaluate all provided `` expressions. If the precondition returns `false`, +/// disable the branch for the remainder of the current call to `select!`. Re-entering `select!` +/// due to a loop clears the "disabled" state. +/// 2. Aggregate the ``s from each branch, including the disabled ones. If the +/// branch is disabled, `` is still evaluated, but the resulting future is not +/// polled. +/// 3. Concurrently await on the results for all remaining ``s. 4. Once an `` returns a value, attempt to apply the value to the provided ``, if +/// the pattern matches, evaluate `` and return. If the pattern **does not** match, +/// disable the current branch and for the remainder of the current call to `select!`. +/// Continue from step 3. 5. If **all** branches are disabled, evaluate the `else` expression. If +/// no else branch is provided, panic. +/// +/// # Runtime characteristics +/// +/// By running all async expressions on the current task, the expressions are +/// able to run **concurrently** but not in **parallel**. This means all +/// expressions are run on the same thread and if one branch blocks the thread, +/// all other expressions will be unable to continue. If parallelism is +/// required, spawn each async expression using [`monoio::spawn`] and pass the +/// join handle to `select!`. +/// +/// [`monoio::spawn`]: crate::spawn +/// +/// # Fairness +/// +/// By default, `select!` randomly picks a branch to check first. This provides +/// some level of fairness when calling `select!` in a loop with branches that +/// are always ready. +/// +/// This behavior can be overridden by adding `biased;` to the beginning of the +/// macro usage. See the examples for details. This will cause `select` to poll +/// the futures in the order they appear from top to bottom. There are a few +/// reasons you may want this: +/// +/// - The random number generation of `monoio::select!` has a non-zero CPU cost +/// - Your futures may interact in a way where known polling order is significant +/// +/// But there is an important caveat to this mode. It becomes your +/// responsibility to ensure that the polling order of your futures is fair. If +/// for example you are selecting between a stream and a shutdown future, and +/// the stream has a huge volume of messages and zero or nearly zero time +/// between them, you should place the shutdown future earlier in the `select!` +/// list to ensure that it is always polled, and will not be ignored due to the +/// stream being constantly ready. +/// +/// # Panics +/// +/// The `select!` macro panics if all branches are disabled **and** there is no +/// provided `else` branch. A branch is disabled when the provided `if` +/// precondition returns `false` **or** when the pattern does not match the +/// result of ``. +/// +/// # Examples +/// +/// Basic select with two branches. +/// +/// ``` +/// async fn do_stuff_async() { +/// // async work +/// } +/// +/// async fn more_async_work() { +/// // more here +/// } +/// +/// #[monoio::main] +/// async fn main() { +/// monoio::select! { +/// _ = do_stuff_async() => { +/// println!("do_stuff_async() completed first") +/// } +/// _ = more_async_work() => { +/// println!("more_async_work() completed first") +/// } +/// }; +/// } +/// ``` +/// +/// Collect the contents of two streams. In this example, we rely on pattern +/// matching and the fact that `stream::iter` is "fused", i.e. once the stream +/// is complete, all calls to `next()` return `None`. +/// +/// Using the same future in multiple `select!` expressions can be done by +/// passing a reference to the future. Doing so requires the future to be +/// [`Unpin`]. A future can be made [`Unpin`] by either using [`Box::pin`] or +/// stack pinning. +/// +/// [`Unpin`]: std::marker::Unpin +/// [`Box::pin`]: std::boxed::Box::pin +/// +/// Using the `biased;` mode to control polling order. +/// +/// ``` +/// #[monoio::main] +/// async fn main() { +/// let mut count = 0u8; +/// +/// loop { +/// monoio::select! { +/// // If you run this example without `biased;`, the polling order is +/// // pseudo-random, and the assertions on the value of count will +/// // (probably) fail. +/// biased; +/// +/// _ = async {}, if count < 1 => { +/// count += 1; +/// assert_eq!(count, 1); +/// } +/// _ = async {}, if count < 2 => { +/// count += 1; +/// assert_eq!(count, 2); +/// } +/// _ = async {}, if count < 3 => { +/// count += 1; +/// assert_eq!(count, 3); +/// } +/// _ = async {}, if count < 4 => { +/// count += 1; +/// assert_eq!(count, 4); +/// } +/// +/// else => { +/// break; +/// } +/// }; +/// } +/// } +/// ``` +/// +/// ## Avoid racy `if` preconditions +/// +/// Given that `if` preconditions are used to disable `select!` branches, some +/// caution must be used to avoid missing values. +/// +/// For example, here is **incorrect** usage of `sleep` with `if`. The objective +/// is to repeatedly run an asynchronous task for up to 50 milliseconds. +/// However, there is a potential for the `sleep` completion to be missed. +/// +/// ```no_run,should_panic +/// use monoio::time::{self, Duration}; +/// +/// async fn some_async_work() { +/// // do work +/// } +/// +/// #[monoio::main(timer_enabled = true)] +/// async fn main() { +/// let mut sleep = std::pin::pin!(time::sleep(Duration::from_millis(50))); +/// +/// while !sleep.is_elapsed() { +/// monoio::select! { +/// _ = &mut sleep, if !sleep.is_elapsed() => { +/// println!("operation timed out"); +/// } +/// _ = some_async_work() => { +/// println!("operation completed"); +/// } +/// } +/// } +/// +/// panic!("This example shows how not to do it!"); +/// } +/// ``` +/// +/// In the above example, `sleep.is_elapsed()` may return `true` even if +/// `sleep.poll()` never returned `Ready`. This opens up a potential race +/// condition where `sleep` expires between the `while !sleep.is_elapsed()` +/// check and the call to `select!` resulting in the `some_async_work()` call to +/// run uninterrupted despite the sleep having elapsed. +/// +/// One way to write the above example without the race would be: +/// +/// ``` +/// use monoio::time::{self, Duration}; +/// +/// async fn some_async_work() { +/// # time::sleep(Duration::from_millis(10)).await; +/// // do work +/// } +/// +/// #[monoio::main(timer_enabled = true)] +/// async fn main() { +/// let mut sleep = std::pin::pin!(time::sleep(Duration::from_millis(50))); +/// +/// loop { +/// monoio::select! { +/// _ = &mut sleep => { +/// println!("operation timed out"); +/// break; +/// } +/// _ = some_async_work() => { +/// println!("operation completed"); +/// } +/// } +/// } +/// } +/// ``` +#[macro_export] +#[cfg_attr(docsrs, doc(cfg(feature = "macros")))] +macro_rules! select { + // Uses a declarative macro to do **most** of the work. While it is possible + // to implement fully with a declarative macro, a procedural macro is used + // to enable improved error messages. + // + // The macro is structured as a tt-muncher. All branches are processed and + // normalized. Once the input is normalized, it is passed to the top-most + // rule. When entering the macro, `@{ }` is inserted at the front. This is + // used to collect the normalized input. + // + // The macro only recurses once per branch. This allows using `select!` + // without requiring the user to increase the recursion limit. + + // All input is normalized, now transform. + (@ { + // The index of the future to poll first (in bias mode), or the RNG + // expression to use to pick a future to poll first. + start=$start:expr; + + // One `_` for each branch in the `select!` macro. Passing this to + // `count!` converts $skip to an integer. + ( $($count:tt)* ) + + // Normalized select branches. `( $skip )` is a set of `_` characters. + // There is one `_` for each select branch **before** this one. Given + // that all input futures are stored in a tuple, $skip is useful for + // generating a pattern to reference the future for the current branch. + // $skip is also used as an argument to `count!`, returning the index of + // the current select branch. + $( ( $($skip:tt)* ) $bind:pat = $fut:expr, if $c:expr => $handle:expr, )+ + + // Fallback expression used when all select branches have been disabled. + ; $else:expr + + }) => {{ + // Enter a context where stable "function-like" proc macros can be used. + // + // This module is defined within a scope and should not leak out of this + // macro. + mod util { + // Generate an enum with one variant per select branch + $crate::select_priv_declare_output_enum!( ( $($count)* ) ); + } + + // `monoio::macros::support` is a public, but doc(hidden) module + // including a re-export of all types needed by this macro. + use $crate::macros::support::Future; + use $crate::macros::support::Pin; + use $crate::macros::support::Poll::{Ready, Pending}; + + const BRANCHES: u32 = $crate::count!( $($count)* ); + + let mut disabled: util::Mask = Default::default(); + + // First, invoke all the pre-conditions. For any that return true, + // set the appropriate bit in `disabled`. + $( + if !$c { + let mask: util::Mask = 1 << $crate::count!( $($skip)* ); + disabled |= mask; + } + )* + + // Create a scope to separate polling from handling the output. This + // adds borrow checker flexibility when using the macro. + let mut output = { + // Safety: Nothing must be moved out of `futures`. This is to + // satisfy the requirement of `Pin::new_unchecked` called below. + let mut futures = ( $( $fut , )+ ); + + $crate::macros::support::poll_fn(|cx| { + // Track if any branch returns pending. If no branch completes + // **or** returns pending, this implies that all branches are + // disabled. + let mut is_pending = false; + + // Choose a starting index to begin polling the futures at. In + // practice, this will either be a pseudo-randomly generated + // number by default, or the constant 0 if `biased;` is + // supplied. + let start = $start; + + for i in 0..BRANCHES { + let branch; + #[allow(clippy::modulo_one)] + { + branch = (start + i) % BRANCHES; + } + match branch { + $( + #[allow(unreachable_code)] + $crate::count!( $($skip)* ) => { + // First, if the future has previously been + // disabled, do not poll it again. This is done + // by checking the associated bit in the + // `disabled` bit field. + let mask = 1 << branch; + + if disabled & mask == mask { + // The future has been disabled. + continue; + } + + // Extract the future for this branch from the + // tuple + let ( $($skip,)* fut, .. ) = &mut futures; + + // Safety: future is stored on the stack above + // and never moved. + let mut fut = unsafe { Pin::new_unchecked(fut) }; + + // Try polling it + let out = match fut.poll(cx) { + Ready(out) => out, + Pending => { + // Track that at least one future is + // still pending and continue polling. + is_pending = true; + continue; + } + }; + + // Disable the future from future polling. + disabled |= mask; + + // The future returned a value, check if matches + // the specified pattern. + #[allow(unused_variables)] + #[allow(unused_mut)] + match &out { + $bind => {} + _ => continue, + } + + // The select is complete, return the value + return Ready($crate::select_variant!(util::Out, ($($skip)*))(out)); + } + )* + _ => unreachable!("reaching this means there probably is an off by one bug"), + } + } + + if is_pending { + Pending + } else { + // All branches have been disabled. + Ready(util::Out::Disabled) + } + }).await + }; + + match output { + $( + $crate::select_variant!(util::Out, ($($skip)*) ($bind)) => $handle, + )* + util::Out::Disabled => $else, + _ => unreachable!("failed to match bind"), + } + }}; + + // ==== Normalize ===== + + // These rules match a single `select!` branch and normalize it for + // processing by the first rule. + + (@ { start=$start:expr; $($t:tt)* } ) => { + // No `else` branch + $crate::select!(@{ start=$start; $($t)*; panic!("all branches are disabled and there is no else branch") }) + }; + (@ { start=$start:expr; $($t:tt)* } else => $else:expr $(,)?) => { + $crate::select!(@{ start=$start; $($t)*; $else }) + }; + (@ { start=$start:expr; ( $($s:tt)* ) $($t:tt)* } $p:pat = $f:expr, if $c:expr => $h:block, $($r:tt)* ) => { + $crate::select!(@{ start=$start; ($($s)* _) $($t)* ($($s)*) $p = $f, if $c => $h, } $($r)*) + }; + (@ { start=$start:expr; ( $($s:tt)* ) $($t:tt)* } $p:pat = $f:expr => $h:block, $($r:tt)* ) => { + $crate::select!(@{ start=$start; ($($s)* _) $($t)* ($($s)*) $p = $f, if true => $h, } $($r)*) + }; + (@ { start=$start:expr; ( $($s:tt)* ) $($t:tt)* } $p:pat = $f:expr, if $c:expr => $h:block $($r:tt)* ) => { + $crate::select!(@{ start=$start; ($($s)* _) $($t)* ($($s)*) $p = $f, if $c => $h, } $($r)*) + }; + (@ { start=$start:expr; ( $($s:tt)* ) $($t:tt)* } $p:pat = $f:expr => $h:block $($r:tt)* ) => { + $crate::select!(@{ start=$start; ($($s)* _) $($t)* ($($s)*) $p = $f, if true => $h, } $($r)*) + }; + (@ { start=$start:expr; ( $($s:tt)* ) $($t:tt)* } $p:pat = $f:expr, if $c:expr => $h:expr ) => { + $crate::select!(@{ start=$start; ($($s)* _) $($t)* ($($s)*) $p = $f, if $c => $h, }) + }; + (@ { start=$start:expr; ( $($s:tt)* ) $($t:tt)* } $p:pat = $f:expr => $h:expr ) => { + $crate::select!(@{ start=$start; ($($s)* _) $($t)* ($($s)*) $p = $f, if true => $h, }) + }; + (@ { start=$start:expr; ( $($s:tt)* ) $($t:tt)* } $p:pat = $f:expr, if $c:expr => $h:expr, $($r:tt)* ) => { + $crate::select!(@{ start=$start; ($($s)* _) $($t)* ($($s)*) $p = $f, if $c => $h, } $($r)*) + }; + (@ { start=$start:expr; ( $($s:tt)* ) $($t:tt)* } $p:pat = $f:expr => $h:expr, $($r:tt)* ) => { + $crate::select!(@{ start=$start; ($($s)* _) $($t)* ($($s)*) $p = $f, if true => $h, } $($r)*) + }; + + // ===== Entry point ===== + + (biased; $p:pat = $($t:tt)* ) => { + $crate::select!(@{ start=0; () } $p = $($t)*) + }; + + ( $p:pat = $($t:tt)* ) => { + // Randomly generate a starting point. This makes `select!` a bit more + // fair and avoids always polling the first future. + $crate::select!(@{ start={ $crate::macros::support::thread_rng_n(BRANCHES) }; () } $p = $($t)*) + }; + () => { + compile_error!("select! requires at least one branch.") + }; +} + +// And here... we manually list out matches for up to 64 branches... I'm not +// happy about it either, but this is how we manage to use a declarative macro! + +#[macro_export] +#[doc(hidden)] +macro_rules! count { + () => { + 0 + }; + (_) => { + 1 + }; + (_ _) => { + 2 + }; + (_ _ _) => { + 3 + }; + (_ _ _ _) => { + 4 + }; + (_ _ _ _ _) => { + 5 + }; + (_ _ _ _ _ _) => { + 6 + }; + (_ _ _ _ _ _ _) => { + 7 + }; + (_ _ _ _ _ _ _ _) => { + 8 + }; + (_ _ _ _ _ _ _ _ _) => { + 9 + }; + (_ _ _ _ _ _ _ _ _ _) => { + 10 + }; + (_ _ _ _ _ _ _ _ _ _ _) => { + 11 + }; + (_ _ _ _ _ _ _ _ _ _ _ _) => { + 12 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _) => { + 13 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 14 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 15 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 16 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 17 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 18 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 19 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 20 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 21 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 22 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 23 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 24 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 25 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 26 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 27 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 28 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 29 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 30 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 31 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 32 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 33 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 34 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 35 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 36 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 37 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 38 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 39 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 40 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 41 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 42 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 43 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 44 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 45 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 46 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 47 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 48 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 49 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 50 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 51 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 52 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 53 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 54 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 55 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 56 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 57 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 58 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 59 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 60 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 61 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 62 + }; + (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) => { + 63 + }; +} + +#[macro_export] +#[doc(hidden)] +macro_rules! select_variant { + ($($p:ident)::*, () $($t:tt)*) => { + $($p)::*::_0 $($t)* + }; + ($($p:ident)::*, (_) $($t:tt)*) => { + $($p)::*::_1 $($t)* + }; + ($($p:ident)::*, (_ _) $($t:tt)*) => { + $($p)::*::_2 $($t)* + }; + ($($p:ident)::*, (_ _ _) $($t:tt)*) => { + $($p)::*::_3 $($t)* + }; + ($($p:ident)::*, (_ _ _ _) $($t:tt)*) => { + $($p)::*::_4 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _) $($t:tt)*) => { + $($p)::*::_5 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_6 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_7 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_8 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_9 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_10 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_11 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_12 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_13 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_14 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_15 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_16 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_17 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_18 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_19 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_20 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_21 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_22 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_23 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_24 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_25 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_26 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_27 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_28 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_29 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_30 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_31 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_32 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_33 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_34 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_35 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_36 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_37 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_38 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_39 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_40 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_41 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_42 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_43 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_44 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_45 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_46 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_47 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_48 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_49 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_50 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_51 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_52 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_53 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_54 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_55 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_56 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_57 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_58 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_59 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_60 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_61 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_62 $($t)* + }; + ($($p:ident)::*, (_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _) $($t:tt)*) => { + $($p)::*::_63 $($t)* + }; +} diff --git a/vendor/monoio/src/macros/support.rs b/vendor/monoio/src/macros/support.rs new file mode 100644 index 000000000..949b6e432 --- /dev/null +++ b/vendor/monoio/src/macros/support.rs @@ -0,0 +1,153 @@ +pub use std::{future::Future, pin::Pin, task::Poll}; + +pub use futures_util_fork::{maybe_done, poll_fn, MaybeDone, PollFn}; + +pub use crate::utils::thread_rng_n; + +mod futures_util_fork { + use core::{fmt, mem, pin::Pin}; + use std::{ + future::Future, + task::{Context, Poll}, + }; + + /// Future for the [`poll_fn`] function. + #[must_use = "futures do nothing unless you `.await` or poll them"] + pub struct PollFn { + f: F, + } + + impl Unpin for PollFn {} + + /// Creates a new future wrapping around a function returning [`Poll`]. + /// + /// Polling the returned future delegates to the wrapped function. + /// + /// # Examples + /// + /// ``` + /// # futures::executor::block_on(async { + /// use futures::{ + /// future::poll_fn, + /// task::{Context, Poll}, + /// }; + /// + /// fn read_line(_cx: &mut Context<'_>) -> Poll { + /// Poll::Ready("Hello, World!".into()) + /// } + /// + /// let read_future = poll_fn(read_line); + /// assert_eq!(read_future.await, "Hello, World!".to_owned()); + /// # }); + /// ``` + pub fn poll_fn(f: F) -> PollFn + where + F: FnMut(&mut Context<'_>) -> Poll, + { + PollFn { f } + } + + impl fmt::Debug for PollFn { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("PollFn").finish() + } + } + + impl Future for PollFn + where + F: FnMut(&mut Context<'_>) -> Poll, + { + type Output = T; + + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + (self.f)(cx) + } + } + + /// A future that may have completed. + /// + /// This is created by the [`maybe_done()`] function. + #[derive(Debug)] + pub enum MaybeDone { + /// A not-yet-completed future + Future(/* #[pin] */ Fut), + /// The output of the completed future + Done(Fut::Output), + /// The empty variant after the result of a [`MaybeDone`] has been + /// taken using the [`take_output`](MaybeDone::take_output) method. + Gone, + } + + impl Unpin for MaybeDone {} + + /// Wraps a future into a `MaybeDone` + /// + /// # Examples + /// + /// ``` + /// # futures::executor::block_on(async { + /// use futures::{future, pin_mut}; + /// + /// let future = future::maybe_done(async { 5 }); + /// pin_mut!(future); + /// assert_eq!(future.as_mut().take_output(), None); + /// let () = future.as_mut().await; + /// assert_eq!(future.as_mut().take_output(), Some(5)); + /// assert_eq!(future.as_mut().take_output(), None); + /// # }); + /// ``` + pub fn maybe_done(future: Fut) -> MaybeDone { + MaybeDone::Future(future) + } + + impl MaybeDone { + /// Returns an [`Option`] containing a mutable reference to the output + /// of the future. The output of this method will be [`Some`] if + /// and only if the inner future has been completed and + /// [`take_output`](MaybeDone::take_output) has not yet been + /// called. + #[inline] + pub fn output_mut(self: Pin<&mut Self>) -> Option<&mut Fut::Output> { + unsafe { + match self.get_unchecked_mut() { + MaybeDone::Done(res) => Some(res), + _ => None, + } + } + } + + /// Attempt to take the output of a `MaybeDone` without driving it + /// towards completion. + #[inline] + pub fn take_output(self: Pin<&mut Self>) -> Option { + match &*self { + Self::Done(_) => {} + Self::Future(_) | Self::Gone => return None, + } + unsafe { + match mem::replace(self.get_unchecked_mut(), Self::Gone) { + MaybeDone::Done(output) => Some(output), + _ => unreachable!(), + } + } + } + } + + impl Future for MaybeDone { + type Output = (); + + fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + unsafe { + match self.as_mut().get_unchecked_mut() { + MaybeDone::Future(f) => { + let res = ready!(Pin::new_unchecked(f).poll(cx)); + self.set(Self::Done(res)); + } + MaybeDone::Done(_) => {} + MaybeDone::Gone => panic!("MaybeDone polled after value taken"), + } + } + Poll::Ready(()) + } + } +} diff --git a/vendor/monoio/src/macros/try_join.rs b/vendor/monoio/src/macros/try_join.rs new file mode 100644 index 000000000..544af2a01 --- /dev/null +++ b/vendor/monoio/src/macros/try_join.rs @@ -0,0 +1,130 @@ +/// Wait on multiple concurrent branches, returning when **all** branches +/// complete with `Ok(_)` or on the first `Err(_)`. +/// +/// The `try_join!` macro must be used inside of async functions, closures, and +/// blocks. +/// +/// Similar to [`join!`], the `try_join!` macro takes a list of async +/// expressions and evaluates them concurrently on the same task. Each async +/// expression evaluates to a future and the futures from each expression are +/// multiplexed on the current task. The `try_join!` macro returns when **all** +/// branches return with `Ok` or when the **first** branch returns with `Err`. +/// +/// [`join!`]: macro@join +/// +/// # Notes +/// +/// The supplied futures are stored inline and does not require allocating a +/// `Vec`. +/// +/// ### Runtime characteristics +/// +/// By running all async expressions on the current task, the expressions are +/// able to run **concurrently** but not in **parallel**. This means all +/// expressions are run on the same thread and if one branch blocks the thread, +/// all other expressions will be unable to continue. If parallelism is +/// required, spawn each async expression using [`monoio::spawn`] and pass the +/// join handle to `try_join!`. +/// +/// [`monoio::spawn`]: crate::spawn +/// +/// # Examples +/// +/// Basic try_join with two branches. +/// +/// ``` +/// async fn do_stuff_async() -> Result<(), &'static str> { +/// // async work +/// # Ok(()) +/// } +/// +/// async fn more_async_work() -> Result<(), &'static str> { +/// // more here +/// # Ok(()) +/// } +/// +/// #[monoio::main] +/// async fn main() { +/// let res = monoio::try_join!(do_stuff_async(), more_async_work()); +/// +/// match res { +/// Ok((first, second)) => { +/// // do something with the values +/// } +/// Err(err) => { +/// println!("processing failed; error = {}", err); +/// } +/// } +/// } +/// ``` +#[macro_export] +#[cfg_attr(docsrs, doc(cfg(feature = "macros")))] +macro_rules! try_join { + (@ { + // One `_` for each branch in the `try_join!` macro. This is not used once + // normalization is complete. + ( $($count:tt)* ) + + // Normalized try_join! branches + $( ( $($skip:tt)* ) $e:expr, )* + + }) => {{ + use $crate::macros::support::{maybe_done, poll_fn, Future, Pin}; + use $crate::macros::support::Poll::{Ready, Pending}; + + // Safety: nothing must be moved out of `futures`. This is to satisfy + // the requirement of `Pin::new_unchecked` called below. + let mut futures = ( $( maybe_done($e), )* ); + + poll_fn(move |cx| { + let mut is_pending = false; + + $( + // Extract the future for this branch from the tuple. + let ( $($skip,)* fut, .. ) = &mut futures; + + // Safety: future is stored on the stack above + // and never moved. + let mut fut = unsafe { Pin::new_unchecked(fut) }; + + // Try polling + if fut.as_mut().poll(cx).is_pending() { + is_pending = true; + } else if fut.as_mut().output_mut().expect("expected completed future").is_err() { + return Ready(Err(fut.take_output().expect("expected completed future").err().unwrap())) + } + )* + + if is_pending { + Pending + } else { + Ready(Ok(($({ + // Extract the future for this branch from the tuple. + let ( $($skip,)* fut, .. ) = &mut futures; + + // Safety: future is stored on the stack above + // and never moved. + let mut fut = unsafe { Pin::new_unchecked(fut) }; + + fut + .take_output() + .expect("expected completed future") + .ok() + .expect("expected Ok(_)") + },)*))) + } + }).await + }}; + + // ===== Normalize ===== + + (@ { ( $($s:tt)* ) $($t:tt)* } $e:expr, $($r:tt)* ) => { + $crate::try_join!(@{ ($($s)* _) $($t)* ($($s)*) $e, } $($r)*) + }; + + // ===== Entry point ===== + + ( $($e:expr),* $(,)?) => { + $crate::try_join!(@{ () } $($e,)*) + }; +} diff --git a/vendor/monoio/src/net/listener_config.rs b/vendor/monoio/src/net/listener_config.rs new file mode 100644 index 000000000..056850b6a --- /dev/null +++ b/vendor/monoio/src/net/listener_config.rs @@ -0,0 +1,91 @@ +/// Custom listener options +#[derive(Debug, Clone, Copy)] +#[non_exhaustive] +pub struct ListenerOpts { + /// Whether to enable reuse_port. + pub reuse_port: bool, + /// Whether to enable reuse_addr. + pub reuse_addr: bool, + /// Backlog size. + pub backlog: i32, + /// Send buffer size or None to use default. + pub send_buf_size: Option, + /// Recv buffer size or None to use default. + pub recv_buf_size: Option, + /// TCP fast open. + pub tcp_fast_open: bool, +} + +impl Default for ListenerOpts { + #[inline] + fn default() -> Self { + Self::new() + } +} + +impl ListenerOpts { + /// Create a default ListenerOpts. + #[inline] + pub const fn new() -> Self { + Self { + reuse_port: true, + reuse_addr: true, + backlog: 1024, + send_buf_size: None, + recv_buf_size: None, + tcp_fast_open: false, + } + } + + /// Enable SO_REUSEPORT + #[must_use] + #[inline] + pub fn reuse_port(mut self, reuse_port: bool) -> Self { + self.reuse_port = reuse_port; + self + } + + /// Enable SO_REUSEADDR + #[must_use] + #[inline] + pub fn reuse_addr(mut self, reuse_addr: bool) -> Self { + self.reuse_addr = reuse_addr; + self + } + + /// Specify backlog + #[must_use] + #[inline] + pub fn backlog(mut self, backlog: i32) -> Self { + self.backlog = backlog; + self + } + + /// Specify SO_SNDBUF + #[must_use] + #[inline] + pub fn send_buf_size(mut self, send_buf_size: usize) -> Self { + self.send_buf_size = Some(send_buf_size); + self + } + + /// Specify SO_RCVBUF + #[must_use] + #[inline] + pub fn recv_buf_size(mut self, recv_buf_size: usize) -> Self { + self.recv_buf_size = Some(recv_buf_size); + self + } + + /// Specify FastOpen. + /// Note: if it is enabled, the connection will be + /// established on first peer data sent, which means + /// data cannot be sent immediately after connection + /// accepted if client does not send something. + #[must_use] + #[inline] + pub fn tcp_fast_open(mut self, fast_open: bool) -> Self { + self.tcp_fast_open = fast_open; + self + } +} diff --git a/vendor/monoio/src/net/mod.rs b/vendor/monoio/src/net/mod.rs new file mode 100644 index 000000000..7aa47a1df --- /dev/null +++ b/vendor/monoio/src/net/mod.rs @@ -0,0 +1,126 @@ +//! Network related +//! Currently, TCP/UnixStream/UnixDatagram are implemented. + +mod listener_config; +pub mod tcp; +pub mod udp; +#[cfg(unix)] +pub mod unix; + +pub use listener_config::ListenerOpts; +#[deprecated(since = "0.2.0", note = "use ListenerOpts")] +pub use listener_config::ListenerOpts as ListenerConfig; +pub use tcp::{TcpConnectOpts, TcpListener, TcpStream}; +#[cfg(unix)] +pub use unix::{Pipe, UnixDatagram, UnixListener, UnixStream}; +#[cfg(windows)] +use { + std::os::windows::prelude::RawSocket, + windows_sys::Win32::{ + Foundation::NO_ERROR, + Networking::WinSock::{ + closesocket, ioctlsocket, socket, WSACleanup, WSAStartup, ADDRESS_FAMILY, FIONBIO, + INVALID_SOCKET, WINSOCK_SOCKET_TYPE, + }, + }, +}; + +// Copied from mio. +#[cfg(unix)] +pub(crate) fn new_socket( + domain: libc::c_int, + socket_type: libc::c_int, +) -> std::io::Result { + #[cfg(any( + target_os = "android", + target_os = "dragonfly", + target_os = "freebsd", + target_os = "illumos", + target_os = "netbsd", + target_os = "openbsd" + ))] + let socket_type = socket_type | libc::SOCK_NONBLOCK | libc::SOCK_CLOEXEC; + + #[cfg(target_os = "linux")] + let socket_type = { + if crate::driver::op::is_legacy() { + socket_type | libc::SOCK_CLOEXEC | libc::SOCK_NONBLOCK + } else { + socket_type | libc::SOCK_CLOEXEC + } + }; + + // Gives a warning for platforms without SOCK_NONBLOCK. + #[allow(clippy::let_and_return)] + #[cfg(unix)] + let socket = crate::syscall!(socket(domain, socket_type, 0)); + + // Mimic `libstd` and set `SO_NOSIGPIPE` on apple systems. + #[cfg(target_vendor = "apple")] + let socket = socket.and_then(|socket| { + crate::syscall!(setsockopt( + socket, + libc::SOL_SOCKET, + libc::SO_NOSIGPIPE, + &1 as *const libc::c_int as *const libc::c_void, + std::mem::size_of::() as libc::socklen_t + )) + .map(|_| socket) + }); + + // Darwin doesn't have SOCK_NONBLOCK or SOCK_CLOEXEC. + #[cfg(any(target_os = "ios", target_os = "macos"))] + let socket = socket.and_then(|socket| { + // For platforms that don't support flags in socket, we need to + // set the flags ourselves. + crate::syscall!(fcntl(socket, libc::F_SETFL, libc::O_NONBLOCK)) + .and_then(|_| { + crate::syscall!(fcntl(socket, libc::F_SETFD, libc::FD_CLOEXEC)).map(|_| socket) + }) + .inspect_err(|_| { + // If either of the `fcntl` calls failed, ensure the socket is + // closed and return the error. + let _ = crate::syscall!(close(socket)); + }) + }); + + socket +} + +#[allow(non_snake_case, missing_docs)] +#[cfg(windows)] +#[inline] +pub fn MAKEWORD(a: u8, b: u8) -> u16 { + (a as u16) | ((b as u16) << 8) +} + +#[cfg(windows)] +pub(crate) fn new_socket( + domain: ADDRESS_FAMILY, + socket_type: WINSOCK_SOCKET_TYPE, +) -> std::io::Result { + let _: i32 = crate::syscall!( + WSAStartup(MAKEWORD(2, 2), std::ptr::null_mut()), + PartialEq::eq, + NO_ERROR as _ + )?; + let socket = crate::syscall!( + socket(domain as _, socket_type, 0), + PartialEq::eq, + INVALID_SOCKET + )?; + crate::syscall!( + ioctlsocket(socket, FIONBIO, &mut 1), + PartialEq::ne, + NO_ERROR as _ + ) + .map(|_: i32| socket as RawSocket) + .inspect_err(|_| { + // If either of the `ioctlsocket` calls failed, ensure the socket is + // closed and return the error. + unsafe { + closesocket(socket); + WSACleanup(); + } + }) +} diff --git a/vendor/monoio/src/net/tcp/listener.rs b/vendor/monoio/src/net/tcp/listener.rs new file mode 100644 index 000000000..09ae6844b --- /dev/null +++ b/vendor/monoio/src/net/tcp/listener.rs @@ -0,0 +1,321 @@ +use std::{ + cell::UnsafeCell, + io, + net::{Ipv4Addr, Ipv6Addr, SocketAddr, SocketAddrV4, SocketAddrV6, ToSocketAddrs}, +}; + +#[cfg(unix)] +use { + libc::{sockaddr_in, sockaddr_in6, AF_INET, AF_INET6}, + std::os::unix::prelude::{AsRawFd, FromRawFd, IntoRawFd, RawFd}, +}; +#[cfg(windows)] +use { + std::os::windows::prelude::{AsRawSocket, FromRawSocket, IntoRawSocket, RawSocket}, + windows_sys::Win32::Networking::WinSock::{ + AF_INET, AF_INET6, SOCKADDR_IN as sockaddr_in, SOCKADDR_IN6 as sockaddr_in6, + }, +}; + +use super::stream::TcpStream; +use crate::{ + driver::{op::Op, shared_fd::SharedFd}, + io::{stream::Stream, CancelHandle}, + net::ListenerOpts, +}; + +/// TcpListener +pub struct TcpListener { + fd: SharedFd, + sys_listener: Option, + meta: UnsafeCell, +} + +impl TcpListener { + #[allow(unreachable_code, clippy::diverging_sub_expression, unused_variables)] + pub(crate) fn from_shared_fd(fd: SharedFd) -> Self { + #[cfg(unix)] + let sys_listener = unsafe { std::net::TcpListener::from_raw_fd(fd.raw_fd()) }; + #[cfg(windows)] + let sys_listener = unsafe { std::net::TcpListener::from_raw_socket(fd.raw_socket()) }; + Self { + fd, + sys_listener: Some(sys_listener), + meta: UnsafeCell::new(ListenerMeta::default()), + } + } + + /// Bind to address with config + pub fn bind_with_config(addr: A, opts: &ListenerOpts) -> io::Result { + let addr = addr + .to_socket_addrs()? + .next() + .ok_or_else(|| io::Error::new(io::ErrorKind::Other, "empty address"))?; + + let domain = if addr.is_ipv6() { + socket2::Domain::IPV6 + } else { + socket2::Domain::IPV4 + }; + let sys_listener = + socket2::Socket::new(domain, socket2::Type::STREAM, Some(socket2::Protocol::TCP))?; + + #[cfg(feature = "legacy")] + Self::set_non_blocking(&sys_listener)?; + + let addr = socket2::SockAddr::from(addr); + #[cfg(unix)] + if opts.reuse_port { + sys_listener.set_reuse_port(true)?; + } + if opts.reuse_addr { + sys_listener.set_reuse_address(true)?; + } + if let Some(send_buf_size) = opts.send_buf_size { + sys_listener.set_send_buffer_size(send_buf_size)?; + } + if let Some(recv_buf_size) = opts.recv_buf_size { + sys_listener.set_recv_buffer_size(recv_buf_size)?; + } + if opts.tcp_fast_open { + #[cfg(any(target_os = "linux", target_os = "android"))] + super::tfo::set_tcp_fastopen(&sys_listener, opts.backlog)?; + #[cfg(any(target_os = "ios", target_os = "macos"))] + let _ = super::tfo::set_tcp_fastopen_force_enable(&sys_listener); + } + sys_listener.bind(&addr)?; + sys_listener.listen(opts.backlog)?; + + #[cfg(any(target_os = "ios", target_os = "macos"))] + if opts.tcp_fast_open { + super::tfo::set_tcp_fastopen(&sys_listener)?; + } + + #[cfg(unix)] + let fd = sys_listener.into_raw_fd(); + + #[cfg(windows)] + let fd = sys_listener.into_raw_socket(); + + Ok(Self::from_shared_fd(SharedFd::new::(fd)?)) + } + + /// Bind to address + pub fn bind(addr: A) -> io::Result { + const DEFAULT_CFG: ListenerOpts = ListenerOpts::new(); + Self::bind_with_config(addr, &DEFAULT_CFG) + } + + /// Accept + pub async fn accept(&self) -> io::Result<(TcpStream, SocketAddr)> { + let op = Op::accept(&self.fd)?; + + // Await the completion of the event + let completion = op.await; + + // Convert fd + let fd = completion.meta.result?; + + // Construct stream + let stream = TcpStream::from_shared_fd(SharedFd::new::(fd as _)?); + + // Construct SocketAddr + let storage = completion.data.addr.0.as_ptr(); + let addr = unsafe { + match (*storage).ss_family as _ { + AF_INET => { + // Safety: if the ss_family field is AF_INET then storage must be a sockaddr_in. + let addr: &sockaddr_in = &*(storage as *const sockaddr_in); + #[cfg(unix)] + let ip = Ipv4Addr::from(addr.sin_addr.s_addr.to_ne_bytes()); + #[cfg(windows)] + let ip = Ipv4Addr::from(addr.sin_addr.S_un.S_addr.to_ne_bytes()); + let port = u16::from_be(addr.sin_port); + SocketAddr::V4(SocketAddrV4::new(ip, port)) + } + AF_INET6 => { + // Safety: if the ss_family field is AF_INET6 then storage must be a + // sockaddr_in6. + let addr: &sockaddr_in6 = &*(storage as *const sockaddr_in6); + #[cfg(unix)] + let ip = Ipv6Addr::from(addr.sin6_addr.s6_addr); + #[cfg(windows)] + let ip = Ipv6Addr::from(addr.sin6_addr.u.Byte); + let port = u16::from_be(addr.sin6_port); + #[cfg(unix)] + let scope_id = addr.sin6_scope_id; + #[cfg(windows)] + let scope_id = addr.Anonymous.sin6_scope_id; + SocketAddr::V6(SocketAddrV6::new(ip, port, addr.sin6_flowinfo, scope_id)) + } + _ => { + return Err(io::ErrorKind::InvalidInput.into()); + } + } + }; + + Ok((stream, addr)) + } + + /// Cancelable accept + pub async fn cancelable_accept(&self, c: CancelHandle) -> io::Result<(TcpStream, SocketAddr)> { + use crate::io::operation_canceled; + + if c.canceled() { + return Err(operation_canceled()); + } + let op = Op::accept(&self.fd)?; + let _guard = c.associate_op(op.op_canceller()); + + // Await the completion of the event + let completion = op.await; + + // Convert fd + let fd = completion.meta.result?; + + // Construct stream + let stream = TcpStream::from_shared_fd(SharedFd::new::(fd as _)?); + + // Construct SocketAddr + let storage = completion.data.addr.0.as_ptr(); + let addr = unsafe { + match (*storage).ss_family as _ { + AF_INET => { + // Safety: if the ss_family field is AF_INET then storage must be a sockaddr_in. + let addr: &sockaddr_in = &*(storage as *const sockaddr_in); + #[cfg(unix)] + let ip = Ipv4Addr::from(addr.sin_addr.s_addr.to_ne_bytes()); + #[cfg(windows)] + let ip = Ipv4Addr::from(addr.sin_addr.S_un.S_addr.to_ne_bytes()); + let port = u16::from_be(addr.sin_port); + SocketAddr::V4(SocketAddrV4::new(ip, port)) + } + AF_INET6 => { + // Safety: if the ss_family field is AF_INET6 then storage must be a + // sockaddr_in6. + let addr: &sockaddr_in6 = &*(storage as *const sockaddr_in6); + #[cfg(unix)] + let ip = Ipv6Addr::from(addr.sin6_addr.s6_addr); + #[cfg(windows)] + let ip = Ipv6Addr::from(addr.sin6_addr.u.Byte); + let port = u16::from_be(addr.sin6_port); + #[cfg(unix)] + let scope_id = addr.sin6_scope_id; + #[cfg(windows)] + let scope_id = addr.Anonymous.sin6_scope_id; + SocketAddr::V6(SocketAddrV6::new(ip, port, addr.sin6_flowinfo, scope_id)) + } + _ => { + return Err(io::ErrorKind::InvalidInput.into()); + } + } + }; + + Ok((stream, addr)) + } + + /// Returns the local address that this listener is bound to. + pub fn local_addr(&self) -> io::Result { + let meta = self.meta.get(); + if let Some(addr) = unsafe { &*meta }.local_addr { + return Ok(addr); + } + self.sys_listener + .as_ref() + .unwrap() + .local_addr() + .inspect(|&addr| { + unsafe { &mut *meta }.local_addr = Some(addr); + }) + } + + #[cfg(feature = "legacy")] + fn set_non_blocking(_socket: &socket2::Socket) -> io::Result<()> { + crate::driver::CURRENT.with(|x| match x { + // TODO: windows ioring support + #[cfg(all(target_os = "linux", feature = "iouring"))] + crate::driver::Inner::Uring(_) => Ok(()), + crate::driver::Inner::Legacy(_) => _socket.set_nonblocking(true), + }) + } + + /// Wait for read readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn readable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_read(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Creates new `TcpListener` from a `std::net::TcpListener`. + pub fn from_std(stdl: std::net::TcpListener) -> io::Result { + #[cfg(unix)] + let fd = stdl.as_raw_fd(); + #[cfg(windows)] + let fd = stdl.as_raw_socket(); + match SharedFd::new::(fd) { + Ok(shared) => { + #[cfg(unix)] + let _ = stdl.into_raw_fd(); + #[cfg(windows)] + let _ = stdl.into_raw_socket(); + Ok(Self::from_shared_fd(shared)) + } + Err(e) => Err(e), + } + } +} + +impl Stream for TcpListener { + type Item = io::Result<(TcpStream, SocketAddr)>; + + #[inline] + async fn next(&mut self) -> Option { + Some(self.accept().await) + } +} + +impl std::fmt::Debug for TcpListener { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("TcpListener").field("fd", &self.fd).finish() + } +} + +#[cfg(unix)] +impl AsRawFd for TcpListener { + #[inline] + fn as_raw_fd(&self) -> RawFd { + self.fd.raw_fd() + } +} + +#[cfg(windows)] +impl AsRawSocket for TcpListener { + #[inline] + fn as_raw_socket(&self) -> RawSocket { + self.fd.raw_socket() + } +} + +impl Drop for TcpListener { + #[inline] + fn drop(&mut self) { + let listener = self.sys_listener.take().unwrap(); + #[cfg(unix)] + let _ = listener.into_raw_fd(); + #[cfg(windows)] + let _ = listener.into_raw_socket(); + } +} + +#[derive(Debug, Default, Clone)] +struct ListenerMeta { + local_addr: Option, +} diff --git a/vendor/monoio/src/net/tcp/mod.rs b/vendor/monoio/src/net/tcp/mod.rs new file mode 100644 index 000000000..ca5fd197f --- /dev/null +++ b/vendor/monoio/src/net/tcp/mod.rs @@ -0,0 +1,14 @@ +#![allow(unreachable_pub)] +//! TCP related. + +mod listener; +mod split; +mod stream; +mod tfo; + +pub use listener::TcpListener; +pub use split::{TcpOwnedReadHalf, TcpOwnedWriteHalf}; +pub use stream::{TcpConnectOpts, TcpStream}; + +#[cfg(feature = "poll-io")] +pub mod stream_poll; diff --git a/vendor/monoio/src/net/tcp/split.rs b/vendor/monoio/src/net/tcp/split.rs new file mode 100644 index 000000000..c8465f107 --- /dev/null +++ b/vendor/monoio/src/net/tcp/split.rs @@ -0,0 +1,56 @@ +use std::{io, net::SocketAddr}; + +use super::TcpStream; +use crate::io::{ + as_fd::{AsReadFd, AsWriteFd, SharedFdWrapper}, + OwnedReadHalf, OwnedWriteHalf, +}; + +/// OwnedReadHalf. +pub type TcpOwnedReadHalf = OwnedReadHalf; +/// OwnedWriteHalf +pub type TcpOwnedWriteHalf = OwnedWriteHalf; + +impl TcpOwnedReadHalf { + /// Returns the remote address that this stream is connected to. + #[inline] + pub fn peer_addr(&self) -> io::Result { + unsafe { &*self.0.get() }.peer_addr() + } + + /// Returns the local address that this stream is bound to. + #[inline] + pub fn local_addr(&self) -> io::Result { + unsafe { &*self.0.get() }.local_addr() + } +} + +impl AsReadFd for TcpOwnedReadHalf { + #[inline] + fn as_reader_fd(&mut self) -> &SharedFdWrapper { + let raw_stream = unsafe { &mut *self.0.get() }; + raw_stream.as_reader_fd() + } +} + +impl TcpOwnedWriteHalf { + /// Returns the remote address that this stream is connected to. + #[inline] + pub fn peer_addr(&self) -> io::Result { + unsafe { &*self.0.get() }.peer_addr() + } + + /// Returns the local address that this stream is bound to. + #[inline] + pub fn local_addr(&self) -> io::Result { + unsafe { &*self.0.get() }.local_addr() + } +} + +impl AsWriteFd for TcpOwnedWriteHalf { + #[inline] + fn as_writer_fd(&mut self) -> &SharedFdWrapper { + let raw_stream = unsafe { &mut *self.0.get() }; + raw_stream.as_writer_fd() + } +} diff --git a/vendor/monoio/src/net/tcp/stream.rs b/vendor/monoio/src/net/tcp/stream.rs new file mode 100644 index 000000000..bc01ec78b --- /dev/null +++ b/vendor/monoio/src/net/tcp/stream.rs @@ -0,0 +1,658 @@ +use std::{ + cell::UnsafeCell, + future::Future, + io, + net::{SocketAddr, ToSocketAddrs}, + time::Duration, +}; + +#[cfg(unix)] +use { + libc::{shutdown, AF_INET, AF_INET6, SHUT_WR, SOCK_STREAM}, + std::os::unix::prelude::{AsRawFd, FromRawFd, IntoRawFd, RawFd}, +}; +#[cfg(windows)] +use { + std::os::windows::prelude::{AsRawSocket, FromRawSocket, IntoRawSocket, RawSocket}, + windows_sys::Win32::Networking::WinSock::{ + shutdown, AF_INET, AF_INET6, SD_SEND as SHUT_WR, SOCK_STREAM, + }, +}; + +use crate::{ + buf::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut}, + driver::{op::Op, shared_fd::SharedFd}, + io::{ + as_fd::{AsReadFd, AsWriteFd, SharedFdWrapper}, + operation_canceled, AsyncReadRent, AsyncWriteRent, CancelHandle, CancelableAsyncReadRent, + CancelableAsyncWriteRent, Split, + }, + BufResult, +}; + +/// Custom tcp connect options +#[derive(Debug, Clone, Copy)] +#[non_exhaustive] +pub struct TcpConnectOpts { + /// TCP fast open. + pub tcp_fast_open: bool, +} + +impl Default for TcpConnectOpts { + #[inline] + fn default() -> Self { + Self::new() + } +} + +impl TcpConnectOpts { + /// Create a default TcpConnectOpts. + #[inline] + pub const fn new() -> Self { + Self { + tcp_fast_open: false, + } + } + + /// Specify FastOpen + /// Note: This option only works for linux 4.1+ + /// and macos/ios 9.0+. + /// If it is enabled, the connection will be + /// established on the first call to write. + #[must_use] + #[inline] + pub fn tcp_fast_open(mut self, fast_open: bool) -> Self { + self.tcp_fast_open = fast_open; + self + } +} +/// TcpStream +pub struct TcpStream { + pub(super) fd: SharedFd, + meta: StreamMeta, +} + +/// TcpStream is safe to split to two parts +unsafe impl Split for TcpStream {} + +impl TcpStream { + pub(crate) fn from_shared_fd(fd: SharedFd) -> Self { + #[cfg(unix)] + let meta = StreamMeta::new(fd.raw_fd()); + #[cfg(windows)] + let meta = StreamMeta::new(fd.raw_socket()); + #[cfg(feature = "zero-copy")] + // enable SOCK_ZEROCOPY + meta.set_zero_copy(); + + Self { fd, meta } + } + + /// Open a TCP connection to a remote host. + /// Note: This function may block the current thread while resolution is + /// performed. + // TODO(chihai): Fix it, maybe spawn_blocking like tokio. + pub async fn connect(addr: A) -> io::Result { + // TODO(chihai): loop for all addrs + let addr = addr + .to_socket_addrs()? + .next() + .ok_or_else(|| io::Error::new(io::ErrorKind::Other, "empty address"))?; + + Self::connect_addr(addr).await + } + + /// Establish a connection to the specified `addr`. + pub async fn connect_addr(addr: SocketAddr) -> io::Result { + const DEFAULT_OPTS: TcpConnectOpts = TcpConnectOpts { + tcp_fast_open: false, + }; + Self::connect_addr_with_config(addr, &DEFAULT_OPTS).await + } + + /// Establish a connection to the specified `addr` with given config. + pub async fn connect_addr_with_config( + addr: SocketAddr, + opts: &TcpConnectOpts, + ) -> io::Result { + let domain = match addr { + SocketAddr::V4(_) => AF_INET, + SocketAddr::V6(_) => AF_INET6, + }; + let socket = crate::net::new_socket(domain, SOCK_STREAM)?; + #[allow(unused_mut)] + let mut tfo = opts.tcp_fast_open; + + if tfo { + #[cfg(any(target_os = "linux", target_os = "android"))] + super::tfo::try_set_tcp_fastopen_connect(&socket); + #[cfg(any(target_os = "ios", target_os = "macos"))] + // if we cannot set force tcp fastopen, we will not use it. + if super::tfo::set_tcp_fastopen_force_enable(&socket).is_err() { + tfo = false; + } + } + let completion = Op::connect(SharedFd::new::(socket)?, addr, tfo)?.await; + completion.meta.result?; + + let stream = TcpStream::from_shared_fd(completion.data.fd); + // wait write ready on epoll branch + if crate::driver::op::is_legacy() { + #[cfg(all(any(target_os = "ios", target_os = "macos"), feature = "legacy"))] + if !tfo { + stream.writable(true).await?; + } else { + // set writable as init state + crate::driver::CURRENT.with(|inner| match inner { + crate::driver::Inner::Legacy(inner) => { + let idx = stream.fd.registered_index().unwrap(); + if let Some(mut readiness) = + unsafe { &mut *inner.get() }.io_dispatch.get(idx) + { + readiness.set_writable(); + } + } + #[allow(unreachable_patterns)] + _ => unreachable!("should never happens"), + }) + } + #[cfg(not(any(target_os = "ios", target_os = "macos")))] + stream.writable(true).await?; + + // getsockopt libc::SO_ERROR + #[cfg(unix)] + let sys_socket = unsafe { std::net::TcpStream::from_raw_fd(stream.fd.raw_fd()) }; + #[cfg(windows)] + let sys_socket = + unsafe { std::net::TcpStream::from_raw_socket(stream.fd.raw_socket()) }; + let err = sys_socket.take_error(); + #[cfg(unix)] + let _ = sys_socket.into_raw_fd(); + #[cfg(windows)] + let _ = sys_socket.into_raw_socket(); + if let Some(e) = err? { + return Err(e); + } + } + Ok(stream) + } + + /// Return the local address that this stream is bound to. + #[inline] + pub fn local_addr(&self) -> io::Result { + self.meta.local_addr() + } + + /// Return the remote address that this stream is connected to. + #[inline] + pub fn peer_addr(&self) -> io::Result { + self.meta.peer_addr() + } + + /// Get the value of the `TCP_NODELAY` option on this socket. + #[inline] + pub fn nodelay(&self) -> io::Result { + self.meta.no_delay() + } + + /// Set the value of the `TCP_NODELAY` option on this socket. + #[inline] + pub fn set_nodelay(&self, nodelay: bool) -> io::Result<()> { + self.meta.set_no_delay(nodelay) + } + + /// Set the value of the `SO_KEEPALIVE` option on this socket. + #[inline] + pub fn set_tcp_keepalive( + &self, + time: Option, + interval: Option, + retries: Option, + ) -> io::Result<()> { + self.meta.set_tcp_keepalive(time, interval, retries) + } + + /// Creates new `TcpStream` from a `std::net::TcpStream`. + pub fn from_std(stream: std::net::TcpStream) -> io::Result { + #[cfg(unix)] + let fd = stream.as_raw_fd(); + #[cfg(windows)] + let fd = stream.as_raw_socket(); + match SharedFd::new::(fd) { + Ok(shared) => { + #[cfg(unix)] + let _ = stream.into_raw_fd(); + #[cfg(windows)] + let _ = stream.into_raw_socket(); + Ok(Self::from_shared_fd(shared)) + } + Err(e) => Err(e), + } + } + + /// Wait for read readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn readable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_read(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Wait for write readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn writable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_write(&self.fd, relaxed).unwrap(); + op.wait().await + } +} + +impl AsReadFd for TcpStream { + #[inline] + fn as_reader_fd(&mut self) -> &SharedFdWrapper { + SharedFdWrapper::new(&self.fd) + } +} + +impl AsWriteFd for TcpStream { + #[inline] + fn as_writer_fd(&mut self) -> &SharedFdWrapper { + SharedFdWrapper::new(&self.fd) + } +} + +#[cfg(unix)] +impl IntoRawFd for TcpStream { + #[inline] + fn into_raw_fd(self) -> RawFd { + self.fd + .try_unwrap() + .expect("unexpected multiple reference to rawfd") + } +} +#[cfg(unix)] +impl AsRawFd for TcpStream { + #[inline] + fn as_raw_fd(&self) -> RawFd { + self.fd.raw_fd() + } +} + +#[cfg(windows)] +impl IntoRawSocket for TcpStream { + #[inline] + fn into_raw_socket(self) -> RawSocket { + self.fd + .try_unwrap() + .expect("unexpected multiple reference to rawfd") + } +} + +#[cfg(windows)] +impl AsRawSocket for TcpStream { + #[inline] + fn as_raw_socket(&self) -> RawSocket { + self.fd.raw_socket() + } +} + +impl std::fmt::Debug for TcpStream { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("TcpStream").field("fd", &self.fd).finish() + } +} + +impl AsyncWriteRent for TcpStream { + #[inline] + fn write(&mut self, buf: T) -> impl Future> { + // Submit the write operation + let op = Op::send(self.fd.clone(), buf).unwrap(); + op.write() + } + + #[inline] + fn writev(&mut self, buf_vec: T) -> impl Future> { + let op = Op::writev(&self.fd, buf_vec).unwrap(); + op.write() + } + + #[inline] + async fn flush(&mut self) -> std::io::Result<()> { + // Tcp stream does not need flush. + Ok(()) + } + + fn shutdown(&mut self) -> impl Future> { + // We could use shutdown op here, which requires kernel 5.11+. + // However, for simplicity, we just close the socket using direct syscall. + #[cfg(unix)] + let fd = self.as_raw_fd(); + #[cfg(windows)] + let fd = self.as_raw_socket() as _; + let res = match unsafe { shutdown(fd, SHUT_WR) } { + -1 => Err(io::Error::last_os_error()), + _ => Ok(()), + }; + async move { res } + } +} + +impl CancelableAsyncWriteRent for TcpStream { + #[inline] + async fn cancelable_write( + &mut self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult { + let fd = self.fd.clone(); + + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::send(fd, buf).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.write().await + } + + #[inline] + async fn cancelable_writev( + &mut self, + buf_vec: T, + c: CancelHandle, + ) -> crate::BufResult { + let fd = self.fd.clone(); + + if c.canceled() { + return (Err(operation_canceled()), buf_vec); + } + + let op = Op::writev(&fd, buf_vec).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.write().await + } + + #[inline] + async fn cancelable_flush(&mut self, _c: CancelHandle) -> io::Result<()> { + // Tcp stream does not need flush. + Ok(()) + } + + fn cancelable_shutdown(&mut self, _c: CancelHandle) -> impl Future> { + // We could use shutdown op here, which requires kernel 5.11+. + // However, for simplicity, we just close the socket using direct syscall. + #[cfg(unix)] + let fd = self.as_raw_fd(); + #[cfg(windows)] + let fd = self.as_raw_socket() as _; + let res = match unsafe { shutdown(fd, SHUT_WR) } { + -1 => Err(io::Error::last_os_error()), + _ => Ok(()), + }; + async move { res } + } +} + +impl AsyncReadRent for TcpStream { + #[inline] + fn read(&mut self, buf: T) -> impl Future> { + // Submit the read operation + let op = Op::recv(self.fd.clone(), buf).unwrap(); + op.read() + } + + #[inline] + fn readv(&mut self, buf: T) -> impl Future> { + // Submit the read operation + let op = Op::readv(self.fd.clone(), buf).unwrap(); + op.read() + } +} + +impl CancelableAsyncReadRent for TcpStream { + #[inline] + async fn cancelable_read( + &mut self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult { + let fd = self.fd.clone(); + + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::recv(fd, buf).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.read().await + } + + #[inline] + async fn cancelable_readv( + &mut self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult { + let fd = self.fd.clone(); + + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::readv(fd, buf).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.read().await + } +} + +#[cfg(all(unix, feature = "legacy", feature = "tokio-compat"))] +impl tokio::io::AsyncRead for TcpStream { + fn poll_read( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + buf: &mut tokio::io::ReadBuf<'_>, + ) -> std::task::Poll> { + unsafe { + let slice = buf.unfilled_mut(); + let raw_buf = crate::buf::RawBuf::new(slice.as_ptr() as *const u8, slice.len()); + let mut recv = Op::recv_raw(&self.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_legacy(&mut recv, cx)); + + std::task::Poll::Ready(ret.result.map(|n| { + buf.assume_init(n as usize); + buf.advance(n as usize); + })) + } + } +} + +#[cfg(all(unix, feature = "legacy", feature = "tokio-compat"))] +impl tokio::io::AsyncWrite for TcpStream { + fn poll_write( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + buf: &[u8], + ) -> std::task::Poll> { + unsafe { + let raw_buf = crate::buf::RawBuf::new(buf.as_ptr(), buf.len()); + let mut send = Op::send_raw(&self.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_legacy(&mut send, cx)); + + std::task::Poll::Ready(ret.result.map(|n| n as usize)) + } + } + + fn poll_flush( + self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + std::task::Poll::Ready(Ok(())) + } + + fn poll_shutdown( + self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + let fd = self.as_raw_fd(); + let res = match unsafe { libc::shutdown(fd, libc::SHUT_WR) } { + -1 => Err(io::Error::last_os_error()), + _ => Ok(()), + }; + std::task::Poll::Ready(res) + } + + fn poll_write_vectored( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + bufs: &[std::io::IoSlice<'_>], + ) -> std::task::Poll> { + unsafe { + let raw_buf = + crate::buf::RawBufVectored::new(bufs.as_ptr() as *const libc::iovec, bufs.len()); + let mut writev = Op::writev_raw(&self.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_legacy(&mut writev, cx)); + + std::task::Poll::Ready(ret.result.map(|n| n as usize)) + } + } + + fn is_write_vectored(&self) -> bool { + true + } +} + +struct StreamMeta { + socket: Option, + meta: UnsafeCell, +} + +#[derive(Debug, Default, Clone)] +struct Meta { + local_addr: Option, + peer_addr: Option, +} + +impl StreamMeta { + #[cfg(unix)] + fn new(fd: RawFd) -> Self { + Self { + socket: unsafe { Some(socket2::Socket::from_raw_fd(fd)) }, + meta: Default::default(), + } + } + + /// When operating files, we should use RawHandle; + /// When operating sockets, we should use RawSocket; + #[cfg(windows)] + fn new(fd: RawSocket) -> Self { + Self { + socket: unsafe { Some(socket2::Socket::from_raw_socket(fd)) }, + meta: Default::default(), + } + } + + fn local_addr(&self) -> io::Result { + let meta = unsafe { &mut *self.meta.get() }; + if let Some(addr) = meta.local_addr { + return Ok(addr); + } + + let ret = self + .socket + .as_ref() + .unwrap() + .local_addr() + .map(|addr| addr.as_socket().expect("tcp socket is expected")); + if let Ok(addr) = ret { + meta.local_addr = Some(addr); + } + ret + } + + fn peer_addr(&self) -> io::Result { + let meta = unsafe { &mut *self.meta.get() }; + if let Some(addr) = meta.peer_addr { + return Ok(addr); + } + + let ret = self + .socket + .as_ref() + .unwrap() + .peer_addr() + .map(|addr| addr.as_socket().expect("tcp socket is expected")); + if let Ok(addr) = ret { + meta.peer_addr = Some(addr); + } + ret + } + + fn no_delay(&self) -> io::Result { + self.socket.as_ref().unwrap().nodelay() + } + + fn set_no_delay(&self, no_delay: bool) -> io::Result<()> { + self.socket.as_ref().unwrap().set_nodelay(no_delay) + } + + #[allow(unused_variables)] + fn set_tcp_keepalive( + &self, + time: Option, + interval: Option, + retries: Option, + ) -> io::Result<()> { + let mut t = socket2::TcpKeepalive::new(); + if let Some(time) = time { + t = t.with_time(time) + } + if let Some(interval) = interval { + t = t.with_interval(interval) + } + #[cfg(unix)] + if let Some(retries) = retries { + t = t.with_retries(retries) + } + self.socket.as_ref().unwrap().set_tcp_keepalive(&t) + } + + #[cfg(feature = "zero-copy")] + fn set_zero_copy(&self) { + #[cfg(target_os = "linux")] + unsafe { + let fd = self.socket.as_ref().unwrap().as_raw_fd(); + let v: libc::c_int = 1; + libc::setsockopt( + fd, + libc::SOL_SOCKET, + libc::SO_ZEROCOPY, + &v as *const _ as *const _, + std::mem::size_of::() as _, + ); + } + } +} + +impl Drop for StreamMeta { + fn drop(&mut self) { + let socket = self.socket.take().unwrap(); + #[cfg(unix)] + let _ = socket.into_raw_fd(); + #[cfg(windows)] + let _ = socket.into_raw_socket(); + } +} diff --git a/vendor/monoio/src/net/tcp/stream_poll.rs b/vendor/monoio/src/net/tcp/stream_poll.rs new file mode 100644 index 000000000..0d1b7bc5d --- /dev/null +++ b/vendor/monoio/src/net/tcp/stream_poll.rs @@ -0,0 +1,197 @@ +//! This module provide a poll-io style interface for TcpStream. + +use std::{io, net::SocketAddr, time::Duration}; + +#[cfg(unix)] +use { + libc::{shutdown, SHUT_WR}, + std::os::fd::AsRawFd, +}; +#[cfg(windows)] +use { + std::os::windows::io::AsRawSocket, + windows_sys::Win32::Networking::WinSock::{shutdown, SD_SEND as SHUT_WR}, +}; + +use super::TcpStream; +use crate::driver::op::Op; + +/// A TcpStream with poll-io style interface. +/// Using this struct, you can use TcpStream in a poll-like way. +/// Underlying, it is based on a uring-based epoll. +#[derive(Debug)] +pub struct TcpStreamPoll(TcpStream); + +impl crate::io::IntoPollIo for TcpStream { + type PollIo = TcpStreamPoll; + + #[inline] + fn try_into_poll_io(self) -> Result { + self.try_into_poll_io() + } +} + +impl TcpStream { + /// Convert to poll-io style TcpStreamPoll + #[inline] + pub fn try_into_poll_io(mut self) -> Result { + match self.fd.cvt_poll() { + Ok(_) => Ok(TcpStreamPoll(self)), + Err(e) => Err((e, self)), + } + } +} + +impl crate::io::IntoCompIo for TcpStreamPoll { + type CompIo = TcpStream; + + #[inline] + fn try_into_comp_io(self) -> Result { + self.try_into_comp_io() + } +} + +impl TcpStreamPoll { + /// Convert to normal TcpStream + #[inline] + pub fn try_into_comp_io(mut self) -> Result { + match self.0.fd.cvt_comp() { + Ok(_) => Ok(self.0), + Err(e) => Err((e, self)), + } + } +} + +impl tokio::io::AsyncRead for TcpStreamPoll { + #[inline] + fn poll_read( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + buf: &mut tokio::io::ReadBuf<'_>, + ) -> std::task::Poll> { + unsafe { + let slice = buf.unfilled_mut(); + let raw_buf = crate::buf::RawBuf::new(slice.as_ptr() as *const u8, slice.len()); + let mut recv = Op::recv_raw(&self.0.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_io(&mut recv, cx)); + + std::task::Poll::Ready(ret.result.map(|n| { + buf.assume_init(n as usize); + buf.advance(n as usize); + })) + } + } +} + +impl tokio::io::AsyncWrite for TcpStreamPoll { + #[inline] + fn poll_write( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + buf: &[u8], + ) -> std::task::Poll> { + unsafe { + let raw_buf = crate::buf::RawBuf::new(buf.as_ptr(), buf.len()); + let mut send = Op::send_raw(&self.0.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_io(&mut send, cx)); + + std::task::Poll::Ready(ret.result.map(|n| n as usize)) + } + } + + #[inline] + fn poll_flush( + self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + std::task::Poll::Ready(Ok(())) + } + + #[inline] + fn poll_shutdown( + self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + #[cfg(unix)] + let fd = self.0.as_raw_fd(); + #[cfg(windows)] + let fd = self.0.as_raw_socket() as _; + let res = match unsafe { shutdown(fd, SHUT_WR) } { + -1 => Err(io::Error::last_os_error()), + _ => Ok(()), + }; + std::task::Poll::Ready(res) + } + + #[inline] + fn poll_write_vectored( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + bufs: &[std::io::IoSlice<'_>], + ) -> std::task::Poll> { + unsafe { + let raw_buf = crate::buf::RawBufVectored::new(bufs.as_ptr() as _, bufs.len()); + let mut writev = Op::writev_raw(&self.0.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_io(&mut writev, cx)); + + std::task::Poll::Ready(ret.result.map(|n| n as usize)) + } + } + + #[inline] + fn is_write_vectored(&self) -> bool { + true + } +} + +impl TcpStreamPoll { + /// Return the local address that this stream is bound to. + #[inline] + pub fn local_addr(&self) -> io::Result { + self.0.local_addr() + } + + /// Return the remote address that this stream is connected to. + #[inline] + pub fn peer_addr(&self) -> io::Result { + self.0.peer_addr() + } + + /// Get the value of the `TCP_NODELAY` option on this socket. + #[inline] + pub fn nodelay(&self) -> io::Result { + self.0.nodelay() + } + + /// Set the value of the `TCP_NODELAY` option on this socket. + #[inline] + pub fn set_nodelay(&self, nodelay: bool) -> io::Result<()> { + self.0.set_nodelay(nodelay) + } + + /// Set the value of the `SO_KEEPALIVE` option on this socket. + #[inline] + pub fn set_tcp_keepalive( + &self, + time: Option, + interval: Option, + retries: Option, + ) -> io::Result<()> { + self.0.set_tcp_keepalive(time, interval, retries) + } +} + +#[cfg(unix)] +impl AsRawFd for TcpStreamPoll { + #[inline] + fn as_raw_fd(&self) -> std::os::unix::io::RawFd { + self.0.as_raw_fd() + } +} + +#[cfg(windows)] +impl AsRawSocket for TcpStreamPoll { + fn as_raw_socket(&self) -> std::os::windows::io::RawSocket { + self.0.as_raw_socket() + } +} diff --git a/vendor/monoio/src/net/tcp/tfo/linux.rs b/vendor/monoio/src/net/tcp/tfo/linux.rs new file mode 100644 index 000000000..bea85ed6b --- /dev/null +++ b/vendor/monoio/src/net/tcp/tfo/linux.rs @@ -0,0 +1,53 @@ +use std::{cell::Cell, io, os::fd::AsRawFd}; + +#[cfg(feature = "unstable")] +#[thread_local] +pub(crate) static TFO_CONNECT_AVAILABLE: Cell = Cell::new(true); + +#[cfg(not(feature = "unstable"))] +thread_local! { + pub(crate) static TFO_CONNECT_AVAILABLE: Cell = const { Cell::new(true) }; +} + +/// Call before listen. +pub(crate) fn set_tcp_fastopen(fd: &S, fast_open: i32) -> io::Result<()> { + crate::syscall!(setsockopt( + fd.as_raw_fd(), + libc::SOL_TCP, + libc::TCP_FASTOPEN, + &fast_open as *const _ as *const libc::c_void, + std::mem::size_of::() as libc::socklen_t + ))?; + Ok(()) +} + +/// Call before connect. +/// Linux 4.1+ only. +pub(crate) fn set_tcp_fastopen_connect(fd: &S) -> io::Result<()> { + const ENABLED: libc::c_int = 0x1; + + crate::syscall!(setsockopt( + fd.as_raw_fd(), + libc::SOL_TCP, + libc::TCP_FASTOPEN_CONNECT, + &ENABLED as *const _ as *const libc::c_void, + std::mem::size_of::() as libc::socklen_t + ))?; + Ok(()) +} + +pub(crate) fn try_set_tcp_fastopen_connect(fd: &S) { + if !TFO_CONNECT_AVAILABLE.get() { + return; + } + match set_tcp_fastopen_connect(fd) { + Ok(_) => (), + Err(e) if e.raw_os_error() == Some(libc::ENOPROTOOPT) => { + TFO_CONNECT_AVAILABLE.set(false); + } + Err(_e) => { + #[cfg(all(debug_assertions, feature = "debug"))] + tracing::warn!("set_tcp_fastopen_connect failed: {}", _e); + } + } +} diff --git a/vendor/monoio/src/net/tcp/tfo/macos.rs b/vendor/monoio/src/net/tcp/tfo/macos.rs new file mode 100644 index 000000000..3465fe541 --- /dev/null +++ b/vendor/monoio/src/net/tcp/tfo/macos.rs @@ -0,0 +1,30 @@ +use std::{io, os::fd::AsRawFd}; + +/// Call before listen. +pub(crate) fn set_tcp_fastopen(fd: &S) -> io::Result<()> { + const ENABLED: libc::c_int = 0x1; + crate::syscall!(setsockopt( + fd.as_raw_fd(), + libc::IPPROTO_TCP, + libc::TCP_FASTOPEN, + &ENABLED as *const _ as *const libc::c_void, + std::mem::size_of::() as libc::socklen_t + ))?; + Ok(()) +} + +/// Force use fastopen. +/// MacOS only. +pub(crate) fn set_tcp_fastopen_force_enable(fd: &S) -> io::Result<()> { + const TCP_FASTOPEN_FORCE_ENABLE: libc::c_int = 0x218; + const ENABLED: libc::c_int = 0x1; + + crate::syscall!(setsockopt( + fd.as_raw_fd(), + libc::IPPROTO_TCP, + TCP_FASTOPEN_FORCE_ENABLE, + &ENABLED as *const _ as *const libc::c_void, + std::mem::size_of::() as libc::socklen_t + ))?; + Ok(()) +} diff --git a/vendor/monoio/src/net/tcp/tfo/mod.rs b/vendor/monoio/src/net/tcp/tfo/mod.rs new file mode 100644 index 000000000..1c20734b7 --- /dev/null +++ b/vendor/monoio/src/net/tcp/tfo/mod.rs @@ -0,0 +1,11 @@ +//! TCP Fast Open + +#[cfg(any(target_os = "ios", target_os = "macos"))] +mod macos; +#[cfg(any(target_os = "ios", target_os = "macos"))] +pub(crate) use macos::{set_tcp_fastopen, set_tcp_fastopen_force_enable}; + +#[cfg(any(target_os = "linux", target_os = "android"))] +mod linux; +#[cfg(any(target_os = "linux", target_os = "android"))] +pub(crate) use linux::{set_tcp_fastopen, try_set_tcp_fastopen_connect}; diff --git a/vendor/monoio/src/net/udp.rs b/vendor/monoio/src/net/udp.rs new file mode 100644 index 000000000..96b8f9d75 --- /dev/null +++ b/vendor/monoio/src/net/udp.rs @@ -0,0 +1,316 @@ +//! UDP impl. + +#[cfg(unix)] +use std::os::unix::prelude::{AsRawFd, FromRawFd, IntoRawFd}; +#[cfg(windows)] +use std::os::windows::prelude::{AsRawSocket, FromRawSocket, IntoRawSocket, RawSocket}; +use std::{ + io, + net::{SocketAddr, ToSocketAddrs}, +}; + +use crate::{ + buf::{IoBuf, IoBufMut}, + driver::{op::Op, shared_fd::SharedFd}, + io::{operation_canceled, CancelHandle, Split}, +}; + +/// A UDP socket. +/// +/// After creating a `UdpSocket` by [`bind`]ing it to a socket address, data can be +/// [sent to] and [received from] any other socket address. +/// +/// Although UDP is a connectionless protocol, this implementation provides an interface +/// to set an address where data should be sent and received from. After setting a remote +/// address with [`connect`], data can be sent to and received from that address with +/// [`send`] and [`recv`]. +#[derive(Debug)] +pub struct UdpSocket { + fd: SharedFd, +} + +/// UdpSocket is safe to split to two parts +unsafe impl Split for UdpSocket {} + +impl UdpSocket { + pub(crate) fn from_shared_fd(fd: SharedFd) -> Self { + Self { fd } + } + + #[cfg(feature = "legacy")] + fn set_non_blocking(_socket: &socket2::Socket) -> io::Result<()> { + crate::driver::CURRENT.with(|x| match x { + // TODO: windows ioring support + #[cfg(all(target_os = "linux", feature = "iouring"))] + crate::driver::Inner::Uring(_) => Ok(()), + crate::driver::Inner::Legacy(_) => _socket.set_nonblocking(true), + }) + } + + /// Creates a UDP socket from the given address. + pub fn bind(addr: A) -> io::Result { + let addr = addr + .to_socket_addrs()? + .next() + .ok_or_else(|| io::Error::new(io::ErrorKind::Other, "empty address"))?; + let domain = if addr.is_ipv6() { + socket2::Domain::IPV6 + } else { + socket2::Domain::IPV4 + }; + let socket = + socket2::Socket::new(domain, socket2::Type::DGRAM, Some(socket2::Protocol::UDP))?; + #[cfg(feature = "legacy")] + Self::set_non_blocking(&socket)?; + + let addr = socket2::SockAddr::from(addr); + socket.bind(&addr)?; + + #[cfg(unix)] + let fd = socket.into_raw_fd(); + #[cfg(windows)] + let fd = socket.into_raw_socket(); + + Ok(Self::from_shared_fd(SharedFd::new::(fd)?)) + } + + /// Receives a single datagram message on the socket. On success, returns the number + /// of bytes read and the origin. + pub async fn recv_from(&self, buf: T) -> crate::BufResult<(usize, SocketAddr), T> { + let op = Op::recv_msg(self.fd.clone(), buf).unwrap(); + op.wait().await + } + + /// Sends data on the socket to the given address. On success, returns the + /// number of bytes written. + pub async fn send_to( + &self, + buf: T, + socket_addr: SocketAddr, + ) -> crate::BufResult { + let op = Op::send_msg(self.fd.clone(), buf, Some(socket_addr)).unwrap(); + op.wait().await + } + + /// Returns the socket address of the remote peer this socket was connected to. + pub fn peer_addr(&self) -> io::Result { + #[cfg(unix)] + let socket = unsafe { socket2::Socket::from_raw_fd(self.fd.as_raw_fd()) }; + #[cfg(windows)] + let socket = unsafe { socket2::Socket::from_raw_socket(self.fd.as_raw_socket()) }; + let addr = socket.peer_addr(); + #[cfg(unix)] + let _ = socket.into_raw_fd(); + #[cfg(windows)] + let _ = socket.into_raw_socket(); + addr? + .as_socket() + .ok_or_else(|| io::ErrorKind::InvalidInput.into()) + } + + /// Returns the socket address that this socket was created from. + pub fn local_addr(&self) -> io::Result { + #[cfg(unix)] + let socket = unsafe { socket2::Socket::from_raw_fd(self.fd.as_raw_fd()) }; + #[cfg(windows)] + let socket = unsafe { socket2::Socket::from_raw_socket(self.fd.as_raw_socket()) }; + let addr = socket.local_addr(); + #[cfg(unix)] + let _ = socket.into_raw_fd(); + #[cfg(windows)] + let _ = socket.into_raw_socket(); + addr? + .as_socket() + .ok_or_else(|| io::ErrorKind::InvalidInput.into()) + } + + /// Connects this UDP socket to a remote address, allowing the `send` and + /// `recv` syscalls to be used to send data and also applies filters to only + /// receive data from the specified address. + pub async fn connect(&self, socket_addr: SocketAddr) -> io::Result<()> { + let op = Op::connect(self.fd.clone(), socket_addr, false)?; + let completion = op.await; + completion.meta.result?; + Ok(()) + } + + /// Sends data on the socket to the remote address to which it is connected. + pub async fn send(&self, buf: T) -> crate::BufResult { + let op = Op::send_msg(self.fd.clone(), buf, None).unwrap(); + op.wait().await + } + + /// Receives a single datagram message on the socket from the remote address to + /// which it is connected. On success, returns the number of bytes read. + pub async fn recv(&self, buf: T) -> crate::BufResult { + let op = Op::recv(self.fd.clone(), buf).unwrap(); + op.read().await + } + + /// Creates new `UdpSocket` from a `std::net::UdpSocket`. + pub fn from_std(socket: std::net::UdpSocket) -> io::Result { + #[cfg(unix)] + let fd = socket.as_raw_fd(); + #[cfg(windows)] + let fd = socket.as_raw_socket(); + match SharedFd::new::(fd) { + Ok(shared) => { + #[cfg(unix)] + let _ = socket.into_raw_fd(); + #[cfg(windows)] + let _ = socket.into_raw_socket(); + Ok(Self::from_shared_fd(shared)) + } + Err(e) => Err(e), + } + } + + /// Set value for the `SO_REUSEADDR` option on this socket. + #[allow(unused_variables)] + pub fn set_reuse_address(&self, reuse: bool) -> io::Result<()> { + #[cfg(unix)] + let r = { + let socket = unsafe { socket2::Socket::from_raw_fd(self.fd.as_raw_fd()) }; + let r = socket.set_reuse_address(reuse); + let _ = socket.into_raw_fd(); + r + }; + #[cfg(windows)] + let r = { + let socket = unsafe { socket2::Socket::from_raw_socket(self.fd.as_raw_socket()) }; + let _ = socket.into_raw_socket(); + Ok(()) + }; + r + } + + /// Set value for the `SO_REUSEPORT` option on this socket. + #[allow(unused_variables)] + pub fn set_reuse_port(&self, reuse: bool) -> io::Result<()> { + #[cfg(unix)] + let r = { + let socket = unsafe { socket2::Socket::from_raw_fd(self.fd.as_raw_fd()) }; + let r = socket.set_reuse_port(reuse); + let _ = socket.into_raw_fd(); + r + }; + #[cfg(windows)] + let r = { + let socket = unsafe { socket2::Socket::from_raw_socket(self.fd.as_raw_socket()) }; + let _ = socket.into_raw_socket(); + Ok(()) + }; + r + } + + /// Wait for read readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn readable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_read(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Wait for write readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn writable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_write(&self.fd, relaxed).unwrap(); + op.wait().await + } +} + +#[cfg(unix)] +impl AsRawFd for UdpSocket { + fn as_raw_fd(&self) -> std::os::fd::RawFd { + self.fd.raw_fd() + } +} + +#[cfg(windows)] +impl AsRawSocket for UdpSocket { + fn as_raw_socket(&self) -> RawSocket { + self.fd.raw_socket() + } +} + +/// Cancelable related methods +impl UdpSocket { + /// Receives a single datagram message on the socket. On success, returns the number + /// of bytes read and the origin. + pub async fn cancelable_recv_from( + &self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult<(usize, SocketAddr), T> { + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::recv_msg(self.fd.clone(), buf).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.wait().await + } + + /// Sends data on the socket to the given address. On success, returns the + /// number of bytes written. + pub async fn cancelable_send_to( + &self, + buf: T, + socket_addr: SocketAddr, + c: CancelHandle, + ) -> crate::BufResult { + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::send_msg(self.fd.clone(), buf, Some(socket_addr)).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.wait().await + } + + /// Sends data on the socket to the remote address to which it is connected. + pub async fn cancelable_send( + &self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult { + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::send_msg(self.fd.clone(), buf, None).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.wait().await + } + + /// Receives a single datagram message on the socket from the remote address to + /// which it is connected. On success, returns the number of bytes read. + pub async fn cancelable_recv( + &self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult { + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::recv(self.fd.clone(), buf).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.read().await + } +} diff --git a/vendor/monoio/src/net/unix/datagram/mod.rs b/vendor/monoio/src/net/unix/datagram/mod.rs new file mode 100644 index 000000000..c0e276857 --- /dev/null +++ b/vendor/monoio/src/net/unix/datagram/mod.rs @@ -0,0 +1,178 @@ +//! Unix datagram related. + +use std::{ + io, + os::unix::{ + net::UnixDatagram as StdUnixDatagram, + prelude::{AsRawFd, IntoRawFd, RawFd}, + }, + path::Path, +}; + +use super::{ + socket_addr::{local_addr, pair, peer_addr, socket_addr}, + SocketAddr, +}; +use crate::{ + buf::{IoBuf, IoBufMut}, + driver::{op::Op, shared_fd::SharedFd}, + net::new_socket, +}; + +/// UnixDatagram +pub struct UnixDatagram { + fd: SharedFd, +} + +impl UnixDatagram { + pub(crate) fn from_shared_fd(fd: SharedFd) -> Self { + Self { fd } + } + + /// Creates a Unix datagram socket bound to the given path. + pub fn bind>(path: P) -> io::Result { + StdUnixDatagram::bind(path).and_then(Self::from_std) + } + + /// Creates a new `UnixDatagram` which is not bound to any address. + pub fn unbound() -> io::Result { + StdUnixDatagram::unbound().and_then(Self::from_std) + } + + /// Creates an unnamed pair of connected sockets. + pub fn pair() -> io::Result<(Self, Self)> { + let (a, b) = pair(libc::SOCK_DGRAM)?; + Ok((Self::from_std(a)?, Self::from_std(b)?)) + } + + /// Connects the socket to the specified address. + pub async fn connect>(path: P) -> io::Result { + let (addr, addr_len) = socket_addr(path.as_ref())?; + Self::inner_connect(addr, addr_len).await + } + + /// Connects the socket to an address. + pub async fn connect_addr(addr: SocketAddr) -> io::Result { + let (addr, addr_len) = addr.into_parts(); + Self::inner_connect(addr, addr_len).await + } + + #[inline(always)] + async fn inner_connect( + sockaddr: libc::sockaddr_un, + socklen: libc::socklen_t, + ) -> io::Result { + let socket = new_socket(libc::AF_UNIX, libc::SOCK_DGRAM)?; + let op = Op::connect_unix(SharedFd::new::(socket)?, sockaddr, socklen)?; + let completion = op.await; + completion.meta.result?; + + Ok(Self::from_shared_fd(completion.data.fd)) + } + + /// Creates new `UnixDatagram` from a `std::os::unix::net::UnixDatagram`. + pub fn from_std(datagram: StdUnixDatagram) -> io::Result { + match SharedFd::new::(datagram.as_raw_fd()) { + Ok(shared) => { + let _ = datagram.into_raw_fd(); + Ok(Self::from_shared_fd(shared)) + } + Err(e) => Err(e), + } + } + + /// Returns the socket address of the local half of this connection. + pub fn local_addr(&self) -> io::Result { + local_addr(self.as_raw_fd()) + } + + /// Returns the socket address of the remote half of this connection. + pub fn peer_addr(&self) -> io::Result { + peer_addr(self.as_raw_fd()) + } + + /// Wait for read readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn readable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_read(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Wait for write readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn writable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_write(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Sends data on the socket to the given address. On success, returns the + /// number of bytes written. + pub async fn send_to>( + &self, + buf: T, + path: P, + ) -> crate::BufResult { + let addr = match crate::net::unix::socket_addr::socket_addr(path.as_ref()) { + Ok(addr) => addr, + Err(e) => return (Err(e), buf), + }; + let op = Op::send_msg_unix( + self.fd.clone(), + buf, + Some(SocketAddr::from_parts(addr.0, addr.1)), + ) + .unwrap(); + op.wait().await + } + + /// Receives a single datagram message on the socket. On success, returns the number + /// of bytes read and the origin. + pub async fn recv_from(&self, buf: T) -> crate::BufResult<(usize, SocketAddr), T> { + let op = Op::recv_msg_unix(self.fd.clone(), buf).unwrap(); + op.wait().await + } + + /// Sends data on the socket to the remote address to which it is connected. + pub async fn send(&self, buf: T) -> crate::BufResult { + let op = Op::send_msg_unix(self.fd.clone(), buf, None).unwrap(); + op.wait().await + } + + /// Receives a single datagram message on the socket from the remote address to + /// which it is connected. On success, returns the number of bytes read. + pub async fn recv(&self, buf: T) -> crate::BufResult { + let op = Op::recv(self.fd.clone(), buf).unwrap(); + op.read().await + } +} + +impl AsRawFd for UnixDatagram { + #[inline] + fn as_raw_fd(&self) -> RawFd { + self.fd.raw_fd() + } +} + +impl std::fmt::Debug for UnixDatagram { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("UnixDatagram") + .field("fd", &self.fd) + .finish() + } +} diff --git a/vendor/monoio/src/net/unix/listener.rs b/vendor/monoio/src/net/unix/listener.rs new file mode 100644 index 000000000..fba686bbb --- /dev/null +++ b/vendor/monoio/src/net/unix/listener.rs @@ -0,0 +1,198 @@ +use std::{ + io, + mem::{ManuallyDrop, MaybeUninit}, + os::unix::prelude::{AsRawFd, FromRawFd, IntoRawFd, RawFd}, + path::Path, +}; + +use super::{socket_addr::SocketAddr, UnixStream}; +use crate::{ + driver::{op::Op, shared_fd::SharedFd}, + io::{stream::Stream, CancelHandle}, + net::ListenerOpts, +}; + +/// UnixListener +pub struct UnixListener { + fd: SharedFd, + sys_listener: Option, +} + +impl UnixListener { + pub(crate) fn from_shared_fd(fd: SharedFd) -> Self { + let sys_listener = unsafe { std::os::unix::net::UnixListener::from_raw_fd(fd.raw_fd()) }; + Self { + fd, + sys_listener: Some(sys_listener), + } + } + + /// Creates a new `UnixListener` bound to the specified socket with custom + /// config. + pub fn bind_with_config>( + path: P, + config: &ListenerOpts, + ) -> io::Result { + let sys_listener = + socket2::Socket::new(socket2::Domain::UNIX, socket2::Type::STREAM, None)?; + let addr = socket2::SockAddr::unix(path)?; + + if config.reuse_port { + sys_listener.set_reuse_port(true)?; + } + if config.reuse_addr { + sys_listener.set_reuse_address(true)?; + } + if let Some(send_buf_size) = config.send_buf_size { + sys_listener.set_send_buffer_size(send_buf_size)?; + } + if let Some(recv_buf_size) = config.recv_buf_size { + sys_listener.set_recv_buffer_size(recv_buf_size)?; + } + + sys_listener.bind(&addr)?; + sys_listener.listen(config.backlog)?; + + let fd = SharedFd::new::(sys_listener.into_raw_fd())?; + + Ok(Self::from_shared_fd(fd)) + } + + /// Creates a new `UnixListener` bound to the specified socket with default + /// config. + pub fn bind>(path: P) -> io::Result { + Self::bind_with_config(path, &ListenerOpts::default()) + } + + /// Accept + pub async fn accept(&self) -> io::Result<(UnixStream, SocketAddr)> { + let op = Op::accept(&self.fd)?; + + // Await the completion of the event + let completion = op.await; + + // Convert fd + let fd = completion.meta.result?; + + // Construct stream + let stream = UnixStream::from_shared_fd(SharedFd::new::(fd as _)?); + + // Construct SocketAddr + let mut storage = unsafe { std::mem::MaybeUninit::assume_init(completion.data.addr.0) }; + let storage: *mut libc::sockaddr_storage = &mut storage as *mut _; + let raw_addr_un: libc::sockaddr_un = unsafe { *storage.cast() }; + let raw_addr_len = completion.data.addr.1; + + let addr = SocketAddr::from_parts(raw_addr_un, raw_addr_len); + + Ok((stream, addr)) + } + + /// Cancelable accept + pub async fn cancelable_accept(&self, c: CancelHandle) -> io::Result<(UnixStream, SocketAddr)> { + use crate::io::operation_canceled; + + if c.canceled() { + return Err(operation_canceled()); + } + let op = Op::accept(&self.fd)?; + let _guard = c.associate_op(op.op_canceller()); + + // Await the completion of the event + let completion = op.await; + + // Convert fd + let fd = completion.meta.result?; + + // Construct stream + let stream = UnixStream::from_shared_fd(SharedFd::new::(fd as _)?); + + // Construct SocketAddr + let mut storage = unsafe { std::mem::MaybeUninit::assume_init(completion.data.addr.0) }; + let storage: *mut libc::sockaddr_storage = &mut storage as *mut _; + let raw_addr_un: libc::sockaddr_un = unsafe { *storage.cast() }; + let raw_addr_len = completion.data.addr.1; + + let addr = SocketAddr::from_parts(raw_addr_un, raw_addr_len); + + Ok((stream, addr)) + } + + /// Wait for read readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn readable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_read(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Creates new `UnixListener` from a `std::os::unix::net::UnixListener`. + pub fn from_std(sys_listener: std::os::unix::net::UnixListener) -> io::Result { + match SharedFd::new::(sys_listener.as_raw_fd()) { + Ok(shared) => Ok(Self { + fd: shared, + sys_listener: Some(sys_listener), + }), + Err(e) => Err(e), + } + } +} + +impl Stream for UnixListener { + type Item = io::Result<(UnixStream, SocketAddr)>; + + #[inline] + async fn next(&mut self) -> Option { + Some(self.accept().await) + } +} + +impl std::fmt::Debug for UnixListener { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("UnixListener") + .field("fd", &self.fd) + .finish() + } +} + +impl IntoRawFd for UnixListener { + #[inline] + fn into_raw_fd(self) -> RawFd { + let mut this = ManuallyDrop::new(self); + #[allow(invalid_value)] + #[allow(clippy::uninit_assumed_init)] + let (mut fd, mut sys_listener) = unsafe { + ( + MaybeUninit::uninit().assume_init(), + MaybeUninit::uninit().assume_init(), + ) + }; + std::mem::swap(&mut this.fd, &mut fd); + std::mem::swap(&mut this.sys_listener, &mut sys_listener); + let _ = sys_listener.take().unwrap().into_raw_fd(); + + fd.try_unwrap() + .expect("unexpected multiple reference to rawfd") + } +} + +impl AsRawFd for UnixListener { + #[inline] + fn as_raw_fd(&self) -> RawFd { + self.fd.raw_fd() + } +} + +impl Drop for UnixListener { + #[inline] + fn drop(&mut self) { + let _ = self.sys_listener.take().unwrap().into_raw_fd(); + } +} diff --git a/vendor/monoio/src/net/unix/mod.rs b/vendor/monoio/src/net/unix/mod.rs new file mode 100644 index 000000000..fa081150c --- /dev/null +++ b/vendor/monoio/src/net/unix/mod.rs @@ -0,0 +1,30 @@ +#![allow(unreachable_pub)] +//! Unix related. + +mod datagram; +mod listener; +mod pipe; +mod socket_addr; +mod split; +mod stream; +mod ucred; + +#[cfg(target_os = "linux")] +mod seq_packet; +pub use datagram::UnixDatagram; +pub use listener::UnixListener; +pub use pipe::{new_pipe, Pipe}; +#[cfg(target_os = "linux")] +pub use seq_packet::{UnixSeqpacket, UnixSeqpacketListener}; +pub use socket_addr::SocketAddr; +pub use split::{UnixOwnedReadHalf, UnixOwnedWriteHalf}; +pub use stream::UnixStream; + +#[cfg(feature = "poll-io")] +pub mod stream_poll; + +pub(crate) fn path_offset(sockaddr: &libc::sockaddr_un) -> usize { + let base = sockaddr as *const _ as usize; + let path = &sockaddr.sun_path as *const _ as usize; + path - base +} diff --git a/vendor/monoio/src/net/unix/pipe.rs b/vendor/monoio/src/net/unix/pipe.rs new file mode 100644 index 000000000..760a0332e --- /dev/null +++ b/vendor/monoio/src/net/unix/pipe.rs @@ -0,0 +1,37 @@ +use std::{io, os::unix::prelude::RawFd}; + +use crate::driver::shared_fd::SharedFd; + +/// Unix pipe. +pub struct Pipe { + #[allow(dead_code)] + pub(crate) fd: SharedFd, +} + +impl Pipe { + pub(crate) fn from_shared_fd(fd: SharedFd) -> Self { + Self { fd } + } + + fn from_raw_fd(fd: RawFd) -> Self { + Self::from_shared_fd(SharedFd::new_without_register(fd)) + } +} + +/// Create a new pair of pipe. +pub fn new_pipe() -> io::Result<(Pipe, Pipe)> { + let mut pipes = [0 as libc::c_int; 2]; + #[cfg(target_os = "linux")] + let flag = { + if crate::driver::op::is_legacy() { + libc::O_NONBLOCK + } else { + 0 + } + }; + #[cfg(target_os = "linux")] + crate::syscall!(pipe2(pipes.as_mut_ptr() as _, flag))?; + #[cfg(not(target_os = "linux"))] + crate::syscall!(pipe(pipes.as_mut_ptr() as _))?; + Ok((Pipe::from_raw_fd(pipes[0]), Pipe::from_raw_fd(pipes[1]))) +} diff --git a/vendor/monoio/src/net/unix/seq_packet/listener.rs b/vendor/monoio/src/net/unix/seq_packet/listener.rs new file mode 100644 index 000000000..0fa3ca5c1 --- /dev/null +++ b/vendor/monoio/src/net/unix/seq_packet/listener.rs @@ -0,0 +1,89 @@ +use std::{ + io, + os::fd::{AsRawFd, RawFd}, + path::Path, +}; + +use super::UnixSeqpacket; +use crate::{ + driver::{op::Op, shared_fd::SharedFd}, + io::stream::Stream, + net::{ + new_socket, + unix::{socket_addr::socket_addr, SocketAddr}, + }, +}; + +const DEFAULT_BACKLOG: libc::c_int = 128; + +/// Listener for UnixSeqpacket +pub struct UnixSeqpacketListener { + fd: SharedFd, +} + +impl UnixSeqpacketListener { + /// Creates a new `UnixSeqpacketListener` bound to the specified path with custom backlog + pub fn bind_with_backlog>(path: P, backlog: libc::c_int) -> io::Result { + let (addr, addr_len) = socket_addr(path.as_ref())?; + let socket = new_socket(libc::AF_UNIX, libc::SOCK_SEQPACKET)?; + crate::syscall!(bind(socket, &addr as *const _ as *const _, addr_len))?; + crate::syscall!(listen(socket, backlog))?; + Ok(Self { + fd: SharedFd::new::(socket)?, + }) + } + + /// Creates a new `UnixSeqpacketListener` bound to the specified path with default backlog(128) + #[inline] + pub fn bind>(path: P) -> io::Result { + Self::bind_with_backlog(path, DEFAULT_BACKLOG) + } + + /// Accept a UnixSeqpacket + pub async fn accept(&self) -> io::Result<(UnixSeqpacket, SocketAddr)> { + let op = Op::accept(&self.fd)?; + + // Await the completion of the event + let completion = op.await; + + // Convert fd + let fd = completion.meta.result?; + + // Construct stream + let stream = UnixSeqpacket::from_shared_fd(SharedFd::new::(fd as _)?); + + // Construct SocketAddr + let mut storage = unsafe { std::mem::MaybeUninit::assume_init(completion.data.addr.0) }; + let storage: *mut libc::sockaddr_storage = &mut storage as *mut _; + let raw_addr_un: libc::sockaddr_un = unsafe { *storage.cast() }; + let raw_addr_len = completion.data.addr.1; + + let addr = SocketAddr::from_parts(raw_addr_un, raw_addr_len); + + Ok((stream, addr)) + } +} + +impl AsRawFd for UnixSeqpacketListener { + #[inline] + fn as_raw_fd(&self) -> RawFd { + self.fd.raw_fd() + } +} + +impl std::fmt::Debug for UnixSeqpacketListener { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("UnixSeqpacketListener") + .field("fd", &self.fd) + .finish() + } +} + +impl Stream for UnixSeqpacketListener { + type Item = io::Result<(UnixSeqpacket, SocketAddr)>; + + #[inline] + async fn next(&mut self) -> Option { + Some(self.accept().await) + } +} diff --git a/vendor/monoio/src/net/unix/seq_packet/mod.rs b/vendor/monoio/src/net/unix/seq_packet/mod.rs new file mode 100644 index 000000000..74f236698 --- /dev/null +++ b/vendor/monoio/src/net/unix/seq_packet/mod.rs @@ -0,0 +1,161 @@ +//! UnixSeqpacket related. +//! Only available on linux. + +use std::{ + io, + os::unix::prelude::{AsRawFd, RawFd}, + path::Path, +}; + +use super::{ + socket_addr::{local_addr, pair, peer_addr, socket_addr}, + SocketAddr, +}; +use crate::{ + buf::{IoBuf, IoBufMut}, + driver::{op::Op, shared_fd::SharedFd}, + net::new_socket, +}; + +mod listener; +pub use listener::UnixSeqpacketListener; + +/// UnixSeqpacket +pub struct UnixSeqpacket { + fd: SharedFd, +} + +impl UnixSeqpacket { + pub(crate) fn from_shared_fd(fd: SharedFd) -> Self { + Self { fd } + } + + /// Creates an unnamed pair of connected sockets. + pub fn pair() -> io::Result<(Self, Self)> { + let (a, b) = pair(libc::SOCK_SEQPACKET)?; + Ok(( + Self::from_shared_fd(SharedFd::new::(a)?), + Self::from_shared_fd(SharedFd::new::(b)?), + )) + } + + /// Connects the socket to the specified address. + pub async fn connect>(path: P) -> io::Result { + let (addr, addr_len) = socket_addr(path.as_ref())?; + Self::inner_connect(addr, addr_len).await + } + + /// Connects the socket to an address. + pub async fn connect_addr(addr: SocketAddr) -> io::Result { + let (addr, addr_len) = addr.into_parts(); + Self::inner_connect(addr, addr_len).await + } + + #[inline(always)] + async fn inner_connect( + sockaddr: libc::sockaddr_un, + socklen: libc::socklen_t, + ) -> io::Result { + let socket = new_socket(libc::AF_UNIX, libc::SOCK_SEQPACKET)?; + let op = Op::connect_unix(SharedFd::new::(socket)?, sockaddr, socklen)?; + let completion = op.await; + completion.meta.result?; + + Ok(Self::from_shared_fd(completion.data.fd)) + } + + /// Returns the socket address of the local half of this connection. + pub fn local_addr(&self) -> io::Result { + local_addr(self.as_raw_fd()) + } + + /// Returns the socket address of the remote half of this connection. + pub fn peer_addr(&self) -> io::Result { + peer_addr(self.as_raw_fd()) + } + + /// Wait for read readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn readable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_read(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Wait for write readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn writable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_write(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Sends data on the socket to the given address. On success, returns the + /// number of bytes written. + pub async fn send_to>( + &self, + buf: T, + path: P, + ) -> crate::BufResult { + let addr = match crate::net::unix::socket_addr::socket_addr(path.as_ref()) { + Ok(addr) => addr, + Err(e) => return (Err(e), buf), + }; + let op = Op::send_msg_unix( + self.fd.clone(), + buf, + Some(SocketAddr::from_parts(addr.0, addr.1)), + ) + .unwrap(); + op.wait().await + } + + /// Receives a single datagram message on the socket. On success, returns the number + /// of bytes read and the origin. + pub async fn recv_from(&self, buf: T) -> crate::BufResult<(usize, SocketAddr), T> { + let op = Op::recv_msg_unix(self.fd.clone(), buf).unwrap(); + op.wait().await + } + + /// Sends data on the socket to the remote address to which it is connected. + pub async fn send(&self, buf: T) -> crate::BufResult { + let op = Op::send_msg_unix(self.fd.clone(), buf, None).unwrap(); + op.wait().await + } + + /// Receives a single datagram message on the socket from the remote address to + /// which it is connected. On success, returns the number of bytes read. + pub async fn recv(&self, buf: T) -> crate::BufResult { + let op = Op::recv(self.fd.clone(), buf).unwrap(); + op.read().await + } +} + +impl AsRawFd for UnixSeqpacket { + #[inline] + fn as_raw_fd(&self) -> RawFd { + self.fd.raw_fd() + } +} + +impl std::fmt::Debug for UnixSeqpacket { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("UnixSeqpacket") + .field("fd", &self.fd) + .finish() + } +} diff --git a/vendor/monoio/src/net/unix/socket_addr.rs b/vendor/monoio/src/net/unix/socket_addr.rs new file mode 100644 index 000000000..38db6bb3e --- /dev/null +++ b/vendor/monoio/src/net/unix/socket_addr.rs @@ -0,0 +1,247 @@ +//! SocketAddr for UDS. +//! Forked from mio. + +use std::{ + ascii, + cmp::Ordering, + ffi::OsStr, + fmt, io, mem, + os::unix::prelude::{FromRawFd, OsStrExt, RawFd}, + path::Path, +}; + +use super::path_offset; + +/// Unix SocketAddr. +/// There is no way to create a [`net::SocketAddr`] so we forked it from mio. +#[derive(Clone)] +pub struct SocketAddr { + sockaddr: libc::sockaddr_un, + socklen: libc::socklen_t, +} + +struct AsciiEscaped<'a>(&'a [u8]); + +enum AddressKind<'a> { + Unnamed, + Pathname(&'a Path), + Abstract(&'a [u8]), +} + +impl SocketAddr { + fn address(&self) -> AddressKind<'_> { + let offset = path_offset(&self.sockaddr); + // Don't underflow in `len` below. + if (self.socklen as usize) < offset { + return AddressKind::Unnamed; + } + let len = self.socklen as usize - offset; + let path = unsafe { &*(&self.sockaddr.sun_path as *const [libc::c_char] as *const [u8]) }; + + // macOS seems to return a len of 16 and a zeroed sun_path for unnamed addresses + if len == 0 || (cfg!(not(any(target_os = "linux", target_os = "android"))) && path[0] == 0) + { + AddressKind::Unnamed + } else if self.sockaddr.sun_path[0] == 0 { + AddressKind::Abstract(&path[1..len]) + } else { + AddressKind::Pathname(OsStr::from_bytes(&path[..len - 1]).as_ref()) + } + } + + #[allow(unused)] + pub(crate) fn new(f: F) -> io::Result + where + F: FnOnce(*mut libc::sockaddr, &mut libc::socklen_t) -> io::Result, + { + let mut sockaddr = { + let sockaddr = mem::MaybeUninit::::zeroed(); + unsafe { sockaddr.assume_init() } + }; + + let raw_sockaddr = &mut sockaddr as *mut libc::sockaddr_un as *mut libc::sockaddr; + let mut socklen = mem::size_of_val(&sockaddr) as libc::socklen_t; + + f(raw_sockaddr, &mut socklen)?; + Ok(SocketAddr::from_parts(sockaddr, socklen)) + } + + pub(crate) fn from_parts( + sockaddr: libc::sockaddr_un, + mut socklen: libc::socklen_t, + ) -> SocketAddr { + fn sun_path_offset(addr: &libc::sockaddr_un) -> usize { + let base: usize = (addr as *const libc::sockaddr_un).cast::<()>() as usize; + let path: usize = (&addr.sun_path as *const libc::c_char).cast::<()>() as usize; + path - base + } + + if socklen == 0 { + // When there is a datagram from unnamed unix socket + // linux returns zero bytes of address + socklen = sun_path_offset(&sockaddr) as libc::socklen_t; // i.e., zero-length address + } else if sockaddr.sun_family != libc::AF_UNIX as libc::sa_family_t { + panic!("file descriptor did not correspond to a Unix socket"); + } + + SocketAddr { sockaddr, socklen } + } + + pub(crate) fn into_parts(self) -> (libc::sockaddr_un, libc::socklen_t) { + (self.sockaddr, self.socklen) + } + + /// Returns `true` if the address is unnamed. + /// + /// Documentation reflected in [`SocketAddr`] + /// + /// [`SocketAddr`]: std::os::unix::net::SocketAddr + #[inline] + pub fn is_unnamed(&self) -> bool { + matches!(self.address(), AddressKind::Unnamed) + } + + /// Returns the contents of this address if it is a `pathname` address. + /// + /// Documentation reflected in [`SocketAddr`] + /// + /// [`SocketAddr`]: std::os::unix::net::SocketAddr + #[inline] + pub fn as_pathname(&self) -> Option<&Path> { + if let AddressKind::Pathname(path) = self.address() { + Some(path) + } else { + None + } + } + + /// Returns the contents of this address if it is an abstract namespace + /// without the leading null byte. + // Link to std::os::unix::net::SocketAddr pending + // https://github.com/rust-lang/rust/issues/85410. + #[inline] + pub fn as_abstract_namespace(&self) -> Option<&[u8]> { + if let AddressKind::Abstract(path) = self.address() { + Some(path) + } else { + None + } + } + + #[inline] + pub(crate) fn as_ptr(&self) -> *const libc::sockaddr_un { + &self.sockaddr as *const _ + } + + #[inline] + pub(crate) fn len(&self) -> libc::socklen_t { + self.socklen + } +} + +impl fmt::Debug for SocketAddr { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + match self.address() { + AddressKind::Unnamed => write!(fmt, "(unnamed)"), + AddressKind::Abstract(name) => write!(fmt, "{} (abstract)", AsciiEscaped(name)), + AddressKind::Pathname(path) => write!(fmt, "{path:?} (pathname)"), + } + } +} + +impl<'a> fmt::Display for AsciiEscaped<'a> { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(fmt, "\"")?; + for byte in self.0.iter().cloned().flat_map(ascii::escape_default) { + write!(fmt, "{}", byte as char)?; + } + write!(fmt, "\"") + } +} + +pub(crate) fn socket_addr(path: &Path) -> io::Result<(libc::sockaddr_un, libc::socklen_t)> { + let sockaddr = mem::MaybeUninit::::zeroed(); + + // This is safe to assume because a `libc::sockaddr_un` filled with `0` + // bytes is properly initialized. + // + // `0` is a valid value for `sockaddr_un::sun_family`; it is + // `libc::AF_UNSPEC`. + // + // `[0; 108]` is a valid value for `sockaddr_un::sun_path`; it begins an + // abstract path. + let mut sockaddr = unsafe { sockaddr.assume_init() }; + + sockaddr.sun_family = libc::AF_UNIX as libc::sa_family_t; + + let bytes = path.as_os_str().as_bytes(); + match (bytes.first(), bytes.len().cmp(&sockaddr.sun_path.len())) { + // Abstract paths don't need a null terminator + (Some(&0), Ordering::Greater) => { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "path must be no longer than libc::sockaddr_un.sun_path", + )); + } + (_, Ordering::Greater) | (_, Ordering::Equal) => { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "path must be shorter than libc::sockaddr_un.sun_path", + )); + } + _ => {} + } + + for (dst, src) in sockaddr.sun_path.iter_mut().zip(bytes.iter()) { + *dst = *src as libc::c_char; + } + + let offset = path_offset(&sockaddr); + let mut socklen = offset + bytes.len(); + + match bytes.first() { + // The struct has already been zeroes so the null byte for pathname + // addresses is already there. + Some(&0) | None => {} + Some(_) => socklen += 1, + } + + Ok((sockaddr, socklen as libc::socklen_t)) +} + +pub(crate) fn pair(flags: libc::c_int) -> io::Result<(T, T)> +where + T: FromRawFd, +{ + #[cfg(any( + target_os = "android", + target_os = "dragonfly", + target_os = "freebsd", + target_os = "illumos", + target_os = "netbsd", + target_os = "openbsd" + ))] + let flags = flags | libc::SOCK_NONBLOCK | libc::SOCK_CLOEXEC; + + #[cfg(target_os = "linux")] + let flags = { + if crate::driver::op::is_legacy() { + flags | libc::SOCK_CLOEXEC | libc::SOCK_NONBLOCK + } else { + flags | libc::SOCK_CLOEXEC + } + }; + + let mut fds = [-1; 2]; + crate::syscall!(socketpair(libc::AF_UNIX, flags, 0, fds.as_mut_ptr()))?; + let pair = unsafe { (T::from_raw_fd(fds[0]), T::from_raw_fd(fds[1])) }; + Ok(pair) +} + +pub(crate) fn local_addr(socket: RawFd) -> io::Result { + SocketAddr::new(|sockaddr, socklen| crate::syscall!(getsockname(socket, sockaddr, socklen))) +} + +pub(crate) fn peer_addr(socket: RawFd) -> io::Result { + SocketAddr::new(|sockaddr, socklen| crate::syscall!(getpeername(socket, sockaddr, socklen))) +} diff --git a/vendor/monoio/src/net/unix/split.rs b/vendor/monoio/src/net/unix/split.rs new file mode 100644 index 000000000..a5503aa13 --- /dev/null +++ b/vendor/monoio/src/net/unix/split.rs @@ -0,0 +1,45 @@ +use std::io; + +use super::{SocketAddr, UnixStream}; +use crate::io::{ + as_fd::{AsReadFd, AsWriteFd, SharedFdWrapper}, + OwnedReadHalf, OwnedWriteHalf, +}; + +/// OwnedReadHalf. +pub type UnixOwnedReadHalf = OwnedReadHalf; + +/// OwnedWriteHalf. +pub type UnixOwnedWriteHalf = OwnedWriteHalf; + +impl UnixOwnedReadHalf { + /// Returns the remote address that this stream is connected to. + #[inline] + pub fn peer_addr(&self) -> io::Result { + let raw_stream = unsafe { &mut *self.0.get() }; + raw_stream.peer_addr() + } + + /// Returns the local address that this stream is bound to. + #[inline] + pub fn local_addr(&self) -> io::Result { + let raw_stream = unsafe { &mut *self.0.get() }; + raw_stream.local_addr() + } +} + +impl AsReadFd for UnixOwnedReadHalf { + #[inline] + fn as_reader_fd(&mut self) -> &SharedFdWrapper { + let raw_stream = unsafe { &mut *self.0.get() }; + raw_stream.as_reader_fd() + } +} + +impl AsWriteFd for UnixOwnedWriteHalf { + #[inline] + fn as_writer_fd(&mut self) -> &SharedFdWrapper { + let raw_stream = unsafe { &mut *self.0.get() }; + raw_stream.as_writer_fd() + } +} diff --git a/vendor/monoio/src/net/unix/stream.rs b/vendor/monoio/src/net/unix/stream.rs new file mode 100644 index 000000000..74f07ef29 --- /dev/null +++ b/vendor/monoio/src/net/unix/stream.rs @@ -0,0 +1,367 @@ +use std::{ + future::Future, + io::{self}, + os::unix::prelude::{AsRawFd, FromRawFd, IntoRawFd, RawFd}, + path::Path, +}; + +use super::{ + socket_addr::{local_addr, pair, peer_addr, socket_addr, SocketAddr}, + ucred::UCred, +}; +use crate::{ + buf::{IoBuf, IoBufMut, IoVecBuf, IoVecBufMut}, + driver::{op::Op, shared_fd::SharedFd}, + io::{ + as_fd::{AsReadFd, AsWriteFd, SharedFdWrapper}, + operation_canceled, AsyncReadRent, AsyncWriteRent, CancelHandle, CancelableAsyncReadRent, + CancelableAsyncWriteRent, Split, + }, + net::new_socket, + BufResult, +}; + +/// UnixStream +pub struct UnixStream { + pub(super) fd: SharedFd, +} + +/// UnixStream is safe to split to two parts +unsafe impl Split for UnixStream {} + +impl UnixStream { + pub(crate) fn from_shared_fd(fd: SharedFd) -> Self { + Self { fd } + } + + /// Connect UnixStream to a path. + pub async fn connect>(path: P) -> io::Result { + let (addr, addr_len) = socket_addr(path.as_ref())?; + Self::inner_connect(addr, addr_len).await + } + + /// Connects the socket to an address. + pub async fn connect_addr(addr: SocketAddr) -> io::Result { + let (addr, addr_len) = addr.into_parts(); + Self::inner_connect(addr, addr_len).await + } + + #[inline(always)] + async fn inner_connect( + sockaddr: libc::sockaddr_un, + socklen: libc::socklen_t, + ) -> io::Result { + let socket = new_socket(libc::AF_UNIX, libc::SOCK_STREAM)?; + let op = Op::connect_unix(SharedFd::new::(socket)?, sockaddr, socklen)?; + let completion = op.await; + completion.meta.result?; + + let stream = Self::from_shared_fd(completion.data.fd); + if crate::driver::op::is_legacy() { + stream.writable(true).await?; + } + // getsockopt + let sys_socket = unsafe { std::os::unix::net::UnixStream::from_raw_fd(stream.fd.raw_fd()) }; + let err = sys_socket.take_error(); + let _ = sys_socket.into_raw_fd(); + if let Some(e) = err? { + return Err(e); + } + Ok(stream) + } + + /// Creates an unnamed pair of connected sockets. + /// + /// Returns two `UnixStream`s which are connected to each other. + pub fn pair() -> io::Result<(Self, Self)> { + let (a, b) = pair(libc::SOCK_STREAM)?; + Ok((Self::from_std(a)?, Self::from_std(b)?)) + } + + /// Returns effective credentials of the process which called `connect` or + /// `pair`. + pub fn peer_cred(&self) -> io::Result { + super::ucred::get_peer_cred(self) + } + + /// Creates new `UnixStream` from a `std::os::unix::net::UnixStream`. + pub fn from_std(stream: std::os::unix::net::UnixStream) -> io::Result { + match SharedFd::new::(stream.as_raw_fd()) { + Ok(shared) => { + let _ = stream.into_raw_fd(); + Ok(Self::from_shared_fd(shared)) + } + Err(e) => Err(e), + } + } + + /// Returns the socket address of the local half of this connection. + pub fn local_addr(&self) -> io::Result { + local_addr(self.as_raw_fd()) + } + + /// Returns the socket address of the remote half of this connection. + pub fn peer_addr(&self) -> io::Result { + peer_addr(self.as_raw_fd()) + } + + /// Wait for read readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn readable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_read(&self.fd, relaxed).unwrap(); + op.wait().await + } + + /// Wait for write readiness. + /// Note: Do not use it before every io. It is different from other runtimes! + /// + /// Everytime call to this method may pay a syscall cost. + /// In uring impl, it will push a PollAdd op; in epoll impl, it will use use + /// inner readiness state; if !relaxed, it will call syscall poll after that. + /// + /// If relaxed, on legacy driver it may return false positive result. + /// If you want to do io by your own, you must maintain io readiness and wait + /// for io ready with relaxed=false. + pub async fn writable(&self, relaxed: bool) -> io::Result<()> { + let op = Op::poll_write(&self.fd, relaxed).unwrap(); + op.wait().await + } +} + +impl AsReadFd for UnixStream { + #[inline] + fn as_reader_fd(&mut self) -> &SharedFdWrapper { + SharedFdWrapper::new(&self.fd) + } +} + +impl AsWriteFd for UnixStream { + #[inline] + fn as_writer_fd(&mut self) -> &SharedFdWrapper { + SharedFdWrapper::new(&self.fd) + } +} + +impl IntoRawFd for UnixStream { + #[inline] + fn into_raw_fd(self) -> RawFd { + self.fd + .try_unwrap() + .expect("unexpected multiple reference to rawfd") + } +} + +impl AsRawFd for UnixStream { + #[inline] + fn as_raw_fd(&self) -> RawFd { + self.fd.raw_fd() + } +} + +impl std::fmt::Debug for UnixStream { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("UnixStream").field("fd", &self.fd).finish() + } +} + +impl AsyncWriteRent for UnixStream { + #[inline] + fn write(&mut self, buf: T) -> impl Future> { + // Submit the write operation + let op = Op::send(self.fd.clone(), buf).unwrap(); + op.write() + } + + #[inline] + fn writev(&mut self, buf_vec: T) -> impl Future> { + let op = Op::writev(&self.fd, buf_vec).unwrap(); + op.write() + } + + #[inline] + async fn flush(&mut self) -> std::io::Result<()> { + // Unix stream does not need flush. + Ok(()) + } + + fn shutdown(&mut self) -> impl Future> { + // We could use shutdown op here, which requires kernel 5.11+. + // However, for simplicity, we just close the socket using direct syscall. + let fd = self.as_raw_fd(); + async move { + match unsafe { libc::shutdown(fd, libc::SHUT_WR) } { + -1 => Err(io::Error::last_os_error()), + _ => Ok(()), + } + } + } +} + +impl CancelableAsyncWriteRent for UnixStream { + #[inline] + async fn cancelable_write( + &mut self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult { + let fd = self.fd.clone(); + + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::send(fd, buf).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.write().await + } + + #[inline] + async fn cancelable_writev( + &mut self, + buf_vec: T, + c: CancelHandle, + ) -> crate::BufResult { + let fd = self.fd.clone(); + + if c.canceled() { + return (Err(operation_canceled()), buf_vec); + } + + let op = Op::writev(&fd, buf_vec).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.write().await + } + + #[inline] + async fn cancelable_flush(&mut self, _c: CancelHandle) -> io::Result<()> { + // Unix stream does not need flush. + Ok(()) + } + + async fn cancelable_shutdown(&mut self, _c: CancelHandle) -> io::Result<()> { + // We could use shutdown op here, which requires kernel 5.11+. + // However, for simplicity, we just close the socket using direct syscall. + let fd = self.as_raw_fd(); + match unsafe { libc::shutdown(fd, libc::SHUT_WR) } { + -1 => Err(io::Error::last_os_error()), + _ => Ok(()), + } + } +} + +impl AsyncReadRent for UnixStream { + #[inline] + fn read(&mut self, buf: T) -> impl Future> { + // Submit the read operation + let op = Op::recv(self.fd.clone(), buf).unwrap(); + op.read() + } + + #[inline] + fn readv(&mut self, buf: T) -> impl Future> { + // Submit the read operation + let op = Op::readv(self.fd.clone(), buf).unwrap(); + op.read() + } +} + +impl CancelableAsyncReadRent for UnixStream { + #[inline] + async fn cancelable_read( + &mut self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult { + let fd = self.fd.clone(); + + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::recv(fd, buf).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.read().await + } + + #[inline] + async fn cancelable_readv( + &mut self, + buf: T, + c: CancelHandle, + ) -> crate::BufResult { + let fd = self.fd.clone(); + + if c.canceled() { + return (Err(operation_canceled()), buf); + } + + let op = Op::readv(fd, buf).unwrap(); + let _guard = c.associate_op(op.op_canceller()); + op.read().await + } +} + +#[cfg(all(unix, feature = "legacy", feature = "tokio-compat"))] +impl tokio::io::AsyncRead for UnixStream { + fn poll_read( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + buf: &mut tokio::io::ReadBuf<'_>, + ) -> std::task::Poll> { + unsafe { + let slice = buf.unfilled_mut(); + let raw_buf = crate::buf::RawBuf::new(slice.as_ptr() as *const u8, slice.len()); + let mut recv = Op::recv_raw(&self.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_legacy(&mut recv, cx)); + + std::task::Poll::Ready(ret.result.map(|n| { + buf.assume_init(n as usize); + buf.advance(n as usize); + })) + } + } +} + +#[cfg(all(unix, feature = "legacy", feature = "tokio-compat"))] +impl tokio::io::AsyncWrite for UnixStream { + fn poll_write( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + buf: &[u8], + ) -> std::task::Poll> { + unsafe { + let raw_buf = crate::buf::RawBuf::new(buf.as_ptr(), buf.len()); + let mut send = Op::send_raw(&self.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_legacy(&mut send, cx)); + + std::task::Poll::Ready(ret.result.map(|n| n as usize)) + } + } + + fn poll_flush( + self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + std::task::Poll::Ready(Ok(())) + } + + fn poll_shutdown( + self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + let fd = self.as_raw_fd(); + let res = match unsafe { libc::shutdown(fd, libc::SHUT_WR) } { + -1 => Err(io::Error::last_os_error()), + _ => Ok(()), + }; + std::task::Poll::Ready(res) + } +} diff --git a/vendor/monoio/src/net/unix/stream_poll.rs b/vendor/monoio/src/net/unix/stream_poll.rs new file mode 100644 index 000000000..f3516dd5f --- /dev/null +++ b/vendor/monoio/src/net/unix/stream_poll.rs @@ -0,0 +1,146 @@ +//! This module provide a poll-io style interface for UnixStream. + +use std::{io, os::fd::AsRawFd}; + +use super::{SocketAddr, UnixStream}; +use crate::driver::op::Op; + +/// A UnixStream with poll-io style interface. +/// Using this struct, you can use UnixStream in a poll-like way. +/// Underlying, it is based on a uring-based epoll. +#[derive(Debug)] +pub struct UnixStreamPoll(UnixStream); + +impl crate::io::IntoPollIo for UnixStream { + type PollIo = UnixStreamPoll; + + #[inline] + fn try_into_poll_io(self) -> Result { + self.try_into_poll_io() + } +} + +impl UnixStream { + /// Convert to poll-io style UnixStreamPoll + #[inline] + pub fn try_into_poll_io(mut self) -> Result { + match self.fd.cvt_poll() { + Ok(_) => Ok(UnixStreamPoll(self)), + Err(e) => Err((e, self)), + } + } +} + +impl crate::io::IntoCompIo for UnixStreamPoll { + type CompIo = UnixStream; + + #[inline] + fn try_into_comp_io(self) -> Result { + self.try_into_comp_io() + } +} + +impl UnixStreamPoll { + /// Convert to normal UnixStream + #[inline] + pub fn try_into_comp_io(mut self) -> Result { + match self.0.fd.cvt_comp() { + Ok(_) => Ok(self.0), + Err(e) => Err((e, self)), + } + } +} + +impl tokio::io::AsyncRead for UnixStreamPoll { + #[inline] + fn poll_read( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + buf: &mut tokio::io::ReadBuf<'_>, + ) -> std::task::Poll> { + unsafe { + let slice = buf.unfilled_mut(); + let raw_buf = crate::buf::RawBuf::new(slice.as_ptr() as *const u8, slice.len()); + let mut recv = Op::recv_raw(&self.0.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_io(&mut recv, cx)); + + std::task::Poll::Ready(ret.result.map(|n| { + buf.assume_init(n as usize); + buf.advance(n as usize); + })) + } + } +} + +impl tokio::io::AsyncWrite for UnixStreamPoll { + #[inline] + fn poll_write( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + buf: &[u8], + ) -> std::task::Poll> { + unsafe { + let raw_buf = crate::buf::RawBuf::new(buf.as_ptr(), buf.len()); + let mut send = Op::send_raw(&self.0.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_io(&mut send, cx)); + + std::task::Poll::Ready(ret.result.map(|n| n as usize)) + } + } + + #[inline] + fn poll_flush( + self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + std::task::Poll::Ready(Ok(())) + } + + #[inline] + fn poll_shutdown( + self: std::pin::Pin<&mut Self>, + _cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + let fd = self.0.as_raw_fd(); + let res = match unsafe { libc::shutdown(fd, libc::SHUT_WR) } { + -1 => Err(io::Error::last_os_error()), + _ => Ok(()), + }; + std::task::Poll::Ready(res) + } + + #[inline] + fn poll_write_vectored( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + bufs: &[std::io::IoSlice<'_>], + ) -> std::task::Poll> { + unsafe { + let raw_buf = + crate::buf::RawBufVectored::new(bufs.as_ptr() as *const libc::iovec, bufs.len()); + let mut writev = Op::writev_raw(&self.0.fd, raw_buf); + let ret = ready!(crate::driver::op::PollLegacy::poll_io(&mut writev, cx)); + + std::task::Poll::Ready(ret.result.map(|n| n as usize)) + } + } + + #[inline] + fn is_write_vectored(&self) -> bool { + true + } +} + +impl UnixStreamPoll { + /// Returns the socket address of the local half of this connection. + #[inline] + pub fn local_addr(&self) -> io::Result { + self.0.local_addr() + } + + /// Returns the socket address of the remote half of this connection. + #[inline] + pub fn peer_addr(&self) -> io::Result { + self.0.peer_addr() + } +} diff --git a/vendor/monoio/src/net/unix/ucred.rs b/vendor/monoio/src/net/unix/ucred.rs new file mode 100644 index 000000000..d7112f065 --- /dev/null +++ b/vendor/monoio/src/net/unix/ucred.rs @@ -0,0 +1,139 @@ +// Forked from tokio. +use libc::{gid_t, pid_t, uid_t}; + +/// Credentials of a process +#[derive(Copy, Clone, Eq, PartialEq, Hash, Debug)] +pub struct UCred { + /// PID (process ID) of the process + pid: Option, + /// UID (user ID) of the process + uid: uid_t, + /// GID (group ID) of the process + gid: gid_t, +} + +impl UCred { + /// Gets UID (user ID) of the process. + #[inline] + pub fn uid(&self) -> uid_t { + self.uid + } + + /// Gets GID (group ID) of the process. + #[inline] + pub fn gid(&self) -> gid_t { + self.gid + } + + /// Gets PID (process ID) of the process. + #[inline] + pub fn pid(&self) -> Option { + self.pid + } +} + +#[cfg(any(target_os = "linux", target_os = "android", target_os = "openbsd"))] +pub(crate) use self::impl_linux::get_peer_cred; +#[cfg(any(target_os = "macos", target_os = "ios"))] +pub(crate) use self::impl_macos::get_peer_cred; + +#[cfg(any(target_os = "macos", target_os = "ios"))] +pub(crate) mod impl_macos { + use std::{ + io, + mem::{size_of, MaybeUninit}, + os::unix::io::AsRawFd, + }; + + use libc::{c_void, getpeereid, getsockopt, pid_t, LOCAL_PEEREPID, SOL_LOCAL}; + + use crate::net::unix::UnixStream; + + pub(crate) fn get_peer_cred(sock: &UnixStream) -> io::Result { + unsafe { + let raw_fd = sock.as_raw_fd(); + + let mut uid = MaybeUninit::uninit(); + let mut gid = MaybeUninit::uninit(); + let mut pid: MaybeUninit = MaybeUninit::uninit(); + let mut pid_size: MaybeUninit = MaybeUninit::new(size_of::() as u32); + + if getsockopt( + raw_fd, + SOL_LOCAL, + LOCAL_PEEREPID, + pid.as_mut_ptr() as *mut c_void, + pid_size.as_mut_ptr(), + ) != 0 + { + return Err(io::Error::last_os_error()); + } + + assert!(pid_size.assume_init() == (size_of::() as u32)); + + let ret = getpeereid(raw_fd, uid.as_mut_ptr(), gid.as_mut_ptr()); + + if ret == 0 { + Ok(super::UCred { + uid: uid.assume_init(), + gid: gid.assume_init(), + pid: Some(pid.assume_init()), + }) + } else { + Err(io::Error::last_os_error()) + } + } + } +} + +#[cfg(any(target_os = "linux", target_os = "android", target_os = "openbsd"))] +pub(crate) mod impl_linux { + use std::{io, mem}; + + #[cfg(target_os = "openbsd")] + use libc::sockpeercred as ucred; + #[cfg(any(target_os = "linux", target_os = "android"))] + use libc::ucred; + use libc::{c_void, getsockopt, socklen_t, SOL_SOCKET, SO_PEERCRED}; + + use crate::net::unix::UnixStream; + + pub(crate) fn get_peer_cred(sock: &UnixStream) -> io::Result { + use std::os::unix::io::AsRawFd; + + unsafe { + let raw_fd = sock.as_raw_fd(); + + let mut ucred = ucred { + pid: 0, + uid: 0, + gid: 0, + }; + + let ucred_size = mem::size_of::(); + + // These paranoid checks should be optimized-out + assert!(mem::size_of::() <= mem::size_of::()); + assert!(ucred_size <= u32::MAX as usize); + + let mut ucred_size = ucred_size as socklen_t; + + let ret = getsockopt( + raw_fd, + SOL_SOCKET, + SO_PEERCRED, + &mut ucred as *mut ucred as *mut c_void, + &mut ucred_size, + ); + if ret == 0 && ucred_size as usize == mem::size_of::() { + Ok(super::UCred { + uid: ucred.uid, + gid: ucred.gid, + pid: Some(ucred.pid), + }) + } else { + Err(io::Error::last_os_error()) + } + } + } +} diff --git a/vendor/monoio/src/runtime.rs b/vendor/monoio/src/runtime.rs new file mode 100644 index 000000000..e4dc90af1 --- /dev/null +++ b/vendor/monoio/src/runtime.rs @@ -0,0 +1,449 @@ +use std::future::Future; + +#[cfg(any(all(target_os = "linux", feature = "iouring"), feature = "legacy"))] +use crate::time::TimeDriver; +#[cfg(all(target_os = "linux", feature = "iouring"))] +use crate::IoUringDriver; +#[cfg(feature = "legacy")] +use crate::LegacyDriver; +use crate::{ + driver::Driver, + scheduler::{LocalScheduler, TaskQueue}, + task::{ + new_task, + waker_fn::{dummy_waker, set_poll, should_poll}, + JoinHandle, + }, + time::driver::Handle as TimeHandle, +}; + +#[cfg(feature = "sync")] +thread_local! { + pub(crate) static DEFAULT_CTX: Context = Context { + thread_id: crate::utils::thread_id::DEFAULT_THREAD_ID, + unpark_cache: std::cell::RefCell::new(fxhash::FxHashMap::default()), + waker_sender_cache: std::cell::RefCell::new(fxhash::FxHashMap::default()), + tasks: Default::default(), + time_handle: None, + blocking_handle: crate::blocking::BlockingHandle::Empty(crate::blocking::BlockingStrategy::Panic), + }; +} + +scoped_thread_local!(pub(crate) static CURRENT: Context); + +pub(crate) struct Context { + /// Owned task set and local run queue + pub(crate) tasks: TaskQueue, + + /// Thread id(not the kernel thread id but a generated unique number) + pub(crate) thread_id: usize, + + /// Thread unpark handles + #[cfg(feature = "sync")] + pub(crate) unpark_cache: + std::cell::RefCell>, + + /// Waker sender cache + #[cfg(feature = "sync")] + pub(crate) waker_sender_cache: + std::cell::RefCell>>, + + /// Time Handle + pub(crate) time_handle: Option, + + /// Blocking Handle + #[cfg(feature = "sync")] + pub(crate) blocking_handle: crate::blocking::BlockingHandle, +} + +impl Context { + #[cfg(feature = "sync")] + pub(crate) fn new(blocking_handle: crate::blocking::BlockingHandle) -> Self { + let thread_id = crate::builder::BUILD_THREAD_ID.with(|id| *id); + + Self { + thread_id, + unpark_cache: std::cell::RefCell::new(fxhash::FxHashMap::default()), + waker_sender_cache: std::cell::RefCell::new(fxhash::FxHashMap::default()), + tasks: TaskQueue::default(), + time_handle: None, + blocking_handle, + } + } + + #[cfg(not(feature = "sync"))] + pub(crate) fn new() -> Self { + let thread_id = crate::builder::BUILD_THREAD_ID.with(|id| *id); + + Self { + thread_id, + tasks: TaskQueue::default(), + time_handle: None, + } + } + + #[allow(unused)] + #[cfg(feature = "sync")] + pub(crate) fn unpark_thread(&self, id: usize) { + use crate::driver::{thread::get_unpark_handle, unpark::Unpark}; + if let Some(handle) = self.unpark_cache.borrow().get(&id) { + handle.unpark(); + return; + } + + if let Some(v) = get_unpark_handle(id) { + // Write back to local cache + let w = v.clone(); + self.unpark_cache.borrow_mut().insert(id, w); + v.unpark(); + } + } + + #[allow(unused)] + #[cfg(feature = "sync")] + pub(crate) fn send_waker(&self, id: usize, w: std::task::Waker) { + use crate::driver::thread::get_waker_sender; + if let Some(sender) = self.waker_sender_cache.borrow().get(&id) { + let _ = sender.send(w); + return; + } + + if let Some(s) = get_waker_sender(id) { + // Write back to local cache + let _ = s.send(w); + self.waker_sender_cache.borrow_mut().insert(id, s); + } + } +} + +/// Monoio runtime +pub struct Runtime { + pub(crate) context: Context, + pub(crate) driver: D, +} + +impl Runtime { + pub(crate) fn new(context: Context, driver: D) -> Self { + Self { context, driver } + } + + /// Block on + pub fn block_on(&mut self, future: F) -> F::Output + where + F: Future, + D: Driver, + { + assert!( + !CURRENT.is_set(), + "Can not start a runtime inside a runtime" + ); + + let waker = dummy_waker(); + let cx = &mut std::task::Context::from_waker(&waker); + + self.driver.with(|| { + CURRENT.set(&self.context, || { + #[cfg(feature = "sync")] + let join = unsafe { spawn_without_static(future) }; + #[cfg(not(feature = "sync"))] + let join = future; + + let mut join = std::pin::pin!(join); + set_poll(); + loop { + loop { + // Consume all tasks(with max round to prevent io starvation) + let mut max_round = self.context.tasks.len() * 2; + while let Some(t) = self.context.tasks.pop() { + t.run(); + if max_round == 0 { + // maybe there's a looping task + break; + } else { + max_round -= 1; + } + } + + // Check main future + while should_poll() { + // check if ready + if let std::task::Poll::Ready(t) = join.as_mut().poll(cx) { + return t; + } + } + + if self.context.tasks.is_empty() { + // No task to execute, we should wait for io blockingly + // Hot path + break; + } + + // Cold path + let _ = self.driver.submit(); + } + + // Wait and Process CQ(the error is ignored for not debug mode) + #[cfg(not(all(debug_assertions, feature = "debug")))] + let _ = self.driver.park(); + + #[cfg(all(debug_assertions, feature = "debug"))] + if let Err(e) = self.driver.park() { + trace!("park error: {:?}", e); + } + } + }) + }) + } +} + +/// Fusion Runtime is a wrapper of io_uring driver or legacy driver based +/// runtime. +#[cfg(feature = "legacy")] +pub enum FusionRuntime<#[cfg(all(target_os = "linux", feature = "iouring"))] L, R> { + /// Uring driver based runtime. + #[cfg(all(target_os = "linux", feature = "iouring"))] + Uring(Runtime), + /// Legacy driver based runtime. + Legacy(Runtime), +} + +/// Fusion Runtime is a wrapper of io_uring driver or legacy driver based +/// runtime. +#[cfg(all(target_os = "linux", feature = "iouring", not(feature = "legacy")))] +pub enum FusionRuntime { + /// Uring driver based runtime. + Uring(Runtime), +} + +#[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] +impl FusionRuntime +where + L: Driver, + R: Driver, +{ + /// Block on + pub fn block_on(&mut self, future: F) -> F::Output + where + F: Future, + { + match self { + FusionRuntime::Uring(inner) => { + info!("Monoio is running with io_uring driver"); + inner.block_on(future) + } + FusionRuntime::Legacy(inner) => { + info!("Monoio is running with legacy driver"); + inner.block_on(future) + } + } + } +} + +#[cfg(all(feature = "legacy", not(all(target_os = "linux", feature = "iouring"))))] +impl FusionRuntime +where + R: Driver, +{ + /// Block on + pub fn block_on(&mut self, future: F) -> F::Output + where + F: Future, + { + match self { + FusionRuntime::Legacy(inner) => inner.block_on(future), + } + } +} + +#[cfg(all(not(feature = "legacy"), all(target_os = "linux", feature = "iouring")))] +impl FusionRuntime +where + R: Driver, +{ + /// Block on + pub fn block_on(&mut self, future: F) -> F::Output + where + F: Future, + { + match self { + FusionRuntime::Uring(inner) => inner.block_on(future), + } + } +} + +// L -> Fusion +#[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] +impl From> for FusionRuntime { + fn from(r: Runtime) -> Self { + Self::Uring(r) + } +} + +// TL -> Fusion +#[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] +impl From>> + for FusionRuntime, TimeDriver> +{ + fn from(r: Runtime>) -> Self { + Self::Uring(r) + } +} + +// R -> Fusion +#[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] +impl From> for FusionRuntime { + fn from(r: Runtime) -> Self { + Self::Legacy(r) + } +} + +// TR -> Fusion +#[cfg(all(target_os = "linux", feature = "iouring", feature = "legacy"))] +impl From>> + for FusionRuntime, TimeDriver> +{ + fn from(r: Runtime>) -> Self { + Self::Legacy(r) + } +} + +// R -> Fusion +#[cfg(all(feature = "legacy", not(all(target_os = "linux", feature = "iouring"))))] +impl From> for FusionRuntime { + fn from(r: Runtime) -> Self { + Self::Legacy(r) + } +} + +// TR -> Fusion +#[cfg(all(feature = "legacy", not(all(target_os = "linux", feature = "iouring"))))] +impl From>> for FusionRuntime> { + fn from(r: Runtime>) -> Self { + Self::Legacy(r) + } +} + +// L -> Fusion +#[cfg(all(target_os = "linux", feature = "iouring", not(feature = "legacy")))] +impl From> for FusionRuntime { + fn from(r: Runtime) -> Self { + Self::Uring(r) + } +} + +// TL -> Fusion +#[cfg(all(target_os = "linux", feature = "iouring", not(feature = "legacy")))] +impl From>> for FusionRuntime> { + fn from(r: Runtime>) -> Self { + Self::Uring(r) + } +} + +/// Spawns a new asynchronous task, returning a [`JoinHandle`] for it. +/// +/// Spawning a task enables the task to execute concurrently to other tasks. +/// There is no guarantee that a spawned task will execute to completion. When a +/// runtime is shutdown, all outstanding tasks are dropped, regardless of the +/// lifecycle of that task. +/// +/// +/// [`JoinHandle`]: super::task::JoinHandle +/// +/// # Examples +/// +/// In this example, a server is started and `spawn` is used to start a new task +/// that processes each received connection. +/// +/// ```no_run +/// #[monoio::main] +/// async fn main() { +/// let handle = monoio::spawn(async { +/// println!("hello from a background task"); +/// }); +/// +/// // Let the task complete +/// handle.await; +/// } +/// ``` +pub fn spawn(future: T) -> JoinHandle +where + T: Future + 'static, + T::Output: 'static, +{ + let (task, join) = new_task( + crate::utils::thread_id::get_current_thread_id(), + future, + LocalScheduler, + ); + + CURRENT.with(|ctx| { + ctx.tasks.push(task); + }); + join +} + +#[cfg(feature = "sync")] +unsafe fn spawn_without_static(future: T) -> JoinHandle +where + T: Future, +{ + use crate::task::new_task_holding; + let (task, join) = new_task_holding( + crate::utils::thread_id::get_current_thread_id(), + future, + LocalScheduler, + ); + + CURRENT.with(|ctx| { + ctx.tasks.push(task); + }); + join +} + +#[cfg(test)] +mod tests { + #[cfg(all(feature = "sync", target_os = "linux", feature = "iouring"))] + #[test] + fn across_thread() { + use futures::channel::oneshot; + + use crate::driver::IoUringDriver; + + let (tx1, rx1) = oneshot::channel::(); + let (tx2, rx2) = oneshot::channel::(); + + std::thread::spawn(move || { + let mut rt = crate::RuntimeBuilder::::new() + .build() + .unwrap(); + rt.block_on(async move { + let n = rx1.await.expect("unable to receive rx1"); + assert!(tx2.send(n).is_ok()); + }); + }); + + let mut rt = crate::RuntimeBuilder::::new() + .build() + .unwrap(); + rt.block_on(async move { + assert!(tx1.send(24).is_ok()); + assert_eq!(rx2.await.expect("unable to receive rx2"), 24); + }); + } + + #[cfg(all(target_os = "linux", feature = "iouring"))] + #[test] + fn timer() { + use crate::driver::IoUringDriver; + let mut rt = crate::RuntimeBuilder::::new() + .enable_timer() + .build() + .unwrap(); + let instant = std::time::Instant::now(); + rt.block_on(async { + crate::time::sleep(std::time::Duration::from_millis(200)).await; + }); + let eps = instant.elapsed().subsec_millis(); + assert!((eps as i32 - 200).abs() < 50); + } +} diff --git a/vendor/monoio/src/scheduler.rs b/vendor/monoio/src/scheduler.rs new file mode 100644 index 000000000..1f0747b2b --- /dev/null +++ b/vendor/monoio/src/scheduler.rs @@ -0,0 +1,74 @@ +use std::{cell::UnsafeCell, collections::VecDeque, marker::PhantomData}; + +use crate::task::{Schedule, Task}; + +pub(crate) struct LocalScheduler; + +impl Schedule for LocalScheduler { + fn schedule(&self, task: Task) { + crate::runtime::CURRENT.with(|cx| cx.tasks.push(task)); + } + + fn yield_now(&self, task: Task) { + crate::runtime::CURRENT.with(|cx| cx.tasks.push_front(task)); + } +} + +pub(crate) struct TaskQueue { + // Local queue. + queue: UnsafeCell>>, + // Make sure the type is `!Send` and `!Sync`. + _marker: PhantomData<*const ()>, +} + +impl Default for TaskQueue { + fn default() -> Self { + Self::new() + } +} + +impl Drop for TaskQueue { + fn drop(&mut self) { + unsafe { + let queue = &mut *self.queue.get(); + while let Some(_task) = queue.pop_front() {} + } + } +} + +impl TaskQueue { + pub(crate) fn new() -> Self { + const DEFAULT_TASK_QUEUE_SIZE: usize = 4096; + Self::new_with_capacity(DEFAULT_TASK_QUEUE_SIZE) + } + pub(crate) fn new_with_capacity(capacity: usize) -> Self { + Self { + queue: UnsafeCell::new(VecDeque::with_capacity(capacity)), + _marker: PhantomData, + } + } + + pub(crate) fn len(&self) -> usize { + unsafe { (*self.queue.get()).len() } + } + + pub(crate) fn is_empty(&self) -> bool { + self.len() == 0 + } + + pub(crate) fn push(&self, runnable: Task) { + unsafe { + (*self.queue.get()).push_back(runnable); + } + } + + pub(crate) fn push_front(&self, runnable: Task) { + unsafe { + (*self.queue.get()).push_front(runnable); + } + } + + pub(crate) fn pop(&self) -> Option> { + unsafe { (*self.queue.get()).pop_front() } + } +} diff --git a/vendor/monoio/src/task/core.rs b/vendor/monoio/src/task/core.rs new file mode 100644 index 000000000..b43633fb2 --- /dev/null +++ b/vendor/monoio/src/task/core.rs @@ -0,0 +1,177 @@ +use std::{ + cell::UnsafeCell, + future::Future, + pin::Pin, + task::{Context, Poll, Waker}, +}; + +use super::{ + raw::{self, Vtable}, + state::State, + utils::UnsafeCellExt, + Schedule, +}; + +#[repr(C)] +pub(crate) struct Cell { + pub(crate) header: Header, + pub(crate) core: Core, + pub(crate) trailer: Trailer, +} + +pub(crate) struct Core { + /// Scheduler used to drive this future + pub(crate) scheduler: S, + /// Either the future or the output + pub(crate) stage: CoreStage, +} +pub(crate) struct CoreStage { + stage: UnsafeCell>, +} + +pub(crate) enum Stage { + Running(T), + Finished(T::Output), + Consumed, +} + +#[repr(C)] +pub(crate) struct Header { + /// State + pub(crate) state: State, + /// Table of function pointers for executing actions on the task. + pub(crate) vtable: &'static Vtable, + /// Thread ID(sync: used for wake task on its thread; sync disabled: do checking) + pub(crate) owner_id: usize, +} + +pub(crate) struct Trailer { + /// Consumer task waiting on completion of this task. + pub(crate) waker: UnsafeCell>, +} + +impl Cell { + /// Allocates a new task cell, containing the header, trailer, and core + /// structures. + pub(crate) fn new(owner_id: usize, future: T, scheduler: S) -> Box> { + Box::new(Cell { + header: Header { + state: State::new(), + vtable: raw::vtable::(), + owner_id, + }, + core: Core { + scheduler, + stage: CoreStage { + stage: UnsafeCell::new(Stage::Running(future)), + }, + }, + trailer: Trailer { + waker: UnsafeCell::new(None), + }, + }) + } +} + +impl CoreStage { + pub(crate) fn with_mut(&self, f: impl FnOnce(*mut Stage) -> R) -> R { + self.stage.with_mut(f) + } + + pub(crate) fn poll(&self, mut cx: Context<'_>) -> Poll { + let res = { + self.with_mut(|ptr| { + // Safety: The caller ensures mutual exclusion to the field. + let future = match unsafe { &mut *ptr } { + Stage::Running(future) => future, + _ => unreachable!("unexpected stage"), + }; + + // Safety: The caller ensures the future is pinned. + let future = unsafe { Pin::new_unchecked(future) }; + + future.poll(&mut cx) + }) + }; + + if res.is_ready() { + self.drop_future_or_output(); + } + + res + } + + /// Drop the future + /// + /// # Safety + /// + /// The caller must ensure it is safe to mutate the `stage` field. + pub(crate) fn drop_future_or_output(&self) { + // Safety: the caller ensures mutual exclusion to the field. + unsafe { + self.set_stage(Stage::Consumed); + } + } + + /// Store the task output + /// + /// # Safety + /// + /// The caller must ensure it is safe to mutate the `stage` field. + pub(crate) fn store_output(&self, output: T::Output) { + // Safety: the caller ensures mutual exclusion to the field. + unsafe { + self.set_stage(Stage::Finished(output)); + } + } + + /// Take the task output + /// + /// # Safety + /// + /// The caller must ensure it is safe to mutate the `stage` field. + pub(crate) fn take_output(&self) -> T::Output { + use std::mem; + + self.with_mut(|ptr| { + // Safety:: the caller ensures mutual exclusion to the field. + match mem::replace(unsafe { &mut *ptr }, Stage::Consumed) { + Stage::Finished(output) => output, + _ => panic!("JoinHandle polled after completion"), + } + }) + } + + unsafe fn set_stage(&self, stage: Stage) { + self.with_mut(|ptr| *ptr = stage) + } +} + +impl Header { + #[allow(unused)] + pub(crate) fn get_owner_id(&self) -> usize { + // safety: If there are concurrent writes, then that write has violated + // the safety requirements on `set_owner_id`. + self.owner_id + } +} + +impl Trailer { + pub(crate) unsafe fn set_waker(&self, waker: Option) { + self.waker.with_mut(|ptr| { + *ptr = waker; + }); + } + + pub(crate) unsafe fn will_wake(&self, waker: &Waker) -> bool { + self.waker + .with(|ptr| (*ptr).as_ref().unwrap().will_wake(waker)) + } + + pub(crate) fn wake_join(&self) { + self.waker.with(|ptr| match unsafe { &*ptr } { + Some(waker) => waker.wake_by_ref(), + None => panic!("waker missing"), + }); + } +} diff --git a/vendor/monoio/src/task/harness.rs b/vendor/monoio/src/task/harness.rs new file mode 100644 index 000000000..db75f386b --- /dev/null +++ b/vendor/monoio/src/task/harness.rs @@ -0,0 +1,450 @@ +use std::{ + future::Future, + panic, + ptr::NonNull, + task::{Context, Poll, Waker}, +}; + +use super::utils::UnsafeCellExt; +use crate::{ + task::{ + core::{Cell, Core, CoreStage, Header, Trailer}, + state::Snapshot, + waker::waker_ref, + Schedule, Task, + }, + utils::thread_id::{try_get_current_thread_id, DEFAULT_THREAD_ID}, +}; + +pub(crate) struct Harness { + cell: NonNull>, +} + +impl Harness +where + T: Future, + S: 'static, +{ + pub(crate) unsafe fn from_raw(ptr: NonNull
) -> Harness { + Harness { + cell: ptr.cast::>(), + } + } + + fn header(&self) -> &Header { + unsafe { &self.cell.as_ref().header } + } + + fn trailer(&self) -> &Trailer { + unsafe { &self.cell.as_ref().trailer } + } + + fn core(&self) -> &Core { + unsafe { &self.cell.as_ref().core } + } +} + +impl Harness +where + T: Future, + S: Schedule, +{ + /// Polls the inner future. + pub(super) fn poll(self) { + trace!("MONOIO DEBUG[Harness]:: poll"); + match self.poll_inner() { + PollFuture::Notified => { + // We should re-schedule the task. + self.header().state.ref_inc(); + self.core().scheduler.yield_now(self.get_new_task()); + } + PollFuture::Complete => { + self.complete(); + } + PollFuture::Done => (), + } + } + + /// Do polland return the status. + /// + /// poll_inner does not take a ref-count. We must make sure the task is + /// alive when call this method + fn poll_inner(&self) -> PollFuture { + // notified -> running + self.header().state.transition_to_running(); + + // poll the future + let waker_ref = waker_ref::(self.header()); + let cx = Context::from_waker(&waker_ref); + let res = poll_future(&self.core().stage, cx); + + if res == Poll::Ready(()) { + return PollFuture::Complete; + } + + use super::state::TransitionToIdle; + match self.header().state.transition_to_idle() { + TransitionToIdle::Ok => PollFuture::Done, + TransitionToIdle::OkNotified => PollFuture::Notified, + } + } + + pub(super) fn dealloc(self) { + trace!("MONOIO DEBUG[Harness]:: dealloc"); + + // Release the join waker, if there is one. + self.trailer().waker.with_mut(drop); + + // Check causality + self.core().stage.with_mut(drop); + + unsafe { + drop(Box::from_raw(self.cell.as_ptr())); + } + } + + #[cfg(feature = "sync")] + pub(super) fn finish(self, val: ::Output) { + trace!("MONOIO DEBUG[Harness]:: finish"); + self.header().state.transition_to_running(); + self.core().stage.store_output(val); + self.complete(); + } + + // ===== join handle ===== + + /// Read the task output into `dst`. + pub(super) fn try_read_output(self, dst: &mut Poll, waker: &Waker) { + trace!("MONOIO DEBUG[Harness]:: try_read_output"); + if can_read_output(self.header(), self.trailer(), waker) { + *dst = Poll::Ready(self.core().stage.take_output()); + } + } + + pub(super) fn drop_join_handle_slow(self) { + trace!("MONOIO DEBUG[Harness]:: drop_join_handle_slow"); + + let mut maybe_panic = None; + + // Try to unset `JOIN_INTEREST`. This must be done as a first step in + // case the task concurrently completed. + if self.header().state.unset_join_interested().is_err() { + // It is our responsibility to drop the output. This is critical as + // the task output may not be `Send` and as such must remain with + // the scheduler or `JoinHandle`. i.e. if the output remains in the + // task structure until the task is deallocated, it may be dropped + // by a Waker on any arbitrary thread. + let panic = panic::catch_unwind(panic::AssertUnwindSafe(|| { + self.core().stage.drop_future_or_output(); + })); + + if let Err(panic) = panic { + maybe_panic = Some(panic); + } + } + + // Drop the `JoinHandle` reference, possibly deallocating the task + self.drop_reference(); + + if let Some(panic) = maybe_panic { + panic::resume_unwind(panic); + } + } + + // ===== waker behavior ===== + + /// This call consumes a ref-count and notifies the task. This will create a + /// new Notified and submit it if necessary. + /// + /// The caller does not need to hold a ref-count besides the one that was + /// passed to this call. + pub(super) fn wake_by_val(self) { + trace!("MONOIO DEBUG[Harness]:: wake_by_val"); + let owner_id = self.header().owner_id; + if is_remote_task(owner_id) { + if self.header().state.transition_to_notified_without_submit() { + self.drop_reference(); + return; + } + // send to target thread + trace!("MONOIO DEBUG[Harness]:: wake_by_val with another thread id"); + #[cfg(feature = "sync")] + { + use crate::task::waker::raw_waker; + let waker = raw_waker::(self.cell.cast::
().as_ptr()); + // # Ref Count: self -> waker + let waker = unsafe { Waker::from_raw(waker) }; + crate::runtime::CURRENT.try_with(|maybe_ctx| match maybe_ctx { + Some(ctx) => { + ctx.send_waker(owner_id, waker); + ctx.unpark_thread(owner_id); + } + None => { + let _ = crate::runtime::DEFAULT_CTX.try_with(|default_ctx| { + crate::runtime::CURRENT.set(default_ctx, || { + crate::runtime::CURRENT.with(|ctx| { + ctx.send_waker(owner_id, waker); + ctx.unpark_thread(owner_id); + }); + }); + }); + } + }); + return; + } + #[cfg(not(feature = "sync"))] + { + panic!("waker can only be sent across threads when `sync` feature enabled"); + } + } + + use super::state::TransitionToNotified; + match self.header().state.transition_to_notified() { + TransitionToNotified::Submit => { + // # Ref Count: self -> task + self.core().scheduler.schedule(self.get_new_task()); + } + TransitionToNotified::DoNothing => { + // # Ref Count: self -> -1 + self.drop_reference(); + } + } + } + + /// This call notifies the task. It will not consume any ref-counts, but the + /// caller should hold a ref-count. This will create a new Notified and + /// submit it if necessary. + pub(super) fn wake_by_ref(&self) { + trace!("MONOIO DEBUG[Harness]:: wake_by_ref"); + let owner_id = self.header().owner_id; + if is_remote_task(owner_id) { + if self.header().state.transition_to_notified_without_submit() { + return; + } + + // send to target thread + trace!("MONOIO DEBUG[Harness]:: wake_by_ref with another thread id"); + #[cfg(feature = "sync")] + { + use crate::task::waker::raw_waker; + let waker = raw_waker::(self.cell.cast::
().as_ptr()); + // We create a new waker so we need to inc ref count. + let waker = unsafe { Waker::from_raw(waker) }; + self.header().state.ref_inc(); + crate::runtime::CURRENT.try_with(|maybe_ctx| match maybe_ctx { + Some(ctx) => { + ctx.send_waker(owner_id, waker); + ctx.unpark_thread(owner_id); + } + None => { + let _ = crate::runtime::DEFAULT_CTX.try_with(|default_ctx| { + crate::runtime::CURRENT.set(default_ctx, || { + crate::runtime::CURRENT.with(|ctx| { + ctx.send_waker(owner_id, waker); + ctx.unpark_thread(owner_id); + }); + }); + }); + } + }); + return; + } + #[cfg(not(feature = "sync"))] + { + panic!("waker can only be sent across threads when `sync` feature enabled"); + } + } + + use super::state::TransitionToNotified; + match self.header().state.transition_to_notified() { + TransitionToNotified::Submit => { + // # Ref Count: +1 -> task + self.header().state.ref_inc(); + self.core().scheduler.schedule(self.get_new_task()); + } + TransitionToNotified::DoNothing => (), + } + } + + pub(super) fn drop_reference(self) { + trace!("MONOIO DEBUG[Harness]:: drop_reference"); + if self.header().state.ref_dec() { + self.dealloc(); + } + } + + // ====== internal ====== + + /// Complete the task. This method assumes that the state is RUNNING. + fn complete(self) { + // The future has completed and its output has been written to the task + // stage. We transition from running to complete. + + let snapshot = self.header().state.transition_to_complete(); + + // We catch panics here in case dropping the future or waking the + // JoinHandle panics. + let _ = panic::catch_unwind(panic::AssertUnwindSafe(|| { + if !snapshot.is_join_interested() { + // The `JoinHandle` is not interested in the output of + // this task. It is our responsibility to drop the + // output. + self.core().stage.drop_future_or_output(); + } else if snapshot.has_join_waker() { + // Notify the join handle. The previous transition obtains the + // lock on the waker cell. + self.trailer().wake_join(); + } + })); + } + + /// Create a new task that holds its own ref-count. + /// + /// # Safety + /// + /// Any use of `self` after this call must ensure that a ref-count to the + /// task holds the task alive until after the use of `self`. Passing the + /// returned Task to any method on `self` is unsound if dropping the Task + /// could drop `self` before the call on `self` returned. + fn get_new_task(&self) -> Task { + // safety: The header is at the beginning of the cell, so this cast is + // safe. + unsafe { Task::from_raw(self.cell.cast()) } + } +} + +fn is_remote_task(owner_id: usize) -> bool { + if owner_id == DEFAULT_THREAD_ID { + return true; + } + match try_get_current_thread_id() { + Some(tid) => owner_id != tid, + None => true, + } +} + +fn can_read_output(header: &Header, trailer: &Trailer, waker: &Waker) -> bool { + // Load a snapshot of the current task state + let snapshot = header.state.load(); + + debug_assert!(snapshot.is_join_interested()); + + if !snapshot.is_complete() { + // The waker must be stored in the task struct. + let res = if snapshot.has_join_waker() { + // There already is a waker stored in the struct. If it matches + // the provided waker, then there is no further work to do. + // Otherwise, the waker must be swapped. + let will_wake = unsafe { + // Safety: when `JOIN_INTEREST` is set, only `JOIN_HANDLE` + // may mutate the `waker` field. + trailer.will_wake(waker) + }; + + if will_wake { + // The task is not complete **and** the waker is up to date, + // there is nothing further that needs to be done. + return false; + } + + // Unset the `JOIN_WAKER` to gain mutable access to the `waker` + // field then update the field with the new join worker. + // + // This requires two atomic operations, unsetting the bit and + // then resetting it. If the task transitions to complete + // concurrently to either one of those operations, then setting + // the join waker fails and we proceed to reading the task + // output. + header + .state + .unset_waker() + .and_then(|snapshot| set_join_waker(header, trailer, waker.clone(), snapshot)) + } else { + set_join_waker(header, trailer, waker.clone(), snapshot) + }; + + match res { + Ok(_) => return false, + Err(snapshot) => { + assert!(snapshot.is_complete()); + } + } + } + true +} + +fn set_join_waker( + header: &Header, + trailer: &Trailer, + waker: Waker, + snapshot: Snapshot, +) -> Result { + assert!(snapshot.is_join_interested()); + assert!(!snapshot.has_join_waker()); + + // Safety: Only the `JoinHandle` may set the `waker` field. When + // `JOIN_INTEREST` is **not** set, nothing else will touch the field. + unsafe { + trailer.set_waker(Some(waker)); + } + + // Update the `JoinWaker` state accordingly + let res = header.state.set_join_waker(); + + // If the state could not be updated, then clear the join waker + if res.is_err() { + unsafe { + trailer.set_waker(None); + } + } + + res +} + +enum PollFuture { + Complete, + Notified, + Done, +} + +/// Poll the future. If the future completes, the output is written to the +/// stage field. +fn poll_future(core: &CoreStage, cx: Context<'_>) -> Poll<()> { + // CHIHAI: For efficiency we do not catch. + + // Poll the future. + // let output = panic::catch_unwind(panic::AssertUnwindSafe(|| { + // struct Guard<'a, T: Future> { + // core: &'a CoreStage, + // } + // impl<'a, T: Future> Drop for Guard<'a, T> { + // fn drop(&mut self) { + // // If the future panics on poll, we drop it inside the panic + // // guard. + // self.core.drop_future_or_output(); + // } + // } + // let guard = Guard { core }; + // let res = guard.core.poll(cx); + // mem::forget(guard); + // res + // })); + let output = core.poll(cx); + + // Prepare output for being placed in the core stage. + let output = match output { + // Ok(Poll::Pending) => return Poll::Pending, + // Ok(Poll::Ready(output)) => Ok(output), + // Err(panic) => Err(JoinError::panic(panic)), + Poll::Pending => return Poll::Pending, + Poll::Ready(output) => output, + }; + + // Catch and ignore panics if the future panics on drop. + // let _ = panic::catch_unwind(panic::AssertUnwindSafe(|| { + // core.store_output(output); + // })); + core.store_output(output); + + Poll::Ready(()) +} diff --git a/vendor/monoio/src/task/join.rs b/vendor/monoio/src/task/join.rs new file mode 100644 index 000000000..4f11add61 --- /dev/null +++ b/vendor/monoio/src/task/join.rs @@ -0,0 +1,70 @@ +use std::{ + future::Future, + marker::PhantomData, + pin::Pin, + task::{Context, Poll}, +}; + +use super::raw::RawTask; + +/// JoinHandle can be used to wait task finished. +/// Note if you drop it directly, task will not be terminated. +pub struct JoinHandle { + raw: RawTask, + _p: PhantomData, +} + +unsafe impl Send for JoinHandle {} +unsafe impl Sync for JoinHandle {} + +impl JoinHandle { + pub(super) fn new(raw: RawTask) -> JoinHandle { + JoinHandle { + raw, + _p: PhantomData, + } + } + + /// Checks if the task associated with this `JoinHandle` has finished. + pub fn is_finished(&self) -> bool { + let state = self.raw.header().state.load(); + state.is_complete() + } +} + +impl Unpin for JoinHandle {} + +impl Future for JoinHandle { + type Output = T; + + fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + let mut ret = Poll::Pending; + + // Try to read the task output. If the task is not yet complete, the + // waker is stored and is notified once the task does complete. + // + // The function must go via the vtable, which requires erasing generic + // types. To do this, the function "return" is placed on the stack + // **before** calling the function and is passed into the function using + // `*mut ()`. + // + // Safety: + // + // The type of `T` must match the task's output type. + unsafe { + self.raw + .try_read_output(&mut ret as *mut _ as *mut (), cx.waker()); + } + ret + } +} + +impl Drop for JoinHandle { + fn drop(&mut self) { + if self.raw.header().state.drop_join_handle_fast().is_ok() { + return; + } + + self.raw.drop_join_handle_slow(); + } +} diff --git a/vendor/monoio/src/task/mod.rs b/vendor/monoio/src/task/mod.rs new file mode 100644 index 000000000..6133347ed --- /dev/null +++ b/vendor/monoio/src/task/mod.rs @@ -0,0 +1,106 @@ +//! Task impl +// Heavily borrowed from tokio. +// Copyright (c) 2021 Tokio Contributors, licensed under the MIT license. + +mod utils; +pub(crate) mod waker_fn; + +mod core; +use self::core::{Cell, Header}; + +mod harness; +use self::harness::Harness; + +mod join; +#[allow(unreachable_pub)] // https://github.com/rust-lang/rust/issues/57411 +pub use self::join::JoinHandle; + +mod raw; +use self::raw::RawTask; + +mod state; + +mod waker; + +use std::{future::Future, marker::PhantomData, ptr::NonNull}; + +/// An owned handle to the task, tracked by ref count, not sendable +#[repr(transparent)] +pub(crate) struct Task { + raw: RawTask, + _p: PhantomData, +} + +impl Task { + unsafe fn from_raw(ptr: NonNull
) -> Task { + Task { + raw: RawTask::from_raw(ptr), + _p: PhantomData, + } + } + + fn header(&self) -> &Header { + self.raw.header() + } + + pub(crate) fn run(self) { + self.raw.poll(); + } + + #[cfg(feature = "sync")] + pub(crate) unsafe fn finish(&mut self, val_slot: *mut ()) { + self.raw.finish(val_slot); + } +} + +impl Drop for Task { + fn drop(&mut self) { + // Decrement the ref count + if self.header().state.ref_dec() { + // Deallocate if this is the final ref count + self.raw.dealloc(); + } + } +} + +pub(crate) trait Schedule: Sized + 'static { + /// Schedule the task + fn schedule(&self, task: Task); + /// Schedule the task to run in the near future, yielding the thread to + /// other tasks. + fn yield_now(&self, task: Task) { + self.schedule(task); + } +} + +pub(crate) fn new_task( + owner_id: usize, + task: T, + scheduler: S, +) -> (Task, JoinHandle) +where + S: Schedule, + T: Future + 'static, + T::Output: 'static, +{ + unsafe { new_task_holding(owner_id, task, scheduler) } +} + +pub(crate) unsafe fn new_task_holding( + owner_id: usize, + task: T, + scheduler: S, +) -> (Task, JoinHandle) +where + S: Schedule, + T: Future, +{ + let raw = RawTask::new::(owner_id, task, scheduler); + let task = Task { + raw, + _p: PhantomData, + }; + let join = JoinHandle::new(raw); + + (task, join) +} diff --git a/vendor/monoio/src/task/raw.rs b/vendor/monoio/src/task/raw.rs new file mode 100644 index 000000000..ccdc8e26f --- /dev/null +++ b/vendor/monoio/src/task/raw.rs @@ -0,0 +1,133 @@ +use std::{ + future::Future, + ptr::NonNull, + task::{Poll, Waker}, +}; + +use crate::task::{Cell, Harness, Header, Schedule}; + +pub(crate) struct RawTask { + ptr: NonNull
, +} + +impl Clone for RawTask { + fn clone(&self) -> Self { + *self + } +} + +impl Copy for RawTask {} + +pub(crate) struct Vtable { + /// Poll the future + pub(crate) poll: unsafe fn(NonNull
), + /// Deallocate the memory + pub(crate) dealloc: unsafe fn(NonNull
), + + /// Read the task output, if complete + pub(crate) try_read_output: unsafe fn(NonNull
, *mut (), &Waker), + + /// The join handle has been dropped + pub(crate) drop_join_handle_slow: unsafe fn(NonNull
), + + /// Set future output + #[cfg(feature = "sync")] + pub(crate) finish: unsafe fn(NonNull
, *mut ()), +} + +/// Get the vtable for the requested `T` and `S` generics. +pub(super) fn vtable() -> &'static Vtable { + &Vtable { + poll: poll::, + dealloc: dealloc::, + try_read_output: try_read_output::, + drop_join_handle_slow: drop_join_handle_slow::, + #[cfg(feature = "sync")] + finish: finish::, + } +} + +impl RawTask { + pub(crate) fn new(owner_id: usize, task: T, scheduler: S) -> RawTask + where + T: Future, + S: Schedule, + { + let ptr = Box::into_raw(Cell::new(owner_id, task, scheduler)); + let ptr = unsafe { NonNull::new_unchecked(ptr as *mut Header) }; + + RawTask { ptr } + } + + pub(crate) unsafe fn from_raw(ptr: NonNull
) -> RawTask { + RawTask { ptr } + } + + pub(crate) fn header(&self) -> &Header { + unsafe { self.ptr.as_ref() } + } + + /// Safety: mutual exclusion is required to call this function. + pub(crate) fn poll(self) { + let vtable = self.header().vtable; + unsafe { (vtable.poll)(self.ptr) } + } + + pub(crate) fn dealloc(self) { + let vtable = self.header().vtable; + unsafe { + (vtable.dealloc)(self.ptr); + } + } + + /// Safety: `dst` must be a `*mut Poll>` where `T` + /// is the future stored by the task. + pub(crate) unsafe fn try_read_output(self, dst: *mut (), waker: &Waker) { + let vtable = self.header().vtable; + (vtable.try_read_output)(self.ptr, dst, waker); + } + + pub(crate) fn drop_join_handle_slow(self) { + let vtable = self.header().vtable; + unsafe { (vtable.drop_join_handle_slow)(self.ptr) } + } + + #[cfg(feature = "sync")] + pub(crate) unsafe fn finish(self, val_slot: *mut ()) { + let vtable = self.header().vtable; + unsafe { (vtable.finish)(self.ptr, val_slot) } + } +} + +unsafe fn poll(ptr: NonNull
) { + let harness = Harness::::from_raw(ptr); + harness.poll(); +} + +unsafe fn dealloc(ptr: NonNull
) { + let harness = Harness::::from_raw(ptr); + harness.dealloc(); +} + +#[cfg(feature = "sync")] +unsafe fn finish(ptr: NonNull
, val: *mut ()) { + let harness = Harness::::from_raw(ptr); + let val = &mut *(val as *mut Option<::Output>); + harness.finish(val.take().unwrap()); +} + +unsafe fn try_read_output( + ptr: NonNull
, + dst: *mut (), + waker: &Waker, +) { + let out = &mut *(dst as *mut Poll); + + let harness = Harness::::from_raw(ptr); + harness.try_read_output(out, waker); +} + +unsafe fn drop_join_handle_slow(ptr: NonNull
) { + let harness = Harness::::from_raw(ptr); + harness.drop_join_handle_slow() +} diff --git a/vendor/monoio/src/task/state.rs b/vendor/monoio/src/task/state.rs new file mode 100644 index 000000000..91aebac59 --- /dev/null +++ b/vendor/monoio/src/task/state.rs @@ -0,0 +1,387 @@ +use std::{ + fmt, + sync::atomic::{ + AtomicUsize, + Ordering::{AcqRel, Acquire, Release}, + }, +}; + +pub(crate) struct State(AtomicUsize); + +/// Current state value +#[derive(Copy, Clone)] +pub(crate) struct Snapshot(usize); + +type UpdateResult = Result; + +/// The task is currently being run. +const RUNNING: usize = 0b0001; + +/// The task is complete. +/// +/// Once this bit is set, it is never unset +const COMPLETE: usize = 0b0010; + +/// Extracts the task's lifecycle value from the state +const LIFECYCLE_MASK: usize = 0b11; + +/// Flag tracking if the task has been pushed into a run queue. +const NOTIFIED: usize = 0b100; + +/// The join handle is still around +#[allow(clippy::unusual_byte_groupings)] // https://github.com/rust-lang/rust-clippy/issues/6556 +const JOIN_INTEREST: usize = 0b1_000; + +/// A join handle waker has been set +#[allow(clippy::unusual_byte_groupings)] // https://github.com/rust-lang/rust-clippy/issues/6556 +const JOIN_WAKER: usize = 0b10_000; + +/// All bits +const STATE_MASK: usize = LIFECYCLE_MASK | NOTIFIED | JOIN_INTEREST | JOIN_WAKER; + +/// Bits used by the ref count portion of the state. +const REF_COUNT_MASK: usize = !STATE_MASK; + +/// Number of positions to shift the ref count +const REF_COUNT_SHIFT: usize = REF_COUNT_MASK.count_zeros() as usize; + +/// One ref count +const REF_ONE: usize = 1 << REF_COUNT_SHIFT; + +/// State a task is initialized with +/// +/// A task is initialized with two references: +/// +/// * A reference for Task. +/// * A reference for the JoinHandle. +/// +/// As the task starts with a `JoinHandle`, `JOIN_INTEREST` is set. +/// As the task starts with a `Notified`, `NOTIFIED` is set. +const INITIAL_STATE: usize = (REF_ONE * 2) | JOIN_INTEREST | NOTIFIED; + +#[must_use] +pub(super) enum TransitionToIdle { + Ok, + OkNotified, +} + +#[must_use] +pub(super) enum TransitionToNotified { + DoNothing, + Submit, +} + +impl State { + pub(crate) fn new() -> Self { + State(AtomicUsize::new(INITIAL_STATE)) + } + + pub(crate) fn load(&self) -> Snapshot { + Snapshot(self.0.load(Acquire)) + } + + pub(crate) fn store(&self, val: Snapshot) { + self.0.store(val.0, Release); + } + + /// Attempt to transition the lifecycle to `Running`. This sets the + /// notified bit to false so notifications during the poll can be detected. + pub(super) fn transition_to_running(&self) { + self.fetch_update_action(|mut curr| { + debug_assert!(curr.is_notified()); + debug_assert!(curr.is_idle()); + curr.set_running(); + curr.unset_notified(); + ((), Some(curr)) + }); + } + + /// Transitions the task from `Running` -> `Idle`. + pub(super) fn transition_to_idle(&self) -> TransitionToIdle { + self.fetch_update_action(|mut curr| { + debug_assert!(curr.is_running()); + curr.unset_running(); + let action = if curr.is_notified() { + TransitionToIdle::OkNotified + } else { + TransitionToIdle::Ok + }; + (action, Some(curr)) + }) + } + + /// Transitions the task from `Running` -> `Complete`. + pub(super) fn transition_to_complete(&self) -> Snapshot { + const DELTA: usize = RUNNING | COMPLETE; + + let prev = Snapshot(self.0.fetch_xor(DELTA, AcqRel)); + debug_assert!(prev.is_running()); + debug_assert!(!prev.is_complete()); + + Snapshot(prev.0 ^ DELTA) + } + + /// Try transitions the state to `NOTIFIED`, but if it cannot do it without submitting, it will + /// return false. In another word, if it returns true, it means we have marked the task notified + /// and do not have to do anything. + pub(crate) fn transition_to_notified_without_submit(&self) -> bool { + self.fetch_update_action(|mut curr| { + if curr.is_running() { + curr.set_notified(); + (true, Some(curr)) + } else if curr.is_complete() || curr.is_notified() { + (true, Some(curr)) + } else { + (false, Some(curr)) + } + }) + } + + /// Transitions the state to `NOTIFIED`. + pub(super) fn transition_to_notified(&self) -> TransitionToNotified { + self.fetch_update_action(|mut curr| { + let action = if curr.is_running() { + curr.set_notified(); + TransitionToNotified::DoNothing + } else if curr.is_complete() || curr.is_notified() { + TransitionToNotified::DoNothing + } else { + curr.set_notified(); + TransitionToNotified::Submit + }; + (action, Some(curr)) + }) + } + + /// Optimistically tries to swap the state assuming the join handle is + /// __immediately__ dropped on spawn + pub(super) fn drop_join_handle_fast(&self) -> Result<(), ()> { + if *self.load() == INITIAL_STATE { + self.store(Snapshot((INITIAL_STATE - REF_ONE) & !JOIN_INTEREST)); + trace!("MONOIO DEBUG[State]: drop_join_handle_fast"); + Ok(()) + } else { + Err(()) + } + } + + /// Try to unset the JOIN_INTEREST flag. + /// + /// Returns `Ok` if the operation happens before the task transitions to a + /// completed state, `Err` otherwise. + pub(super) fn unset_join_interested(&self) -> UpdateResult { + self.fetch_update(|curr| { + assert!(curr.is_join_interested()); + + if curr.is_complete() { + return None; + } + + let mut next = curr; + next.unset_join_interested(); + + Some(next) + }) + } + + /// Set the `JOIN_WAKER` bit. + /// + /// Returns `Ok` if the bit is set, `Err` otherwise. This operation fails if + /// the task has completed. + pub(super) fn set_join_waker(&self) -> UpdateResult { + self.fetch_update(|curr| { + assert!(curr.is_join_interested()); + assert!(!curr.has_join_waker()); + + if curr.is_complete() { + return None; + } + + let mut next = curr; + next.set_join_waker(); + + Some(next) + }) + } + + /// Unsets the `JOIN_WAKER` bit. + /// + /// Returns `Ok` has been unset, `Err` otherwise. This operation fails if + /// the task has completed. + pub(super) fn unset_waker(&self) -> UpdateResult { + self.fetch_update(|curr| { + assert!(curr.is_join_interested()); + assert!(curr.has_join_waker()); + + if curr.is_complete() { + return None; + } + + let mut next = curr; + next.unset_join_waker(); + + Some(next) + }) + } + + pub(crate) fn ref_inc(&self) { + use std::{process, sync::atomic::Ordering::Relaxed}; + + let prev = Snapshot(self.0.fetch_add(REF_ONE, Relaxed)); + + trace!( + "MONOIO DEBUG[State]: ref_inc {}, ptr: {:p}", + prev.ref_count() + 1, + self + ); + + // If the reference count overflowed, abort. + if prev.0 > isize::MAX as usize { + process::abort(); + } + } + + /// Returns `true` if the task should be released. + pub(crate) fn ref_dec(&self) -> bool { + let prev = Snapshot(self.0.fetch_sub(REF_ONE, AcqRel)); + debug_assert!(prev.ref_count() >= 1); + trace!( + "MONOIO DEBUG[State]: ref_dec {}, ptr: {:p}", + prev.ref_count() - 1, + self + ); + prev.ref_count() == 1 + } + + fn fetch_update_action(&self, mut f: F) -> T + where + F: FnMut(Snapshot) -> (T, Option), + { + let mut curr = self.load(); + + loop { + let (output, next) = f(curr); + let next = match next { + Some(next) => next, + None => return output, + }; + + let res = self.0.compare_exchange(curr.0, next.0, AcqRel, Acquire); + + match res { + Ok(_) => return output, + Err(actual) => curr = Snapshot(actual), + } + } + } + + fn fetch_update(&self, mut f: F) -> Result + where + F: FnMut(Snapshot) -> Option, + { + let mut curr = self.load(); + + loop { + let next = match f(curr) { + Some(next) => next, + None => return Err(curr), + }; + + let res = self.0.compare_exchange(curr.0, next.0, AcqRel, Acquire); + + match res { + Ok(_) => return Ok(next), + Err(actual) => curr = Snapshot(actual), + } + } + } +} + +impl std::ops::Deref for Snapshot { + type Target = usize; + + fn deref(&self) -> &Self::Target { + &self.0 + } +} + +impl Snapshot { + /// Returns `true` if the task is in an idle state. + pub(super) fn is_idle(self) -> bool { + self.0 & (RUNNING | COMPLETE) == 0 + } + + /// Returns `true` if the task has been flagged as notified. + pub(super) fn is_notified(self) -> bool { + self.0 & NOTIFIED == NOTIFIED + } + + fn unset_notified(&mut self) { + self.0 &= !NOTIFIED + } + + fn set_notified(&mut self) { + self.0 |= NOTIFIED + } + + pub(super) fn is_running(self) -> bool { + self.0 & RUNNING == RUNNING + } + + fn set_running(&mut self) { + self.0 |= RUNNING; + } + + fn unset_running(&mut self) { + self.0 &= !RUNNING; + } + + /// Returns `true` if the task's future has completed execution. + pub(super) fn is_complete(self) -> bool { + self.0 & COMPLETE == COMPLETE + } + + pub(super) fn is_join_interested(self) -> bool { + self.0 & JOIN_INTEREST == JOIN_INTEREST + } + + fn unset_join_interested(&mut self) { + self.0 &= !JOIN_INTEREST + } + + pub(super) fn has_join_waker(self) -> bool { + self.0 & JOIN_WAKER == JOIN_WAKER + } + + fn set_join_waker(&mut self) { + self.0 |= JOIN_WAKER; + } + + fn unset_join_waker(&mut self) { + self.0 &= !JOIN_WAKER + } + + pub(super) fn ref_count(self) -> usize { + (self.0 & REF_COUNT_MASK) >> REF_COUNT_SHIFT + } +} + +impl fmt::Debug for State { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + let snapshot = self.load(); + snapshot.fmt(fmt) + } +} + +impl fmt::Debug for Snapshot { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt.debug_struct("Snapshot") + .field("is_running", &self.is_running()) + .field("is_complete", &self.is_complete()) + .field("is_notified", &self.is_notified()) + .field("is_join_interested", &self.is_join_interested()) + .field("has_join_waker", &self.has_join_waker()) + .field("ref_count", &self.ref_count()) + .finish() + } +} diff --git a/vendor/monoio/src/task/utils.rs b/vendor/monoio/src/task/utils.rs new file mode 100644 index 000000000..1e74cdd6c --- /dev/null +++ b/vendor/monoio/src/task/utils.rs @@ -0,0 +1,16 @@ +use std::cell::UnsafeCell; + +pub(crate) trait UnsafeCellExt { + fn with(&self, f: impl FnOnce(*const T) -> R) -> R; + fn with_mut(&self, f: impl FnOnce(*mut T) -> R) -> R; +} + +impl UnsafeCellExt for UnsafeCell { + fn with(&self, f: impl FnOnce(*const T) -> R) -> R { + f(self.get()) + } + + fn with_mut(&self, f: impl FnOnce(*mut T) -> R) -> R { + f(self.get()) + } +} diff --git a/vendor/monoio/src/task/waker.rs b/vendor/monoio/src/task/waker.rs new file mode 100644 index 000000000..59c968be1 --- /dev/null +++ b/vendor/monoio/src/task/waker.rs @@ -0,0 +1,103 @@ +use std::{ + future::Future, + marker::PhantomData, + mem::ManuallyDrop, + ops, + ptr::NonNull, + task::{RawWaker, RawWakerVTable, Waker}, +}; + +use super::{core::Header, harness::Harness, Schedule}; + +pub(super) struct WakerRef<'a, S: 'static> { + waker: ManuallyDrop, + _p: PhantomData<(&'a Header, S)>, +} + +/// Returns a `WakerRef` which avoids having to pre-emptively increase the +/// refcount if there is no need to do so. +pub(super) fn waker_ref(header: &Header) -> WakerRef<'_, S> +where + T: Future, + S: Schedule, +{ + // `Waker::will_wake` uses the VTABLE pointer as part of the check. This + // means that `will_wake` will always return false when using the current + // task's waker. (discussion at rust-lang/rust#66281). + // + // To fix this, we use a single vtable. Since we pass in a reference at this + // point and not an *owned* waker, we must ensure that `drop` is never + // called on this waker instance. This is done by wrapping it with + // `ManuallyDrop` and then never calling drop. + let waker = unsafe { ManuallyDrop::new(Waker::from_raw(raw_waker::(header))) }; + + WakerRef { + waker, + _p: PhantomData, + } +} + +impl ops::Deref for WakerRef<'_, S> { + type Target = Waker; + + fn deref(&self) -> &Waker { + &self.waker + } +} + +unsafe fn clone_waker(ptr: *const ()) -> RawWaker +where + T: Future, + S: Schedule, +{ + let header = ptr as *const Header; + trace!("MONOIO DEBUG[Waker]: clone_waker"); + (*header).state.ref_inc(); + raw_waker::(header) +} + +unsafe fn drop_waker(ptr: *const ()) +where + T: Future, + S: Schedule, +{ + let ptr = NonNull::new_unchecked(ptr as *mut Header); + let harness = Harness::::from_raw(ptr); + harness.drop_reference(); +} + +unsafe fn wake_by_val(ptr: *const ()) +where + T: Future, + S: Schedule, +{ + let ptr = NonNull::new_unchecked(ptr as *mut Header); + let harness = Harness::::from_raw(ptr); + harness.wake_by_val(); +} + +// Wake without consuming the waker +unsafe fn wake_by_ref(ptr: *const ()) +where + T: Future, + S: Schedule, +{ + let ptr = NonNull::new_unchecked(ptr as *mut Header); + let harness = Harness::::from_raw(ptr); + harness.wake_by_ref(); +} + +pub(super) fn raw_waker(header: *const Header) -> RawWaker +where + T: Future, + S: Schedule, +{ + let ptr = header as *const (); + let vtable = &RawWakerVTable::new( + clone_waker::, + wake_by_val::, + wake_by_ref::, + drop_waker::, + ); + RawWaker::new(ptr, vtable) +} diff --git a/vendor/monoio/src/task/waker_fn.rs b/vendor/monoio/src/task/waker_fn.rs new file mode 100644 index 000000000..da4aae85b --- /dev/null +++ b/vendor/monoio/src/task/waker_fn.rs @@ -0,0 +1,47 @@ +use core::task::{RawWaker, RawWakerVTable, Waker}; +use std::cell::Cell; + +/// Creates a waker that does nothing. +/// +/// This `Waker` is useful for polling a `Future` to check whether it is +/// `Ready`, without doing any additional work. +pub(crate) fn dummy_waker() -> Waker { + fn raw_waker() -> RawWaker { + // the pointer is never dereferenced, so null is ok + RawWaker::new(std::ptr::null::<()>(), vtable()) + } + + fn vtable() -> &'static RawWakerVTable { + &RawWakerVTable::new( + |_| raw_waker(), + |_| { + set_poll(); + }, + |_| { + set_poll(); + }, + |_| {}, + ) + } + + unsafe { Waker::from_raw(raw_waker()) } +} + +#[cfg(feature = "unstable")] +#[thread_local] +static SHOULD_POLL: Cell = Cell::new(true); + +#[cfg(not(feature = "unstable"))] +thread_local! { + static SHOULD_POLL: Cell = const { Cell::new(true) }; +} + +#[inline] +pub(crate) fn should_poll() -> bool { + SHOULD_POLL.replace(false) +} + +#[inline] +pub(crate) fn set_poll() { + SHOULD_POLL.set(true); +} diff --git a/vendor/monoio/src/time/clock.rs b/vendor/monoio/src/time/clock.rs new file mode 100644 index 000000000..410d2d458 --- /dev/null +++ b/vendor/monoio/src/time/clock.rs @@ -0,0 +1,24 @@ +//! Source of time abstraction. +//! +//! By default, `std::time::Instant::now()` is used. However, when the +//! `test-util` feature flag is enabled, the values returned for `now()` are +//! configurable. + +use crate::time::Instant; + +#[derive(Default, Debug, Clone)] +pub(crate) struct Clock {} + +pub(crate) fn now() -> Instant { + Instant::from_std(std::time::Instant::now()) +} + +impl Clock { + pub(crate) fn new() -> Clock { + Clock {} + } + + pub(crate) fn now(&self) -> Instant { + now() + } +} diff --git a/vendor/monoio/src/time/driver/entry.rs b/vendor/monoio/src/time/driver/entry.rs new file mode 100644 index 000000000..2ab2337ae --- /dev/null +++ b/vendor/monoio/src/time/driver/entry.rs @@ -0,0 +1,580 @@ +//! Timer state structures. +//! +//! This module contains the heart of the intrusive timer implementation, and as +//! such the structures inside are full of tricky concurrency and unsafe code. +//! +//! # Ground rules +//! +//! The heart of the timer implementation here is the `TimerShared` structure, +//! shared between the `TimerEntry` and the driver. Generally, we permit access +//! to `TimerShared` ONLY via either 1) a mutable reference to `TimerEntry` or +//! 2) a held driver lock. +//! +//! It follows from this that any changes made while holding BOTH 1 and 2 will +//! be reliably visible, regardless of ordering. This is because of the acq/rel +//! fences on the driver lock ensuring ordering with 2, and rust mutable +//! reference rules for 1 (a mutable reference to an object can't be passed +//! between threads without an acq/rel barrier, and same-thread we have local +//! happens-before ordering). +//! +//! # State field +//! +//! Each timer has a state field associated with it. This field contains either +//! the current scheduled time, or a special flag value indicating its state. +//! This state can either indicate that the timer is on the 'pending' queue (and +//! thus will be fired with an `Ok(())` result soon) or that it has already been +//! fired/deregistered. +//! +//! This single state field allows for code that is firing the timer to +//! synchronize with any racing `reset` calls reliably. +//! +//! # Cached vs true timeouts +//! +//! To allow for the use case of a timeout that is periodically reset before +//! expiration to be as lightweight as possible, we support optimistically +//! lock-free timer resets, in the case where a timer is rescheduled to a later +//! point than it was originally scheduled for. +//! +//! This is accomplished by lazily rescheduling timers. That is, we update the +//! state field field with the true expiration of the timer from the holder of +//! the [`TimerEntry`]. When the driver services timers (ie, whenever it's +//! walking lists of timers), it checks this "true when" value, and reschedules +//! based on it. +//! +//! We do, however, also need to track what the expiration time was when we +//! originally registered the timer; this is used to locate the right linked +//! list when the timer is being cancelled. This is referred to as the "cached +//! when" internally. +//! +//! There is of course a race condition between timer reset and timer +//! expiration. If the driver fails to observe the updated expiration time, it +//! could trigger expiration of the timer too early. However, because +//! `mark_pending` performs a compare-and-swap, it will identify this race and +//! refuse to mark the timer as pending. + +use std::{ + cell::{Cell, RefCell, UnsafeCell}, + marker::PhantomPinned, + pin::Pin, + ptr::NonNull, + task::{Context, Poll, Waker}, +}; + +use super::Handle; +use crate::{time::Instant, utils::linked_list}; + +type TimerResult = Result<(), crate::time::error::Error>; + +const STATE_DEREGISTERED: u64 = u64::MAX; +const STATE_PENDING_FIRE: u64 = STATE_DEREGISTERED - 1; +const STATE_MIN_VALUE: u64 = STATE_PENDING_FIRE; + +/// This structure holds the current shared state of the timer - its scheduled +/// time (if registered), or otherwise the result of the timer completing, as +/// well as the registered waker. +/// +/// Generally, the StateCell is only permitted to be accessed from two contexts: +/// Either a thread holding the corresponding &mut TimerEntry, or a thread +/// holding the timer driver lock. The write actions on the StateCell amount to +/// passing "ownership" of the StateCell between these contexts; moving a timer +/// from the TimerEntry to the driver requires _both_ holding the &mut +/// TimerEntry and the driver lock, while moving it back (firing the timer) +/// requires only the driver lock. +pub(super) struct StateCell { + /// Holds either the scheduled expiration time for this timer, or (if the + /// timer has been fired and is unregistered), `u64::MAX`. + state: Cell, + /// If the timer is fired (an Acquire order read on state shows + /// `u64::MAX`), holds the result that should be returned from + /// polling the timer. Otherwise, the contents are unspecified and reading + /// without holding the driver lock is undefined behavior. + result: UnsafeCell, + /// The currently-registered waker + waker: CachePadded>>, +} + +impl Default for StateCell { + fn default() -> Self { + Self::new() + } +} + +impl std::fmt::Debug for StateCell { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "StateCell({:?})", self.read_state()) + } +} + +impl StateCell { + fn new() -> Self { + Self { + state: Cell::new(STATE_DEREGISTERED), + result: UnsafeCell::new(Ok(())), + waker: CachePadded(RefCell::new(None)), + } + } + + /// Returns the current expiration time, or None if not currently scheduled. + fn when(&self) -> Option { + let cur_state = self.state.get(); + + if cur_state == u64::MAX { + None + } else { + Some(cur_state) + } + } + + /// If the timer is completed, returns the result of the timer. Otherwise, + /// returns None and registers the waker. + fn poll(&self, waker: &Waker) -> Poll { + // We must register first. This ensures that either `fire` will + // observe the new waker, or we will observe a racing fire to have set + // the state, or both. + let mut w = self.waker.0.borrow_mut(); + if w.is_none() { + let _ = w.insert(waker.clone()); + } + + self.read_state() + } + + fn read_state(&self) -> Poll { + let cur_state = self.state.get(); + + if cur_state == STATE_DEREGISTERED { + // SAFETY: The driver has fired this timer; this involves writing + // the result, and then writing (with release ordering) the state + // field. + Poll::Ready(unsafe { *self.result.get() }) + } else { + Poll::Pending + } + } + + /// Marks this timer as being moved to the pending list, if its scheduled + /// time is not after `not_after`. + /// + /// If the timer is scheduled for a time after not_after, returns an Err + /// containing the current scheduled time. + /// + /// SAFETY: Must hold the driver lock. + unsafe fn mark_pending(&self, not_after: u64) -> Result<(), u64> { + // Quick initial debug check to see if the timer is already fired. Since + // firing the timer can only happen with the driver lock held, we know + // we shouldn't be able to "miss" a transition to a fired state, even + // with relaxed ordering. + let cur_state = self.state.get(); + + debug_assert!(cur_state < STATE_MIN_VALUE); + + if cur_state > not_after { + return Err(cur_state); + } + + self.state.set(STATE_PENDING_FIRE); + Ok(()) + } + + /// Fires the timer, setting the result to the provided result. + /// + /// Returns: + /// * `Some(waker) - if fired and a waker needs to be invoked once the driver lock is released + /// * `None` - if fired and a waker does not need to be invoked, or if already fired + /// + /// SAFETY: The driver lock must be held. + unsafe fn fire(&self, result: TimerResult) -> Option { + // Quick initial check to see if the timer is already fired. Since + // firing the timer can only happen with the driver lock held, we know + // we shouldn't be able to "miss" a transition to a fired state, even + // with relaxed ordering. + let cur_state = self.state.get(); + if cur_state == STATE_DEREGISTERED { + return None; + } + + // SAFETY: We assume the driver lock is held and the timer is not + // fired, so only the driver is accessing this field. + // + // We perform a release-ordered store to state below, to ensure this + // write is visible before the state update is visible. + unsafe { + let p = self.result.get(); + *p = result; + } + + self.state.set(STATE_DEREGISTERED); + + self.waker.0.take() + } + + /// Marks the timer as registered (poll will return None) and sets the + /// expiration time. + /// + /// While this function is memory-safe, it should only be called from a + /// context holding both `&mut TimerEntry` and the driver lock. + fn set_expiration(&self, timestamp: u64) { + debug_assert!(timestamp < STATE_MIN_VALUE); + + // We can use relaxed ordering because we hold the driver lock and will + // fence when we release the lock. + self.state.set(timestamp); + } + + /// Attempts to adjust the timer to a new timestamp. + /// + /// If the timer has already been fired, is pending firing, or the new + /// timestamp is earlier than the old timestamp, (or occasionally + /// spuriously) returns Err without changing the timer's state. In this + /// case, the timer must be deregistered and re-registered. + fn extend_expiration(&self, new_timestamp: u64) -> Result<(), ()> { + let state = self.state.get(); + if new_timestamp < state || state >= STATE_MIN_VALUE { + return Err(()); + } + self.state.set(new_timestamp); + Ok(()) + } + + /// Returns true if the state of this timer indicates that the timer might + /// be registered with the driver. This check is performed with relaxed + /// ordering, but is conservative - if it returns false, the timer is + /// definitely _not_ registered. + pub(super) fn might_be_registered(&self) -> bool { + self.state.get() != u64::MAX + } +} + +/// A timer entry. +/// +/// This is the handle to a timer that is controlled by the requester of the +/// timer. As this participates in intrusive data structures, it must be pinned +/// before polling. +#[derive(Debug)] +pub(super) struct TimerEntry { + /// Arc reference to the driver. We can only free the driver after + /// deregistering everything from their respective timer wheels. + driver: Handle, + /// Shared inner structure; this is part of an intrusive linked list, and + /// therefore other references can exist to it while mutable references to + /// Entry exist. + /// + /// This is manipulated only under the inner mutex. TODO: Can we use loom + /// cells for this? + inner: UnsafeCell, + /// Initial deadline for the timer. This is used to register on the first + /// poll, as we can't register prior to being pinned. + initial_deadline: Option, + /// Ensure the type is !Unpin + _m: std::marker::PhantomPinned, +} + +/// An TimerHandle is the (non-enforced) "unique" pointer from the driver to the +/// timer entry. Generally, at most one TimerHandle exists for a timer at a time +/// (enforced by the timer state machine). +/// +/// SAFETY: An TimerHandle is essentially a raw pointer, and the usual caveats +/// of pointer safety apply. In particular, TimerHandle does not itself enforce +/// that the timer does still exist; however, normally an TimerHandle is created +/// immediately before registering the timer, and is consumed when firing the +/// timer, to help minimize mistakes. Still, because TimerHandle cannot enforce +/// memory safety, all operations are unsafe. +#[derive(Debug)] +pub(crate) struct TimerHandle { + inner: NonNull, +} + +pub(super) type EntryList = crate::utils::linked_list::LinkedList; + +/// The shared state structure of a timer. This structure is shared between the +/// frontend (`Entry`) and driver backend. +/// +/// Note that this structure is located inside the `TimerEntry` structure. +#[derive(Debug)] +pub(crate) struct TimerShared { + /// Current state. This records whether the timer entry is currently under + /// the ownership of the driver, and if not, its current state (not + /// complete, fired, error, etc). + state: StateCell, + + /// Data manipulated by the driver thread itself, only. + driver_state: CachePadded, + + _p: PhantomPinned, +} + +impl TimerShared { + pub(super) fn new() -> Self { + Self { + state: StateCell::default(), + driver_state: CachePadded(TimerSharedPadded::new()), + _p: PhantomPinned, + } + } + + /// Gets the cached time-of-expiration value + pub(super) fn cached_when(&self) -> u64 { + // Cached-when is only accessed under the driver lock, so we can use relaxed + self.driver_state.0.cached_when.get() + } + + /// Gets the true time-of-expiration value, and copies it into the cached + /// time-of-expiration value. + /// + /// SAFETY: Must be called with the driver lock held, and when this entry is + /// not in any timer wheel lists. + pub(super) unsafe fn sync_when(&self) -> u64 { + let true_when = self.true_when(); + + self.driver_state.0.cached_when.set(true_when); + + true_when + } + + /// Sets the cached time-of-expiration value. + /// + /// SAFETY: Must be called with the driver lock held, and when this entry is + /// not in any timer wheel lists. + unsafe fn set_cached_when(&self, when: u64) { + self.driver_state.0.cached_when.set(when); + } + + /// Returns the true time-of-expiration value, with relaxed memory ordering. + pub(super) fn true_when(&self) -> u64 { + self.state.when().expect("Timer already fired") + } + + /// Sets the true time-of-expiration value, even if it is less than the + /// current expiration or the timer is deregistered. + /// + /// SAFETY: Must only be called with the driver lock held and the entry not + /// in the timer wheel. + pub(super) unsafe fn set_expiration(&self, t: u64) { + self.state.set_expiration(t); + self.driver_state.0.cached_when.set(t); + } + + /// Sets the true time-of-expiration only if it is after the current. + pub(super) fn extend_expiration(&self, t: u64) -> Result<(), ()> { + self.state.extend_expiration(t) + } + + /// Returns a TimerHandle for this timer. + pub(super) fn handle(&self) -> TimerHandle { + TimerHandle { + inner: NonNull::from(self), + } + } + + /// Returns true if the state of this timer indicates that the timer might + /// be registered with the driver. This check is performed with relaxed + /// ordering, but is conservative - if it returns false, the timer is + /// definitely _not_ registered. + pub(super) fn might_be_registered(&self) -> bool { + self.state.might_be_registered() + } +} + +/// Additional shared state between the driver and the timer which is cache +/// padded. This contains the information that the driver thread accesses most +/// frequently to minimize contention. In particular, we move it away from the +/// waker, as the waker is updated on every poll. +struct TimerSharedPadded { + /// The expiration time for which this entry is currently registered. + /// Generally owned by the driver, but is accessed by the entry when not + /// registered. + cached_when: Cell, + + /// The true expiration time. Set by the timer future, read by the driver. + true_when: Cell, + + /// A link within the doubly-linked list of timers on a particular level and + /// slot. Valid only if state is equal to Registered. + /// + /// Only accessed under the entry lock. + pointers: UnsafeCell>, +} + +impl std::fmt::Debug for TimerSharedPadded { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("TimerSharedPadded") + .field("when", &self.true_when) + .field("cached_when", &self.cached_when) + .finish() + } +} + +impl TimerSharedPadded { + fn new() -> Self { + Self { + cached_when: Cell::new(0), + true_when: Cell::new(0), + pointers: UnsafeCell::new(linked_list::Pointers::new()), + } + } +} + +unsafe impl linked_list::Link for TimerShared { + type Handle = TimerHandle; + + type Target = TimerShared; + + fn as_raw(handle: &Self::Handle) -> NonNull { + handle.inner + } + + unsafe fn from_raw(ptr: NonNull) -> Self::Handle { + TimerHandle { inner: ptr } + } + + unsafe fn pointers( + target: NonNull, + ) -> NonNull> { + unsafe { NonNull::new(target.as_ref().driver_state.0.pointers.get()).unwrap() } + } +} + +// ===== impl Entry ===== + +impl TimerEntry { + pub(crate) fn new(handle: &Handle, deadline: Instant) -> Self { + let driver = handle.clone(); + + Self { + driver, + inner: UnsafeCell::new(TimerShared::new()), + initial_deadline: Some(deadline), + _m: std::marker::PhantomPinned, + } + } + + fn inner(&self) -> &TimerShared { + unsafe { &*self.inner.get() } + } + + pub(crate) fn is_elapsed(&self) -> bool { + !self.inner().state.might_be_registered() && self.initial_deadline.is_none() + } + + /// Cancels and deregisters the timer. This operation is irreversible. + pub(crate) fn cancel(self: Pin<&mut Self>) { + // We need to perform an acq/rel fence with the driver thread, and the + // simplest way to do so is to grab the driver lock. + // + // Why is this necessary? We're about to release this timer's memory for + // some other non-timer use. However, we've been doing a bunch of + // relaxed (or even non-atomic) writes from the driver thread, and we'll + // be doing more from _this thread_ (as this memory is interpreted as + // something else). + // + // It is critical to ensure that, from the point of view of the driver, + // those future non-timer writes happen-after the timer is fully fired, + // and from the purpose of this thread, the driver's writes all + // happen-before we drop the timer. This in turn requires us to perform + // an acquire-release barrier in _both_ directions between the driver + // and dropping thread. + // + // The lock acquisition in clear_entry serves this purpose. All of the + // driver manipulations happen with the lock held, so we can just take + // the lock and be sure that this drop happens-after everything the + // driver did so far and happens-before everything the driver does in + // the future. While we have the lock held, we also go ahead and + // deregister the entry if necessary. + unsafe { self.driver.clear_entry(NonNull::from(self.inner())) }; + } + + pub(crate) fn reset(mut self: Pin<&mut Self>, new_time: Instant) { + unsafe { self.as_mut().get_unchecked_mut() }.initial_deadline = None; + + let tick = self.driver.time_source().deadline_to_tick(new_time); + + if self.inner().extend_expiration(tick).is_ok() { + return; + } + + unsafe { + self.driver.reregister(tick, self.inner().into()); + } + } + + #[allow(clippy::needless_pass_by_ref_mut)] + pub(crate) fn poll_elapsed( + mut self: Pin<&mut Self>, + cx: &mut Context<'_>, + ) -> Poll> { + if let Some(deadline) = self.initial_deadline { + self.as_mut().reset(deadline); + } + + let this = unsafe { self.get_unchecked_mut() }; + + this.inner().state.poll(cx.waker()) + } +} + +impl TimerHandle { + pub(super) unsafe fn cached_when(&self) -> u64 { + unsafe { self.inner.as_ref().cached_when() } + } + + pub(super) unsafe fn sync_when(&self) -> u64 { + unsafe { self.inner.as_ref().sync_when() } + } + + /// Forcibly sets the true and cached expiration times to the given tick. + /// + /// SAFETY: The caller must ensure that the handle remains valid, the driver + /// lock is held, and that the timer is not in any wheel linked lists. + pub(super) unsafe fn set_expiration(&self, tick: u64) { + self.inner.as_ref().set_expiration(tick); + } + + /// Attempts to mark this entry as pending. If the expiration time is after + /// `not_after`, however, returns an Err with the current expiration time. + /// + /// If an `Err` is returned, the `cached_when` value will be updated to this + /// new expiration time. + /// + /// SAFETY: The caller must ensure that the handle remains valid, the driver + /// lock is held, and that the timer is not in any wheel linked lists. + /// After returning Ok, the entry must be added to the pending list. + pub(super) unsafe fn mark_pending(&self, not_after: u64) -> Result<(), u64> { + match self.inner.as_ref().state.mark_pending(not_after) { + Ok(()) => { + // mark this as being on the pending queue in cached_when + self.inner.as_ref().set_cached_when(u64::MAX); + Ok(()) + } + Err(tick) => { + self.inner.as_ref().set_cached_when(tick); + Err(tick) + } + } + } + + /// Attempts to transition to a terminal state. If the state is already a + /// terminal state, does nothing. + /// + /// Because the entry might be dropped after the state is moved to a + /// terminal state, this function consumes the handle to ensure we don't + /// access the entry afterwards. + /// + /// Returns the last-registered waker, if any. + /// + /// SAFETY: The driver lock must be held while invoking this function, and + /// the entry must not be in any wheel linked lists. + pub(super) unsafe fn fire(self, completed_state: TimerResult) -> Option { + self.inner.as_ref().state.fire(completed_state) + } +} + +impl Drop for TimerEntry { + fn drop(&mut self) { + unsafe { Pin::new_unchecked(self) }.as_mut().cancel() + } +} + +#[cfg_attr(target_arch = "x86_64", repr(align(128)))] +#[cfg_attr(not(target_arch = "x86_64"), repr(align(64)))] +#[derive(Debug, Default)] +struct CachePadded(T); diff --git a/vendor/monoio/src/time/driver/handle.rs b/vendor/monoio/src/time/driver/handle.rs new file mode 100644 index 000000000..4b1d18030 --- /dev/null +++ b/vendor/monoio/src/time/driver/handle.rs @@ -0,0 +1,60 @@ +use std::{fmt, rc::Rc}; + +use crate::time::driver::ClockTime; + +/// Handle to time driver instance. +#[derive(Clone)] +pub(crate) struct Handle { + time_source: ClockTime, + inner: Rc, +} + +impl Handle { + /// Creates a new timer `Handle` from a shared `Inner` timer state. + pub(super) fn new(inner: Rc) -> Self { + let time_source = inner.state.borrow_mut().time_source.clone(); + Handle { time_source, inner } + } + + /// Returns the time source associated with this handle + pub(super) fn time_source(&self) -> &ClockTime { + &self.time_source + } + + /// Access the driver's inner structure + pub(super) fn get(&self) -> &super::Inner { + &self.inner + } +} + +impl Handle { + /// Tries to get a handle to the current timer. + /// + /// # Panics + /// + /// This function panics if there is no current timer set. + /// + /// It can be triggered when `Builder::enable_timer()` or + /// `Builder::enable_all()` are not included in the builder. + /// + /// It can also panic whenever a timer is created outside of a + /// Monoio runtime. That is why `rt.block_on(delay_for(...))` will panic, + /// since the function is executed outside of the runtime. + /// Whereas `rt.block_on(async {delay_for(...).await})` doesn't panic. + /// And this is because wrapping the function on an async makes it lazy, + /// and so gets executed inside the runtime successfully without + /// panicking. + pub(crate) fn current() -> Self { + crate::runtime::CURRENT.with(|c| { + c.time_handle.clone().expect( + "unable to get time handle, maybe you have not enable_timer on creating runtime?", + ) + }) + } +} + +impl fmt::Debug for Handle { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "Handle") + } +} diff --git a/vendor/monoio/src/time/driver/mod.rs b/vendor/monoio/src/time/driver/mod.rs new file mode 100644 index 000000000..3887dfe23 --- /dev/null +++ b/vendor/monoio/src/time/driver/mod.rs @@ -0,0 +1,367 @@ +// Currently, rust warns when an unsafe fn contains an unsafe {} block. However, +// in the future, this will change to the reverse. For now, suppress this +// warning and generally stick with being explicit about unsafety. +#![allow(unused_unsafe)] + +//! Time driver + +mod entry; +use self::entry::{EntryList, TimerEntry, TimerHandle, TimerShared}; + +mod handle; +pub(crate) use self::handle::Handle; + +mod wheel; + +pub(super) mod sleep; + +use std::{cell::RefCell, fmt, io, num::NonZeroU64, ptr::NonNull, rc::Rc}; + +use crate::{ + driver::Driver, + time::{error::Error, Clock, Duration, Instant}, +}; + +/// Time implementation that drives [`Sleep`][sleep], [`Interval`][interval], +/// and [`Timeout`][timeout]. +/// +/// A `Driver` instance tracks the state necessary for managing time and +/// notifying the [`Sleep`][sleep] instances once their deadlines are reached. +/// +/// It is expected that a single instance manages many individual +/// [`Sleep`][sleep] instances. The `Driver` implementation is thread-safe and, +/// as such, is able to handle callers from across threads. +/// +/// After creating the `Driver` instance, the caller must repeatedly call `park` +/// or `park_timeout`. The time driver will perform no work unless `park` or +/// `park_timeout` is called repeatedly. +/// +/// The driver has a resolution of one millisecond. Any unit of time that falls +/// between milliseconds are rounded up to the next millisecond. +/// +/// When an instance is dropped, any outstanding [`Sleep`][sleep] instance that +/// has not elapsed will be notified with an error. At this point, calling +/// `poll` on the [`Sleep`][sleep] instance will result in panic. +/// +/// # Implementation +/// +/// The time driver is based on the [paper by Varghese and Lauck][paper]. +/// +/// A hashed timing wheel is a vector of slots, where each slot handles a time +/// slice. As time progresses, the timer walks over the slot for the current +/// instant, and processes each entry for that slot. When the timer reaches the +/// end of the wheel, it starts again at the beginning. +/// +/// The implementation maintains six wheels arranged in a set of levels. As the +/// levels go up, the slots of the associated wheel represent larger intervals +/// of time. At each level, the wheel has 64 slots. Each slot covers a range of +/// time equal to the wheel at the lower level. At level zero, each slot +/// represents one millisecond of time. +/// +/// The wheels are: +/// +/// * Level 0: 64 x 1 millisecond slots. +/// * Level 1: 64 x 64 millisecond slots. +/// * Level 2: 64 x ~4 second slots. +/// * Level 3: 64 x ~4 minute slots. +/// * Level 4: 64 x ~4 hour slots. +/// * Level 5: 64 x ~12 day slots. +/// +/// When the timer processes entries at level zero, it will notify all the +/// `Sleep` instances as their deadlines have been reached. For all higher +/// levels, all entries will be redistributed across the wheel at the next level +/// down. Eventually, as time progresses, entries with [`Sleep`][sleep] +/// instances will either be canceled (dropped) or their associated entries will +/// reach level zero and be notified. +/// +/// [paper]: http://www.cs.columbia.edu/~nahum/w6998/papers/ton97-timing-wheels.pdf +/// [sleep]: crate::time::Sleep +/// [timeout]: crate::time::Timeout +/// [interval]: crate::time::Interval +#[derive(Debug)] +pub struct TimeDriver { + /// Timing backend in use + time_source: ClockTime, + + /// Shared state + pub(crate) handle: Handle, + + /// Parker to delegate to + park: D, +} + +/// A structure which handles conversion from Instants to u64 timestamps. +#[derive(Debug, Clone)] +struct ClockTime { + clock: super::clock::Clock, + start_time: Instant, +} + +impl ClockTime { + pub(self) fn new(clock: Clock) -> Self { + Self { + start_time: clock.now(), + clock, + } + } + + pub(self) fn deadline_to_tick(&self, t: Instant) -> u64 { + // Round up to the end of a ms + self.instant_to_tick(t + Duration::from_nanos(999_999)) + } + + pub(self) fn instant_to_tick(&self, t: Instant) -> u64 { + // round up + let dur: Duration = t + .checked_duration_since(self.start_time) + .unwrap_or_else(|| Duration::from_secs(0)); + let ms = dur.as_millis(); + + ms.try_into().expect("Duration too far into the future") + } + + pub(self) fn tick_to_duration(&self, t: u64) -> Duration { + Duration::from_millis(t) + } + + pub(self) fn now(&self) -> u64 { + self.instant_to_tick(self.clock.now()) + } +} + +/// Timer state shared between `Driver`, `Handle`, and `Registration`. +struct Inner { + // The state is split like this so `Handle` can access `is_shutdown` without locking the mutex + pub(super) state: RefCell, +} + +/// Time state shared which must be protected by a `Mutex` +struct InnerState { + /// Timing backend in use + time_source: ClockTime, + + /// The last published timer `elapsed` value. + elapsed: u64, + + /// The earliest time at which we promise to wake up without unparking + next_wake: Option, + + /// Timer wheel + wheel: wheel::Wheel, +} + +// ===== impl Driver ===== + +impl TimeDriver +where + D: Driver + 'static, +{ + /// Creates a new `Driver` instance that uses `park` to block the current + /// thread and `time_source` to get the current time and convert to ticks. + /// + /// Specifying the source of time is useful when testing. + pub(crate) fn new(park: D, clock: Clock) -> TimeDriver { + let time_source = ClockTime::new(clock); + + let inner = Inner::new(time_source.clone()); + + TimeDriver { + time_source, + handle: Handle::new(Rc::new(inner)), + park, + } + } + + fn park_internal(&self, limit: Option) -> io::Result<()> { + let mut inner_state = self.handle.get().state.borrow_mut(); + + let next_wake = inner_state.wheel.next_expiration_time(); + inner_state.next_wake = + next_wake.map(|t| NonZeroU64::new(t).unwrap_or_else(|| NonZeroU64::new(1).unwrap())); + drop(inner_state); + + match next_wake { + Some(when) => { + let now = self.time_source.now(); + // Note that we effectively round up to 1ms here - this avoids + // very short-duration microsecond-resolution sleeps that the OS + // might treat as zero-length. + let mut duration = self.time_source.tick_to_duration(when.saturating_sub(now)); + + if duration > Duration::from_millis(0) { + if let Some(limit) = limit { + duration = std::cmp::min(limit, duration); + } + + self.park.park_timeout(duration)?; + } else { + self.park.park_timeout(Duration::from_secs(0))?; + } + } + None => { + if let Some(duration) = limit { + self.park.park_timeout(duration)?; + } else { + self.park.park()?; + } + } + } + + // Process pending timers after waking up + self.handle.process(); + + Ok(()) + } +} + +impl Handle { + /// Runs timer related logic, and returns the next wakeup time + pub(self) fn process(&self) { + let now = self.time_source().now(); + + self.process_at_time(now) + } + + pub(self) fn process_at_time(&self, mut now: u64) { + let mut state = self.get().state.borrow_mut(); + + if now < state.elapsed { + // Time went backwards! This normally shouldn't happen as the Rust language + // guarantees that an Instant is monotonic, but can happen when running + // Linux in a VM on a Windows host due to std incorrectly trusting the + // hardware clock to be monotonic. + // + // See for more information. + now = state.elapsed; + } + while let Some(entry) = state.wheel.poll(now) { + if let Some(waker) = unsafe { entry.fire(Ok(())) } { + waker.wake(); + } + } + state.elapsed = state.wheel.elapsed(); + state.next_wake = state + .wheel + .poll_at() + .map(|t| NonZeroU64::new(t).unwrap_or_else(|| NonZeroU64::new(1).unwrap())); + } + + /// Removes a registered timer from the driver. + /// + /// The timer will be moved to the cancelled state. Wakers will _not_ be + /// invoked. If the timer is already completed, this function is a no-op. + /// + /// This function always acquires the driver lock, even if the entry does + /// not appear to be registered. + /// + /// SAFETY: The timer must not be registered with some other driver, and + /// `add_entry` must not be called concurrently. + pub(self) unsafe fn clear_entry(&self, entry: NonNull) { + unsafe { + let mut state = self.get().state.borrow_mut(); + if entry.as_ref().might_be_registered() { + state.wheel.remove(entry); + } + + entry.as_ref().handle().fire(Ok(())); + } + } + + /// Removes and re-adds an entry to the driver. + /// + /// SAFETY: The timer must be either unregistered, or registered with this + /// driver. No other threads are allowed to concurrently manipulate the + /// timer at all (the current thread should hold an exclusive reference to + /// the `TimerEntry`) + pub(self) unsafe fn reregister(&self, new_tick: u64, entry: NonNull) { + let waker = unsafe { + let mut state = self.get().state.borrow_mut(); + + // We may have raced with a firing/deregistration, so check before + // deregistering. + if unsafe { entry.as_ref().might_be_registered() } { + state.wheel.remove(entry); + } + + // Now that we have exclusive control of this entry, mint a handle to reinsert + // it. + let entry = entry.as_ref().handle(); + + entry.set_expiration(new_tick); + + // Note: We don't have to worry about racing with some other resetting + // thread, because add_entry and reregister require exclusive control of + // the timer entry. + match unsafe { state.wheel.insert(entry) } { + Ok(_) => None, + Err((entry, super::error::InsertError::Elapsed)) => unsafe { entry.fire(Ok(())) }, + } + }; + + // The timer was fired synchronously as a result of the reregistration. + // Wake the waker; this is needed because we might reset _after_ a poll, + // and otherwise the task won't be awoken to poll again. + if let Some(waker) = waker { + waker.wake(); + } + } +} + +impl Driver for TimeDriver +where + D: Driver + 'static, +{ + fn with(&self, f: impl FnOnce() -> R) -> R { + self.park.with(f) + } + + fn submit(&self) -> io::Result<()> { + self.park.submit() + } + + fn park(&self) -> io::Result<()> { + self.park_internal(None) + } + + #[cfg(feature = "sync")] + type Unpark = D::Unpark; + + fn park_timeout(&self, duration: Duration) -> io::Result<()> { + self.park_internal(Some(duration)) + } + + #[cfg(feature = "sync")] + fn unpark(&self) -> Self::Unpark { + self.park.unpark() + } +} + +impl Drop for TimeDriver +where + D: 'static, +{ + fn drop(&mut self) { + // self.shutdown(); + } +} + +// ===== impl Inner ===== + +impl Inner { + pub(self) fn new(time_source: ClockTime) -> Self { + Inner { + state: RefCell::new(InnerState { + time_source, + elapsed: 0, + next_wake: None, + wheel: wheel::Wheel::new(), + }), + } + } +} + +impl fmt::Debug for Inner { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt.debug_struct("Inner").finish() + } +} diff --git a/vendor/monoio/src/time/driver/sleep.rs b/vendor/monoio/src/time/driver/sleep.rs new file mode 100644 index 000000000..36d11341a --- /dev/null +++ b/vendor/monoio/src/time/driver/sleep.rs @@ -0,0 +1,284 @@ +use std::{ + future::Future, + pin::Pin, + task::{self, Poll}, +}; + +use pin_project_lite::pin_project; + +use crate::time::{ + driver::{Handle, TimerEntry}, + error::Error, + Duration, Instant, +}; + +/// Waits until `deadline` is reached. +/// +/// No work is performed while awaiting on the sleep future to complete. `Sleep` +/// operates at millisecond granularity and should not be used for tasks that +/// require high-resolution timers. +/// +/// To run something regularly on a schedule, see [`interval`]. +/// +/// # Cancellation +/// +/// Canceling a sleep instance is done by dropping the returned future. No +/// additional cleanup work is required. +/// +/// # Examples +/// +/// Wait 100ms and print "100 ms have elapsed". +/// +/// ``` +/// use monoio::time::{sleep_until, Duration, Instant}; +/// +/// #[monoio::main(timer_enabled = true)] +/// async fn main() { +/// sleep_until(Instant::now() + Duration::from_millis(100)).await; +/// println!("100 ms have elapsed"); +/// } +/// ``` +/// +/// See the documentation for the [`Sleep`] type for more examples. +/// +/// [`Sleep`]: struct@crate::time::Sleep +/// [`interval`]: crate::time::interval() +// Alias for old name in 0.x +#[cfg_attr(docsrs, doc(alias = "delay_until"))] +pub fn sleep_until(deadline: Instant) -> Sleep { + Sleep::new_timeout(deadline) +} + +/// Waits until `duration` has elapsed. +/// +/// Equivalent to `sleep_until(Instant::now() + duration)`. An asynchronous +/// analog to `std::thread::sleep`. +/// +/// No work is performed while awaiting on the sleep future to complete. `Sleep` +/// operates at millisecond granularity and should not be used for tasks that +/// require high-resolution timers. +/// +/// To run something regularly on a schedule, see [`interval`]. +/// +/// The maximum duration for a sleep is 68719476734 milliseconds (approximately +/// 2.2 years). +/// +/// # Cancellation +/// +/// Canceling a sleep instance is done by dropping the returned future. No +/// additional cleanup work is required. +/// +/// # Examples +/// +/// Wait 100ms and print "100 ms have elapsed". +/// +/// ``` +/// use monoio::time::{sleep, Duration}; +/// +/// #[monoio::main(timer_enabled = true)] +/// async fn main() { +/// sleep(Duration::from_millis(100)).await; +/// println!("100 ms have elapsed"); +/// } +/// ``` +/// +/// See the documentation for the [`Sleep`] type for more examples. +/// +/// [`Sleep`]: struct@crate::time::Sleep +/// [`interval`]: crate::time::interval() +// Alias for old name in 0.x +#[cfg_attr(docsrs, doc(alias = "delay_for"))] +#[cfg_attr(docsrs, doc(alias = "wait"))] +pub fn sleep(duration: Duration) -> Sleep { + match Instant::now().checked_add(duration) { + Some(deadline) => sleep_until(deadline), + None => sleep_until(Instant::far_future()), + } +} + +pin_project! { + /// Future returned by [`sleep`](sleep) and [`sleep_until`](sleep_until). + /// + /// This type does not implement the `Unpin` trait, which means that if you + /// use it with [`select!`] or by calling `poll`, you have to pin it first. + /// If you use it with `.await`, this does not apply. + /// + /// # Examples + /// + /// Wait 100ms and print "100 ms have elapsed". + /// + /// ``` + /// use monoio::time::{sleep, Duration}; + /// + /// #[monoio::main(timer_enabled = true)] + /// async fn main() { + /// sleep(Duration::from_millis(100)).await; + /// println!("100 ms have elapsed"); + /// } + /// ``` + /// + /// Use with [`select!`]. Pinning the `Sleep` with [`monoio::pin!`] is + /// necessary when the same `Sleep` is selected on multiple times. + /// ```no_run + /// use monoio::time::{self, Duration, Instant}; + /// + /// #[monoio::main(timer_enabled = true)] + /// async fn main() { + /// let sleep = time::sleep(Duration::from_millis(10)); + /// monoio::pin!(sleep); + /// + /// loop { + /// monoio::select! { + /// () = &mut sleep => { + /// println!("timer elapsed"); + /// sleep.as_mut().reset(Instant::now() + Duration::from_millis(50)); + /// }, + /// } + /// } + /// } + /// ``` + /// Use in a struct with boxing. By pinning the `Sleep` with a `Box`, the + /// `HasSleep` struct implements `Unpin`, even though `Sleep` does not. + /// ``` + /// use std::future::Future; + /// use std::pin::Pin; + /// use std::task::{Context, Poll}; + /// use monoio::time::Sleep; + /// + /// struct HasSleep { + /// sleep: Pin>, + /// } + /// + /// impl Future for HasSleep { + /// type Output = (); + /// + /// fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<()> { + /// self.sleep.as_mut().poll(cx) + /// } + /// } + /// ``` + /// Use in a struct with pin projection. This method avoids the `Box`, but + /// the `HasSleep` struct will not be `Unpin` as a consequence. + /// ``` + /// use std::future::Future; + /// use std::pin::Pin; + /// use std::task::{Context, Poll}; + /// use monoio::time::Sleep; + /// use pin_project_lite::pin_project; + /// + /// pin_project! { + /// struct HasSleep { + /// #[pin] + /// sleep: Sleep, + /// } + /// } + /// + /// impl Future for HasSleep { + /// type Output = (); + /// + /// fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<()> { + /// self.project().sleep.poll(cx) + /// } + /// } + /// ``` + /// + /// [`select!`]: ../macro.select.html + /// [`monoio::pin!`]: ../macro.pin.html + #[cfg_attr(docsrs, doc(alias = "Delay"))] + #[derive(Debug)] + #[must_use = "futures do nothing unless you `.await` or poll them"] + pub struct Sleep { + deadline: Instant, + + // The link between the `Sleep` instance and the timer that drives it. + #[pin] + entry: TimerEntry, + } +} + +impl Sleep { + pub(crate) fn new_timeout(deadline: Instant) -> Sleep { + let handle = Handle::current(); + let entry = TimerEntry::new(&handle, deadline); + + Sleep { deadline, entry } + } + + pub(crate) fn far_future() -> Sleep { + Self::new_timeout(Instant::far_future()) + } + + /// Returns the instant at which the future will complete. + pub fn deadline(&self) -> Instant { + self.deadline + } + + /// Returns `true` if `Sleep` has elapsed. + /// + /// A `Sleep` instance is elapsed when the requested duration has elapsed. + pub fn is_elapsed(&self) -> bool { + self.entry.is_elapsed() + } + + /// Resets the `Sleep` instance to a new deadline. + /// + /// Calling this function allows changing the instant at which the `Sleep` + /// future completes without having to create new associated state. + /// + /// This function can be called both before and after the future has + /// completed. + /// + /// To call this method, you will usually combine the call with + /// [`Pin::as_mut`], which lets you call the method without consuming the + /// `Sleep` itself. + /// + /// # Example + /// + /// ``` + /// use monoio::time::{Duration, Instant}; + /// + /// # #[monoio::main(timer_enabled = true)] + /// # async fn main() { + /// let sleep = monoio::time::sleep(Duration::from_millis(10)); + /// monoio::pin!(sleep); + /// + /// sleep + /// .as_mut() + /// .reset(Instant::now() + Duration::from_millis(20)); + /// # } + /// ``` + /// + /// See also the top-level examples. + /// + /// [`Pin::as_mut`]: fn@std::pin::Pin::as_mut + pub fn reset(self: Pin<&mut Self>, deadline: Instant) { + let me = self.project(); + me.entry.reset(deadline); + *me.deadline = deadline; + } + + fn poll_elapsed(self: Pin<&mut Self>, cx: &mut task::Context<'_>) -> Poll> { + let me = self.project(); + me.entry.poll_elapsed(cx) + } +} + +impl Future for Sleep { + type Output = (); + + fn poll(mut self: Pin<&mut Self>, cx: &mut task::Context<'_>) -> Poll { + // `poll_elapsed` can return an error in two cases: + // + // - AtCapacity: this is a pathological case where far too many sleep instances have been + // scheduled. + // - Shutdown: No timer has been setup, which is a mis-use error. + // + // Both cases are extremely rare, and pretty accurately fit into + // "logic errors", so we just panic in this case. A user couldn't + // really do much better if we passed the error onwards. + match ready!(self.as_mut().poll_elapsed(cx)) { + Ok(()) => Poll::Ready(()), + Err(e) => panic!("timer error: {e}"), + } + } +} diff --git a/vendor/monoio/src/time/driver/wheel/level.rs b/vendor/monoio/src/time/driver/wheel/level.rs new file mode 100644 index 000000000..3d53aec11 --- /dev/null +++ b/vendor/monoio/src/time/driver/wheel/level.rs @@ -0,0 +1,277 @@ +use std::{fmt, ptr::NonNull}; + +use crate::time::driver::{EntryList, TimerHandle, TimerShared}; + +/// Wheel for a single level in the timer. This wheel contains 64 slots. +pub(crate) struct Level { + level: usize, + slot_range: u64, + level_range: u64, + + /// Bit field tracking which slots currently contain entries. + /// + /// Using a bit field to track slots that contain entries allows avoiding a + /// scan to find entries. This field is updated when entries are added or + /// removed from a slot. + /// + /// The least-significant bit represents slot zero. + occupied: u64, + + /// Slots. We access these via the EntryInner `current_list` as well, so + /// this needs to be an UnsafeCell. + slot: [EntryList; LEVEL_MULT], +} + +/// Indicates when a slot must be processed next. +#[derive(Debug)] +pub(crate) struct Expiration { + /// The level containing the slot. + pub(crate) level: usize, + + /// The slot index. + pub(crate) slot: usize, + + /// The instant at which the slot needs to be processed. + pub(crate) deadline: u64, +} + +/// Level multiplier. +/// +/// Being a power of 2 is very important. +const LEVEL_MULT: usize = 64; + +impl Level { + pub(crate) fn new(level: usize) -> Level { + // A value has to be Copy in order to use syntax like: + // let stack = Stack::default(); + // ... + // slots: [stack; 64], + // + // Alternatively, since Stack is Default one can + // use syntax like: + // let slots: [Stack; 64] = Default::default(); + // + // However, that is only supported for arrays of size + // 32 or fewer. So in our case we have to explicitly + // invoke the constructor for each array element. + let ctor = EntryList::default; + + Level { + level, + slot_range: slot_range(level), + level_range: level_range(level), + occupied: 0, + slot: [ + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ctor(), + ], + } + } + + /// Finds the slot that needs to be processed next and returns the slot and + /// `Instant` at which this slot must be processed. + pub(crate) fn next_expiration(&self, now: u64) -> Option { + // Use the `occupied` bit field to get the index of the next slot that + // needs to be processed. + let slot = match self.next_occupied_slot(now) { + Some(slot) => slot, + None => return None, + }; + + // From the slot index, calculate the `Instant` at which it needs to be + // processed. This value *must* be in the future with respect to `now`. + + // Compute the start date of the current level by masking the low bits + // of `now` (`level_range` is a power of 2). + let level_start = now & !(self.level_range - 1); + let mut deadline = level_start + slot as u64 * self.slot_range; + + if deadline <= now { + // A timer is in a slot "prior" to the current time. This can occur + // because we do not have an infinite hierarchy of timer levels, and + // eventually a timer scheduled for a very distant time might end up + // being placed in a slot that is beyond the end of all of the + // arrays. + // + // To deal with this, we first limit timers to being scheduled no + // more than MAX_DURATION ticks in the future; that is, they're at + // most one rotation of the top level away. Then, we force timers + // that logically would go into the top+1 level, to instead go into + // the top level's slots. + // + // What this means is that the top level's slots act as a + // pseudo-ring buffer, and we rotate around them indefinitely. If we + // compute a deadline before now, and it's the top level, it + // therefore means we're actually looking at a slot in the future. + debug_assert_eq!(self.level, super::NUM_LEVELS - 1); + + deadline += self.level_range; + } + + debug_assert!( + deadline >= now, + "deadline={:016X}; now={:016X}; level={}; lr={:016X}, sr={:016X}, slot={}; \ + occupied={:b}", + deadline, + now, + self.level, + self.level_range, + self.slot_range, + slot, + self.occupied + ); + + Some(Expiration { + level: self.level, + slot, + deadline, + }) + } + + fn next_occupied_slot(&self, now: u64) -> Option { + if self.occupied == 0 { + return None; + } + + // Get the slot for now using Maths + let now_slot = (now / self.slot_range) as usize; + let occupied = self.occupied.rotate_right(now_slot as u32); + let zeros = occupied.trailing_zeros() as usize; + let slot = (zeros + now_slot) % 64; + + Some(slot) + } + + pub(crate) unsafe fn add_entry(&mut self, item: TimerHandle) { + let slot = slot_for(item.cached_when(), self.level); + + self.slot[slot].push_front(item); + + self.occupied |= occupied_bit(slot); + } + + pub(crate) unsafe fn remove_entry(&mut self, item: NonNull) { + let slot = slot_for(unsafe { item.as_ref().cached_when() }, self.level); + + unsafe { self.slot[slot].remove(item) }; + if self.slot[slot].is_empty() { + // The bit is currently set + debug_assert!(self.occupied & occupied_bit(slot) != 0); + + // Unset the bit + self.occupied ^= occupied_bit(slot); + } + } + + pub(crate) fn take_slot(&mut self, slot: usize) -> EntryList { + self.occupied &= !occupied_bit(slot); + + std::mem::take(&mut self.slot[slot]) + } +} + +impl fmt::Debug for Level { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + fmt.debug_struct("Level") + .field("occupied", &self.occupied) + .finish() + } +} + +fn occupied_bit(slot: usize) -> u64 { + 1 << slot +} + +fn slot_range(level: usize) -> u64 { + LEVEL_MULT.pow(level as u32) as u64 +} + +fn level_range(level: usize) -> u64 { + LEVEL_MULT as u64 * slot_range(level) +} + +/// Convert a duration (milliseconds) and a level to a slot position +fn slot_for(duration: u64, level: usize) -> usize { + ((duration >> (level * 6)) % LEVEL_MULT as u64) as usize +} + +#[cfg(all(test, not(loom)))] +mod test { + use super::*; + + #[test] + fn test_slot_for() { + for pos in 0..64 { + assert_eq!(pos as usize, slot_for(pos, 0)); + } + + for level in 1..5 { + for pos in level..64 { + let a = pos * 64_usize.pow(level as u32); + assert_eq!(pos, slot_for(a as u64, level)); + } + } + } +} diff --git a/vendor/monoio/src/time/driver/wheel/mod.rs b/vendor/monoio/src/time/driver/wheel/mod.rs new file mode 100644 index 000000000..5003e83b2 --- /dev/null +++ b/vendor/monoio/src/time/driver/wheel/mod.rs @@ -0,0 +1,345 @@ +use crate::time::{ + driver::{TimerHandle, TimerShared}, + error::InsertError, +}; + +mod level; +use std::ptr::NonNull; + +pub(crate) use self::level::Expiration; +use self::level::Level; +use super::EntryList; + +/// Timing wheel implementation. +/// +/// This type provides the hashed timing wheel implementation that backs `Timer` +/// and `DelayQueue`. +/// +/// The structure is generic over `T: Stack`. This allows handling timeout data +/// being stored on the heap or in a slab. In order to support the latter case, +/// the slab must be passed into each function allowing the implementation to +/// lookup timer entries. +/// +/// See `Timer` documentation for some implementation notes. +#[derive(Debug)] +pub(crate) struct Wheel { + /// The number of milliseconds elapsed since the wheel started. + elapsed: u64, + + /// Timer wheel. + /// + /// Levels: + /// + /// * 1 ms slots / 64 ms range + /// * 64 ms slots / ~ 4 sec range + /// * ~ 4 sec slots / ~ 4 min range + /// * ~ 4 min slots / ~ 4 hr range + /// * ~ 4 hr slots / ~ 12 day range + /// * ~ 12 day slots / ~ 2 yr range + levels: Box<[Level; NUM_LEVELS]>, + + /// Entries queued for firing + pending: EntryList, +} + +/// Number of levels. Each level has 64 slots. By using 6 levels with 64 slots +/// each, the timer is able to track time up to 2 years into the future with a +/// precision of 1 millisecond. +const NUM_LEVELS: usize = 6; + +/// The maximum duration of a `Sleep` +pub(super) const MAX_DURATION: u64 = (1 << (6 * NUM_LEVELS)) - 1; + +impl Wheel { + /// Create a new timing wheel + pub(crate) fn new() -> Wheel { + let levels = Box::new([ + Level::new(0), + Level::new(1), + Level::new(2), + Level::new(3), + Level::new(4), + Level::new(5), + ]); + + Wheel { + elapsed: 0, + levels, + pending: EntryList::new(), + } + } + + /// Return the number of milliseconds that have elapsed since the timing + /// wheel's creation. + pub(crate) fn elapsed(&self) -> u64 { + self.elapsed + } + + /// Insert an entry into the timing wheel. + /// + /// # Arguments + /// + /// * `item`: The item to insert into the wheel. + /// + /// # Return + /// + /// Returns `Ok` when the item is successfully inserted, `Err` otherwise. + /// + /// `Err(Elapsed)` indicates that `when` represents an instant that has + /// already passed. In this case, the caller should fire the timeout + /// immediately. + /// + /// `Err(Invalid)` indicates an invalid `when` argument as been supplied. + /// + /// # Safety + /// + /// This function registers item into an intrusive linked list. The caller + /// must ensure that `item` is pinned and will not be dropped without first + /// being deregistered. + pub(crate) unsafe fn insert( + &mut self, + item: TimerHandle, + ) -> Result { + let when = item.sync_when(); + + if when <= self.elapsed { + return Err((item, InsertError::Elapsed)); + } + + // Get the level at which the entry should be stored + let level = self.level_for(when); + + unsafe { + self.levels.get_unchecked_mut(level).add_entry(item); + } + + debug_assert!({ + unsafe { self.levels.get_unchecked(level) } + .next_expiration(self.elapsed) + .map(|e| e.deadline >= self.elapsed) + .unwrap_or(true) + }); + + Ok(when) + } + + /// Remove `item` from the timing wheel. + pub(crate) unsafe fn remove(&mut self, item: NonNull) { + unsafe { + let when = item.as_ref().cached_when(); + if when == u64::MAX { + self.pending.remove(item); + } else { + debug_assert!( + self.elapsed <= when, + "elapsed={}; when={}", + self.elapsed, + when + ); + + let level = self.level_for(when); + + self.levels.get_unchecked_mut(level).remove_entry(item); + } + } + } + + /// Instant at which to poll + pub(crate) fn poll_at(&self) -> Option { + self.next_expiration().map(|expiration| expiration.deadline) + } + + /// Advances the timer up to the instant represented by `now`. + pub(crate) fn poll(&mut self, now: u64) -> Option { + loop { + if let Some(handle) = self.pending.pop_back() { + return Some(handle); + } + + match self.next_expiration() { + Some(ref expiration) if expiration.deadline <= now => { + self.process_expiration(expiration); + + self.set_elapsed(expiration.deadline); + } + _ => { + // in this case the poll did not indicate an expiration + // _and_ we were not able to find a next expiration in + // the current list of timers. advance to the poll's + // current time and do nothing else. + self.set_elapsed(now); + break; + } + } + } + + self.pending.pop_back() + } + + /// Returns the instant at which the next timeout expires. + fn next_expiration(&self) -> Option { + if !self.pending.is_empty() { + // Expire immediately as we have things pending firing + return Some(Expiration { + level: 0, + slot: 0, + deadline: self.elapsed, + }); + } + + // Check all levels + for level in 0..NUM_LEVELS { + if let Some(expiration) = self.levels[level].next_expiration(self.elapsed) { + // There cannot be any expirations at a higher level that happen + // before this one. + debug_assert!(self.no_expirations_before(level + 1, expiration.deadline)); + + return Some(expiration); + } + } + + None + } + + /// Returns the tick at which this timer wheel next needs to perform some + /// processing, or None if there are no timers registered. + pub(super) fn next_expiration_time(&self) -> Option { + self.next_expiration().map(|ex| ex.deadline) + } + + /// Used for debug assertions + fn no_expirations_before(&self, start_level: usize, before: u64) -> bool { + let mut res = true; + + for l2 in start_level..NUM_LEVELS { + if let Some(e2) = self.levels[l2].next_expiration(self.elapsed) { + if e2.deadline < before { + res = false; + } + } + } + + res + } + + /// iteratively find entries that are between the wheel's current + /// time and the expiration time. for each in that population either + /// queue it for notification (in the case of the last level) or tier + /// it down to the next level (in all other cases). + pub(crate) fn process_expiration(&mut self, expiration: &Expiration) { + // Note that we need to take _all_ of the entries off the list before + // processing any of them. This is important because it's possible that + // those entries might need to be reinserted into the same slot. + // + // This happens only on the highest level, when an entry is inserted + // more than MAX_DURATION into the future. When this happens, we wrap + // around, and process some entries a multiple of MAX_DURATION before + // they actually need to be dropped down a level. We then reinsert them + // back into the same position; we must make sure we don't then process + // those entries again or we'll end up in an infinite loop. + let mut entries = self.take_entries(expiration); + + while let Some(item) = entries.pop_back() { + if expiration.level == 0 { + debug_assert_eq!(unsafe { item.cached_when() }, expiration.deadline); + } + + // Try to expire the entry; this is cheap (doesn't synchronize) if + // the timer is not expired, and updates cached_when. + match unsafe { item.mark_pending(expiration.deadline) } { + Ok(()) => { + // Item was expired + self.pending.push_front(item); + } + Err(expiration_tick) => { + let level = level_for(expiration.deadline, expiration_tick); + unsafe { + self.levels.get_unchecked_mut(level).add_entry(item); + } + } + } + } + } + + fn set_elapsed(&mut self, when: u64) { + assert!( + self.elapsed <= when, + "elapsed={:?}; when={:?}", + self.elapsed, + when + ); + + if when > self.elapsed { + self.elapsed = when; + } + } + + /// Obtains the list of entries that need processing for the given + /// expiration. + fn take_entries(&mut self, expiration: &Expiration) -> EntryList { + unsafe { self.levels.get_unchecked_mut(expiration.level) }.take_slot(expiration.slot) + } + + fn level_for(&self, when: u64) -> usize { + level_for(self.elapsed, when) + } +} + +fn level_for(elapsed: u64, when: u64) -> usize { + const SLOT_MASK: u64 = (1 << 6) - 1; + + // Mask in the trailing bits ignored by the level calculation in order to cap + // the possible leading zeros + let mut masked = elapsed ^ when | SLOT_MASK; + + if masked >= MAX_DURATION { + // Fudge the timer into the top level + masked = MAX_DURATION - 1; + } + + let leading_zeros = masked.leading_zeros() as usize; + let significant = 63 - leading_zeros; + + significant / 6 +} + +#[cfg(all(test, not(loom)))] +mod test { + use super::*; + + #[test] + fn test_level_for() { + for pos in 0..64 { + assert_eq!(0, level_for(0, pos), "level_for({pos}) -- binary = {pos:b}"); + } + + for level in 1..5 { + for pos in level..64 { + let a = pos * 64_usize.pow(level as u32); + assert_eq!( + level, + level_for(0, a as u64), + "level_for({a}) -- binary = {a:b}" + ); + + if pos > level { + let a = a - 1; + assert_eq!( + level, + level_for(0, a as u64), + "level_for({a}) -- binary = {a:b}" + ); + } + + if pos < 64 { + let a = a + 1; + assert_eq!( + level, + level_for(0, a as u64), + "level_for({a}) -- binary = {a:b}" + ); + } + } + } + } +} diff --git a/vendor/monoio/src/time/driver/wheel/stack.rs b/vendor/monoio/src/time/driver/wheel/stack.rs new file mode 100644 index 000000000..e7ed137f5 --- /dev/null +++ b/vendor/monoio/src/time/driver/wheel/stack.rs @@ -0,0 +1,112 @@ +use super::{Item, OwnedItem}; +use crate::time::driver::Entry; + +use std::ptr; + +/// A doubly linked stack +#[derive(Debug)] +pub(crate) struct Stack { + head: Option, +} + +impl Default for Stack { + fn default() -> Stack { + Stack { head: None } + } +} + +impl Stack { + pub(crate) fn is_empty(&self) -> bool { + self.head.is_none() + } + + pub(crate) fn push(&mut self, entry: OwnedItem) { + // Get a pointer to the entry to for the prev link + let ptr: *const Entry = &*entry as *const _; + + // Remove the old head entry + let old = self.head.take(); + + unsafe { + // Ensure the entry is not already in a stack. + debug_assert!((*entry.next_stack.get()).is_none()); + debug_assert!((*entry.prev_stack.get()).is_null()); + + if let Some(ref entry) = old.as_ref() { + debug_assert!({ + // The head is not already set to the entry + ptr != &***entry as *const _ + }); + + // Set the previous link on the old head + *entry.prev_stack.get() = ptr; + } + + // Set this entry's next pointer + *entry.next_stack.get() = old; + } + + // Update the head pointer + self.head = Some(entry); + } + + /// Pops an item from the stack + pub(crate) fn pop(&mut self) -> Option { + let entry = self.head.take(); + + unsafe { + if let Some(entry) = entry.as_ref() { + self.head = (*entry.next_stack.get()).take(); + + if let Some(entry) = self.head.as_ref() { + *entry.prev_stack.get() = ptr::null(); + } + + *entry.prev_stack.get() = ptr::null(); + } + } + + entry + } + + pub(crate) fn remove(&mut self, entry: &Item) { + unsafe { + // Ensure that the entry is in fact contained by the stack + debug_assert!({ + // This walks the full linked list even if an entry is found. + let mut next = self.head.as_ref(); + let mut contains = false; + + while let Some(n) = next { + if entry as *const _ == &**n as *const _ { + debug_assert!(!contains); + contains = true; + } + + next = (*n.next_stack.get()).as_ref(); + } + + contains + }); + + // Unlink `entry` from the next node + let next = (*entry.next_stack.get()).take(); + + if let Some(next) = next.as_ref() { + (*next.prev_stack.get()) = *entry.prev_stack.get(); + } + + // Unlink `entry` from the prev node + + if let Some(prev) = (*entry.prev_stack.get()).as_ref() { + *prev.next_stack.get() = next; + } else { + // It is the head + self.head = next; + } + + // Unset the prev pointer + *entry.prev_stack.get() = ptr::null(); + } + } +} diff --git a/vendor/monoio/src/time/error.rs b/vendor/monoio/src/time/error.rs new file mode 100644 index 000000000..3640a2cd3 --- /dev/null +++ b/vendor/monoio/src/time/error.rs @@ -0,0 +1,118 @@ +//! Time error types. + +use std::{error, fmt}; + +use self::Kind::*; + +/// Errors encountered by the timer implementation. +/// +/// Currently, there are two different errors that can occur: +/// +/// * `shutdown` occurs when a timer operation is attempted, but the timer instance has been +/// dropped. In this case, the operation will never be able to complete and the `shutdown` error +/// is returned. This is a permanent error, i.e., once this error is observed, timer operations +/// will never succeed in the future. +/// +/// * `at_capacity` occurs when a timer operation is attempted, but the timer instance is currently +/// handling its maximum number of outstanding sleep instances. In this case, the operation is not +/// able to be performed at the current moment, and `at_capacity` is returned. This is a transient +/// error, i.e., at some point in the future, if the operation is attempted again, it might +/// succeed. Callers that observe this error should attempt to [shed load]. One way to do this +/// would be dropping the future that issued the timer operation. +/// +/// [shed load]: https://en.wikipedia.org/wiki/Load_Shedding +#[derive(Debug, Copy, Clone)] +pub struct Error(Kind); + +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +#[repr(u8)] +pub(crate) enum Kind { + Shutdown = 1, + AtCapacity = 2, + Invalid = 3, +} + +impl From for Error { + fn from(k: Kind) -> Self { + Error(k) + } +} + +/// Error returned by `Timeout`. +#[derive(Debug, PartialEq, Eq)] +pub struct Elapsed(()); + +#[derive(Debug)] +pub(crate) enum InsertError { + Elapsed, +} + +// ===== impl Error ===== + +impl Error { + /// Creates an error representing a shutdown timer. + pub fn shutdown() -> Error { + Error(Shutdown) + } + + /// Returns `true` if the error was caused by the timer being shutdown. + pub fn is_shutdown(&self) -> bool { + matches!(self.0, Kind::Shutdown) + } + + /// Creates an error representing a timer at capacity. + pub fn at_capacity() -> Error { + Error(AtCapacity) + } + + /// Returns `true` if the error was caused by the timer being at capacity. + pub fn is_at_capacity(&self) -> bool { + matches!(self.0, Kind::AtCapacity) + } + + /// Create an error representing a misconfigured timer. + pub fn invalid() -> Error { + Error(Invalid) + } + + /// Returns `true` if the error was caused by the timer being misconfigured. + pub fn is_invalid(&self) -> bool { + matches!(self.0, Kind::Invalid) + } +} + +impl error::Error for Error {} + +impl fmt::Display for Error { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + use self::Kind::*; + let descr = match self.0 { + Shutdown => "the timer is shutdown, must be called from the context of Monoio runtime", + AtCapacity => "timer is at capacity and cannot create a new entry", + Invalid => "timer duration exceeds maximum duration", + }; + write!(fmt, "{descr}") + } +} + +// ===== impl Elapsed ===== + +impl Elapsed { + pub(crate) fn new() -> Self { + Elapsed(()) + } +} + +impl fmt::Display for Elapsed { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + "deadline has elapsed".fmt(fmt) + } +} + +impl std::error::Error for Elapsed {} + +impl From for std::io::Error { + fn from(_err: Elapsed) -> std::io::Error { + std::io::ErrorKind::TimedOut.into() + } +} diff --git a/vendor/monoio/src/time/instant.rs b/vendor/monoio/src/time/instant.rs new file mode 100644 index 000000000..771cbb9a4 --- /dev/null +++ b/vendor/monoio/src/time/instant.rs @@ -0,0 +1,219 @@ +#![allow(clippy::trivially_copy_pass_by_ref)] + +use std::{fmt, ops, time::Duration}; + +/// A measurement of a monotonically nondecreasing clock. +/// Opaque and useful only with `Duration`. +/// +/// Instants are always guaranteed to be no less than any previously measured +/// instant when created, and are often useful for tasks such as measuring +/// benchmarks or timing how long an operation takes. +/// +/// Note, however, that instants are not guaranteed to be **steady**. In other +/// words, each tick of the underlying clock may not be the same length (e.g. +/// some seconds may be longer than others). An instant may jump forwards or +/// experience time dilation (slow down or speed up), but it will never go +/// backwards. +/// +/// Instants are opaque types that can only be compared to one another. There is +/// no method to get "the number of seconds" from an instant. Instead, it only +/// allows measuring the duration between two instants (or comparing two +/// instants). +/// +/// The size of an `Instant` struct may vary depending on the target operating +/// system. +/// +/// # Note +/// +/// This type wraps the inner `std` variant and is used to align the Monoio +/// clock for uses of `now()`. This can be useful for testing where you can +/// take advantage of `time::pause()` and `time::advance()`. +#[derive(Clone, Copy, Eq, PartialEq, PartialOrd, Ord, Hash)] +pub struct Instant { + std: std::time::Instant, +} + +impl Instant { + /// Returns an instant corresponding to "now". + /// + /// # Examples + /// + /// ``` + /// use monoio::time::Instant; + /// + /// let now = Instant::now(); + /// ``` + pub fn now() -> Instant { + variant::now() + } + + /// Create a `monoio::time::Instant` from a `std::time::Instant`. + pub fn from_std(std: std::time::Instant) -> Instant { + Instant { std } + } + + pub(crate) fn far_future() -> Instant { + // Roughly 30 years from now. + // API does not provide a way to obtain max `Instant` + // or convert specific date in the future to instant. + // 1000 years overflows on macOS, 100 years overflows on FreeBSD. + Self::now() + Duration::from_secs(86400 * 365 * 30) + } + + /// Convert the value into a `std::time::Instant`. + pub fn into_std(self) -> std::time::Instant { + self.std + } + + /// Returns the amount of time elapsed from another instant to this one. + /// + /// # Panics + /// + /// This function will panic if `earlier` is later than `self`. + pub fn duration_since(&self, earlier: Instant) -> Duration { + self.std.duration_since(earlier.std) + } + + /// Returns the amount of time elapsed from another instant to this one, or + /// None if that instant is later than this one. + /// + /// # Examples + /// + /// ``` + /// use monoio::time::{sleep, Duration, Instant}; + /// + /// #[monoio::main(timer_enabled = true)] + /// async fn main() { + /// let now = Instant::now(); + /// sleep(Duration::new(0, 50)).await; + /// let new_now = Instant::now(); + /// println!("{:?}", new_now.checked_duration_since(now)); + /// println!("{:?}", now.checked_duration_since(new_now)); // None + /// } + /// ``` + pub fn checked_duration_since(&self, earlier: Instant) -> Option { + self.std.checked_duration_since(earlier.std) + } + + /// Returns the amount of time elapsed from another instant to this one, or + /// zero duration if that instant is later than this one. + /// + /// # Examples + /// + /// ``` + /// use monoio::time::{sleep, Duration, Instant}; + /// + /// #[monoio::main(timer_enabled = true)] + /// async fn main() { + /// let now = Instant::now(); + /// sleep(Duration::new(0, 10)).await; + /// let new_now = Instant::now(); + /// println!("{:?}", new_now.saturating_duration_since(now)); + /// println!("{:?}", now.saturating_duration_since(new_now)); // 0ns + /// } + /// ``` + pub fn saturating_duration_since(&self, earlier: Instant) -> Duration { + self.std.saturating_duration_since(earlier.std) + } + + /// Returns the amount of time elapsed since this instant was created. + /// + /// # Panics + /// + /// This function may panic if the current time is earlier than this + /// instant, which is something that can happen if an `Instant` is + /// produced synthetically. + /// + /// # Examples + /// + /// ``` + /// use monoio::time::{sleep, Duration, Instant}; + /// + /// #[monoio::main(timer_enabled = true)] + /// async fn main() { + /// let instant = Instant::now(); + /// let three_secs = Duration::from_millis(30); + /// sleep(three_secs).await; + /// assert!(instant.elapsed() >= three_secs); + /// } + /// ``` + pub fn elapsed(&self) -> Duration { + Instant::now() - *self + } + + /// Returns `Some(t)` where `t` is the time `self + duration` if `t` can be + /// represented as `Instant` (which means it's inside the bounds of the + /// underlying data structure), `None` otherwise. + pub fn checked_add(&self, duration: Duration) -> Option { + self.std.checked_add(duration).map(Instant::from_std) + } + + /// Returns `Some(t)` where `t` is the time `self - duration` if `t` can be + /// represented as `Instant` (which means it's inside the bounds of the + /// underlying data structure), `None` otherwise. + pub fn checked_sub(&self, duration: Duration) -> Option { + self.std.checked_sub(duration).map(Instant::from_std) + } +} + +impl From for Instant { + fn from(time: std::time::Instant) -> Instant { + Instant::from_std(time) + } +} + +impl From for std::time::Instant { + fn from(time: Instant) -> std::time::Instant { + time.into_std() + } +} + +impl ops::Add for Instant { + type Output = Instant; + + fn add(self, other: Duration) -> Instant { + Instant::from_std(self.std + other) + } +} + +impl ops::AddAssign for Instant { + fn add_assign(&mut self, rhs: Duration) { + *self = *self + rhs; + } +} + +impl ops::Sub for Instant { + type Output = Duration; + + fn sub(self, rhs: Instant) -> Duration { + self.std - rhs.std + } +} + +impl ops::Sub for Instant { + type Output = Instant; + + fn sub(self, rhs: Duration) -> Instant { + Instant::from_std(self.std.checked_sub(rhs).unwrap()) + } +} + +impl ops::SubAssign for Instant { + fn sub_assign(&mut self, rhs: Duration) { + *self = *self - rhs; + } +} + +impl fmt::Debug for Instant { + fn fmt(&self, fmt: &mut fmt::Formatter<'_>) -> fmt::Result { + self.std.fmt(fmt) + } +} + +mod variant { + use super::Instant; + + pub(super) fn now() -> Instant { + Instant::from_std(std::time::Instant::now()) + } +} diff --git a/vendor/monoio/src/time/interval.rs b/vendor/monoio/src/time/interval.rs new file mode 100644 index 000000000..2a28802d0 --- /dev/null +++ b/vendor/monoio/src/time/interval.rs @@ -0,0 +1,447 @@ +use std::{ + future::Future, + pin::Pin, + task::{Context, Poll}, +}; + +use crate::{ + macros::support::poll_fn, + time::{sleep_until, Duration, Instant, Sleep}, +}; + +/// Creates new [`Interval`] that yields with interval of `period`. The first +/// tick completes immediately. The default [`MissedTickBehavior`] is +/// [`Burst`](MissedTickBehavior::Burst), but this can be configured +/// by calling [`set_missed_tick_behavior`](Interval::set_missed_tick_behavior). +/// +/// An interval will tick indefinitely. At any time, the [`Interval`] value can +/// be dropped. This cancels the interval. +/// +/// This function is equivalent to +/// [`interval_at(Instant::now(), period)`](interval_at). +/// +/// # Panics +/// +/// This function panics if `period` is zero. +/// +/// # Examples +/// +/// ``` +/// use monoio::time::{self, Duration}; +/// +/// #[monoio::main(timer_enabled = true)] +/// async fn main() { +/// let mut interval = time::interval(Duration::from_millis(10)); +/// +/// interval.tick().await; // ticks immediately +/// interval.tick().await; // ticks after 10ms +/// interval.tick().await; // ticks after 10ms +/// +/// // approximately 20ms have elapsed. +/// } +/// ``` +/// +/// A simple example using `interval` to execute a task every two seconds. +/// +/// The difference between `interval` and [`sleep`] is that an [`Interval`] +/// measures the time since the last tick, which means that [`.tick().await`] +/// may wait for a shorter time than the duration specified for the interval +/// if some time has passed between calls to [`.tick().await`]. +/// +/// If the tick in the example below was replaced with [`sleep`], the task +/// would only be executed once every three seconds, and not every two +/// seconds. +/// +/// ``` +/// use monoio::time; +/// +/// async fn task_that_takes_a_second() { +/// println!("hello"); +/// time::sleep(time::Duration::from_secs(1)).await +/// } +/// +/// #[monoio::main(timer_enabled = true)] +/// async fn main() { +/// let mut interval = time::interval(time::Duration::from_secs(2)); +/// for _i in 0..5 { +/// interval.tick().await; +/// task_that_takes_a_second().await; +/// } +/// } +/// ``` +/// +/// [`sleep`]: crate::time::sleep() +/// [`.tick().await`]: Interval::tick +pub fn interval(period: Duration) -> Interval { + assert!(period > Duration::new(0, 0), "`period` must be non-zero."); + + interval_at(Instant::now(), period) +} + +/// Creates new [`Interval`] that yields with interval of `period` with the +/// first tick completing at `start`. The default [`MissedTickBehavior`] is +/// [`Burst`](MissedTickBehavior::Burst), but this can be configured +/// by calling [`set_missed_tick_behavior`](Interval::set_missed_tick_behavior). +/// +/// An interval will tick indefinitely. At any time, the [`Interval`] value can +/// be dropped. This cancels the interval. +/// +/// # Panics +/// +/// This function panics if `period` is zero. +/// +/// # Examples +/// +/// ``` +/// use monoio::time::{interval_at, Duration, Instant}; +/// +/// #[monoio::main(timer_enabled = true)] +/// async fn main() { +/// let start = Instant::now() + Duration::from_millis(50); +/// let mut interval = interval_at(start, Duration::from_millis(10)); +/// +/// interval.tick().await; // ticks after 50ms +/// interval.tick().await; // ticks after 10ms +/// interval.tick().await; // ticks after 10ms +/// +/// // approximately 70ms have elapsed. +/// } +/// ``` +pub fn interval_at(start: Instant, period: Duration) -> Interval { + assert!(period > Duration::new(0, 0), "`period` must be non-zero."); + + Interval { + delay: Box::pin(sleep_until(start)), + period, + missed_tick_behavior: Default::default(), + } +} + +/// Defines the behavior of an [`Interval`] when it misses a tick. +/// +/// Sometimes, an [`Interval`]'s tick is missed. For example, consider the +/// following: +/// +/// ``` +/// use monoio::time::{self, Duration}; +/// # async fn task_that_takes_one_to_three_millis() {} +/// +/// #[monoio::main(timer_enabled = true)] +/// async fn main() { +/// // ticks every 2 milliseconds +/// let mut interval = time::interval(Duration::from_millis(2)); +/// for _ in 0..5 { +/// interval.tick().await; +/// // if this takes more than 2 milliseconds, a tick will be delayed +/// task_that_takes_one_to_three_millis().await; +/// } +/// } +/// ``` +/// +/// Generally, a tick is missed if too much time is spent without calling +/// [`Interval::tick()`]. +/// +/// By default, when a tick is missed, [`Interval`] fires ticks as quickly as it +/// can until it is "caught up" in time to where it should be. +/// `MissedTickBehavior` can be used to specify a different behavior for +/// [`Interval`] to exhibit. Each variant represents a different strategy. +/// +/// Note that because the executor cannot guarantee exact precision with timers, +/// these strategies will only apply when the delay is greater than 5 +/// milliseconds. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum MissedTickBehavior { + /// Tick as fast as possible until caught up. + /// + /// When this strategy is used, [`Interval`] schedules ticks "normally" (the + /// same as it would have if the ticks hadn't been delayed), which results + /// in it firing ticks as fast as possible until it is caught up in time to + /// where it should be. Unlike [`Delay`] and [`Skip`], the ticks yielded + /// when `Burst` is used (the [`Instant`]s that [`tick`](Interval::tick) + /// yields) aren't different than they would have been if a tick had not + /// been missed. Like [`Skip`], and unlike [`Delay`], the ticks may be + /// shortened. + /// + /// This looks something like this: + /// ```text + /// Expected ticks: | 1 | 2 | 3 | 4 | 5 | 6 | + /// Actual ticks: | work -----| delay | work | work | work -| work -----| + /// ``` + /// + /// In code: + /// + /// ``` + /// use monoio::time::{interval, Duration}; + /// # async fn task_that_takes_200_millis() {} + /// + /// # #[monoio::main(timer_enabled = true)] + /// # async fn main() { + /// let mut interval = interval(Duration::from_millis(50)); + /// + /// task_that_takes_200_millis().await; + /// // The `Interval` has missed a tick + /// + /// // Since we have exceeded our timeout, this will resolve immediately + /// interval.tick().await; + /// + /// // Since we are more than 100ms after the start of `interval`, this will + /// // also resolve immediately. + /// interval.tick().await; + /// + /// // Also resolves immediately, because it was supposed to resolve at + /// // 150ms after the start of `interval` + /// interval.tick().await; + /// + /// // Resolves immediately + /// interval.tick().await; + /// + /// // Since we have gotten to 200ms after the start of `interval`, this + /// // will resolve after 50ms + /// interval.tick().await; + /// # } + /// ``` + /// + /// This is the default behavior when [`Interval`] is created with + /// [`interval`] and [`interval_at`]. + /// + /// [`Delay`]: MissedTickBehavior::Delay + /// [`Skip`]: MissedTickBehavior::Skip + Burst, + + /// Tick at multiples of `period` from when [`tick`] was called, rather than + /// from `start`. + /// + /// When this strategy is used and [`Interval`] has missed a tick, instead + /// of scheduling ticks to fire at multiples of `period` from `start` (the + /// time when the first tick was fired), it schedules all future ticks to + /// happen at a regular `period` from the point when [`tick`] was called. + /// Unlike [`Burst`] and [`Skip`], ticks are not shortened, and they aren't + /// guaranteed to happen at a multiple of `period` from `start` any longer. + /// + /// This looks something like this: + /// ```text + /// Expected ticks: | 1 | 2 | 3 | 4 | 5 | 6 | + /// Actual ticks: | work -----| delay | work -----| work -----| work -----| + /// ``` + /// + /// In code: + /// + /// ``` + /// use monoio::time::{interval, Duration, MissedTickBehavior}; + /// # async fn task_that_takes_more_than_50_millis() {} + /// + /// # #[monoio::main(timer_enabled = true)] + /// # async fn main() { + /// let mut interval = interval(Duration::from_millis(50)); + /// interval.set_missed_tick_behavior(MissedTickBehavior::Delay); + /// + /// task_that_takes_more_than_50_millis().await; + /// // The `Interval` has missed a tick + /// + /// // Since we have exceeded our timeout, this will resolve immediately + /// interval.tick().await; + /// + /// // But this one, rather than also resolving immediately, as might happen + /// // with the `Burst` or `Skip` behaviors, will not resolve until + /// // 50ms after the call to `tick` up above. That is, in `tick`, when we + /// // recognize that we missed a tick, we schedule the next tick to happen + /// // 50ms (or whatever the `period` is) from right then, not from when + /// // were were *supposed* to tick + /// interval.tick().await; + /// # } + /// ``` + /// + /// [`Burst`]: MissedTickBehavior::Burst + /// [`Skip`]: MissedTickBehavior::Skip + /// [`tick`]: Interval::tick + Delay, + + /// Skip missed ticks and tick on the next multiple of `period` from + /// `start`. + /// + /// When this strategy is used, [`Interval`] schedules the next tick to fire + /// at the next-closest tick that is a multiple of `period` away from + /// `start` (the point where [`Interval`] first ticked). Like [`Burst`], all + /// ticks remain multiples of `period` away from `start`, but unlike + /// [`Burst`], the ticks may not be *one* multiple of `period` away from the + /// last tick. Like [`Delay`], the ticks are no longer the same as they + /// would have been if ticks had not been missed, but unlike [`Delay`], and + /// like [`Burst`], the ticks may be shortened to be less than one `period` + /// away from each other. + /// + /// This looks something like this: + /// ```text + /// Expected ticks: | 1 | 2 | 3 | 4 | 5 | 6 | + /// Actual ticks: | work -----| delay | work ---| work -----| work -----| + /// ``` + /// + /// In code: + /// + /// ``` + /// use monoio::time::{interval, Duration, MissedTickBehavior}; + /// # async fn task_that_takes_75_millis() {} + /// + /// # #[monoio::main(timer_enabled = true)] + /// # async fn main() { + /// let mut interval = interval(Duration::from_millis(50)); + /// interval.set_missed_tick_behavior(MissedTickBehavior::Skip); + /// + /// task_that_takes_75_millis().await; + /// // The `Interval` has missed a tick + /// + /// // Since we have exceeded our timeout, this will resolve immediately + /// interval.tick().await; + /// + /// // This one will resolve after 25ms, 100ms after the start of + /// // `interval`, which is the closest multiple of `period` from the start + /// // of `interval` after the call to `tick` up above. + /// interval.tick().await; + /// # } + /// ``` + /// + /// [`Burst`]: MissedTickBehavior::Burst + /// [`Delay`]: MissedTickBehavior::Delay + Skip, +} + +impl MissedTickBehavior { + /// If a tick is missed, this method is called to determine when the next + /// tick should happen. + fn next_timeout(&self, timeout: Instant, now: Instant, period: Duration) -> Instant { + match self { + Self::Burst => timeout + period, + Self::Delay => now + period, + Self::Skip => { + now + period + - Duration::from_nanos( + ((now - timeout).as_nanos() % period.as_nanos()) + .try_into() + // This operation is practically guaranteed not to + // fail, as in order for it to fail, `period` would + // have to be longer than `now - timeout`, and both + // would have to be longer than 584 years. + // + // If it did fail, there's not a good way to pass + // the error along to the user, so we just panic. + .expect( + "too much time has elapsed since the interval was supposed to tick", + ), + ) + } + } + } +} + +impl Default for MissedTickBehavior { + /// Returns [`MissedTickBehavior::Burst`]. + /// + /// For most usecases, the [`Burst`] strategy is what is desired. + /// Additionally, to preserve backwards compatibility, the [`Burst`] + /// strategy must be the default. For these reasons, + /// [`MissedTickBehavior::Burst`] is the default for [`MissedTickBehavior`]. + /// See [`Burst`] for more details. + /// + /// [`Burst`]: MissedTickBehavior::Burst + fn default() -> Self { + Self::Burst + } +} + +/// Interval returned by [`interval`] and [`interval_at`] +/// +/// This type allows you to wait on a sequence of instants with a certain +/// duration between each instant. Unlike calling [`sleep`] in a loop, this lets +/// you count the time spent between the calls to [`sleep`] as well. +#[derive(Debug)] +pub struct Interval { + /// Future that completes the next time the `Interval` yields a value. + delay: Pin>, + + /// The duration between values yielded by `Interval`. + period: Duration, + + /// The strategy `Interval` should use when a tick is missed. + missed_tick_behavior: MissedTickBehavior, +} + +impl Interval { + /// Completes when the next instant in the interval has been reached. + /// + /// # Examples + /// + /// ``` + /// use std::time::Duration; + /// + /// use monoio::time; + /// + /// #[monoio::main(timer_enabled = true)] + /// async fn main() { + /// let mut interval = time::interval(Duration::from_millis(10)); + /// + /// interval.tick().await; + /// interval.tick().await; + /// interval.tick().await; + /// + /// // approximately 20ms have elapsed. + /// } + /// ``` + pub async fn tick(&mut self) -> Instant { + poll_fn(|cx| self.poll_tick(cx)).await + } + + /// Poll for the next instant in the interval to be reached. + /// + /// This method can return the following values: + /// + /// * `Poll::Pending` if the next instant has not yet been reached. + /// * `Poll::Ready(instant)` if the next instant has been reached. + /// + /// When this method returns `Poll::Pending`, the current task is scheduled + /// to receive a wakeup when the instant has elapsed. Note that on multiple + /// calls to `poll_tick`, only the [`Waker`](std::task::Waker) from the + /// [`Context`] passed to the most recent call is scheduled to receive a + /// wakeup. + pub fn poll_tick(&mut self, cx: &mut Context<'_>) -> Poll { + // Wait for the delay to be done + ready!(Pin::new(&mut self.delay).poll(cx)); + + // Get the time when we were scheduled to tick + let timeout = self.delay.deadline(); + + let now = Instant::now(); + + // If a tick was not missed, and thus we are being called before the + // next tick is due, just schedule the next tick normally, one `period` + // after `timeout` + // + // However, if a tick took excessively long and we are now behind, + // schedule the next tick according to how the user specified with + // `MissedTickBehavior` + let next = if now > timeout + Duration::from_millis(5) { + self.missed_tick_behavior + .next_timeout(timeout, now, self.period) + } else { + timeout + self.period + }; + + self.delay.as_mut().reset(next); + + // Return the time when we were scheduled to tick + Poll::Ready(timeout) + } + + /// Returns the [`MissedTickBehavior`] strategy currently being used. + pub fn missed_tick_behavior(&self) -> MissedTickBehavior { + self.missed_tick_behavior + } + + /// Sets the [`MissedTickBehavior`] strategy that should be used. + pub fn set_missed_tick_behavior(&mut self, behavior: MissedTickBehavior) { + self.missed_tick_behavior = behavior; + } + + /// Returns the period of the interval. + pub fn period(&self) -> Duration { + self.period + } +} diff --git a/vendor/monoio/src/time/mod.rs b/vendor/monoio/src/time/mod.rs new file mode 100644 index 000000000..008a71333 --- /dev/null +++ b/vendor/monoio/src/time/mod.rs @@ -0,0 +1,114 @@ +//! Utilities for tracking time. +//! +//! This module provides a number of types for executing code after a set period +//! of time. +//! +//! * [`Sleep`] is a future that does no work and completes at a specific [`Instant`] in time. +//! +//! * [`Interval`] is a stream yielding a value at a fixed period. It is initialized with a +//! [`Duration`] and repeatedly yields each time the duration elapses. +//! +//! * [`Timeout`]: Wraps a future or stream, setting an upper bound to the amount of time it is +//! allowed to execute. If the future or stream does not complete in time, then it is canceled and +//! an error is returned. +//! +//! These types are sufficient for handling a large number of scenarios +//! involving time. +//! +//! These types must be used from within the context of the +//! [`Runtime`](crate::runtime::Runtime). +//! +//! # Examples +//! +//! Wait 100ms and print "100 ms have elapsed" +//! +//! ``` +//! use std::time::Duration; +//! +//! use monoio::time::sleep; +//! +//! #[monoio::main(timer_enabled = true)] +//! async fn main() { +//! sleep(Duration::from_millis(100)).await; +//! println!("100 ms have elapsed"); +//! } +//! ``` +//! +//! Require that an operation takes no more than 1s. +//! +//! ``` +//! use monoio::time::{timeout, Duration}; +//! +//! async fn long_future() { +//! // do work here +//! } +//! +//! # async fn dox() { +//! let res = timeout(Duration::from_secs(1), long_future()).await; +//! +//! if res.is_err() { +//! println!("operation timed out"); +//! } +//! # } +//! ``` +//! +//! A simple example using [`interval`] to execute a task every two seconds. +//! +//! The difference between [`interval`] and [`sleep`] is that an [`interval`] +//! measures the time since the last tick, which means that `.tick().await` may +//! wait for a shorter time than the duration specified for the interval +//! if some time has passed between calls to `.tick().await`. +//! +//! If the tick in the example below was replaced with [`sleep`], the task +//! would only be executed once every three seconds, and not every two +//! seconds. +//! +//! ``` +//! use monoio::time; +//! +//! async fn task_that_takes_a_second() { +//! println!("hello"); +//! time::sleep(time::Duration::from_secs(1)).await +//! } +//! +//! #[monoio::main(timer_enabled = true)] +//! async fn main() { +//! let mut interval = time::interval(time::Duration::from_secs(2)); +//! for _i in 0..5 { +//! interval.tick().await; +//! task_that_takes_a_second().await; +//! } +//! } +//! ``` +//! +//! [`interval`]: crate::time::interval() + +// Heavily borrowed from tokio. +// Copyright (c) 2021 Tokio Contributors, licensed under the MIT license. + +mod clock; +pub(crate) use self::clock::Clock; + +pub(crate) mod driver; + +#[doc(inline)] +pub use driver::{ + sleep::{sleep, sleep_until, Sleep}, + TimeDriver, +}; + +pub mod error; + +mod instant; +pub use self::instant::Instant; + +mod interval; +pub use interval::{interval, interval_at, Interval, MissedTickBehavior}; + +mod timeout; +// Re-export for convenience +#[doc(no_inline)] +pub use std::time::Duration; + +#[doc(inline)] +pub use timeout::{timeout, timeout_at, Timeout}; diff --git a/vendor/monoio/src/time/timeout.rs b/vendor/monoio/src/time/timeout.rs new file mode 100644 index 000000000..40a4bed86 --- /dev/null +++ b/vendor/monoio/src/time/timeout.rs @@ -0,0 +1,119 @@ +//! Allows a future to execute for a maximum amount of time. +//! +//! See [`Timeout`] documentation for more details. +//! +//! [`Timeout`]: struct@Timeout + +use std::{ + future::Future, + pin::Pin, + task::{self, Poll}, +}; + +use pin_project_lite::pin_project; + +use crate::time::{error::Elapsed, sleep_until, Duration, Instant, Sleep}; + +/// Require a `Future` to complete before the specified duration has elapsed. +/// +/// If the future completes before the duration has elapsed, then the completed +/// value is returned. Otherwise, an error is returned and the future is +/// canceled. +/// +/// # Cancelation +/// +/// Cancelling a timeout is done by dropping the future. No additional cleanup +/// or other work is required. +/// +/// The original future may be obtained by calling [`Timeout::into_inner`]. This +/// consumes the `Timeout`. +pub fn timeout(duration: Duration, future: T) -> Timeout +where + T: Future, +{ + let deadline = Instant::now().checked_add(duration); + let delay = match deadline { + Some(deadline) => Sleep::new_timeout(deadline), + None => Sleep::far_future(), + }; + Timeout::new_with_delay(future, delay) +} + +/// Require a `Future` to complete before the specified instant in time. +/// +/// If the future completes before the instant is reached, then the completed +/// value is returned. Otherwise, an error is returned. +/// +/// # Cancelation +/// +/// Cancelling a timeout is done by dropping the future. No additional cleanup +/// or other work is required. +/// +/// The original future may be obtained by calling [`Timeout::into_inner`]. This +/// consumes the `Timeout`. +pub fn timeout_at(deadline: Instant, future: T) -> Timeout +where + T: Future, +{ + let delay = sleep_until(deadline); + + Timeout { + value: future, + delay, + } +} + +pin_project! { + /// Future returned by [`timeout`](timeout) and [`timeout_at`](timeout_at). + #[must_use = "futures do nothing unless you `.await` or poll them"] + #[derive(Debug)] + pub struct Timeout { + #[pin] + value: T, + #[pin] + delay: Sleep, + } +} + +impl Timeout { + pub(crate) fn new_with_delay(value: T, delay: Sleep) -> Timeout { + Timeout { value, delay } + } + + /// Gets a reference to the underlying value in this timeout. + pub fn get_ref(&self) -> &T { + &self.value + } + + /// Gets a mutable reference to the underlying value in this timeout. + pub fn get_mut(&mut self) -> &mut T { + &mut self.value + } + + /// Consumes this timeout, returning the underlying value. + pub fn into_inner(self) -> T { + self.value + } +} + +impl Future for Timeout +where + T: Future, +{ + type Output = Result; + + fn poll(self: Pin<&mut Self>, cx: &mut task::Context<'_>) -> Poll { + let me = self.project(); + + // First, try polling the future + if let Poll::Ready(v) = me.value.poll(cx) { + return Poll::Ready(Ok(v)); + } + + // Now check the timer + match me.delay.poll(cx) { + Poll::Ready(()) => Poll::Ready(Err(Elapsed::new())), + Poll::Pending => Poll::Pending, + } + } +} diff --git a/vendor/monoio/src/utils/bind_to_cpu_set.rs b/vendor/monoio/src/utils/bind_to_cpu_set.rs new file mode 100644 index 000000000..644383552 --- /dev/null +++ b/vendor/monoio/src/utils/bind_to_cpu_set.rs @@ -0,0 +1,48 @@ +/// Bind error +#[cfg(unix)] +pub type BindError = nix::Result; + +/// Bind error +#[cfg(windows)] +pub type BindError = std::io::Result; + +/// Bind current thread to given cpus +#[cfg(any(target_os = "android", target_os = "dragonfly", target_os = "linux"))] +pub fn bind_to_cpu_set(cpus: impl IntoIterator) -> BindError<()> { + let mut cpuset = nix::sched::CpuSet::new(); + for cpu in cpus { + cpuset.set(cpu)?; + } + let pid = nix::unistd::Pid::from_raw(0); + nix::sched::sched_setaffinity(pid, &cpuset) +} + +/// Bind current thread to given cpus(but not works for non-linux) +#[cfg(all( + unix, + not(any(target_os = "android", target_os = "dragonfly", target_os = "linux")) +))] +pub fn bind_to_cpu_set(_: impl IntoIterator) -> BindError<()> { + Ok(()) +} + +/// Bind current thread to given cpus +#[cfg(windows)] +pub fn bind_to_cpu_set(_: impl IntoIterator) -> BindError<()> { + Ok(()) +} + +#[cfg(all(test, feature = "utils"))] +mod tests { + use super::*; + + #[test] + fn bind_cpu() { + assert!(bind_to_cpu_set(Some(0)).is_ok()); + #[cfg(all( + unix, + any(target_os = "android", target_os = "dragonfly", target_os = "linux") + ))] + assert!(bind_to_cpu_set(Some(100000)).is_err()); + } +} diff --git a/vendor/monoio/src/utils/box_into_inner.rs b/vendor/monoio/src/utils/box_into_inner.rs new file mode 100644 index 000000000..68b598e20 --- /dev/null +++ b/vendor/monoio/src/utils/box_into_inner.rs @@ -0,0 +1,11 @@ +pub(crate) trait IntoInner { + /// Consumes the allocation, returning the value. + fn consume(self) -> T; +} + +impl IntoInner for Box { + #[inline] + fn consume(self) -> T { + *self + } +} diff --git a/vendor/monoio/src/utils/ctrlc.rs b/vendor/monoio/src/utils/ctrlc.rs new file mode 100644 index 000000000..f72b7f215 --- /dev/null +++ b/vendor/monoio/src/utils/ctrlc.rs @@ -0,0 +1,64 @@ +//! Forked from https://github.com/kennytm/async-ctrlc/blob/master/src/lib.rs + +use std::{ + future::Future, + marker::PhantomData, + pin::Pin, + ptr::null_mut, + sync::atomic::{AtomicBool, AtomicPtr, Ordering}, + task::{Context, Poll, Waker}, +}; + +use ctrlc::set_handler; +pub use ctrlc::Error; + +use crate::driver::{unpark::Unpark, UnparkHandle}; + +static WAKER: AtomicPtr = AtomicPtr::new(null_mut()); +static ACTIVE: AtomicBool = AtomicBool::new(false); + +/// A future which is fulfilled when the program receives the Ctrl+C signal. +#[derive(Debug)] +pub struct CtrlC { + // Make it not Send or Sync since the signal handler holds an UnparkHandle + // of current thread. + // If users want to wake other threads, they should do it with channel manually. + _private: PhantomData<*const ()>, +} + +impl Future for CtrlC { + type Output = (); + fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { + if ACTIVE.swap(false, Ordering::SeqCst) { + Poll::Ready(()) + } else { + let new_waker = Box::new(cx.waker().clone()); + let old_waker_ptr = WAKER.swap(Box::into_raw(new_waker), Ordering::SeqCst); + if !old_waker_ptr.is_null() { + let _ = unsafe { Box::from_raw(old_waker_ptr) }; + } + Poll::Pending + } + } +} + +impl CtrlC { + /// Creates a new `CtrlC` future. + /// + /// There should be at most one `CtrlC` instance in the whole program. The + /// second call to `Ctrl::new()` would return an error. + pub fn new() -> Result { + let unpark_handler = UnparkHandle::current(); + set_handler(move || { + ACTIVE.store(true, Ordering::SeqCst); + let waker_ptr = WAKER.swap(null_mut(), Ordering::SeqCst); + if !waker_ptr.is_null() { + unsafe { Box::from_raw(waker_ptr) }.wake(); + } + let _ = unpark_handler.unpark(); + })?; + Ok(CtrlC { + _private: PhantomData, + }) + } +} diff --git a/vendor/monoio/src/utils/linked_list.rs b/vendor/monoio/src/utils/linked_list.rs new file mode 100644 index 000000000..2a4cec7fd --- /dev/null +++ b/vendor/monoio/src/utils/linked_list.rs @@ -0,0 +1,608 @@ +//! Linked list impl. +//! +//! An intrusive double linked list of data +//! +//! The data structure supports tracking pinned nodes. Most of the data +//! structure's APIs are `unsafe` as they require the caller to ensure the +//! specified node is actually contained by the list. +// Heavily borrowed from tokio. +// Copyright (c) 2021 Tokio Contributors, licensed under the MIT license. +#![allow(unused)] + +use core::{ + cell::UnsafeCell, + fmt, + marker::{PhantomData, PhantomPinned}, + mem::ManuallyDrop, + ptr::{self, NonNull}, +}; + +/// An intrusive linked list. +/// +/// Currently, the list is not emptied on drop. It is the caller's +/// responsibility to ensure the list is empty before dropping it. +pub(crate) struct LinkedList { + /// Linked list head + head: Option>, + + /// Linked list tail + tail: Option>, + + /// Node type marker. + _marker: PhantomData<*const L>, +} + +/// Defines how a type is tracked within a linked list. +/// +/// In order to support storing a single type within multiple lists, accessing +/// the list pointers is decoupled from the entry type. +/// +/// # Safety +/// +/// Implementations must guarantee that `Target` types are pinned in memory. In +/// other words, when a node is inserted, the value will not be moved as long as +/// it is stored in the list. +pub(crate) unsafe trait Link { + /// Handle to the list entry. + /// + /// This is usually a pointer-ish type. + type Handle; + + /// Node type + type Target; + + /// Convert the handle to a raw pointer without consuming the handle + #[allow(clippy::wrong_self_convention)] + fn as_raw(handle: &Self::Handle) -> NonNull; + + /// Convert the raw pointer to a handle + unsafe fn from_raw(ptr: NonNull) -> Self::Handle; + + /// Return the pointers for a node + unsafe fn pointers(target: NonNull) -> NonNull>; +} + +/// Previous / next pointers +pub(crate) struct Pointers { + inner: UnsafeCell>, +} +/// We do not want the compiler to put the `noalias` attribute on mutable +/// references to this type, so the type has been made `!Unpin` with a +/// `PhantomPinned` field. +/// +/// Additionally, we never access the `prev` or `next` fields directly, as any +/// such access would implicitly involve the creation of a reference to the +/// field, which we want to avoid since the fields are not `!Unpin`, and would +/// hence be given the `noalias` attribute if we were to do such an access. +/// As an alternative to accessing the fields directly, the `Pointers` type +/// provides getters and setters for the two fields, and those are implemented +/// using raw pointer casts and offsets, which is valid since the struct is +/// #[repr(C)]. +/// +/// See this link for more information: +/// +#[repr(C)] +struct PointersInner { + /// The previous node in the list. null if there is no previous node. + /// + /// This field is accessed through pointer manipulation, so it is not dead + /// code. + #[allow(dead_code)] + prev: Option>, + + /// The next node in the list. null if there is no previous node. + /// + /// This field is accessed through pointer manipulation, so it is not dead + /// code. + #[allow(dead_code)] + next: Option>, + + /// This type is !Unpin due to the heuristic from: + /// + _pin: PhantomPinned, +} + +// ===== impl LinkedList ===== + +impl LinkedList { + /// Creates an empty linked list. + pub(crate) const fn new() -> LinkedList { + LinkedList { + head: None, + tail: None, + _marker: PhantomData, + } + } +} + +impl LinkedList { + /// Adds an element first in the list. + pub(crate) fn push_front(&mut self, val: L::Handle) { + // The value should not be dropped, it is being inserted into the list + let val = ManuallyDrop::new(val); + let ptr = L::as_raw(&val); + assert_ne!(self.head, Some(ptr)); + unsafe { + L::pointers(ptr).as_mut().set_next(self.head); + L::pointers(ptr).as_mut().set_prev(None); + + if let Some(head) = self.head { + L::pointers(head).as_mut().set_prev(Some(ptr)); + } + + self.head = Some(ptr); + + if self.tail.is_none() { + self.tail = Some(ptr); + } + } + } + + /// Removes the last element from a list and returns it, or None if it is + /// empty. + pub(crate) fn pop_back(&mut self) -> Option { + unsafe { + let last = self.tail?; + self.tail = L::pointers(last).as_ref().get_prev(); + + if let Some(prev) = L::pointers(last).as_ref().get_prev() { + L::pointers(prev).as_mut().set_next(None); + } else { + self.head = None + } + + L::pointers(last).as_mut().set_prev(None); + L::pointers(last).as_mut().set_next(None); + + Some(L::from_raw(last)) + } + } + + /// Returns whether the linked list does not contain any node + pub(crate) fn is_empty(&self) -> bool { + if self.head.is_some() { + return false; + } + + assert!(self.tail.is_none()); + true + } + + /// Removes the specified node from the list + /// + /// # Safety + /// + /// The caller **must** ensure that `node` is currently contained by + /// `self` or not contained by any other list. + pub(crate) unsafe fn remove(&mut self, node: NonNull) -> Option { + if let Some(prev) = L::pointers(node).as_ref().get_prev() { + debug_assert_eq!(L::pointers(prev).as_ref().get_next(), Some(node)); + L::pointers(prev) + .as_mut() + .set_next(L::pointers(node).as_ref().get_next()); + } else { + if self.head != Some(node) { + return None; + } + + self.head = L::pointers(node).as_ref().get_next(); + } + + if let Some(next) = L::pointers(node).as_ref().get_next() { + debug_assert_eq!(L::pointers(next).as_ref().get_prev(), Some(node)); + L::pointers(next) + .as_mut() + .set_prev(L::pointers(node).as_ref().get_prev()); + } else { + // This might be the last item in the list + if self.tail != Some(node) { + return None; + } + + self.tail = L::pointers(node).as_ref().get_prev(); + } + + L::pointers(node).as_mut().set_next(None); + L::pointers(node).as_mut().set_prev(None); + + Some(L::from_raw(node)) + } +} + +impl fmt::Debug for LinkedList { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("LinkedList") + .field("head", &self.head) + .field("tail", &self.tail) + .finish() + } +} + +impl LinkedList { + pub(crate) fn last(&self) -> Option<&L::Target> { + let tail = self.tail.as_ref()?; + unsafe { Some(&*tail.as_ptr()) } + } +} + +impl Default for LinkedList { + fn default() -> Self { + Self::new() + } +} + +// ===== impl DrainFilter ===== + +pub(crate) struct DrainFilter<'a, T: Link, F> { + list: &'a mut LinkedList, + filter: F, + curr: Option>, +} + +impl LinkedList { + pub(crate) fn drain_filter(&mut self, filter: F) -> DrainFilter<'_, T, F> + where + F: FnMut(&mut T::Target) -> bool, + { + let curr = self.head; + DrainFilter { + curr, + filter, + list: self, + } + } +} + +impl<'a, T, F> Iterator for DrainFilter<'a, T, F> +where + T: Link, + F: FnMut(&mut T::Target) -> bool, +{ + type Item = T::Handle; + + fn next(&mut self) -> Option { + while let Some(curr) = self.curr { + // safety: the pointer references data contained by the list + self.curr = unsafe { T::pointers(curr).as_ref() }.get_next(); + + // safety: the value is still owned by the linked list. + if (self.filter)(unsafe { &mut *curr.as_ptr() }) { + return unsafe { self.list.remove(curr) }; + } + } + + None + } +} + +// ===== impl Pointers ===== + +impl Pointers { + /// Create a new set of empty pointers + pub(crate) fn new() -> Pointers { + Pointers { + inner: UnsafeCell::new(PointersInner { + prev: None, + next: None, + _pin: PhantomPinned, + }), + } + } + + fn get_prev(&self) -> Option> { + // SAFETY: prev is the first field in PointersInner, which is #[repr(C)]. + unsafe { + let inner = self.inner.get(); + let prev = inner as *const Option>; + ptr::read(prev) + } + } + fn get_next(&self) -> Option> { + // SAFETY: next is the second field in PointersInner, which is #[repr(C)]. + unsafe { + let inner = self.inner.get(); + let prev = inner as *const Option>; + let next = prev.add(1); + ptr::read(next) + } + } + + fn set_prev(&mut self, value: Option>) { + // SAFETY: prev is the first field in PointersInner, which is #[repr(C)]. + unsafe { + let inner = self.inner.get(); + let prev = inner as *mut Option>; + ptr::write(prev, value); + } + } + fn set_next(&mut self, value: Option>) { + // SAFETY: next is the second field in PointersInner, which is #[repr(C)]. + unsafe { + let inner = self.inner.get(); + let prev = inner as *mut Option>; + let next = prev.add(1); + ptr::write(next, value); + } + } +} + +impl fmt::Debug for Pointers { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + let prev = self.get_prev(); + let next = self.get_next(); + f.debug_struct("Pointers") + .field("prev", &prev) + .field("next", &next) + .finish() + } +} + +#[cfg(test)] +mod tests { + use std::pin::Pin; + + use super::*; + + #[derive(Debug)] + struct Entry { + pointers: Pointers, + val: i32, + } + + unsafe impl<'a> Link for &'a Entry { + type Handle = Pin<&'a Entry>; + type Target = Entry; + + fn as_raw(handle: &Pin<&'_ Entry>) -> NonNull { + NonNull::from(handle.get_ref()) + } + + unsafe fn from_raw(ptr: NonNull) -> Pin<&'a Entry> { + Pin::new_unchecked(&*ptr.as_ptr()) + } + + unsafe fn pointers(mut target: NonNull) -> NonNull> { + NonNull::from(&mut target.as_mut().pointers) + } + } + + fn entry(val: i32) -> Pin> { + Box::pin(Entry { + pointers: Pointers::new(), + val, + }) + } + + fn ptr(r: &Pin>) -> NonNull { + r.as_ref().get_ref().into() + } + + fn collect_list(list: &mut LinkedList<&'_ Entry, <&'_ Entry as Link>::Target>) -> Vec { + let mut ret = vec![]; + + while let Some(entry) = list.pop_back() { + ret.push(entry.val); + } + + ret + } + + fn push_all<'a>( + list: &mut LinkedList<&'a Entry, <&'_ Entry as Link>::Target>, + entries: &[Pin<&'a Entry>], + ) { + for entry in entries.iter() { + list.push_front(*entry); + } + } + + macro_rules! assert_clean { + ($e:ident) => {{ + assert!($e.pointers.get_next().is_none()); + assert!($e.pointers.get_prev().is_none()); + }}; + } + + macro_rules! assert_ptr_eq { + ($a:expr, $b:expr) => {{ + // Deal with mapping a Pin<&mut T> -> Option> + assert_eq!(Some($a.as_ref().get_ref().into()), $b) + }}; + } + + #[test] + fn const_new() { + const _: LinkedList<&Entry, <&Entry as Link>::Target> = LinkedList::new(); + } + + #[test] + fn push_and_drain() { + let a = entry(5); + let b = entry(7); + let c = entry(31); + + let mut list = LinkedList::new(); + assert!(list.is_empty()); + + list.push_front(a.as_ref()); + assert!(!list.is_empty()); + list.push_front(b.as_ref()); + list.push_front(c.as_ref()); + + let items: Vec = collect_list(&mut list); + assert_eq!([5, 7, 31].to_vec(), items); + + assert!(list.is_empty()); + } + + #[test] + fn push_pop_push_pop() { + let a = entry(5); + let b = entry(7); + + let mut list = LinkedList::<&Entry, <&Entry as Link>::Target>::new(); + + list.push_front(a.as_ref()); + + let entry = list.pop_back().unwrap(); + assert_eq!(5, entry.val); + assert!(list.is_empty()); + + list.push_front(b.as_ref()); + + let entry = list.pop_back().unwrap(); + assert_eq!(7, entry.val); + + assert!(list.is_empty()); + assert!(list.pop_back().is_none()); + } + + #[test] + fn remove_by_address() { + let a = entry(5); + let b = entry(7); + let c = entry(31); + + unsafe { + // Remove first + let mut list = LinkedList::new(); + + push_all(&mut list, &[c.as_ref(), b.as_ref(), a.as_ref()]); + assert!(list.remove(ptr(&a)).is_some()); + assert_clean!(a); + // `a` should be no longer there and can't be removed twice + assert!(list.remove(ptr(&a)).is_none()); + assert!(!list.is_empty()); + + assert!(list.remove(ptr(&b)).is_some()); + assert_clean!(b); + // `b` should be no longer there and can't be removed twice + assert!(list.remove(ptr(&b)).is_none()); + assert!(!list.is_empty()); + + assert!(list.remove(ptr(&c)).is_some()); + assert_clean!(c); + // `b` should be no longer there and can't be removed twice + assert!(list.remove(ptr(&c)).is_none()); + assert!(list.is_empty()); + } + + unsafe { + // Remove middle + let mut list = LinkedList::new(); + + push_all(&mut list, &[c.as_ref(), b.as_ref(), a.as_ref()]); + + assert!(list.remove(ptr(&a)).is_some()); + assert_clean!(a); + + assert_ptr_eq!(b, list.head); + assert_ptr_eq!(c, b.pointers.get_next()); + assert_ptr_eq!(b, c.pointers.get_prev()); + + let items = collect_list(&mut list); + assert_eq!([31, 7].to_vec(), items); + } + + unsafe { + // Remove middle + let mut list = LinkedList::new(); + + push_all(&mut list, &[c.as_ref(), b.as_ref(), a.as_ref()]); + + assert!(list.remove(ptr(&b)).is_some()); + assert_clean!(b); + + assert_ptr_eq!(c, a.pointers.get_next()); + assert_ptr_eq!(a, c.pointers.get_prev()); + + let items = collect_list(&mut list); + assert_eq!([31, 5].to_vec(), items); + } + + unsafe { + // Remove last + // Remove middle + let mut list = LinkedList::new(); + + push_all(&mut list, &[c.as_ref(), b.as_ref(), a.as_ref()]); + + assert!(list.remove(ptr(&c)).is_some()); + assert_clean!(c); + + assert!(b.pointers.get_next().is_none()); + assert_ptr_eq!(b, list.tail); + + let items = collect_list(&mut list); + assert_eq!([7, 5].to_vec(), items); + } + + unsafe { + // Remove first of two + let mut list = LinkedList::new(); + + push_all(&mut list, &[b.as_ref(), a.as_ref()]); + + assert!(list.remove(ptr(&a)).is_some()); + + assert_clean!(a); + + // a should be no longer there and can't be removed twice + assert!(list.remove(ptr(&a)).is_none()); + + assert_ptr_eq!(b, list.head); + assert_ptr_eq!(b, list.tail); + + assert!(b.pointers.get_next().is_none()); + assert!(b.pointers.get_prev().is_none()); + + let items = collect_list(&mut list); + assert_eq!([7].to_vec(), items); + } + + unsafe { + // Remove last of two + let mut list = LinkedList::new(); + + push_all(&mut list, &[b.as_ref(), a.as_ref()]); + + assert!(list.remove(ptr(&b)).is_some()); + + assert_clean!(b); + + assert_ptr_eq!(a, list.head); + assert_ptr_eq!(a, list.tail); + + assert!(a.pointers.get_next().is_none()); + assert!(a.pointers.get_prev().is_none()); + + let items = collect_list(&mut list); + assert_eq!([5].to_vec(), items); + } + + unsafe { + // Remove last item + let mut list = LinkedList::new(); + + push_all(&mut list, &[a.as_ref()]); + + assert!(list.remove(ptr(&a)).is_some()); + assert_clean!(a); + + assert!(list.head.is_none()); + assert!(list.tail.is_none()); + let items = collect_list(&mut list); + assert!(items.is_empty()); + } + + unsafe { + // Remove missing + let mut list = LinkedList::<&Entry, <&Entry as Link>::Target>::new(); + + list.push_front(b.as_ref()); + list.push_front(a.as_ref()); + + assert!(list.remove(ptr(&c)).is_none()); + } + } +} diff --git a/vendor/monoio/src/utils/mod.rs b/vendor/monoio/src/utils/mod.rs new file mode 100644 index 000000000..4b2deb0cd --- /dev/null +++ b/vendor/monoio/src/utils/mod.rs @@ -0,0 +1,25 @@ +//! Common utils + +pub(crate) mod box_into_inner; +pub(crate) mod linked_list; +#[allow(dead_code)] +pub(crate) mod slab; +#[allow(dead_code)] +pub(crate) mod thread_id; +pub(crate) mod uring_detect; + +mod rand; +pub use rand::thread_rng_n; +pub use uring_detect::detect_uring; + +pub use crate::driver::op::is_legacy; + +#[cfg(feature = "signal")] +mod ctrlc; +#[cfg(feature = "signal")] +pub use self::ctrlc::{CtrlC, Error as CtrlCError}; + +#[cfg(feature = "utils")] +mod bind_to_cpu_set; +#[cfg(feature = "utils")] +pub use bind_to_cpu_set::{bind_to_cpu_set, BindError}; diff --git a/vendor/monoio/src/utils/rand.rs b/vendor/monoio/src/utils/rand.rs new file mode 100644 index 000000000..4ef2d67e4 --- /dev/null +++ b/vendor/monoio/src/utils/rand.rs @@ -0,0 +1,88 @@ +//! Fast random number generate +//! +//! Implement xorshift64+: 2 32-bit xorshift sequences added together. +//! Shift triplet `[17,7,16]` was calculated as indicated in Marsaglia's +//! Xorshift paper: +//! This generator passes the SmallCrush suite, part of TestU01 framework: +//! +// Heavily borrowed from tokio. +// Copyright (c) 2021 Tokio Contributors, licensed under the MIT license. +use std::cell::Cell; + +#[derive(Debug)] +pub(crate) struct FastRand { + one: Cell, + two: Cell, +} + +impl FastRand { + /// Initialize a new, thread-local, fast random number generator. + pub(crate) fn new(seed: u64) -> FastRand { + let one = (seed >> 32) as u32; + let mut two = seed as u32; + + if two == 0 { + // This value cannot be zero + two = 1; + } + + FastRand { + one: Cell::new(one), + two: Cell::new(two), + } + } + + pub(crate) fn fastrand_n(&self, n: u32) -> u32 { + // This is similar to fastrand() % n, but faster. + // See https://lemire.me/blog/2016/06/27/a-fast-alternative-to-the-modulo-reduction/ + let mul = (self.fastrand() as u64).wrapping_mul(n as u64); + (mul >> 32) as u32 + } + + fn fastrand(&self) -> u32 { + let mut s1 = self.one.get(); + let s0 = self.two.get(); + + s1 ^= s1 << 17; + s1 = s1 ^ s0 ^ s1 >> 7 ^ s0 >> 16; + + self.one.set(s0); + self.two.set(s1); + + s0.wrapping_add(s1) + } +} + +/// Used by the select macro and `StreamMap` +pub fn thread_rng_n(n: u32) -> u32 { + thread_local! { + static THREAD_RNG: FastRand = FastRand::new(seed()); + } + + THREAD_RNG.with(|rng| rng.fastrand_n(n)) +} + +use std::{ + collections::hash_map::RandomState, + hash::BuildHasher, + sync::atomic::{AtomicU32, Ordering::Relaxed}, +}; + +static COUNTER: AtomicU32 = AtomicU32::new(1); + +fn seed() -> u64 { + let rand_state = RandomState::new(); + rand_state.hash_one(COUNTER.fetch_add(1, Relaxed)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rand() { + for _ in 0..100 { + assert!(thread_rng_n(10) < 10); + } + } +} diff --git a/vendor/monoio/src/utils/slab.rs b/vendor/monoio/src/utils/slab.rs new file mode 100644 index 000000000..651823dd2 --- /dev/null +++ b/vendor/monoio/src/utils/slab.rs @@ -0,0 +1,402 @@ +//! Slab. +//! Part of code and design forked from tokio. + +use std::{ + mem::MaybeUninit, + ops::{Deref, DerefMut}, +}; + +/// Pre-allocated storage for a uniform data type +#[derive(Default)] +pub(crate) struct Slab { + // pages of continued memory + pages: [Option>; NUM_PAGES], + // cached write page id + w_page_id: usize, + // current generation + generation: u32, +} + +const NUM_PAGES: usize = 26; +const PAGE_INITIAL_SIZE: usize = 64; +const COMPACT_INTERVAL: u32 = 2048; + +impl Slab { + /// Create a new slab. + pub(crate) const fn new() -> Slab { + Slab { + pages: [ + None, None, None, None, None, None, None, None, None, None, None, None, None, None, + None, None, None, None, None, None, None, None, None, None, None, None, + ], + w_page_id: 0, + generation: 0, + } + } + + /// Get slab len. + #[allow(unused)] + pub(crate) fn len(&self) -> usize { + self.pages.iter().fold(0, |acc, page| match page { + Some(page) => acc + page.used, + None => acc, + }) + } + + pub(crate) fn get(&mut self, key: usize) -> Option> { + let page_id = get_page_id(key); + // here we make 2 mut ref so we must make it safe. + let slab = unsafe { &mut *(self as *mut Slab) }; + let page = match unsafe { self.pages.get_unchecked_mut(page_id) } { + Some(page) => page, + None => return None, + }; + let index = key - page.prev_len; + match page.get_entry_mut(index) { + None => None, + Some(entry) => match entry { + Entry::Vacant(_) => None, + Entry::Occupied(_) => Some(Ref { slab, page, index }), + }, + } + } + + /// Insert an element into slab. The key is returned. + /// Note: If the slab is out of slot, it will panic. + pub(crate) fn insert(&mut self, val: T) -> usize { + let begin_id = self.w_page_id; + for i in begin_id..NUM_PAGES { + unsafe { + let page = match self.pages.get_unchecked_mut(i) { + Some(page) => page, + None => { + let page = Page::new( + PAGE_INITIAL_SIZE << i, + (PAGE_INITIAL_SIZE << i) - PAGE_INITIAL_SIZE, + ); + let r = self.pages.get_unchecked_mut(i); + *r = Some(page); + r.as_mut().unwrap_unchecked() + } + }; + if let Some(slot) = page.alloc() { + page.set(slot, val); + self.w_page_id = i; + return slot + page.prev_len; + } + } + } + panic!("out of slot"); + } + + /// Remove an element from slab. + #[allow(unused)] + pub(crate) fn remove(&mut self, key: usize) -> Option { + let page_id = get_page_id(key); + let page = match unsafe { self.pages.get_unchecked_mut(page_id) } { + Some(page) => page, + None => return None, + }; + let val = page.remove(key - page.prev_len); + self.mark_remove(); + val + } + + pub(crate) fn mark_remove(&mut self) { + // compact + self.generation = self.generation.wrapping_add(1); + if self.generation % COMPACT_INTERVAL == 0 { + // reset write page index + self.w_page_id = 0; + // find the last allocated page and try to drop + if let Some((id, last_page)) = self + .pages + .iter_mut() + .enumerate() + .rev() + .find_map(|(id, p)| p.as_mut().map(|p| (id, p))) + { + if last_page.is_empty() && id > 0 { + unsafe { + *self.pages.get_unchecked_mut(id) = None; + } + } + } + } + } +} + +// Forked from tokio. +fn get_page_id(key: usize) -> usize { + const POINTER_WIDTH: u32 = std::mem::size_of::() as u32 * 8; + const PAGE_INDEX_SHIFT: u32 = PAGE_INITIAL_SIZE.trailing_zeros() + 1; + + let slot_shifted = (key.saturating_add(PAGE_INITIAL_SIZE)) >> PAGE_INDEX_SHIFT; + ((POINTER_WIDTH - slot_shifted.leading_zeros()) as usize).min(NUM_PAGES - 1) +} + +/// Ref point to a valid slot. +pub(crate) struct Ref<'a, T> { + slab: &'a mut Slab, + page: &'a mut Page, + index: usize, +} + +impl<'a, T> Ref<'a, T> { + #[allow(unused)] + pub(crate) fn remove(self) -> T { + // # Safety + // We make sure the index is valid. + let val = unsafe { self.page.remove(self.index).unwrap_unchecked() }; + self.slab.mark_remove(); + val + } +} + +impl<'a, T> AsRef for Ref<'a, T> { + fn as_ref(&self) -> &T { + // # Safety + // We make sure the index is valid. + unsafe { self.page.get(self.index).unwrap_unchecked() } + } +} + +impl<'a, T> AsMut for Ref<'a, T> { + fn as_mut(&mut self) -> &mut T { + // # Safety + // We make sure the index is valid. + unsafe { self.page.get_mut(self.index).unwrap_unchecked() } + } +} + +impl<'a, T> Deref for Ref<'a, T> { + type Target = T; + + fn deref(&self) -> &Self::Target { + self.as_ref() + } +} + +impl<'a, T> DerefMut for Ref<'a, T> { + fn deref_mut(&mut self) -> &mut Self::Target { + self.as_mut() + } +} + +enum Entry { + Vacant(usize), + Occupied(T), +} + +impl Entry { + fn as_ref(&self) -> Option<&T> { + match self { + Entry::Vacant(_) => None, + Entry::Occupied(inner) => Some(inner), + } + } + + fn as_mut(&mut self) -> Option<&mut T> { + match self { + Entry::Vacant(_) => None, + Entry::Occupied(inner) => Some(inner), + } + } + + fn is_vacant(&self) -> bool { + matches!(self, Entry::Vacant(_)) + } + + unsafe fn unwrap_unchecked(self) -> T { + match self { + Entry::Vacant(_) => std::hint::unreachable_unchecked(), + Entry::Occupied(inner) => inner, + } + } +} + +struct Page { + // continued buffer of fixed size + slots: Box<[MaybeUninit>]>, + // number of occupied slots + used: usize, + // number of initialized slots + initialized: usize, + // next slot to write + next: usize, + // sum of previous page's slots count + prev_len: usize, +} + +impl Page { + fn new(size: usize, prev_len: usize) -> Self { + let mut buffer = Vec::with_capacity(size); + unsafe { buffer.set_len(size) }; + let slots = buffer.into_boxed_slice(); + Self { + slots, + used: 0, + initialized: 0, + next: 0, + prev_len, + } + } + + fn is_empty(&self) -> bool { + self.used == 0 + } + + fn is_full(&self) -> bool { + self.used == self.slots.len() + } + + // alloc a slot + // Safety: after slot is allocated, the caller must guarantee it will be + // initialized + unsafe fn alloc(&mut self) -> Option { + let next = self.next; + if self.is_full() { + // current page is full + debug_assert_eq!(next, self.slots.len(), "next should eq to slots.len()"); + return None; + } else if next >= self.initialized { + // the slot to write is not initialized + debug_assert_eq!(next, self.initialized, "next should eq to initialized"); + self.initialized += 1; + self.next += 1; + } else { + // the slot has already been initialized + // it must be Vacant + let slot = self.slots.get_unchecked(next).assume_init_ref(); + match slot { + Entry::Vacant(next_slot) => { + self.next = *next_slot; + } + _ => std::hint::unreachable_unchecked(), + } + } + self.used += 1; + Some(next) + } + + // set value of the slot + // Safety: the slot must returned by Self::alloc. + unsafe fn set(&mut self, slot: usize, val: T) { + let slot = self.slots.get_unchecked_mut(slot); + *slot = MaybeUninit::new(Entry::Occupied(val)); + } + + fn get(&self, slot: usize) -> Option<&T> { + if slot >= self.initialized { + return None; + } + unsafe { self.slots.get_unchecked(slot).assume_init_ref() }.as_ref() + } + + fn get_mut(&mut self, slot: usize) -> Option<&mut T> { + if slot >= self.initialized { + return None; + } + unsafe { self.slots.get_unchecked_mut(slot).assume_init_mut() }.as_mut() + } + + fn get_entry_mut(&mut self, slot: usize) -> Option<&mut Entry> { + if slot >= self.initialized { + return None; + } + unsafe { Some(self.slots.get_unchecked_mut(slot).assume_init_mut()) } + } + + fn remove(&mut self, slot: usize) -> Option { + if slot >= self.initialized { + return None; + } + unsafe { + let slot_mut = self.slots.get_unchecked_mut(slot).assume_init_mut(); + if slot_mut.is_vacant() { + return None; + } + let val = std::mem::replace(slot_mut, Entry::Vacant(self.next)); + self.next = slot; + self.used -= 1; + + Some(val.unwrap_unchecked()) + } + } +} + +impl Drop for Page { + fn drop(&mut self) { + let mut to_drop = std::mem::take(&mut self.slots).into_vec(); + + unsafe { + if self.is_empty() { + // fast drop if empty + to_drop.set_len(0); + } else { + // slow drop + to_drop.set_len(self.initialized); + std::mem::transmute::>>, Vec>>(to_drop); + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn insert_get_remove_one() { + let mut slab = Slab::default(); + let key = slab.insert(10); + assert_eq!(slab.get(key).unwrap().as_mut(), &10); + assert_eq!(slab.remove(key), Some(10)); + assert!(slab.get(key).is_none()); + assert_eq!(slab.len(), 0); + } + + #[test] + fn insert_get_remove_many() { + let mut slab = Slab::new(); + let mut keys = vec![]; + + for i in 0..10 { + for j in 0..10 { + let val = (i * 10) + j; + + let key = slab.insert(val); + keys.push((key, val)); + assert_eq!(slab.get(key).unwrap().as_mut(), &val); + } + + for (key, val) in keys.drain(..) { + assert_eq!(val, slab.remove(key).unwrap()); + } + } + } + + #[test] + fn get_not_exist() { + let mut slab = Slab::::new(); + assert!(slab.get(0).is_none()); + assert!(slab.get(1).is_none()); + assert!(slab.get(usize::MAX).is_none()); + assert!(slab.remove(0).is_none()); + assert!(slab.remove(1).is_none()); + assert!(slab.remove(usize::MAX).is_none()); + } + + #[test] + fn insert_remove_big() { + let mut slab = Slab::default(); + let keys = (0..1_000_000).map(|i| slab.insert(i)).collect::>(); + keys.iter().zip(0..1_000_000).for_each(|(key, val)| { + assert_eq!(slab.remove(*key).unwrap(), val); + }); + keys.iter().for_each(|key| { + assert!(slab.get(*key).is_none()); + }); + assert_eq!(slab.len(), 0); + } +} diff --git a/vendor/monoio/src/utils/thread_id.rs b/vendor/monoio/src/utils/thread_id.rs new file mode 100644 index 000000000..dda01502b --- /dev/null +++ b/vendor/monoio/src/utils/thread_id.rs @@ -0,0 +1,21 @@ +use std::sync::atomic::{AtomicUsize, Ordering::Relaxed}; + +// thread id begins from 16. +// 0 is default thread +// 1-15 are unused +static ID_GEN: AtomicUsize = AtomicUsize::new(16); + +pub(crate) const DEFAULT_THREAD_ID: usize = 0; + +/// Used to generate thread id. +pub(crate) fn gen_id() -> usize { + ID_GEN.fetch_add(1, Relaxed) +} + +pub(crate) fn get_current_thread_id() -> usize { + crate::runtime::CURRENT.with(|ctx| ctx.thread_id) +} + +pub(crate) fn try_get_current_thread_id() -> Option { + crate::runtime::CURRENT.try_with(|maybe_ctx| maybe_ctx.map(|ctx| ctx.thread_id)) +} diff --git a/vendor/monoio/src/utils/uring_detect.rs b/vendor/monoio/src/utils/uring_detect.rs new file mode 100644 index 000000000..f41b4f4e4 --- /dev/null +++ b/vendor/monoio/src/utils/uring_detect.rs @@ -0,0 +1,85 @@ +//! Detect if current platform support io_uring. + +#[cfg(all(target_os = "linux", feature = "iouring"))] +macro_rules! err_to_false { + ($e: expr) => { + match $e { + Ok(x) => x, + Err(_) => { + return false; + } + } + }; +} +#[cfg(all(target_os = "linux", feature = "iouring"))] +fn detect_uring_inner() -> bool { + let val = std::env::var("MONOIO_FORCE_LEGACY_DRIVER"); + match val { + Ok(v) if matches!(v.to_ascii_lowercase().as_str(), "1" | "true" | "yes") => { + return false; + } + _ => {} + } + + use io_uring::opcode::*; + auto_const_array::auto_const_array! { + const USED_OP: [u8; _] = [ + Accept::CODE, + AsyncCancel::CODE, + Close::CODE, + Connect::CODE, + Fsync::CODE, + OpenAt::CODE, + PollAdd::CODE, + ProvideBuffers::CODE, + Read::CODE, + Readv::CODE, + Recv::CODE, + Send::CODE, + SendMsg::CODE, + RecvMsg::CODE, + #[cfg(feature = "splice")] + Splice::CODE, + Timeout::CODE, + Write::CODE, + Writev::CODE, + ]; + } + + let uring = err_to_false!(io_uring::IoUring::new(2)); + let mut probe = io_uring::Probe::new(); + err_to_false!(uring.submitter().register_probe(&mut probe)); + USED_OP.iter().all(|op| probe.is_supported(*op)) +} + +/// Detect if current platform supports our needed uring ops. +#[cfg(all(target_os = "linux", feature = "iouring"))] +pub fn detect_uring() -> bool { + static mut URING_SUPPORTED: bool = false; + static INIT: std::sync::Once = std::sync::Once::new(); + + unsafe { + INIT.call_once(|| { + URING_SUPPORTED = detect_uring_inner(); + }); + URING_SUPPORTED + } +} + +/// Detect if current platform supports our needed uring ops. +#[cfg(not(all(target_os = "linux", feature = "iouring")))] +pub fn detect_uring() -> bool { + false +} + +#[cfg(test)] +mod tests { + #[cfg(all(target_os = "linux", feature = "iouring"))] + #[test] + fn test_detect() { + assert!( + super::detect_uring(), + "io_uring or ops not supported on current platform" + ) + } +} diff --git a/vendor/monoio/tests/buf_writter.rs b/vendor/monoio/tests/buf_writter.rs new file mode 100644 index 000000000..7d6f43108 --- /dev/null +++ b/vendor/monoio/tests/buf_writter.rs @@ -0,0 +1,31 @@ +use monoio::{ + io::{AsyncReadRent, AsyncWriteRent, BufReader, BufWriter, Splitable}, + net::{TcpListener, TcpStream}, +}; + +#[monoio::test_all] +async fn ensure_buf_writter_write_properly() { + let srv = TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = srv.local_addr().unwrap(); + + monoio::spawn(async move { + let stream = TcpStream::connect(&addr).await.unwrap(); + let (_, stream_write) = stream.into_split(); + + let mut buf_w = BufWriter::new(stream_write); + assert!(buf_w.write(b"1").await.0.is_ok()); + assert!(buf_w.write(b"2").await.0.is_ok()); + assert!(buf_w.write(b"3").await.0.is_ok()); + assert!(buf_w.flush().await.is_ok()); + }); + + let (stream, _) = srv.accept().await.unwrap(); + let (rd, _) = stream.into_split(); + + let s: Vec = Vec::with_capacity(16); + let mut buf = BufReader::new(rd); + let (size, s) = buf.read(s).await; + + assert!(size.is_ok()); + assert_eq!(s, b"123"); +} diff --git a/vendor/monoio/tests/ctrlc_legacy.rs b/vendor/monoio/tests/ctrlc_legacy.rs new file mode 100644 index 000000000..2a0fd53ba --- /dev/null +++ b/vendor/monoio/tests/ctrlc_legacy.rs @@ -0,0 +1,14 @@ +#[cfg(feature = "signal")] +#[monoio::test(driver = "legacy")] +async fn test_ctrlc_legacy() { + use libc::{getpid, kill, SIGINT}; + use monoio::utils::CtrlC; + + let c = CtrlC::new().unwrap(); + std::thread::spawn(|| unsafe { + std::thread::sleep(std::time::Duration::from_millis(500)); + kill(getpid(), SIGINT); + }); + + c.await; +} diff --git a/vendor/monoio/tests/ctrlc_uring.rs b/vendor/monoio/tests/ctrlc_uring.rs new file mode 100644 index 000000000..602b963d9 --- /dev/null +++ b/vendor/monoio/tests/ctrlc_uring.rs @@ -0,0 +1,14 @@ +#[cfg(feature = "signal")] +#[monoio::test(driver = "uring")] +async fn test_ctrlc_uring() { + use libc::{getpid, kill, SIGINT}; + use monoio::utils::CtrlC; + + let c = CtrlC::new().unwrap(); + std::thread::spawn(|| unsafe { + std::thread::sleep(std::time::Duration::from_millis(500)); + kill(getpid(), SIGINT); + }); + + c.await; +} diff --git a/vendor/monoio/tests/fs_create_dir.rs b/vendor/monoio/tests/fs_create_dir.rs new file mode 100644 index 000000000..9b543e136 --- /dev/null +++ b/vendor/monoio/tests/fs_create_dir.rs @@ -0,0 +1,137 @@ +#![cfg(all(unix, feature = "mkdirat"))] + +use monoio::fs; +use tempfile::tempdir; + +#[monoio::test_all] +async fn create_single_dirctory() { + let temp_dir = tempdir().unwrap(); + let path = temp_dir.path().join("test"); + + fs::create_dir(&path).await.unwrap(); + + assert!(path.exists()); + + std::fs::remove_dir(&path).unwrap(); + + assert!(!path.exists()); + + fs::create_dir_all(&path).await.unwrap(); + + assert!(path.exists()); +} + +#[monoio::test_all] +async fn create_nested_directories() { + let temp_dir = tempdir().unwrap(); + let path = temp_dir.path().join("test/foo/bar"); + + fs::create_dir_all(&path).await.unwrap(); + + assert!(path.exists()); +} + +#[monoio::test_all] +async fn create_existing_directory() { + let temp_dir = tempdir().unwrap(); + + fs::create_dir_all(temp_dir.path()).await.unwrap(); +} + +#[monoio::test_all] +async fn create_invalid_path() { + let temp_dir = tempdir().unwrap(); + + let mut path = temp_dir.path().display().to_string(); + path += "invalid_dir/\0"; + + let res = fs::create_dir_all(path).await; + + assert!(res.is_err()); +} + +#[monoio::test_all] +async fn create_directory_with_special_characters() { + let temp_dir = tempdir().unwrap(); + + let path = temp_dir.path().join("foo/😀"); + + fs::create_dir_all(&path).await.unwrap(); + + assert!(path.exists()); +} + +#[monoio::test_all] +async fn create_directory_where_file_exists() { + let temp_file = tempfile::NamedTempFile::new().unwrap(); + fs::write(temp_file.path(), "foo bar").await.0.unwrap(); + + let res = fs::create_dir(temp_file.path()).await; + + assert!(res.is_err()); + + let res = fs::create_dir_all(temp_file.path()).await; + + assert!(res.is_err()); +} + +#[monoio::test_all] +async fn create_directory_with_symlink() { + let temp_dir = tempdir().unwrap(); + + let target = temp_dir.path().join("foo"); + + fs::create_dir_all(&target).await.unwrap(); + + let link = temp_dir.path().join("bar"); + let to_create = link.join("nested"); + + std::os::unix::fs::symlink(&target, &link).unwrap(); + + fs::create_dir_all(&to_create).await.unwrap(); + + assert!(to_create.exists()); + assert!(target.join("nested").exists()); +} + +#[monoio::test_all] +async fn create_very_long_path() { + let temp_dir = tempdir().unwrap(); + + let mut path = temp_dir.path().to_path_buf(); + for _ in 0..255 { + path.push("a/"); + } + + fs::create_dir_all(&path).await.unwrap(); + + assert!(path.exists()); +} + +#[monoio::test_all] +async fn create_directory_with_permission_issue() { + use std::os::unix::fs::PermissionsExt; + + let temp_dir = tempdir().unwrap(); + + let target = temp_dir.path().join("foo"); + + fs::create_dir_all(&target).await.unwrap(); + + // use `std`'s due to the `monoio`'s `set_permissions` is not implement. + let mut perm = std::fs::metadata(&target).unwrap().permissions(); + perm.set_mode(0o400); + + std::fs::set_permissions(&target, perm.clone()).unwrap(); + + let path = target.join("bar"); + let res = fs::create_dir_all(&path).await; + assert!(res.is_err()); + + perm.set_mode(0o700); + std::fs::set_permissions(&target, perm).unwrap(); + + fs::create_dir_all(&path).await.unwrap(); + + assert!(path.exists()); +} diff --git a/vendor/monoio/tests/fs_file.rs b/vendor/monoio/tests/fs_file.rs new file mode 100644 index 000000000..6e24e9a27 --- /dev/null +++ b/vendor/monoio/tests/fs_file.rs @@ -0,0 +1,210 @@ +// todo fix these CI in windows +#![cfg(not(windows))] +use std::io::prelude::*; +#[cfg(unix)] +use std::os::unix::io::{AsRawFd, FromRawFd, RawFd}; +#[cfg(windows)] +use std::os::windows::io::{AsRawHandle, FromRawHandle, RawHandle as RawFd}; + +use monoio::fs::File; +use tempfile::NamedTempFile; + +const HELLO: &[u8] = b"hello world..."; + +async fn read_hello(file: &File) { + let buf = Vec::with_capacity(1024); + let (res, buf) = file.read_at(buf, 0).await; + let n = res.unwrap(); + + assert!(n > 0 && n <= HELLO.len()); + assert_eq!(&buf, &HELLO[..n]); +} + +#[monoio::test_all] +async fn basic_read() { + let mut tempfile = tempfile(); + tempfile.write_all(HELLO).unwrap(); + tempfile.as_file_mut().sync_data().unwrap(); + + let file = File::open(tempfile.path()).await.unwrap(); + read_hello(&file).await; +} + +#[monoio::test_all] +async fn basic_read_exact() { + let mut tempfile = tempfile(); + tempfile.write_all(HELLO).unwrap(); + tempfile.as_file_mut().sync_data().unwrap(); + + let file = File::open(tempfile.path()).await.unwrap(); + let buf = Vec::with_capacity(HELLO.len()); + let (res, buf) = file.read_exact_at(buf, 0).await; + res.unwrap(); + assert_eq!(&buf[..], HELLO); + + let buf = Vec::with_capacity(HELLO.len() * 2); + let (res, _) = file.read_exact_at(buf, 0).await; + assert_eq!(res.unwrap_err().kind(), std::io::ErrorKind::UnexpectedEof); +} + +#[monoio::test_all] +async fn basic_write() { + let tempfile = tempfile(); + + let file = File::create(tempfile.path()).await.unwrap(); + file.write_at(HELLO, 0).await.0.unwrap(); + file.sync_all().await.unwrap(); + + let file = std::fs::read(tempfile.path()).unwrap(); + assert_eq!(file, HELLO); +} + +#[monoio::test_all] +async fn basic_write_all() { + let tempfile = tempfile(); + + let file = File::create(tempfile.path()).await.unwrap(); + file.write_all_at(HELLO, 0).await.0.unwrap(); + file.sync_all().await.unwrap(); + + let file = std::fs::read(tempfile.path()).unwrap(); + assert_eq!(file, HELLO); +} + +#[monoio::test(driver = "uring")] +async fn cancel_read() { + let mut tempfile = tempfile(); + tempfile.write_all(HELLO).unwrap(); + tempfile.as_file_mut().sync_data().unwrap(); + + let file = File::open(tempfile.path()).await.unwrap(); + + // Poll the future once, then cancel it + poll_once(async { read_hello(&file).await }).await; + + read_hello(&file).await; +} + +#[monoio::test_all] +async fn explicit_close() { + let mut tempfile = tempfile(); + tempfile.write_all(HELLO).unwrap(); + tempfile.as_file_mut().sync_data().unwrap(); + + let file = File::open(tempfile.path()).await.unwrap(); + #[cfg(unix)] + let fd = file.as_raw_fd(); + #[cfg(windows)] + let fd = file.as_raw_handle(); + + file.close().await.unwrap(); + + assert_invalid_fd(fd, tempfile.as_file().metadata().unwrap()); +} + +#[monoio::test_all] +async fn drop_open() { + let tempfile = tempfile(); + + // Do something else + let file_w = File::create(tempfile.path()).await.unwrap(); + file_w.write_at(HELLO, 0).await.0.unwrap(); + file_w.sync_all().await.unwrap(); + + let file = std::fs::read(tempfile.path()).unwrap(); + assert_eq!(file, HELLO); + drop(file_w); +} + +#[test] +fn drop_off_runtime() { + let tempfile = tempfile(); + #[cfg(all(target_os = "linux", feature = "iouring"))] + let file = monoio::start::(async { + File::open(tempfile.path()).await.unwrap() + }); + #[cfg(not(all(target_os = "linux", feature = "iouring")))] + let file = monoio::start::(async { + File::open(tempfile.path()).await.unwrap() + }); + + #[cfg(unix)] + let fd = file.as_raw_fd(); + #[cfg(windows)] + let fd = file.as_raw_handle(); + drop(file); + + assert_invalid_fd(fd, tempfile.as_file().metadata().unwrap()); +} + +#[monoio::test_all] +async fn sync_doesnt_kill_anything() { + let tempfile = tempfile(); + + let file = File::create(tempfile.path()).await.unwrap(); + file.sync_all().await.unwrap(); + file.sync_data().await.unwrap(); + file.write_at(&b"foo"[..], 0).await.0.unwrap(); + file.sync_all().await.unwrap(); + file.sync_data().await.unwrap(); +} + +fn tempfile() -> NamedTempFile { + NamedTempFile::new().expect("unable to create tempfile") +} + +#[allow(unused)] +async fn poll_once(future: impl std::future::Future) { + use std::{pin::pin, task::Poll}; + + use futures::future::poll_fn; + + let mut future = pin!(future); + poll_fn(|cx| { + assert!(future.as_mut().poll(cx).is_pending()); + Poll::Ready(()) + }) + .await; +} + +fn assert_invalid_fd(fd: RawFd, base: std::fs::Metadata) { + use std::fs::File; + #[cfg(unix)] + let f = unsafe { File::from_raw_fd(fd) }; + #[cfg(windows)] + let f = unsafe { File::from_raw_handle(fd) }; + + let meta = f.metadata(); + std::mem::forget(f); + + if let Ok(meta) = meta { + if !meta.is_file() { + return; + } + + #[cfg(unix)] + { + use std::os::unix::fs::MetadataExt; + let inode = meta.ino(); + let actual = base.ino(); + if inode == actual { + panic!(); + } + } + } +} + +#[monoio::test_all] +async fn file_from_std() { + let tempfile = tempfile(); + let std_file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create(true) + .open(tempfile.path()) + .unwrap(); + let file = File::from_std(std_file).unwrap(); + file.write_at(HELLO, 0).await.0.unwrap(); + file.sync_all().await.unwrap(); + read_hello(&file).await; +} diff --git a/vendor/monoio/tests/fs_metadata.rs b/vendor/monoio/tests/fs_metadata.rs new file mode 100644 index 000000000..0f1c40c1d --- /dev/null +++ b/vendor/monoio/tests/fs_metadata.rs @@ -0,0 +1,82 @@ +#![cfg(unix)] + +use std::io::Write; + +#[monoio::test_all] +async fn basic_file_metadata() { + let mut file = tempfile::NamedTempFile::new().unwrap(); + + assert_eq!(file.write(b"foo bar").unwrap(), 7); + + let m_file = monoio::fs::File::open(file.path()).await.unwrap(); + + let m_meta = monoio::fs::metadata(file.path()).await.unwrap(); + let mf_meta = m_file.metadata().await.unwrap(); + let std_meta = std::fs::metadata(file.path()).unwrap(); + + assert_eq!(m_meta.len(), std_meta.len()); + assert_eq!(mf_meta.len(), std_meta.len()); + + assert_eq!(m_meta.modified().unwrap(), std_meta.modified().unwrap()); + assert_eq!(mf_meta.modified().unwrap(), std_meta.modified().unwrap()); + + assert_eq!(m_meta.accessed().unwrap(), std_meta.accessed().unwrap()); + assert_eq!(mf_meta.accessed().unwrap(), std_meta.accessed().unwrap()); + + #[cfg(target_os = "linux")] + assert_eq!(m_meta.created().unwrap(), std_meta.created().unwrap()); + #[cfg(target_os = "linux")] + assert_eq!(mf_meta.created().unwrap(), std_meta.created().unwrap()); + + assert_eq!(m_meta.is_file(), std_meta.is_file()); + assert_eq!(mf_meta.is_file(), std_meta.is_file()); + + assert_eq!(m_meta.is_dir(), std_meta.is_dir()); + assert_eq!(mf_meta.is_dir(), std_meta.is_dir()); +} + +#[monoio::test_all] +async fn dir_metadata() { + let dir = tempfile::tempdir().unwrap(); + + let m_meta = monoio::fs::metadata(dir.path()).await.unwrap(); + let std_meta = std::fs::metadata(dir.path()).unwrap(); + + assert_eq!(m_meta.len(), std_meta.len()); + + assert_eq!(m_meta.modified().unwrap(), std_meta.modified().unwrap()); + + assert_eq!(m_meta.accessed().unwrap(), std_meta.accessed().unwrap()); + + #[cfg(target_os = "linux")] + assert_eq!(m_meta.created().unwrap(), std_meta.created().unwrap()); + + assert_eq!(m_meta.is_file(), std_meta.is_file()); + + assert_eq!(m_meta.is_dir(), std_meta.is_dir()); +} + +#[monoio::test_all] +async fn symlink_metadata() { + let dir = tempfile::tempdir().unwrap(); + let link = dir.path().join("link"); + std::os::unix::fs::symlink(dir.path(), &link).unwrap(); + + let m_meta = monoio::fs::symlink_metadata(&link).await.unwrap(); + let std_meta = std::fs::symlink_metadata(&link).unwrap(); + + assert_eq!(m_meta.len(), std_meta.len()); + + assert_eq!(m_meta.modified().unwrap(), std_meta.modified().unwrap()); + + assert_eq!(m_meta.accessed().unwrap(), std_meta.accessed().unwrap()); + + #[cfg(target_os = "linux")] + assert_eq!(m_meta.created().unwrap(), std_meta.created().unwrap()); + + assert_eq!(m_meta.is_file(), std_meta.is_file()); + + assert_eq!(m_meta.is_dir(), std_meta.is_dir()); + + assert_eq!(m_meta.is_symlink(), std_meta.is_symlink()); +} diff --git a/vendor/monoio/tests/fs_rename.rs b/vendor/monoio/tests/fs_rename.rs new file mode 100644 index 000000000..fb98837dc --- /dev/null +++ b/vendor/monoio/tests/fs_rename.rs @@ -0,0 +1,78 @@ +#![cfg(all(unix, feature = "renameat"))] + +use std::{fs::Permissions, os::unix::fs::PermissionsExt}; + +#[monoio::test_all] +async fn rename_file_in_the_same_directory() { + let temp_dir = tempfile::tempdir().unwrap(); + let file = tempfile::NamedTempFile::new_in(temp_dir.path()).unwrap(); + + let old_file_path = file.path(); + let new_file_path = temp_dir.path().join("test-file"); + + let result = monoio::fs::rename(old_file_path, &new_file_path).await; + assert!(result.is_ok()); + + assert!(new_file_path.exists()); + assert!(!old_file_path.exists()); +} + +#[monoio::test_all] +async fn rename_file_in_different_directory() { + let temp_dir1 = tempfile::tempdir().unwrap(); + let temp_dir2 = tempfile::tempdir().unwrap(); + let file = tempfile::NamedTempFile::new_in(temp_dir1.path()).unwrap(); + + let old_file_path = file.path(); + let new_file_path = temp_dir2.path().join("test-file"); + + let result = monoio::fs::rename(old_file_path, &new_file_path).await; + assert!(result.is_ok()); + + assert!(new_file_path.exists()); + assert!(!old_file_path.exists()); +} + +#[monoio::test_all] +async fn mv_file_in_different_directory() { + let temp_dir1 = tempfile::tempdir().unwrap(); + let temp_dir2 = tempfile::tempdir().unwrap(); + let file = tempfile::NamedTempFile::new_in(temp_dir1.path()).unwrap(); + + let old_file_path = file.path(); + let old_file_name = old_file_path.file_name().unwrap(); + let new_file_path = temp_dir2.path().join(old_file_name); + + let result = monoio::fs::rename(old_file_path, &new_file_path).await; + assert!(result.is_ok()); + + assert!(new_file_path.exists()); + assert!(!old_file_path.exists()); +} + +#[monoio::test_all] +async fn rename_inexist_file() { + let temp_dir = tempfile::tempdir().unwrap(); + + let old_file_path = temp_dir.path().join("inexist.txt"); + let new_file_path = temp_dir.path().join("renamed.txt"); + + let result = monoio::fs::rename(old_file_path, new_file_path).await; + + assert!(result.is_err()); +} + +#[monoio::test_all] +async fn rename_file_without_permission() { + let temp_dir = tempfile::tempdir().unwrap(); + let temp_file = tempfile::NamedTempFile::new_in(&temp_dir).unwrap(); + + std::fs::set_permissions(temp_dir.path(), Permissions::from_mode(0o0)).unwrap(); + + let old_file_path = temp_file.path(); + let new_file_path = temp_dir.path().join("test-file"); + + let result = monoio::fs::rename(old_file_path, &new_file_path).await; + + assert!(result.is_err()); +} diff --git a/vendor/monoio/tests/fs_unlink.rs b/vendor/monoio/tests/fs_unlink.rs new file mode 100644 index 000000000..4a3b2514a --- /dev/null +++ b/vendor/monoio/tests/fs_unlink.rs @@ -0,0 +1,38 @@ +#![cfg(all(unix, feature = "unlinkat", feature = "mkdirat"))] + +use std::{io, path::PathBuf}; + +use monoio::fs::{self, File}; +use tempfile::tempdir; + +async fn create_file(path: &PathBuf) -> io::Result<()> { + let file = File::create(path).await?; + file.close().await?; + Ok(()) +} + +#[monoio::test_all] +async fn remove_file() { + let dir = tempdir().unwrap(); + let target = dir.path().join("test"); + + create_file(&target).await.unwrap(); + fs::remove_file(&target).await.unwrap(); + assert!(File::open(&target).await.is_err()); + assert!(fs::remove_file(&target).await.is_err()); +} + +#[monoio::test_all] +async fn remove_dir() { + let dir = tempdir().unwrap(); + let target = dir.path().join("test"); + + fs::create_dir(&target).await.unwrap(); + let path = target.join("file"); + create_file(&path).await.unwrap(); + assert!(fs::remove_dir(&target).await.is_err()); // dir is not empty + fs::remove_file(&path).await.unwrap(); + fs::remove_dir(&target).await.unwrap(); + assert!(create_file(&path).await.is_err()); // dir has been removed + assert!(fs::remove_dir(&target).await.is_err()); +} diff --git a/vendor/monoio/tests/tcp_accept.rs b/vendor/monoio/tests/tcp_accept.rs new file mode 100644 index 000000000..df6236c3b --- /dev/null +++ b/vendor/monoio/tests/tcp_accept.rs @@ -0,0 +1,31 @@ +use std::net::{IpAddr, SocketAddr}; + +use monoio::net::{TcpListener, TcpStream}; + +macro_rules! test_accept { + ($(($ident:ident, $target:expr),)*) => { + $( + #[monoio::test_all] + async fn $ident() { + let listener = TcpListener::bind($target).unwrap(); + let addr = listener.local_addr().unwrap(); + let (tx, rx) = local_sync::oneshot::channel(); + monoio::spawn(async move { + let (socket, _) = listener.accept().await.unwrap(); + assert!(tx.send(socket).is_ok()); + }); + let cli = TcpStream::connect(&addr).await.unwrap(); + let srv = rx.await.unwrap(); + assert_eq!(cli.local_addr().unwrap(), srv.peer_addr().unwrap()); + } + )* + } +} + +test_accept! { + (ip_str, "127.0.0.1:0"), + (host_str, "localhost:0"), + (socket_addr, "127.0.0.1:0".parse::().unwrap()), + (str_port_tuple, ("127.0.0.1", 0)), + (ip_port_tuple, ("127.0.0.1".parse::().unwrap(), 0)), +} diff --git a/vendor/monoio/tests/tcp_connect.rs b/vendor/monoio/tests/tcp_connect.rs new file mode 100644 index 000000000..29c9a191e --- /dev/null +++ b/vendor/monoio/tests/tcp_connect.rs @@ -0,0 +1,178 @@ +use std::net::{IpAddr, SocketAddr}; + +use monoio::net::{TcpListener, TcpStream}; + +macro_rules! test_connect_ip { + ($(($ident:ident, $target:expr, $addr_f:path),)*) => { + $( + #[monoio::test_all] + async fn $ident() { + let listener = TcpListener::bind($target).unwrap(); + let addr = listener.local_addr().unwrap(); + assert!($addr_f(&addr)); + + let (tx, rx) = local_sync::oneshot::channel(); + + monoio::spawn(async move { + let (socket, addr) = listener.accept().await.unwrap(); + assert_eq!(addr, socket.peer_addr().unwrap()); + assert!(tx.send(socket).is_ok()); + }); + + let mine = TcpStream::connect(&addr).await.unwrap(); + let theirs = rx.await.unwrap(); + + assert_eq!(mine.local_addr().unwrap(), theirs.peer_addr().unwrap()); + assert_eq!(theirs.local_addr().unwrap(), mine.peer_addr().unwrap()); + } + )* + } +} + +test_connect_ip! { + (connect_v4, "127.0.0.1:0", SocketAddr::is_ipv4), +} + +#[cfg(not(all( + target_os = "linux", + any( + target_arch = "x86", + target_arch = "aarch64", + target_arch = "arm", + target_arch = "riscv64", + target_arch = "s390x" + ) +)))] +test_connect_ip! { + (connect_v6, "[::1]:0", SocketAddr::is_ipv6), +} + +macro_rules! test_connect { + ($(($ident:ident, $mapping:tt),)*) => { + $( + #[monoio::test_all] + async fn $ident() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + #[allow(clippy::redundant_closure_call)] + let addr = $mapping(&listener); + + let server = async { + assert!(listener.accept().await.is_ok()); + }; + + let client = async { + assert!(TcpStream::connect(addr).await.is_ok()); + }; + + monoio::join!(server, client); + } + )* + } +} + +test_connect! { + (ip_string, (|listener: &TcpListener| { + format!("127.0.0.1:{}", listener.local_addr().unwrap().port()) + })), + (ip_str, (|listener: &TcpListener| { + let s = format!("127.0.0.1:{}", listener.local_addr().unwrap().port()); + let slice: &str = &*Box::leak(s.into_boxed_str()); + slice + })), + (ip_port_tuple, (|listener: &TcpListener| { + let addr = listener.local_addr().unwrap(); + (addr.ip(), addr.port()) + })), + (ip_port_tuple_ref, (|listener: &TcpListener| { + let addr = listener.local_addr().unwrap(); + let tuple_ref: &(IpAddr, u16) = &*Box::leak(Box::new((addr.ip(), addr.port()))); + tuple_ref + })), + (ip_str_port_tuple, (|listener: &TcpListener| { + let addr = listener.local_addr().unwrap(); + ("127.0.0.1", addr.port()) + })), +} + +#[monoio::test_all(timer_enabled = true)] +async fn connect_timeout_dst() { + let drop_flag = DropFlag::default(); + let drop_flag_copy = drop_flag.clone(); + { + let connect = async move { + let _unused = drop_flag_copy; + TcpStream::connect("1.1.1.1:1").await + }; + + let res = monoio::select! { + _ = connect => { false } + _ = monoio::time::sleep(std::time::Duration::from_secs(1)) => { true } + }; + assert!(res); + } + drop_flag.assert_dropped(); +} + +#[monoio::test_all] +async fn connect_invalid_dst() { + assert!(TcpStream::connect("127.0.0.1:1").await.is_err()); +} + +#[monoio::test_all(timer_enabled = true)] +async fn cancel_read() { + use monoio::io::CancelableAsyncReadRent; + + let mut s = TcpStream::connect("rsproxy.cn:80").await.unwrap(); + let buf = vec![0; 20]; + + let canceler = monoio::io::Canceller::new(); + let handle = canceler.handle(); + monoio::spawn(async move { + monoio::time::sleep(std::time::Duration::from_millis(100)).await; + canceler.cancel(); + }); + let (res, _) = s.cancelable_read(buf, handle).await; + assert!(res.is_err()); +} + +#[monoio::test_all(timer_enabled = true)] +async fn cancel_select() { + use std::pin::pin; + + use monoio::io::CancelableAsyncReadRent; + + let mut s = TcpStream::connect("rsproxy.cn:80").await.unwrap(); + let buf = vec![0; 20]; + + let canceler = monoio::io::Canceller::new(); + let handle = canceler.handle(); + + let mut timer = pin!(monoio::time::sleep(std::time::Duration::from_millis(100))); + let mut recv = pin!(s.cancelable_read(buf, handle)); + + monoio::select! { + _ = &mut timer => { + canceler.cancel(); + let (res, _buf) = recv.await; + assert!(res.is_err()); + }, + _ = &mut recv => { + // process data + } + } +} + +#[derive(Default, Clone)] +struct DropFlag(std::rc::Rc>); + +impl Drop for DropFlag { + fn drop(&mut self) { + *self.0.borrow_mut() = true; + } +} + +impl DropFlag { + fn assert_dropped(&self) { + assert!(*self.0.borrow()); + } +} diff --git a/vendor/monoio/tests/tcp_echo.rs b/vendor/monoio/tests/tcp_echo.rs new file mode 100644 index 000000000..d3aca3a1b --- /dev/null +++ b/vendor/monoio/tests/tcp_echo.rs @@ -0,0 +1,124 @@ +use monoio::{ + io::{self, AsyncReadRentExt, AsyncWriteRentExt, Splitable}, + net::{TcpListener, TcpStream}, +}; + +#[monoio::test_all] +async fn echo_server() { + const ITER: usize = 1024; + + let (tx, rx) = local_sync::oneshot::channel(); + + let srv = TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = srv.local_addr().unwrap(); + + let msg = "foo bar baz"; + let iov_msg = "iovec_is_so_good"; + monoio::spawn(async move { + let mut stream = TcpStream::connect(&addr).await.unwrap(); + + let mut buf_vec_to_write: Option>> = Some(vec![ + iov_msg.as_bytes()[..9].into(), + iov_msg.as_bytes()[9..].into(), + ]); + for _ in 0..ITER { + // write + assert!(stream.write_all(msg).await.0.is_ok()); + + // read + let buf = Box::new([0; 11]); + let (res, buf) = stream.read_exact(buf).await; + assert!(res.is_ok()); + assert_eq!(res.unwrap(), 11); + assert_eq!(&buf[..], msg.as_bytes()); + + // writev + let buf_vec: monoio::buf::VecBuf = buf_vec_to_write.take().unwrap().into(); + let (res, buf_vec) = stream.write_vectored_all(buf_vec).await; + let raw_vec: Vec> = buf_vec.into(); + assert!(res.is_ok()); + assert_eq!(res.unwrap(), iov_msg.len()); + buf_vec_to_write = Some(raw_vec); + + // readv + let buf_vec: monoio::buf::VecBuf = vec![vec![0; 3], vec![0; iov_msg.len() - 3]].into(); + let (res, buf_vec) = stream.read_vectored_exact(buf_vec).await; + assert!(res.is_ok()); + assert_eq!(res.unwrap(), iov_msg.len()); + let raw_vec: Vec> = buf_vec.into(); + assert_eq!(&raw_vec[0], &iov_msg.as_bytes()[..3]); + assert_eq!(&raw_vec[1], &iov_msg.as_bytes()[3..]); + } + + assert!(tx.send(()).is_ok()); + }); + + let (stream, _) = srv.accept().await.unwrap(); + let (mut rd, mut wr) = stream.into_split(); + + // todo fix these CI in windows + #[cfg(not(windows))] + { + let n = io::copy(&mut rd, &mut wr).await.unwrap(); + assert_eq!(n, (ITER * (msg.len() + iov_msg.len())) as u64); + + assert!(rx.await.is_ok()); + } +} + +#[monoio::test_all(timer_enabled = true)] +async fn rw_able() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let listener_addr = listener.local_addr().unwrap(); + + monoio::select! { + _ = monoio::time::sleep(std::time::Duration::from_millis(50)) => {}, + _ = listener.readable(false) => { + panic!("unexpected readable"); + } + } + let mut active = TcpStream::connect(listener_addr).await.unwrap(); + + assert!(active.writable(false).await.is_ok()); + assert!(listener.readable(false).await.is_ok()); + let (conn, _) = listener.accept().await.unwrap(); + monoio::select! { + _ = monoio::time::sleep(std::time::Duration::from_millis(50)) => {}, + _ = conn.readable(false) => { + panic!("unexpected readable"); + } + _ = active.readable(false) => { + panic!("unexpected readable"); + } + _ = listener.readable(false) => { + // even listener's inner readiness state is ready, we will check it again + panic!("unexpected readable"); + } + } + let (res, _) = active.write_all("MSG").await; + assert!(res.is_ok()); + assert!(conn.readable(false).await.is_ok()); +} + +#[monoio::test_all] +async fn echo_tfo() { + use std::net::SocketAddr; + + let bind_addr = "127.0.0.1:0".parse::().unwrap(); + let opts = monoio::net::ListenerOpts::default().tcp_fast_open(true); + let listener = TcpListener::bind_with_config(bind_addr, &opts).unwrap(); + let addr = listener.local_addr().unwrap(); + let (tx, rx) = local_sync::oneshot::channel(); + monoio::spawn(async move { + let (mut socket, active_addr) = listener.accept().await.unwrap(); + socket.read_exact(vec![0; 2]).await.0.unwrap(); + assert!(tx.send(active_addr).is_ok()); + }); + let opts = monoio::net::TcpConnectOpts::default().tcp_fast_open(true); + let mut active = TcpStream::connect_addr_with_config(addr, &opts) + .await + .unwrap(); + active.write_all(b"hi").await.0.unwrap(); + let active_addr = rx.await.unwrap(); + assert_eq!(active.local_addr().unwrap(), active_addr); +} diff --git a/vendor/monoio/tests/tcp_into_split.rs b/vendor/monoio/tests/tcp_into_split.rs new file mode 100644 index 000000000..4fc10124b --- /dev/null +++ b/vendor/monoio/tests/tcp_into_split.rs @@ -0,0 +1,126 @@ +use std::{ + io::{Error, ErrorKind, Read, Result, Write}, + net, thread, +}; + +use monoio::{ + io::{AsyncReadRent, AsyncWriteRentExt, Splitable}, + net::{TcpListener, TcpStream}, + try_join, +}; + +#[monoio::test_all] +async fn split() -> Result<()> { + const MSG: &[u8] = b"split"; + + let listener = TcpListener::bind("127.0.0.1:0")?; + let addr = listener.local_addr()?; + + let (stream1, (mut stream2, _)) = try_join! { + TcpStream::connect(&addr), + listener.accept(), + }?; + let (mut read_half, mut write_half) = stream1.into_split(); + + let ((), (), ()) = try_join! { + async { + let len = stream2.write_all(MSG).await.0?; + assert_eq!(len, MSG.len()); + + let read_buf = vec![0u8; 32]; + let (read_res, read_buf) = stream2.read(read_buf).await; + assert_eq!(read_res.unwrap(), MSG.len()); + assert_eq!(&read_buf[..MSG.len()], MSG); + Result::Ok(()) + }, + async { + let len = write_half.write_all(MSG).await.0?; + assert_eq!(len, MSG.len()); + Ok(()) + }, + async { + let read_buf = vec![0u8; 32]; + let (read_res, read_buf) = read_half.read(read_buf).await; + assert_eq!(read_res.unwrap(), MSG.len()); + assert_eq!(&read_buf[..MSG.len()], MSG); + Ok(()) + }, + }?; + + Ok(()) +} + +#[monoio::test_all(enable_timer = true)] +async fn reunite() -> Result<()> { + let listener = net::TcpListener::bind("127.0.0.1:0")?; + let addr = listener.local_addr()?; + + let handle = thread::spawn(move || { + drop(listener.accept().unwrap()); + drop(listener.accept().unwrap()); + }); + + let stream1 = TcpStream::connect(&addr).await?; + let (read1, write1) = stream1.into_split(); + + let stream2 = TcpStream::connect(&addr).await?; + let (_, write2) = stream2.into_split(); + + let read1 = match read1.reunite(write2) { + Ok(_) => panic!("Reunite should not succeed"), + Err(err) => err.0, + }; + + read1.reunite(write1).expect("Reunite should succeed"); + + handle.join().unwrap(); + Ok(()) +} + +/// Test that dropping the write half actually closes the stream. +#[monoio::test_all(enable_timer = true, entries = 1024)] +async fn drop_write() -> Result<()> { + const MSG: &[u8] = b"split"; + + let listener = net::TcpListener::bind("127.0.0.1:0")?; + let addr = listener.local_addr()?; + + let handle = thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + stream.write_all(MSG).unwrap(); + + let mut read_buf = [0u8; 32]; + let res = match stream.read(&mut read_buf) { + Ok(0) => Ok(()), + Ok(len) => Err(Error::new( + ErrorKind::Other, + format!("Unexpected read: {len} bytes."), + )), + Err(err) => Err(err), + }; + + drop(stream); + + res + }); + + let stream = TcpStream::connect(&addr).await?; + let (mut read_half, write_half) = stream.into_split(); + + let read_buf = vec![0u8; 32]; + let (read_res, read_buf) = read_half.read(read_buf).await; + assert_eq!(read_res.unwrap(), MSG.len()); + assert_eq!(&read_buf[..MSG.len()], MSG); + // drop it while the read is in progress + monoio::spawn(async move { + monoio::time::sleep(std::time::Duration::from_millis(10)).await; + drop(write_half); + }); + match read_half.read(read_buf).await.0 { + Ok(0) => {} + Ok(len) => panic!("Unexpected read: {len} bytes."), + Err(err) => panic!("Unexpected error: {err}."), + } + handle.join().unwrap().unwrap(); + Ok(()) +} diff --git a/vendor/monoio/tests/tcp_split.rs b/vendor/monoio/tests/tcp_split.rs new file mode 100644 index 000000000..4342b4563 --- /dev/null +++ b/vendor/monoio/tests/tcp_split.rs @@ -0,0 +1,38 @@ +use std::{ + io::{Read, Result, Write}, + thread, +}; + +use monoio::{ + io::{AsyncReadRent, AsyncWriteRentExt, Splitable}, + net::TcpStream, +}; + +#[monoio::test_all] +async fn split() -> Result<()> { + const MSG: &[u8] = b"split"; + + let listener = std::net::TcpListener::bind("127.0.0.1:0")?; + let addr = listener.local_addr()?; + + let handle = thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + stream.write_all(MSG).unwrap(); + + let mut read_buf = [0u8; 32]; + let read_len = stream.read(&mut read_buf).unwrap(); + assert_eq!(&read_buf[..read_len], MSG); + }); + + let stream = TcpStream::connect(&addr).await?; + let (mut read_half, mut write_half) = stream.into_split(); + + let read_buf = Box::new([0u8; 32]); + let (read_res, buf) = read_half.read(read_buf).await; + assert_eq!(read_res.unwrap(), MSG.len()); + assert_eq!(&buf[..MSG.len()], MSG); + + write_half.write_all(MSG).await.0?; + handle.join().unwrap(); + Ok(()) +} diff --git a/vendor/monoio/tests/udp.rs b/vendor/monoio/tests/udp.rs new file mode 100644 index 000000000..595e6398b --- /dev/null +++ b/vendor/monoio/tests/udp.rs @@ -0,0 +1,98 @@ +use monoio::net::udp::UdpSocket; + +#[monoio::test_all] +async fn connect() { + const MSG: &str = "foo bar baz"; + + let passive = UdpSocket::bind("127.0.0.1:0").unwrap(); + let passive_addr = passive.local_addr().unwrap(); + + let active = UdpSocket::bind("127.0.0.1:0").unwrap(); + let active_addr = active.local_addr().unwrap(); + + active.connect(passive_addr).await.unwrap(); + active.send(MSG).await.0.unwrap(); + + let (res, buffer) = passive.recv(Vec::with_capacity(20)).await; + res.unwrap(); + assert_eq!(MSG.as_bytes(), &buffer); + assert_eq!(active.local_addr().unwrap(), active_addr); + assert_eq!(active.peer_addr().unwrap(), passive_addr); +} + +#[monoio::test_all] +async fn send_to() { + const MSG: &str = "foo bar baz"; + + macro_rules! must_success { + ($r: expr, $expect_addr: expr) => { + let res = $r; + assert_eq!(res.0.unwrap().1, $expect_addr); + assert_eq!(res.1, MSG.as_bytes()); + }; + } + + let passive1 = UdpSocket::bind("127.0.0.1:0").unwrap(); + let passive1_addr = passive1.local_addr().unwrap(); + + let passive01 = std::net::UdpSocket::bind("127.0.0.1:0").unwrap(); + let passive01_addr = passive01.local_addr().unwrap(); + + let passive2 = UdpSocket::bind("127.0.0.1:0").unwrap(); + let passive2_addr = passive2.local_addr().unwrap(); + + let passive3 = UdpSocket::bind("127.0.0.1:0").unwrap(); + let passive3_addr = passive3.local_addr().unwrap(); + + let active = UdpSocket::bind("127.0.0.1:0").unwrap(); + let active_addr = active.local_addr().unwrap(); + + active.send_to(MSG, passive01_addr).await.0.unwrap(); + active.send_to(MSG, passive1_addr).await.0.unwrap(); + active.send_to(MSG, passive2_addr).await.0.unwrap(); + active.send_to(MSG, passive3_addr).await.0.unwrap(); + + must_success!(passive1.recv_from(vec![0; 20]).await, active_addr); + must_success!(passive2.recv_from(vec![0; 20]).await, active_addr); + must_success!(passive3.recv_from(vec![0; 20]).await, active_addr); +} + +#[monoio::test_all(timer_enabled = true)] +async fn rw_able() { + const MSG: &str = "foo bar baz"; + + let passive = UdpSocket::bind("127.0.0.1:0").unwrap(); + let passive_addr = passive.local_addr().unwrap(); + + let active = UdpSocket::bind("127.0.0.1:0").unwrap(); + + assert!(active.writable(false).await.is_ok()); + monoio::select! { + _ = monoio::time::sleep(std::time::Duration::from_millis(50)) => {}, + _ = passive.readable(false) => { + panic!("unexpected readable"); + } + } + + active.connect(passive_addr).await.unwrap(); + active.send(MSG).await.0.unwrap(); + assert!(passive.readable(false).await.is_ok()); +} + +#[monoio::test_all(timer_enabled = true)] +async fn cancel_recv_from() { + let passive = UdpSocket::bind("127.0.0.1:0").unwrap(); + let canceller = monoio::io::Canceller::new(); + let recv = passive.cancelable_recv_from(vec![0; 20], canceller.handle()); + let mut recv = std::pin::pin!(recv); + + monoio::select! { + _ = monoio::time::sleep(std::time::Duration::from_millis(50)) => { + canceller.cancel(); + assert!(recv.await.0.is_err()); + }, + _ = &mut recv => { + panic!("unexpected readable"); + } + } +} diff --git a/vendor/monoio/tests/uds_cred.rs b/vendor/monoio/tests/uds_cred.rs new file mode 100644 index 000000000..717d6589d --- /dev/null +++ b/vendor/monoio/tests/uds_cred.rs @@ -0,0 +1,17 @@ +#![cfg(unix)] +use libc::{getegid, geteuid}; +use monoio::net::UnixStream; + +#[monoio::test_all] +async fn test_socket_pair() { + let (a, b) = UnixStream::pair().unwrap(); + let cred_a = a.peer_cred().unwrap(); + let cred_b = b.peer_cred().unwrap(); + assert_eq!(cred_a, cred_b); + + let uid = unsafe { geteuid() }; + let gid = unsafe { getegid() }; + + assert_eq!(cred_a.uid(), uid); + assert_eq!(cred_a.gid(), gid); +} diff --git a/vendor/monoio/tests/uds_split.rs b/vendor/monoio/tests/uds_split.rs new file mode 100644 index 000000000..350a51c45 --- /dev/null +++ b/vendor/monoio/tests/uds_split.rs @@ -0,0 +1,44 @@ +#![cfg(unix)] +use monoio::{ + io::{AsyncReadRent, AsyncReadRentExt, AsyncWriteRent, AsyncWriteRentExt, Splitable}, + net::UnixStream, +}; + +/// Checks that `UnixStream` can be split into a read half and a write half +/// using `UnixStream::split` and `UnixStream::split_mut`. +/// +/// Verifies that the implementation of `AsyncWrite::poll_shutdown` shutdowns +/// the stream for writing by reading to the end of stream on the other side of +/// the connection. +#[monoio::test_all(entries = 1024)] +async fn split() -> std::io::Result<()> { + let (a, b) = UnixStream::pair()?; + + let (mut a_read, mut a_write) = a.into_split(); + let (mut b_read, mut b_write) = b.into_split(); + + let (a_response, b_response) = futures::future::try_join( + send_recv_all(&mut a_read, &mut a_write, b"A"), + send_recv_all(&mut b_read, &mut b_write, b"B"), + ) + .await?; + + assert_eq!(a_response, b"B"); + assert_eq!(b_response, b"A"); + + Ok(()) +} + +async fn send_recv_all( + read: &mut R, + write: &mut W, + input: &'static [u8], +) -> std::io::Result> { + write.write_all(input).await.0?; + write.shutdown().await?; + + let output = Vec::with_capacity(2); + let (res, buf) = read.read_exact(output).await; + assert_eq!(res.unwrap_err().kind(), std::io::ErrorKind::UnexpectedEof); + Ok(buf) +} diff --git a/vendor/monoio/tests/uds_stream.rs b/vendor/monoio/tests/uds_stream.rs new file mode 100644 index 000000000..5d6c37726 --- /dev/null +++ b/vendor/monoio/tests/uds_stream.rs @@ -0,0 +1,65 @@ +#![cfg(unix)] +use futures::future::try_join; +use monoio::{ + io::{AsyncReadRent, AsyncReadRentExt, AsyncWriteRent, AsyncWriteRentExt}, + net::{UnixListener, UnixStream}, +}; + +#[monoio::test_all] +async fn accept_read_write() -> std::io::Result<()> { + let dir = tempfile::Builder::new() + .prefix("monoio-uds-tests") + .tempdir() + .unwrap(); + let sock_path = dir.path().join("connect.sock"); + + let listener = UnixListener::bind(&sock_path)?; + + let accept = listener.accept(); + let connect = UnixStream::connect(&sock_path); + let ((mut server, _), client) = try_join(accept, connect).await?; + + // testing into_raw_fd and from_raw_fd + #[cfg(unix)] + use std::os::fd::{FromRawFd, IntoRawFd}; + #[cfg(unix)] + let fd = client.into_raw_fd(); + #[cfg(unix)] + let client = + UnixStream::from_std(unsafe { std::os::unix::net::UnixStream::from_raw_fd(fd) }).unwrap(); + + let mut client = client; + let write_len = client.write_all(b"hello").await.0?; + assert_eq!(write_len, 5); + drop(client); + + let buf = Box::new([0u8; 5]); + let (res, buf) = server.read_exact(buf).await; + assert_eq!(res.unwrap(), 5); + assert_eq!(&buf[..], b"hello"); + let len = server.read(buf).await.0?; + assert_eq!(len, 0); + Ok(()) +} + +#[monoio::test_all] +async fn shutdown() -> std::io::Result<()> { + let dir = tempfile::Builder::new() + .prefix("monoio-uds-tests") + .tempdir() + .unwrap(); + let sock_path = dir.path().join("connect.sock"); + + let listener = UnixListener::bind(&sock_path)?; + + let accept = listener.accept(); + let connect = UnixStream::connect(&sock_path); + let ((mut server, _), mut client) = try_join(accept, connect).await?; + + // Shut down the client + client.shutdown().await?; + // Read from the server should return 0 to indicate the channel has been closed. + let n = server.read(Box::new([0u8; 1])).await.0?; + assert_eq!(n, 0); + Ok(()) +} diff --git a/vendor/monoio/tests/unix_datagram.rs b/vendor/monoio/tests/unix_datagram.rs new file mode 100644 index 000000000..67fec8834 --- /dev/null +++ b/vendor/monoio/tests/unix_datagram.rs @@ -0,0 +1,45 @@ +#![cfg(unix)] +use monoio::net::unix::UnixDatagram; + +#[monoio::test_all] +async fn accept_send_recv() -> std::io::Result<()> { + let dir = tempfile::Builder::new() + .prefix("monoio-unix-datagram-tests") + .tempdir() + .unwrap(); + let sock_path = dir.path().join("dgram.sock"); + + let dgram1 = UnixDatagram::bind(&sock_path)?; + let dgram2 = UnixDatagram::connect(&sock_path).await?; + + dgram2.send(b"hello").await.0.unwrap(); + let (_res, buf) = dgram1.recv_from(vec![0; 100]).await; + assert_eq!(buf, b"hello"); + assert!(_res.unwrap().1.is_unnamed()); + + let dgram3 = UnixDatagram::unbound()?; + dgram3.send_to(b"hello2", &sock_path).await.0.unwrap(); + let (res, buf) = dgram1.recv(vec![0; 100]).await; + assert_eq!(buf, b"hello2"); + assert_eq!(res.unwrap(), 6); + Ok(()) +} + +#[monoio::test_all] +async fn addr_type() -> std::io::Result<()> { + let dir = tempfile::Builder::new() + .prefix("monoio-unix-datagram-tests") + .tempdir() + .unwrap(); + let sock_path1 = dir.path().join("dgram_addr1.sock"); + let sock_path2 = dir.path().join("dgram_addr2.sock"); + + let dgram1 = UnixDatagram::bind(&sock_path1)?; + let dgram2 = UnixDatagram::bind(&sock_path2)?; + + dgram1.send_to(b"hello", sock_path2).await.0.unwrap(); + let (_res, buf) = dgram2.recv_from(vec![0; 100]).await; + assert_eq!(buf, b"hello"); + assert_eq!(_res.unwrap().1.as_pathname(), Some(sock_path1.as_path())); + Ok(()) +} diff --git a/vendor/monoio/tests/unix_seqpacket.rs b/vendor/monoio/tests/unix_seqpacket.rs new file mode 100644 index 000000000..65fc21efa --- /dev/null +++ b/vendor/monoio/tests/unix_seqpacket.rs @@ -0,0 +1,22 @@ +#[cfg(target_os = "linux")] +#[monoio::test_all] +async fn test_seqpacket() -> std::io::Result<()> { + use monoio::net::unix::{UnixSeqpacket, UnixSeqpacketListener}; + + let dir = tempfile::Builder::new() + .prefix("monoio-unix-seqpacket-tests") + .tempdir() + .unwrap(); + let sock_path = dir.path().join("seqpacket.sock"); + + let listener = UnixSeqpacketListener::bind(&sock_path).unwrap(); + monoio::spawn(async move { + let (conn, _addr) = listener.accept().await.unwrap(); + let (res, buf) = conn.recv(vec![0; 100]).await; + assert_eq!(res.unwrap(), 5); + assert_eq!(buf, b"hello"); + }); + let conn = UnixSeqpacket::connect(&sock_path).await.unwrap(); + conn.send(b"hello").await.0.unwrap(); + Ok(()) +} diff --git a/vendor/monoio/tests/zero_copy.rs b/vendor/monoio/tests/zero_copy.rs new file mode 100644 index 000000000..6abed191c --- /dev/null +++ b/vendor/monoio/tests/zero_copy.rs @@ -0,0 +1,63 @@ +#[cfg(all(target_os = "linux", feature = "splice"))] +#[monoio::test_all] +async fn zero_copy_for_tcp() { + use monoio::{ + buf::IoBufMut, + io::{zero_copy, AsyncReadRentExt, AsyncWriteRentExt, Splitable}, + net::TcpStream, + }; + + const MSG: &[u8] = b"copy for split"; + let srv = monoio::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let (mut c_tx, mut c_rx) = local_sync::oneshot::channel::<()>(); + let addr = srv.local_addr().unwrap(); + monoio::spawn(async move { + let stream = TcpStream::connect(&addr).await.unwrap(); + let (mut rx, mut tx) = stream.into_split(); + tx.write_all(MSG).await.0.unwrap(); + let buf = Vec::::with_capacity(MSG.len()).slice_mut(0..MSG.len()); + let (res, buf) = rx.read_exact(buf).await; + let buf = buf.into_inner(); + res.unwrap(); + assert_eq!(&buf, MSG); + c_rx.close(); + }); + let (conn, _) = srv.accept().await.unwrap(); + let (mut rx, mut tx) = conn.into_split(); + assert_eq!(zero_copy(&mut rx, &mut tx).await.unwrap(), MSG.len() as u64); + c_tx.closed().await; +} + +#[cfg(all(target_os = "linux", feature = "splice"))] +#[monoio::test_all] +async fn zero_copy_for_uds() { + use monoio::{ + buf::IoBufMut, + io::{zero_copy, AsyncReadRentExt, AsyncWriteRentExt, Splitable}, + net::UnixStream, + }; + + const MSG: &[u8] = b"copy for split"; + let dir = tempfile::Builder::new() + .prefix("monoio-uds-tests") + .tempdir() + .unwrap(); + let sock_path = dir.path().join("zero_copy.sock"); + let srv = monoio::net::UnixListener::bind(&sock_path).unwrap(); + let (mut c_tx, mut c_rx) = local_sync::oneshot::channel::<()>(); + monoio::spawn(async move { + let stream = UnixStream::connect(&sock_path).await.unwrap(); + let (mut rx, mut tx) = stream.into_split(); + tx.write_all(MSG).await.0.unwrap(); + let buf = Vec::::with_capacity(MSG.len()).slice_mut(0..MSG.len()); + let (res, buf) = rx.read_exact(buf).await; + let buf = buf.into_inner(); + res.unwrap(); + assert_eq!(&buf, MSG); + c_rx.close(); + }); + let (conn, _) = srv.accept().await.unwrap(); + let (mut rx, mut tx) = conn.into_split(); + assert_eq!(zero_copy(&mut rx, &mut tx).await.unwrap(), MSG.len() as u64); + c_tx.closed().await; +}