Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/reference/configuration/_pro-replication-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ replication:
- `url` β€” WebSocket URL peers use to connect to this instance
- `databases` β€” Databases to replicate; _Default_: `"*"` (all). Each entry supports `name`, `replicateTo`, and `sharded`
- `routes` β€” Peer nodes; URL strings or `{hostname, port, startTime, revokedCertificates}` objects
- `privateUrls` <VersionBadge version="v5.5.0" /> β€” Addresses where peers on a private network can reach this instance, as `{url, network}` objects. `url` must be a `wss://` URL with no credentials; `network` is a tag you choose for the network (for example a VPC id and region). A peer that dials from the same network uses this URL instead of `url`. See [Private Network Addresses](../replication/overview.md#private-network-addresses)
- `network` <VersionBadge version="v5.5.0" /> β€” The network this instance dials peers from, in addition to the networks in its own `privateUrls`. Needed only for an instance that is not itself reachable on that network
- `privateUrls` <VersionBadge version="v5.4.0" /> β€” Addresses where peers on a private network can reach this instance, as `{url, network}` objects. `url` must be a `wss://` URL with no credentials; `network` is a tag you choose for the network (for example a VPC id and region). A peer that dials from the same network uses this URL instead of `url`. See [Private Network Addresses](../replication/overview.md#private-network-addresses)
- `network` <VersionBadge version="v5.4.0" /> β€” The network this instance dials peers from, in addition to the networks in its own `privateUrls`. Needed only for an instance that is not itself reachable on that network
- `port` β€” Replication port
- `securePort` β€” Secure replication port; _Default_: `9933` (changed from `9925` in v4.5.0)
- `threads` β€” Worker threads dedicated to replication, which then own the replication port and every subscription and load no application code; requires `port` or `securePort`; see [Dedicated Replication Threads](../replication/overview.md#dedicated-replication-threads); _Default_: `0` (replication runs on the HTTP worker threads) (Added in: v5.4.0)
- `threads` β€” Worker threads dedicated to replication, which then own the replication port and every subscription and load no application code; an explicit positive value requires `port` or `securePort`; see [Dedicated Replication Threads](../replication/overview.md#dedicated-replication-threads); _Default_: half of `threads.count`, rounded down, so `0` below two HTTP worker threads; an omitted value falls back to `0`, with a warning, when there is no dedicated replication port or an application registers a hook these threads cannot run; set `0` to keep replication on the HTTP worker threads (Added in: v5.4.0)
- `recordLocks` β€” Cluster-wide `lock()` on replicated databases, and from v5.4.0 lock-backed conditional REST writes; set it on every node, then apply a home map with `record_lock_apply_homes`; see [Cluster Record Locks](../replication/overview.md#cluster-record-locks). Fabric turns it on itself. Requires RocksDB; _Default_: `false` (Added in: v5.3.0)
- `enableRootCAs` β€” Verify against Node.js Mozilla CA store; _Default_: `true`
- `pingInterval` β€” Interval between replication keep-alive pings (ms); _Default_: `30000`
Expand Down
2 changes: 1 addition & 1 deletion docs/reference/replication/clustering.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ Returns an array of status objects from the cluster, including active WebSocket
| `lastReceivedRemoteTime` | Timestamp (from the originating node) of the last received transaction |
| `lastReceivedLocalTime` | Local time when the last transaction was received. A gap between this and `lastReceivedRemoteTime` suggests the node is catching up |
| `sendingMessage` | Timestamp of the transaction actively being sent. Absent when waiting for the next transaction |
| `path` | Which address this node dialed for the connection: `public` (the peer's `url`), `private` (a [private network address](./overview.md#private-network-addresses)), or `public-fallback` (the peer has a private address on this node's network, but it is not connecting). Each worker thread notices an outage and recovers on its own, so sockets to one peer that different threads handle can differ for the length of an outage. Absent while disconnected or before the peer has identified itself. Added in v5.5.0 |
| `path` | Which address this node dialed for the connection: `public` (the peer's `url`), `private` (a [private network address](./overview.md#private-network-addresses)), or `public-fallback` (the peer has a private address on this node's network, but it is not connecting). Each worker thread notices an outage and recovers on its own, so sockets to one peer that different threads handle can differ for the length of an outage. Absent while disconnected or before the peer has identified itself. Added in v5.4.0 |

While a database is taking its initial full copy, these timing fields hold the literal string `"Copying"` rather than a timestamp. Anything that parses them as dates needs to handle that value, since a node taking a full copy is exactly the node you are most likely to be watching.

Expand Down
8 changes: 5 additions & 3 deletions docs/reference/replication/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ As of v5.2, this full-mesh, bidirectional auto-connect behavior applies to nodes

### Private Network Addresses

<VersionBadge version="v5.5.0" />
<VersionBadge version="v5.4.0" />

Some providers bill traffic between public addresses even inside one region, while traffic over a private network (a VLAN or VPC) is free or cheaper. Nodes that share a private network can replicate over it:

Expand Down Expand Up @@ -105,7 +105,7 @@ Transactions are replicated atomically, which may span multiple tables. You can

<VersionBadge version="v5.4.0" />

By default, replication runs on the same worker threads that serve application requests. Catching up a peer that was offline (streaming its backlog, a full copy, back-pressure handling) can then compete with your application for those threads. Setting `replication.threads` starts that many additional worker threads that run replication only:
When replication runs on the same worker threads that serve application requests, catching up a peer that was offline (streaming its backlog, a full copy, back-pressure handling) competes with your application for those threads. `replication.threads` starts that many additional worker threads that run replication only. It defaults to half of `threads.count`, rounded down, so a node with two or more HTTP worker threads and a dedicated replication port gets them without configuration; set it explicitly to choose the count:

```yaml
replication:
Expand All @@ -117,10 +117,12 @@ With dedicated replication threads:

- Only the replication threads listen on the replication port, and they own every subscription to other nodes, so both sending and receiving replication data happen off the application threads.
- The replication threads load no application code. Anything your application installs on a table at runtime is not present there, so applications that rely on it should keep `replication.threads: 0`:
- `setResidency`: replicated writes that carry no residency list are placed by the default residency instead of by your callback.
- `setResidencyById`: nodes outside a record's residency receive an invalidated, partial copy of it that holds the record's indexed attribute values.
- `setComputedAttribute` on an `@indexed` computed attribute: a replication thread cannot maintain that index, so it refuses to receive the database that contains the table. It logs the table and attributes and keeps retrying, and nothing else in that database replicates to this node until `replication.threads` is set back to `0`. Computed attributes defined with `@computed(from: "...")` are not affected.
- Reads that fetch a missing record from another node still run on the thread serving the request.
- A dedicated replication port is required: set `replication.securePort` (default `9933`) or `replication.port`, different from the HTTP and operations API ports. Harper refuses to start otherwise.
- A dedicated replication port is required: set `replication.securePort` (default `9933`) or `replication.port`, different from the HTTP and operations API ports. With an explicit `replication.threads`, Harper refuses to start otherwise; with the default, it logs a warning and starts no replication threads.
- When `replication.threads` is not set and an application registers `setResidency`, `setResidencyById` or `setComputedAttribute` on an indexed attribute at startup, no replication threads start, and a warning names the tables. These hooks are deprecated in v5.4.0.
- `threads.count: 0` (no worker threads) ignores the setting.
- On macOS and Windows, which lack `SO_REUSEPORT`, only one replication thread listens on the replication port; the others carry outbound subscriptions.
- Changing the number of threads takes effect only on a full `restart` of Harper; `restart_service` does not start or stop replication threads. An unscoped `restart_service` does restart the existing replication threads, and deploying or dropping a component does not.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
section: 'Replication'
---

### Cloned Applications Keep Their node_modules

A clone used to rebuild each filesystem-only application's `node_modules` with `npm install`, so a package whose only copy was in `node_modules` (a `file:./node_modules/...` dependency, or a private workspace package in a Next.js standalone build) was lost, and the clone still reported `Available` and served `500` errors. A clone now copies each such application with its `node_modules` from a leader running 5.2.3 or later. A clone from an older leader, or one authenticated by certificate, rebuilds with npm as before and logs that `node_modules`-only packages will be missing. An application that still fails after three attempts stops clone setup, and the node stays `Unavailable`. A request to the leader that receives no data for 10 minutes now fails instead of waiting forever. This is also in 5.3.3. See [harper-pro#1050](https://github.com/HarperFast/harper-pro/pull/1050).

A clone that cannot save its attempt marker (`.cloneAttempt.json`) now stops with an error naming the path before it contacts the leader. Before, it cloned without the safeguards replication applies to a clone. See [harper-pro#1250](https://github.com/HarperFast/harper-pro/pull/1250).
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
section: 'Replication'
order: 60
---

### Cluster Record Locks Stay Opt-In

Cluster record locks remain off unless `replication.recordLocks: true` is set. Harper Fabric turns them on for its clusters; a self-hosted cluster opts in. While they are off, a cluster `lock()` on a replicated table answers `503`, and the message now names the fix: set `replication.recordLocks: true` on every node, restart each one, and apply a home map with `record_lock_apply_homes`. With locks on and no home map applied yet, the `503` names `record_lock_apply_homes` too. Conditional REST writes do not need cluster record locks: while they are off, those writes are checked without a lock, as in 5.3 (see [Conditional Writes Without Cluster Record Locks](#conditional-writes-without-cluster-record-locks)). See [Cluster Record Locks](/reference/v5/replication/overview#cluster-record-locks), [harper-pro#1316](https://github.com/HarperFast/harper-pro/pull/1316) and [harper-pro#1350](https://github.com/HarperFast/harper-pro/pull/1350).
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ order: 40

### Dedicated Replication Threads

`replication.threads` (default `0`) starts worker threads that run replication only. They own the replication port and every subscription, so a peer catching up no longer competes with application requests. These threads load no application code, which limits applications that use `setResidencyById` or `setComputedAttribute` on indexed attributes. See [Dedicated Replication Threads](/reference/v5/replication/overview#dedicated-replication-threads).
Nodes with two or more HTTP worker threads now start worker threads that run replication only: `replication.threads` defaults to half of `threads.count`, rounded down. These threads own the replication port and every subscription, so a peer catching up no longer competes with application requests. They need a replication port of their own (`replication.securePort` or `replication.port`). Without one, an omitted `replication.threads` falls back to `0` with a warning, and an explicit value stops startup.

The threads load no application code, which limits applications that use `setResidency`, `setResidencyById` or `setComputedAttribute` on an indexed attribute. Those hooks are deprecated in 5.4 and warn when registered. When `replication.threads` is omitted and an application registers one at startup, no replication threads start and a warning names the tables. Set `replication.threads: 0` to keep replication on the HTTP worker threads. See [Dedicated Replication Threads](/reference/v5/replication/overview#dedicated-replication-threads).
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
section: 'Container Images'
order: 5
---

### Harper Pro Images Run Node 26

The default `harperfast/harper-pro` tags, which Fabric clusters also use, now run Node 26, the current Active LTS line, instead of Node 24. The `-node22`, `-node24` and `-node26` tags are still published, so a deployment that needs to stay on Node 24 can pin `-node24`. The `-pc` (pointer-compression) image follows the same default. The OpenShift image (`harperfast/harper-pro-openshift`) stays on Node 24, since its UBI base has no Node 26 stream yet. See [harper-pro#1119](https://github.com/HarperFast/harper-pro/pull/1119).
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
section: 'Memory'
order: 20
---

### Harper Pro Workers Start With a Smaller Heap

Harper Pro now builds with core's comment-free production output, which saves about 10 MB of JavaScript heap in each worker. Its workers also load LMDB, `node-forge` and lodash only when a feature that needs them runs. This adds to the [smaller per-thread baseline](#smaller-per-thread-baseline) in core. See [harper-pro#992](https://github.com/HarperFast/harper-pro/pull/992) and [harper-pro#995](https://github.com/HarperFast/harper-pro/pull/995).
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
section: 'Known Issues'
---

### Removing a Node Can Lose Its Undelivered Writes

`remove_node` does not wait for the removed node's writes to reach a peer. A write the node accepted just before its removal, or while a link to its peers was down, can be lost, and nothing reports it. This applies to a node that has stopped too, if it stopped before delivering those writes. On a sharded table, a node that owns none of a record's slots holds a write it accepted until it delivers it to the owners, so removing a [replica node](#replica-nodes-in-a-spare-shard) can lose writes the same way; scale-in of replica nodes is not supported yet. Tracked in [harper-pro#1348](https://github.com/HarperFast/harper-pro/issues/1348).
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
section: 'Container Images'
order: 20
---

### The npm Package Ships Its Tested Dependencies

`@harperfast/harper-pro` on npm now bundles the JavaScript dependency tree it was tested with, instead of publishing an `npm-shrinkwrap.json`, which npm 12 ignores and a tarball install never reads. `npm install -g @harperfast/harper-pro` then installs the same bundled JavaScript tree on npm 10, 11 and 12; the native packages `re2` and `@datadog/pprof` are installed separately, and `re2` and both packages' own dependencies still resolve from version ranges. The package is larger (about 41 MB gzipped, up from 18.5 MB). npm 12 skips dependency install scripts by default, and without them the WAF component does not load; install with `npm install -g --allow-scripts=re2 @harperfast/harper-pro`, as the [README](https://github.com/HarperFast/harper-pro#installing-harper-pro) shows. This is also in 5.3.2. See [harper-pro#981](https://github.com/HarperFast/harper-pro/pull/981).
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
section: 'Replication'
order: 14
---

### Replication Over Private Network Addresses

Nodes that share a private network (a VLAN or VPC) can now replicate over it instead of their public addresses, which some providers bill even within a region. A node lists its private addresses in `replication.privateUrls`, each a `wss://` URL with a `network` tag, and a peer whose own `privateUrls` or `replication.network` names the same tag dials that address instead of the node's `url`. The connection keeps the public URL's TLS server name and certificate, so certificates and SNI proxies need no change. If the private address does not connect, the link uses `url` and checks the private address again later. `cluster_status` reports each connection's `path`: `public`, `private` or `public-fallback`. Older nodes ignore the new settings and keep dialing `url`. See [Private Network Addresses](/reference/v5/replication/overview#private-network-addresses) and [harper-pro#1294](https://github.com/HarperFast/harper-pro/pull/1294).
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
section: 'Replication'
order: 62
---

### Cluster Record Locks Across Node Restarts and Stopped Nodes

Cluster record locks stay available while a node restarts:

- **A restarted lock home recalls its delegations.** A home that restarted used to answer `503` on every key it homes for about six minutes while it waited out every delegation its previous run could have issued. It now asks the other members of its ring and ends that wait as soon as they answer. When they cannot be reached, it still waits. ([harper-pro#1059](https://github.com/HarperFast/harper-pro/pull/1059))
- **Keys a restarted node held are released.** Keys a node held for others stayed granted on their home until the lease ran out, and the node's own home keys failed with `503` while it rejoined. The restart now retires the node's old grants, and other nodes answer "recovering" instead of failing. ([harper-pro#1193](https://github.com/HarperFast/harper-pro/pull/1193))
- **A lock waits out a restarting peer.** A `lock()` that needs an up-to-date check from a node that is restarting, including a worker restart, keeps asking until its own timeout instead of failing with `503` on the first refusal or after 10 seconds. ([harper-pro#1196](https://github.com/HarperFast/harper-pro/pull/1196), [harper-pro#1205](https://github.com/HarperFast/harper-pro/pull/1205))

`record_lock_apply_homes` accepts `fenced`, a list of nodes you attest are stopped outside Harper, so a home-map change can complete past a node that is gone for good instead of failing with `503` or `409`. It needs a majority of the old map's nodes to answer. When it refuses, its `409` lists the per-node steps that recover. See [Cluster Record Locks](/reference/v5/replication/overview#cluster-record-locks), [harper-pro#1065](https://github.com/HarperFast/harper-pro/pull/1065) and [harper-pro#1305](https://github.com/HarperFast/harper-pro/pull/1305).
Loading
Loading