feat(openfga): add psk auth to config - #656
Open
NJona wants to merge 11 commits into
Open
Conversation
Until now a bootstrapped project was implicitly a single data center: its nodes, gateway IPs, config paths and domains all lived directly on CodesphereEnvironment. Multi-DC support needs more than one of each, so this introduces the DataCenter type that holds everything which must differ per data center, while project-level state (project, VPC, jumpbox, shared postgres node, registry) stays on the environment. BuildDataCenters derives the layout from the flags: one entry today, and with --multi-dc a second one that shares the first's PostgreSQL server. The primary data center keeps an empty resource-name suffix, so every name, path and domain a single-DC bootstrap produces is unchanged. Nothing consumes the layout yet — the callers are migrated in the following commits. Two mechanisms keep that migration safe: - ensureDataCenters derives the layout on first use and adopts state a caller passed through the legacy top-level environment fields, so every entry point works whether or not Bootstrap ran first, including infra files written before multi-DC support. - mirrorPrimaryDataCenter projects the primary data center back onto those fields before the infra file is written, so cleanup and restart-vms keep reading what they always have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
The VM list was a package-level global describing one data center. It becomes a function of the environment: the project-shared jumpbox and postgres VMs plus each data center's three Ceph and three k0s nodes, whose names carry that data center's suffix. A single-DC bootstrap therefore still gets exactly ceph-1..3 and k0s-1..3. EnsureComputeInstances routes each instance into its own data center by the definition's DataCenterID instead of inferring placement from the VM's tag, and sorts each data center's nodes independently — the install config assigns roles by index. restart-vms resolves the valid VM names from the infra file's data center layout, so a node of a second data center can be restarted by name. It now also reads the infra file when --project-id and --zone are given, best-effort, since only the file knows the layout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Migrates the infrastructure steps from the single implicit data center to the layout: root login and host configuration run over every data center's nodes, each data center reserves its own gateway, public gateway and SSH proxy IP under a suffixed name, and each gets its own k0s configuration script patching its own gateway services. InstallCodesphere and RunK0sConfigScript now loop over the data centers in ascending order and log a step per data center. The order matters once there is more than one: the primary data center's install creates the database, roles and schema the others reuse, and a k0s script can only patch gateway services an install has already created. The install command moves into the exported InstallCommand, since the CLI prints it for the operator when the bootstrap does not install itself, and it now names the data center's own config, vault and age key. EnsureHostsConfigured also creates /etc/codesphere/secrets up front on every node: the installer uploads a data center's age key to that fixed path but only creates its own configured secrets.baseDir, so for a data center whose baseDir differs the upload target would not exist. Behaviour for a single data center is unchanged — same VM names, same IP names, same script, same install command. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Workspaces resolve per data center, and so does the platform: the frontend asks <dc-id>.<codesphere.domain> for the configuration of the data center a workspace lives in. OMS only created cs.<base-domain> and its wildcard, both pointing at the first data center's gateway, so with more than one data center the second one's endpoint resolved to the first's gateway, which has no route for that host — every browser request for it was reset, and since the frontend fetches that config before rendering, the whole UI failed. EnsureDNSRecords now creates, per data center, its workspace hosting names and SSH proxy name pointing at its own public gateway and SSH proxy, plus <dc-id>.cs.<base-domain> and its wildcard pointing at its own platform gateway. The per-data-center platform names are only created when there is more than one data center: a single one is the primary, which cs.<base-domain> already resolves to. The records that were created are recorded in the infra file, so cleanup deletes exactly those. DeleteDNSRecordSets therefore takes the record list instead of a base domain, and cleanup falls back to deriving the names for infra files written before this and for a cleanup driven only by --project-id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Config and vault generation worked on b.Env.InstallConfig and the single b.icg. Both become per data center: every step takes the DataCenter it generates for and reads its nodes, IPs, paths and config manager from there. The exported Ensure*/Update* entry points keep their signatures and operate on the primary data center, so step names and existing callers are unchanged. Two details worth pointing out: - secrets.baseDir is now the data center's own directory. The installer resolves the vault it reads and writes back from that path, so sharing it between data centers would let one data center's kubernetes and ceph steps overwrite the other's kubeconfig and Ceph credentials. - the local config, vault and remote paths come from the data center, so a second data center writes config-dc2.yaml rather than overwriting config.yaml. Pure refactoring: a single-data-center bootstrap produces the same config and vault as before. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
The registry was written straight into the single install config, so which registry the nodes pull from was decided per config rather than per project. It now resolves onto the environment (ContainerRegistryURL plus credentials) and updateInstallConfig applies it to every data center's config and vault. All three registry types go through the same field, which also fixes the artifact registry never recording its URI on the create path. EnsureLocalContainerRegistry is split in two, because it early-returned when the registry was already running and thereby skipped distributing the registry certificate. A re-run that adds a data center hits exactly that path, and its nodes would then fail every pull with "certificate signed by unknown authority". Starting the registry stays conditional; distributing the certificate now always runs, over every data center's cluster nodes, which is safe because it is idempotent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Wires up multi-data-center support: --multi-dc bootstraps a second data center in the same project, sharing the VPC, jumpbox and PostgreSQL server but running its own Kubernetes and Ceph cluster. The secondary data center's config and vault are derived from the primary's after its secrets exist, which is why Bootstrap handles it in a loop after the primary's config is written: - the config is cloned and its data-center-scoped fields cleared, so the installer's dataCenters topology, the domains and the shared registry are inherited while the ingress CA and cephadm key are regenerated; - the vault is derived through DeriveDataCenterVault, keeping the postgres roles and token keys and dropping the per-cluster secrets; - postgres becomes mode: external pointing at the shared server's internal IP, which is what its certificate's only SAN carries, and "postgres" is appended to operations.skip in the config so manual re-runs on the jumpbox skip it too. A post-generation check fails the bootstrap if secrets that must match across data centers diverged, or if a per-cluster secret was inherited instead of regenerated — both would otherwise surface long after the fact, against a live shared database. Installs run strictly sequentially in ascending data center order, since the primary's install creates the database, roles and schema the others reuse. validateMultiDC rejects the combinations that cannot work (--write-config=false, an explicit --datacenter-id, an empty datacenter name), and the CLI prints one install command per data center in the required order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Covers a full --multi-dc bootstrap in one spec: two data centers with their own nodes, IPs, configs and vaults, the second one pointed at the first's PostgreSQL server, both told about the topology, and the secrets partitioned the way a shared database and separate clusters require. This is the test that pins the invariants the individual commits only enforce locally. Also documents multi-DC on GCP in the installation guide: the quota footprint, the install order, the two config/vault/secrets-dir paths, the topology keys and the DNS scheme, including the per-data-center platform records the frontend depends on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Signed-off-by: NJona <25478046+NJona@users.noreply.github.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
Signed-off-by: Jona Neef <Jona.Neef.97@gmail.com>
NJona
force-pushed
the
multi-dc-10-multi-dc-tests-docs
branch
from
August 7, 2026 16:03
355417a to
cc56549
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For multi dc we need to expose openfga from one dc to be reused by all others (one central required component). This can now be done by enabling gateway and setting the host.
Everything in this PR was already validated manually in a multi dc setup.
Related PRs that were used for the testing: