Server v3.0.0#2635
Open
johngrimes wants to merge 95 commits into
Open
Conversation
Add viewReference support, Reference-typed patient/group with spec cardinality, Bundle unwrapping in the resource parameter, and 422 for semantically invalid ViewDefinitions on the run operation. Across the run, export, and sqlquery-run operations, reject an explicitly unsupported _format and the unsupported source parameter rather than silently ignoring them. Add a creation-time start timestamp to the async job for later export-manifest timing.
Support view.viewReference resolution to a stored ViewDefinition with per-part exclusivity and presence checks. Rewrite the completion manifest to the SQL on FHIR shape (exportId, status, echoed clientTrackingId and _format, export timing fields, and one output per view with name and location parts), dropping the Bulk Data-style fields. Return a Parameters kick-off acknowledgement for SQL on FHIR async operations while leaving the Bulk Data kick-off body unchanged.
…ery 422 Declare the SQL on FHIR spec canonical OperationDefinition URLs for the viewdefinition-run, viewdefinition-export, and sqlquery-run operations in the CapabilityStatement, and stop serving the Pathling-authored OperationDefinitions for them. Add a regression test pinning the existing 422 response for an unmappable FHIR column type under _format=fhir on sqlquery-run, and integration coverage for the source and _format rejections across all levels and transports.
Update the admin UI manifest parser to read the spec-shaped output location parts (one or more per output) instead of a single url part, emitting one download entry per file so a view that produced several files lists every file. Add an end-to-end check that an invalid custom ViewDefinition surfaces the 422 OperationOutcome message.
Update the run, export, and sqlquery-run operation docs for the new viewReference support, Reference-typed patient/group, Bundle unwrapping, the spec-shaped export manifest and kick-off acknowledgement, the source and unsupported-format rejections, the 400/404/422 status semantics, and the spec conformance canonicals. Credit the additional author on the significantly changed sqlquery providers.
Align the admin UI export end-to-end mocks with the new completion manifest: output download files are carried in location parts, and the non-spec transactionTime and requiresAccessToken fields are removed, so the card-listing tests exercise the location-reading parser.
Strip media-type parameters (e.g. text/csv;charset=utf-8) before matching an explicit _format value across the run, export, and sqlquery-run strict parsers, so a supported media type with parameters is treated as that format rather than rejected with 400, matching the spec's content-negotiation edge case.
The strict SQL validator's expression allow-list omitted the named and positional parameter placeholder expressions, so any parameterised query (using ":name") failed static validation. This blocked runtime parameter binding for $sqlquery-run even though the binding machinery was already present. Add NamedParameter and PosParameter to the allow-list. They are leaf, unevaluable value placeholders bound at execution time, so they carry no code-execution risk.
Implement the SQL on FHIR $sqlquery-export operation, the asynchronous counterpart to $sqlquery-run, at the system, type, and instance levels. Clients can run one or more SQL queries against materialised ViewDefinition tables in the background and download each result as files, following the FHIR Asynchronous Request Pattern. Each query produces one output. Table sources can be supplied at request time, inline or by reference, in addition to being read from server storage. Supported formats are NDJSON (default), CSV, and Parquet, with patient, group, and _since filtering. A failed query fails the whole export. To avoid duplication, extract the shared asynchronous-export machinery into a new operations/export package (a generic completion-manifest builder, a file writer, and a filtered data-source builder) reused by both export operations, and refactor $sqlquery-run onto a shared SqlQueryPipeline that the export also uses. The Admin UI gains an export affordance on the SQL query result card, backed by a reusable export-job card shared with the ViewDefinition export. The operation is declared in the CapabilityStatement against the spec canonical and gated by a new sqlQueryExportEnabled configuration flag. Resolves #2633.
Replace the boolean redirectOnComplete marker on the server's asynchronous machinery with an AsyncPattern enum (STANDARD_ASYNC_PATTERN, BULK_DATA), so the @AsyncSupported annotation reads as a deliberate choice between two named patterns rather than the presence or absence of one behaviour. Correct the Javadoc that misattributed the redirect-based completion to a "SQL on FHIR unify-async" specification, citing the HL7 Asynchronous Interaction Request Pattern instead. Behaviour is unchanged; the existing async tests pass with only their identifiers updated.
isAnyExportEnabled now also covers the SQL on FHIR asynchronous export operations, so the all-exports-disabled case must also disable the viewdefinition-export and sqlquery-export flags, both of which default to enabled.
The SQL on FHIR export integration tests added to the server module push the full build-and-test suite past the previous 30-minute job limit, causing the run to be cancelled before completion. Raise the timeout on the test, pre-release, and release server workflows so the suite has room to finish.
The disabled-operation assertion called doesNotContain on its own, which passes vacuously if the operation list is empty. Assert the still-enabled sqlquery-run operation is present first, so the test proves the list is populated and only sqlquery-export was dropped.
johngrimes
added
release
Resolves twelve Netty advisories reported against 4.1.133.Final, spanning HTTP/2 denial-of-service, TLS hostname verification bypass, and DNS resolver issues. All are fixed within the 4.1.x line at 4.1.135.Final, so no move to the 4.2.x series is required. The Netty BOM remains imported ahead of Spring Boot's so the pinned version continues to win.
Addresses CVE-2026-55471, an XML external entity vulnerability in XsltUtilities.saxonTransform via an unhardened Saxon TransformerFactory. HAPI FHIR 8.10.0 still bundles 6.9.4.1, so the existing override of org.hl7.fhir.r4 and org.hl7.fhir.utilities is raised from 6.9.7 to the patched 6.9.10.
Resolves a batch of react-router security advisories by moving to the latest 7.x release. None of the vulnerable code paths (RSC, SSR, data routers, the __manifest endpoint) are reachable from the admin UI, which uses declarative routing only, but upgrading is a low-risk minor bump that clears the findings and lets the react-router suppressions be removed from the Trivy ignore list.
Restore the server image's ability to run as either the FHIR server or a Spark Kubernetes executor, selected by the container's first argument. Spark on Kubernetes launches executor pods with the single argument "executor" and no command override, so the role selection lives in the image entrypoint. A Bash dry-run test covers the role-selection logic without Docker, a network, or a cluster.
Set the dispatching entrypoint as the image's entry behaviour in both the docker and dockerPreRelease profiles, copy it into the image via Jib extraDirectories with executable permissions, and remove the now-dead jvmFlags (Jib ignores them once an explicit entrypoint is set). Run the dispatch test during the test phase of these profiles, gated by skipTests.
The citation pointed to the Spark 4.0.0 entrypoint; the runtime is Spark 4.0.2, whose executor branch is identical.
Restore the dual-role server image for Spark on Kubernetes
The Varnish frontend cache previously inherited Varnish's default 60s first-byte timeout, which caused long-running synchronous Pathling queries to fail with a 503 before the server could respond. Expose the timeout as a PATHLING_FIRST_BYTE_TIMEOUT environment variable, substituted into the backend definition, and surface it through the Helm chart as pathlingCache.firstByteTimeout (defaulting to the previous 60s).
The kick-off requests relied on the default 5-second blocking response timeout, which loaded CI runners exceed during Spark warmup. The shared base class now applies the same 60-second timeout used by the other server integration tests, inherited by all subclasses.
Detect and repair Delta table schema drift after encoder upgrades
The vulnerability requires case-insensitive property matching combined with per-property @JsonIgnoreProperties, neither of which is used in Pathling. No fixed jackson-databind version has been published to Maven Central yet, so suppression is the only way to unblock the security scan. The suppression should be removed once 2.22.1 is released.
Brings in Spring Security 6.5.11, which fixes CVE-2026-47838 (X.509 client certificate impersonation). Pathling does not use X.509 client certificate authentication, but upgrading pre-empts the advisory failing the security scan once the CI vulnerability database updates.
Inject the driver pod name into the Spark configuration via the downward API so that executor pods are created with an ownerReference to the driver pod. Previously, restarting the deployment orphaned completed executor pods, and their on-demand scratch PVCs blocked new executors from claiming volumes.
Introduce a shared validator that walks a Spark result schema and rejects any unresolved (VOID/NullType) type - top-level or nested in structs, arrays, and maps - with an InvalidUserInputError that names every offending field path and suggests both remediations (explicit CAST or a different output format).
Validate the result schema at the top of the Parquet streaming path so that $sqlquery-run and $sqlquery-instance-run return a 400 naming the offending column(s) instead of a 500, before any temporary directory is created.
Validate the result schema before writing Parquet in the view export executor so that a $viewdefinition-export job containing an unresolved (VOID) column fails with a clear diagnostic in its polled status, rather than an opaque internal Spark error. Non-Parquet formats are unaffected.
Explain in the sql-run and view-export operation docs that Parquet cannot represent unresolved (VOID) columns, and describe the 400 (sync) and failed-job (async) errors along with the two remediations.
Replace inline fully-qualified Stream/Collectors references with imports in the Parquet VOID-column export test helper.
Reject VOID columns in Parquet output with a clear error
Carve the plain DESCRIBE [TABLE] <label> form out of the SQL validator's blanket command rejection so consumers can discover the columns and types of a registered view before querying it (issue #2651). Validation is complete at parse time, with a defence-in-depth check on the analysed plan; every unsafe DESCRIBE variant remains rejected.
Add unit and integration coverage for the DESCRIBE QUERY form, whose inner query is validated exactly like a directly submitted query, returns the projected column schema without scanning data, and supports bound parameter markers.
Assert that every unsafe DESCRIBE and SHOW variant is rejected with a 400 OperationOutcome through the endpoint, guarding against a carve-out that is too wide.
Add a schema-introspection section to the $sqlquery-run documentation covering the two allowed DESCRIBE forms, the result columns, and the rejected variants, and expand the SqlValidator class Javadoc to explain the parse-time carve-out and eager-command-execution rationale.
Schema introspection through $sqlquery-run (DESCRIBE)
Broaden the SQL validator's expression allow-list by the four Catalyst nodes required to accept read-only SELECT queries that use explicit window frames and named windows: the three SpecialFrameBoundary markers (CurrentRow, UnboundedPreceding, UnboundedFollowing) and the named-window reference (UnresolvedWindowExpression). Resolves #2649 and #2650.
Permitting named-window references (UnresolvedWindowExpression) let WINDOW w AS (...) queries reach the validator, but the strict expression walk never descended into the window definition body, which is held in a map that is neither a tree child nor reported by plan.expressions(). A disallowed function hidden in a named window's PARTITION BY or ORDER BY therefore bypassed validation. Walk each named-window specification explicitly so those functions are rejected exactly as in the inline OVER (...) form, with a matching defence-in-depth carve-out on the analysed walk.
The maxRows and timeoutSeconds limits are enforced in the shared query execution path reached by both $sqlquery-run and $sqlquery-export, but the documentation described them as applying only to $sqlquery-run and suggested the asynchronous path as a way to avoid the timeout. Correct the scope, document the previously undocumented limits for export, and remove the misleading escape-hatch wording.
Exercises the defence-in-depth branch that descends into a WithWindowDefinition reaching the analysed-plan walk, covering both a clean definition body and one hiding a reflection-style function.
Accept window functions and frames in $sqlquery-run
johngrimes
marked this pull request as ready for review
July 15, 2026 00:42
Provides the PathOutputCommitProtocol and BindingParquetOutputCommitter classes needed to commit Spark writes to an S3 object store without rename-based commits, enabling a MinIO-backed warehouse for the CSIRO deployments.
Note the Spark commit-protocol classes required for Parquet writes to use the magic committer, and that the bindings now ship with the server image.
Pin patched versions of build-time and dev-server-only transitive dependencies flagged by Trivy. None were reachable in the deployed static site, but fixing is preferable to suppression where patches exist. Covers websocket-driver, brace-expansion, fast-uri, shell-quote, svgo, js-yaml and webpack-dev-server.
Extend JobDirectoryFileSystem with deleteJobDirectory, a strict primitive that removes a per-job directory and its contents on the warehouse filesystem, resolved from the warehouse URI rather than the process default. Add a test-only StubFileSystem (a RawLocalFileSystem reporting the stub scheme) so the helper can be exercised against a non-default warehouse filesystem while fs.defaultFS stays file:///, reproducing the production scheme split as a fast unit test. Refs #2666
Rewire ExportFileWriter to delegate per-job directory creation and deletion to JobDirectoryFileSystem, which resolves the filesystem from the warehouse URI, instead of resolving the process default filesystem and operating on a warehouse-scheme path. This fixes the Wrong FS failure for $viewdefinition-export and $sqlquery-export on non-default warehouse filesystems. The best-effort delete contract (log and swallow) is retained in the writer; the Spark dataset-write methods are unchanged. A new regression test exercises the writer against a stub:// warehouse; it reproduced Wrong FS against the pre-fix code. Refs #2666
Rewire the bulk $export executor to create its per-job directory through JobDirectoryFileSystem, which resolves the filesystem from the warehouse URI, rather than resolving the process default filesystem and operating on a warehouse-scheme path. This fixes the Wrong FS failure for bulk $export on non-default warehouse filesystems. The now-redundant database path and Spark session dependencies are dropped from the executor. A new regression test runs a system-level export against a stub:// warehouse; it reproduced Wrong FS against the pre-fix executor. Refs #2666
Add tests exercising the error branches where per-job directory creation fails, so the shared writer and the bulk executor are shown to surface an InternalErrorException rather than the raw filesystem exception. Lifts new code coverage above the project quality gate threshold. Refs #2666
Fix async export writes on non-default warehouse filesystems
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.