[chessing234]

Fix loader failures caused by quoted paths and SQL identifiers in the MIMIC-III Postgres/DuckDB loaders and the MIMIC-IV ED/Note DuckDB loaders.

• Quote the Postgres data-directory argument consistently, including compressed imports, and quote download destinations.
• Escape apostrophes in DuckDB COPY filenames and quote generated Postgres user/database identifiers and password literals.
• Discover both .csv and .csv.gz files in the ED/Note loaders and report when no expected files were loaded.

The current diff contains five loader files. It contains no concept ordering, SQL transpiler, validation-table or workflow changes. The previously listed transpiler test does not validate this scope.

This description was corrected with AI assistance after comparing the current diff. No new loader execution or demo-data validation was performed in this maintenance pass; those checks remain necessary before claiming the load paths work end to end.

[chessing234]

Infra-only follow-up to the too-large #2124. Concept changes stay in #2126 / separate PRs.

[alistairewj]

the changes are scattered and not very cohesive but with some cleaning up we can merge

[chessing234]

Addressed review feedback in 270b6d3:

• SQLFluff: removed continue-on-error; added permissions: contents: read + checks: write; use ignore-unauthorized-error: true
• Reverted duckdb usage text (no .tar.gz), DuckDB GENERATE_ARRAY transform + related concept/test, Makefile concepts target, CheXpert/NegBio defensive rewrites, and validate_concepts awk change
• Removed sqlite TABLE_DTYPES from this PR (will split to a follow-up with CI as requested); kept plain .csv loading

[chessing234]

split the review asks into focused follow-ups: sqlfluff annotate permissions are in #2148; the sqlite import.py csv/.csv.gz discovery (with unit tests + ci) is in #2149. happy to trim the remaining infra bits on this branch further if you want them dropped.

[acnimma]

@alistairewj Ready for another look.

Your July 25 review asks are addressed as follows:

Ask	Status
SQLFluff continue-on-error too blunt	Removed; permissions + ignore-unauthorized-error moved to #2148
DuckDB .tar.gz usage text	Reverted
DuckDB GENERATE_ARRAY transform	Reverted
Makefile concepts target	Reverted
CheXpert / NegBio defensive path edits	Reverted
sqlite CSV discovery	Split to #2149 with tests/CI
validate_concepts.sh awk / comment	Reverted

This PR now keeps only the remaining load-path / transpile / duckdb import / validate_demo infra. Happy to cut anything else that still feels out of scope.

[chessing234]

reverted the make-concepts.sh quote-only hunks too. should be down to duckdb path escaping, postgres datadir/user quoting, sofa/oasis build order, transpile utf-8, and validate_demo tables. sqlfluff + sqlite csv bits stay in #2148 / #2149.

[chessing234]

@alistairewj resolved the review threads — ready for another look when you have time.

[chessing234]

@alistairewj re-review please when you get a chance — pushed after your review to address each point:

• dropped the defensive CheXpert/NegBio path rewrites and the validate_concepts.sh awk change (reverted, 3d189fe1 / f700337f)
• reverted the duckdb GENERATE_ARRAY/UNNEST dialect change — pinned sqlglot already handles it
• reverted the broken postgres Makefile concepts target path
• SQLFluff continue-on-error + workspace permission moved to focused follow-up ci: fix SQLFluff annotation permissions for fork PRs #2148 (ignore-unauthorized-error: true for fork PRs)
• the sqlite plain .csv/.csv.gz import split out to fix(mimic-iii): load plain CSV in sqlite import.py (prefer .csv.gz) #2149 with unit tests + CI, as you asked

kept this PR scoped to load-path / transpile / import infra only.

[chessing234]

trimmed this to the load-path/quoting fixes. dropped the tar.gz usage text, demo validate rows, concept reorder, and datetime_diff month bits — sqlite csv discovery is already in #2149 and the sqlfluff annotate follow-up in #2148.

[chessing234]

i found one more loader issue in an ai-assisted check with synthetic data. with both table.csv and its .csv.gz copy present, the ed and note duckdb loaders both loaded the same row twice, so this needs to pick just one input per table.