v5.2.0: DD tool improvements and dump-based graph distribution - #34
Merged
Conversation
The temp Neo4j instance used for creating filtered graph dumps (dd-only and per-facility) was being OOM-killed by per-user cgroup memory limits on service/login nodes. The 6.4GB graph dump requires mmap pages that exceed the ~12GB of free cgroup headroom. Changes: - Add SLURM dispatch via srun for temp Neo4j lifecycle - Script passed on stdin (bash -s) to avoid shared-fs visibility issues - Readiness probe uses cypher-shell instead of HTTP for Bolt reliability - Trap cleanup kills Neo4j PID before removing temp dir - Property-based deletion (facility_id IS NOT NULL) instead of hard-coded label list — self-maintaining as new labels are added - Symlink source dump instead of copying (saves 60s I/O) - Batch size increased to 50000 rows per transaction - Falls back to local temp Neo4j when already on a compute node BREAKING CHANGE: graph export on HPC requires SLURM access
The hatch build hook could hang indefinitely when gen-pydantic subprocess stalled on NFS (linkml_runtime SchemaView initialization). Changes: - Add 120s timeout to all gen-pydantic subprocess calls - Pass stdin=subprocess.DEVNULL to prevent interactive hangs - Drop --force flag in build hook (use freshness checks instead) - Set _IMAS_CODEX_BUILD=1 env var to skip schema daemon thread during builds (avoids import-lock contention) - Add schema_context_data.py to build hook output validation (runtime dependency that must not be skipped) - Skip redundant schema reference/context generation calls (already handled inside build_models.main())
Symlinks to GPFS paths don't resolve inside Apptainer when the target is outside the explicit bind mounts. Bind the source dump file directly as /dumps/neo4j.dump:ro for the load step.
The graph push builds archives in a temp dir. On HPC, $TMPDIR resolves to /run/user/ (per-user tmpfs) which is not visible from SLURM compute nodes. Use the Neo4j profile's data_dir (on GPFS) as the temp base when srun is available.
Design plan for replacing the stop→dump→load→delete→dump approach with a live Cypher export → CSV → neo4j-admin import → dump pipeline. Includes benchmarks from production graph: 205K DD nodes export in 13s, 559K relationships in 9s, import+dump projected at ~2 min (vs 10-20 min current). Covers DD-only and per-facility variants, index recreation strategy, edge cases (COCOS integer IDs, dual-target HAS_COORDINATE), and SLURM integration.
Replace fragile dump-filter-dump pipeline with direct CSV export from live Neo4j. Zero production downtime — no graph stop/start needed for DD-only and per-facility variants. Pipeline: export_dd_only_csv() → CSVs + DDL + import.sh → tar.gz Load side: import_from_csv() or import.sh (Docker) - export_rebuild.py: full export pipeline (nodes, rels, indexes, metadata) - data.py: graph_load handles both CSV and legacy dump formats - release.py: DD-only and per-facility use CSV export, full uses dump - Dockerfile: Stage 4 handles CSV via import.sh + DDL in pre-start step Tested: 203K nodes, 559K rels exported in ~34s (vs 10-20 min old pipeline) Archive: ~25 MB compressed (vs ~900 MB dump)
Update graph-quality job to detect CSV archives (import.sh + csv/) and use neo4j-admin import instead of database load. Adds DDL execution step for creating indexes after import. Supports both CSV and legacy dump formats.
The service container sets NEO4J_AUTH=neo4j/imas-codex on first start. After neo4j-admin import (which only replaces the neo4j database, not system), the password persists. Use imas-codex instead of default neo4j. Also adds failure counting and error reporting for DDL execution.
The package is not yet published to PyPI, so uv cannot resolve it in CI environments. Remove from both dev deps and [tool.uv.sources]. Developers install manually: uv pip install -e ../imas-standard-names
The CI was reimplementing the import command by parsing CSV filenames, but extracted relationship types incorrectly — rels_IN_VERSION_IMASNodeChange_DDVersion.csv became type IN_VERSION_IMASNodeChange_DDVersion instead of IN_VERSION. Use the archive's import.sh which has correct types baked in.
graph-quality.yml pulls imas-codex-graph (full dump) instead of imas-codex-graph-dd (CSV). Full dumps have zero edge cases — no property discovery, type inference, or filtering issues. RC releases now also push the full dump to GHCR so CI can pull it. DD-only CSV is still pushed for the container image.
The --set-default migration ran AFTER killing ControlMaster sockets, updating SSH config, and stopping tunnels — breaking connectivity to the old node. Move migration to run first while connections are warm. Also surface SSH stderr in the error message for easier diagnosis.
…iants BREAKING CHANGE: CSV-based graph distribution removed after 7 RCs of cascading failures. All graph variants (full, dd-only, per-facility) now use the proven dump-filter-dump pipeline via temp Neo4j. - Delete export_rebuild.py (970 lines) and its feature plan - Remove CSV import branches from CI workflows and Dockerfile - CI tests against full graph dump (imas-codex-graph) - Container builds use dd-only dump (imas-codex-graph-dd) - Simplify Dockerfile: remove DDL execution step (dumps include indexes)
The '^[0-9v]' pattern matched 'v-direct-test' (a stale test artifact) and filtered out 'latest' (starts with 'l'). Replace with semver-aware pattern that only matches proper version tags or 'latest'.
After 'cd /tmp/graph-pull' and 'rm -rf /tmp/graph-pull', the CWD no longer exists. Java VM cannot initialize: 'getcwd() failed'. Add 'cd /' before neo4j-admin database load to restore a valid CWD.
- Free ~15 GB more on CI runner by removing hostedtoolcache, swift, boost - Merge graph load + Neo4j recovery into single RUN to eliminate ~5 GB intermediate layer that persisted in buildx storage - Root cause: smoke-test failed with 'No space left on device' because 2 GB dump expands to ~7 GB (data + WAL) across multiple layers
Remove pre-start recovery from Docker build — the 2.3 GB WAL creation caused 'no space left on device' on CI runners (38 GB free, but buildx layers + 2 GB dump + 5 GB expanded data + 2.3 GB WAL exceeded limit). Recovery now happens at container startup (entrypoint.sh waits 180s). First startup takes ~10-30s longer but the image is ~2.3 GB smaller.
Fixed Cypher queries using non-existent IMASNodeChange properties: - change.semantic_change_type → change.change_type - change.version → v.id (via IN_VERSION relationship) - change.summary → old_value/new_value pair Updated formatters to render version changes correctly. Fixed _get_version_context to filter null OPTIONAL MATCH results.
# Conflicts: # imas_codex/cli/host.py # uv.lock
Simon-McIntosh
temporarily deployed
to
github-pages
April 9, 2026 22:25 — with 
GitHub Pages
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DD Tool Improvements
get_ids_structure: New graph-backed IDS structure analysis toolcore_profiles.profiles_1d.electrons.temperaturecheck_dd_pathsnow suggests corrections for misspelled pathsfetch_dd_pathsandsearch_dd_pathscorrectly render version change historyimas-*todd-*prefix for clarityInfrastructure
Bug Fixes
IMASNodeChangepropertiesgetcwd()error in Docker build neo4j-admin callsimas-standard-namesdependencyCI Status (v5.2.0-rc10)