treeshr: pluggable per-scheme backend for tree-file I/O (TreeFileIo)

[sammuli]

Summary

This adds a small extension point so the local tree-file I/O path in treeshr
can be served by a pluggable backend selected by URL scheme, instead of always
calling the raw open/read/write/lseek/close syscalls. It lets
out-of-tree transports (e.g. trees served from an HTTP endpoint or an object
store) be provided as separate loadable libraries with no new dependency
in MDSplus core.

The design intentionally mirrors the existing mdsip transport-plugin pattern
(IoRoutines / LoadIo in mdstcpip/): where mdsip loads MdsIp<PROTOCOL> and
calls its exported Io symbol, this loads MdsTreeFile<SCHEME> and calls its
exported FileIo symbol.

How it works

• A new vtable, TreeFileIo (treeshr/treefileio.h), groups the operations the
  MDS_IO_* layer performs on a local fd: open/close/read/write/lseek/lock
  plus an mmap_capable flag.
• GetTreeFileIo(filename) in RemoteAccess.c:
  • returns a built-in default backend (thin wrappers over the same syscalls
    as today) when the path has no scheme:// prefix;
  • otherwise resolves MdsTreeFile<SCHEME> via LibFindImageSymbol_C, caches
    the result per scheme, and fails closed (open fails) on an unknown or
    unloadable scheme rather than treating "scheme://…" as a literal path.
• fdinfo_t carries the resolved backend, and every local-branch MDS_IO_*
  operation dispatches through it.
• MapFile keys its mmap-vs-buffered decision off the backend's mmap_capable
  flag (new MDS_IO_MMAP_CAPABLE): non-mmap-able backends take the
  calloc+read path while ordinary local files keep using mmap. Defensive
  null-outs after free in close_top_tree/MapFile guard against double-free
  on the buffered path.

Behavior preservation

With no URL scheme present, the code path is the default backend, i.e.
byte-for-byte the current syscall behavior — no functional change for existing
trees, and no new build/link dependency. The mdsip remote (conid >= 0) path
is untouched.

Tests

• Adds treeshr/testing/TreeFileIoTest.c, a dependency-free test, and
  treeshr/testing/MdsTreeFileTESTMEM.c, a tiny in-tree reference backend
  (scheme testmem://) loaded the same way a real backend would be. It covers:
  default-path round-trip, scheme dispatch + per-scheme cache reuse,
  short-read/EOF semantics, fail-closed on an unknown scheme, and the
  mmap_capable plumbing.

• Built with CMake against this branch and run via ctest; the full
  treeshr/testing suite passes:

  TreeDeleteNodeTest .... Passed
  TreeSegmentTest ....... Passed
  TreeFileIoTest ........ Passed
  100% tests passed, 0 failed out of 3
  

Writing a backend

A backend for scheme x:// is a shared library libMdsTreeFileX (scheme
uppercased) exporting const TreeFileIo *FileIo(void) with default symbol
visibility. It is discovered lazily on first open of an x:// path. The contract
is documented in treeshr/treefileio.h.

Notes for reviewers

• The image-name (MdsTreeFile<SCHEME>) and symbol (FileIo) conventions mirror
  the MdsIp<PROTOCOL>/Io precedent but are of course open to bikeshedding.
• The contract comment notes __declspec(dllexport) for Windows; the in-tree
  reference backend and tests have only been exercised on Linux so far.
• treefileio.h is currently an internal treeshr header (not installed); if you
  want third-party backends to build against an installed SDK, it can be added to
  the install set.

Addendum by @mwinkel-dev

This PR is related to the Fusion Data Project (FDP) that @sammuli is leading.

[mwinkel-dev]

Hi @sammuli -- The next post is not directed to you, but rather to Jenkins.

[mwinkel-dev]

Retest this please

[WhoBrokeTheBuild]

This will take some time to review, this is a large change and we need to understand exactly how customers will be impacted. We've done extensive experiments with providing a URL-based interface for treeshr, and I'm not sure if this implementation is what we want to include. Truthfully, I believe we should have been included in the planning process for this before an implementation was chosen.

[sammuli]

@WhoBrokeTheBuild I'm not at all married to this implementation. It should work ok, but I'm thinking of it as more of a strawman if there are other considerations I'm ignorant of. If there's already a URL-based interface I'm completely open to piggybacking on that. Totally happy to defer to your judgement here.

[mwinkel-dev]

I'm surprised by the 45 failing tests on RHEL8 x86-64, so I am launching a second build just to see if there was a glitch with Jenkins.

[mwinkel-dev]

Retest this please

[sammuli]

Heads-up on the commit churn: I pushed one small follow-up (a8a851f) that adds treefileio.h to the Debian/RedHat devel.noarch manifests — I'd installed the header but missed listing it in the dev-package file lists. Apologies if that aborted a build you were mid-way through investigating, @mwinkel-dev; the current head to test is a8a851f.

[sammuli]

Retest this please

[WhoBrokeTheBuild]

Retest this please (only works for developers)

[mwinkel-dev]

Hi @sammuli -- The second build also failed 45 tests on RHEL8 x86-64. A spot check of a few errors indicates that the tests are not loading numpy.

This is an excerpt from the logs for test 86.

                 . . .
    Start 86: python/MDSplus/tests/connection-io
                 . . .
 MDSIP_CLIENT_LOCAL_LOGFILE=set:/opt/jenkins/workspace/MDSplus_PR-3062@6/workspace-rhel-8-x86_64/build/python/MDSplus/tests/mdsip-local-connection-io-85.log
86: Test timeout computed to be: 1500
86: Traceback (most recent call last):
86:   File "/opt/jenkins/workspace/MDSplus_PR-3062@6/python/MDSplus/tests/connection_case.py", line 32, in <module>
86:     from MDSplus import Connection, GetMany
86:   File "/opt/jenkins/workspace/MDSplus_PR-3062@6/python/MDSplus/__init__.py", line 55, in <module>
86:     import numpy
86: ModuleNotFoundError: No module named 'numpy'
1/1 Test #86: python/MDSplus/tests/connection-io ...***Failed    0.07 sec

Because these tests are only failing on RHEL8, there is a chance that the problem is with our test environment for that platform. I will investigate that on Tuesday.

[mwinkel-dev]

Hi @sammuli -- Thanks for committing the manifest changes. The most recent build no longer fails on the "Packaging" steps.

However, RHEL8 still has problems because of the numpy issue. Although I will look into the NumPy problem on Tuesday, I won't be making any changes to the build / test system. Modifications to the RHEL8 system will be deferred a few days until Stephen is available. (Because of the risk of breaking all builds, we proceed cautiously with changes to the build/test system.)

[mwinkel-dev]

The RHEL8 Docker's image now has three versions of Python: 2.7, 3.11 and 3.12. If running Python3.11, the import numpy statement works fine. However, if running Python3.12, the import fails because it can't find numpy.

A conjecture is that the Dockerfile that builds the RHEL8 image recently changed to now also install the Python3.12 version. Jenkins is apparently using Python3.12, which is why the RHEL8 builds fail on both Jenkins and a developer workstation. Further investigation is needed to identify the root cause of the missing NumPy.

[mwinkel-dev]

This PR did not cause the NumPy problems on RHEL8. It is just coincidence that the RHEL8 container's configuration changed around the same time this PR was submitted.

[mwinkel-dev]

Jenkins has just been fixed, so rebuilding this PR.

[mwinkel-dev]

Retest this please

[mwinkel-dev]

Retest this please