Fix bad reads of nested cmpd/vlen types in JNI

[mattjala]

• Fixed H5DreadVL failing to read data for pre-allocated compound-of-vlen datasets

When H5DreadVL is called with a H5T_COMPOUND memory type whose members include variable-length data, and the caller pre-allocates each row slot in the outer Object[] buffer, then the read completes without throwing an error, but each pre-allocated row ArrayList comes back empty.

If the caller follows the alternate calling pattern, leaving each row slot null and letting the native code allocate the per-row record, then the same read works correctly.

The root cause is the H5T_COMPOUND case in translate_rbuf. This function walks each row's compound, decodes each member into a Java object via translate_atomic_rbuf, and stores the result in the per-row ArrayList. It branches on a found_jList flag based on the result of GetObjectArrayElement(), i.e. whether the caller pre-allocated the row slot.

Later on, this flag is used to decide between using the .add() method of ArrayList (if found_jList is false), or trying to directly set an element on a builtin java array (if found_jList is true).

The latter case is wrong in two ways.

1. Type mismatch. jList is always an ArrayList, either fetched from ret_buf[i] (where each slot is an ArrayList) or newly created a few lines earlier. The attempt to directly set an element using the built-in array syntax silently no-ops.
2. Wrong index variable. Even if jList were a Java array, the index supplied is i (the outer row counter), not x (the inner member counter).

There are currently no tests covering H5DreadVL() for H5T_COMPOUND in the src-jni test tree. This was discovered in the context of HDFView attempting to read compound-of-sequence datasets. HDFView's H5Datatype.allocateArray pre-fills each row slot with new ArrayList<>() before handing the buffer to H5DreadVL, which triggered the bug.

My fix is to replace the use the ArrayList.add() method in every case, and drop the found_jList tracking since jList is always an ArrayList and we always want to use the .add() method. This also causes it to match the pattern of the H5T_VLEN branch, which works for both pre-allocated and non-pre-allocated slots.

To test this case, I added test.TestH5D.testH5Dread_compound_of_vlen_preallocated. For the sake of regression testing, I also added testH5Dread_compound_of_vlen which tests the same thing with non-pre-allocated row slots, although this test passed even before I made any changes to the source code.

I also fixed a very similar problem in H5DwriteVL that led to a SIGSEGV when writing compound of variable-length sequences. Specifically, translate_atomic_wbuf treats in_obj as a java array when it is an ArrayList, producing a garbage length value that eventually leads to a crash.

I changed the H5T_VLEN case to behave similarly to the H5T_ARRAY and H5T_COMPOUND cases and use mToArray on the provided ArrayList to get an array object for the array operations. The test verifying the fix is testH5Dwrite_compound_of_vlen.

---

translate_rbuf's H5T_VLEN case had a similar bug where when found_jList was set to false due to an entry in ret_buf being null. ret_buf.add() would then be invoked on an array of objects without the list .add() method. This would occur whenever a read was invoked of a vlen sequence with a null (non-preallocated) entry. The pre-existing tests only tested the pre-allocated cases.

I removed the use of the found_jList flag, since it conflated the passing of an unallocated slot with ret_buf not being an array. Instead, it now uses ret_buflen == 0 as the check to match the pattern in H5T_INTEGER and other branches.

The test for this fix is testH5Dread_vlen_of_compound_nullslot.

---

translate_atomic_rebuf had two issues related to handling of nested compounds. First, it discarded recursive return values, resulting in the construction of empty lists. Secondly, its member offset (char_buf + i * typeSize + memb_offset) was incorrect. In this case, i was the member index and memberSize was the entire cmpd size, so the offset would be erroneously large. It seems like this came from copying of the offset computation from translate_rbuf, which had to advance over entire elements of compound data. This error was duplicated on the write side in translate_atomic_wbuf's H5T_COMPOUND case (h5util.c:4611).

I changed translate_atomic_rbuf to capture the resultant object, and dropped the i * typeSize term in both routines.
The new test verifying the fix is testH5Dread_vlen_of_nested_compound.

[Copilot]

Pull request overview

This PR fixes multiple JNI translation bugs affecting H5DreadVL/H5DwriteVL for nested datatype combinations (notably compound members containing VLEN, and VLEN of nested compounds), which previously could yield empty Java ArrayList results or crash (SIGSEGV). It also adds regression tests in the Java JNI test suite to cover these cases.

Changes:

• Fix compound read translation to always append members via ArrayList.add() (instead of incorrectly attempting array element assignment when the caller pre-allocates row slots).
• Fix VLEN write translation to convert ArrayList inputs via toArray() before performing array-style operations.
• Add new JUnit tests for compound-of-VLEN reads (preallocated and non-preallocated), VLEN-of-compound reads with null slots, and VLEN-of-nested-compound reads; update expected test output.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
java/src-jni/test/TestH5D.java	Adds helper + new regression tests covering nested compound/VLEN read/write scenarios.
java/src-jni/test/testfiles/JUnit-TestH5D.txt	Updates expected JUnit output list/count for the added tests.
java/src-jni/jni/h5util.c	Adjusts JNI buffer translation logic for compound/VLEN read/write paths to correctly use ArrayList semantics and correct member offsets.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.