GH-47918: [Format] Clarify that empty compressed buffers can omit the length header

[pitrou]

Rationale for this change

The 8-byte length header is considered optional for empty compressed buffers in Arrow C++ and other implementations.

Are these changes tested?

Not applicable.

Are there any user-facing changes?

No.

• GitHub Issue: [Format] Clarify that a compressed IPC buffer can be empty for empty data #47918

[github-actions]

⚠️ GitHub issue #47918 has been automatically assigned in GitHub to PR creator.

[alamb]

Thanks @pitrou -- I think this seems like a reasonable change to the format (if it is one) and since the C/C++ implementation has been doing it I think it could/should become the defacto standard

[alamb]

Do you know if there is any data file in https://github.com/apache/arrow-testing that has this property (empty body that omits the 8-byte length)?

I would like to make sure we test that the rust reader can read it

[paleolimbot]

I'm also interested in such a file. It might be nice in this comment to take a stance on whether an implementation should do this for new implementors that don't have an opinion. It seems like the rationale is that you save 8 bytes per column in most cases (since this would probably get used for every null buffer in compressed output)...does anybody need that level of optimization?

[pitrou]

I don't know about the rationale, I agree that we probably don't care about such an optimization.

[pitrou]

To answer the questions:

1. I don't think the 0-length exception was actually discussed. It was briefly mentioned in @wesm 's original proposal ("Compress each non-zero-length constituent Buffer ..."), but that didn't trigger any particular reaction.
2. I've just checked: the compressed streams in https://github.com/apache/arrow-testing/tree/master/data/arrow-ipc-stream/integration/2.0.0-compression (in particular generated_lz4.stream and generated_zstd.stream) have a 0-length buffer at index 0 of their record batches (it's the null bitmap of the first column, which doesn't have any nulls)

So, such IPC streams have already been emitted by the C++ IPC writer, and quite commonly so it seems (a non-null batch column is sufficient, though a zero-sized batch would also trigger it).

[pitrou]

We could add compressed files with empty batches in the arrow-testing repo if the non-null column case is not sufficient to exercise all readers, by the way.

[pitrou]

Speaking of which, I don't think our integration tests actually exercise compression expect with these predefined files.