Improvements to RucioFileCatalogClient, compatibility of older client with Rucio 41 server

[maxnoe]

BEGINRELEASENOTES

• FIX: RucioFileCatalog when using rucio clients <41 with rucio server >= 41
  ENDRELEASENOTES

This makes a couple of small updates to the RucioFileCatalogClient triggered by issues we discovered in the CTAO integration test environment when running DIRAC with a rucio client version 40 but a rucio server version 41.

In Rucio 41, a long-standing bug ignoring the plugin parameter for get_metadata_bulk was fixed, which in prior versions resulted in the DID_COLUMN metadata being returned regardless of the plugin= parameter.

However, this breaks the RFC in the combination of a rucio 40 client with a rucio 41 server (a combination which is allowed according to Rucio's support policy of newer servers keeping compatibility with older clients), because get_metadata_bulk in rucio 40 uses plugin='JSON" as the default. This worked before because plugin was ignored, however, with a Rucio 41 server, plugin is correctly respected, which results in the JSON metadata being returned where the RFC actually expects the DID_COLUMN metadata (e.g. containing size in bytes, checksums, did type).

Using Rucio client 41 with Rucio server 41 works, because also the default plugin for get_metadata_bulk was changed to DID_COLUMN.

Here, it is better to be explicit though: everywhere DIRAC needs the basic did metadata, DID_COLUMN is explicitly added.

I also changed a couple of calls of get_did_metadata to get_did, which is cheaper to execute as it does not involve the metadata plugin machinery.

[maxnoe]

@cserf

[cserf]

LGTM. The PR is marked as draft, so do you want to had more changes ?

[maxnoe]

I left it draft for now because I didn't come around to testing all changes yet in our integration environment.

I tested most and it works fine, but e.g. the data removal methods I didn't test yet as they are not convered in our integration test suite.

[fstagni]

Hi, I can give the final review and merge this one, but I am just wondering if this is something that you would like to backport. If only this PR is merged, your changes will only be in v9.1.X release. This is true also for #8757

[maxnoe]

This is fine, we are are only using the RucioFileCatalog in the new system, which is on Diracv9 + DiracX, the legacy v8 deployment is using the DFC.

[fstagni]

Yes, but even for that there are v9.0.X (created from rel-v9r0 branch) and v9.1.X (created from integration branch) release branches. After popular request, it was agreed sometime ago that v9.0.X would be the release that could be used to upgrade. If CTAO wants to go directly to v9.1, fine for me.

[maxnoe]

We are currently on 9.1.9 with the new system