[Feature]: Use ProxyManager for obtaining X509 proxy for Rucio?

[maxnoe]

User Story

Some of the DIRAC server components need access to Rucio when using RucioFileCatalogClient.

The natural authentication for now here would be an X509 robot cert indentifying the dirac server, e.g. the same proxy used for submitting pilots to CEs.

However, this would require that components using the FileCatalog request a proxy when needed via the ProxyManager.

This would include I think the WMS Optimizers, Transformation Agent and potentially other components.

@volodymyrss @natthan-pigoux

Feature Description

Have DIRAC server components that need access to rucio optionally use the ProxyManager to obtain a voms proxy for accessing rucio.

Definition of Done

No response

Alternatives Considered

No response

Related Issues

No response

Additional Context

No response

[chaen]

what components are we talking about ? If it's agents, wouldn't a Shifter proxy solve your issue ?

[maxnoe]

At least the WMS Optimizer uses it to locate the data, that's completely on the server side I think.

What is a "Shifter proxy"?

[chaen]

Shifter Proxy is a concept by which you define a role, like DataManager, and you associate it a User and Group. Then any agent can take as configuration option shifterProxy with the values you defined (i.e. DataManager) and it will download and use that specific proxy for its operations.
I've just checked, it seems to be possible for Optimizers as well.

The Shifters are defined in /Operations/Defaults/Shifter. I see we have some docs but it is clearly lacking some concept definition....

So in principle, what you want should just work if you set this shifterProxy option in your executor.

[maxnoe]

Is the x509 proxy for this available as a file on the filesystem?

We would need to configure the rucio.cfg or an environment variable RUCIO_CLIENT_PROXY to point to it on disk for that to work.

[chaen]

it ends up on the work directory of the agent as .shifterCred I believe.

This method ends up being called.

https://github.com/DIRACGrid/DIRAC/blob/integration/src/DIRAC/Core/Utilities/Shifter.py#L54

One option could be to update it to also export the Rucio environment variable, but why isn't rucio reading the standard X509_USER_PROXY variable (which would make more sense to me)

[maxnoe]

It is, there are a couple of sources for it:
https://github.com/rucio/rucio/blob/5548639e324a7278cb7dcc77e81d0f087476e98c/lib/rucio/client/baseclient.py#L338-L347

RUCIO_CLIENT_PROXY is specific to rucio and takes precedence if both RUCIO_CLIENT_PROXY and X509_USER_PROXY are defined.

[chaen]

OK, so I believe setting the shifter option should meet your need ?

[maxnoe]

Great! I'll test that tomorrow and will report back here

[maxnoe]

We configured the shifterProxy for a DIRAC component that needs to talk to Rucio, in this case the Optimizers, however, we see in the logs:

2026-09-02T12:51:32,604503Z WorkloadManagement/Optimizers/WorkloadManagement/InputData/InputData/JID_5 [140418957993536] INFO: Processing input data
2026-09-02T12:51:32,604730Z WorkloadManagement/Optimizers/FileCatalogFactory [140418957993536] DEBUG: Creating FileCatalog client of type RucioFileCatalog
2026-09-02T12:51:32,604775Z WorkloadManagement/Optimizers [140418957993536] DEBUG: Trying to load Resources.Catalog.RucioFileCatalogClient
2026-09-02T12:51:32,604913Z WorkloadManagement/Optimizers [140418957993536] DEBUG: Trying to load CTADIRAC.Resources.Catalog.RucioFileCatalogClient
2026-09-02T12:51:32,604992Z WorkloadManagement/Optimizers [140418957993536] DEBUG: Trying to load DIRAC.Resources.Catalog.RucioFileCatalogClient
2026-09-02T12:51:32,660061Z WorkloadManagement/Optimizers/DIRAC.Resources.Catalog.RucioFileCatalogClient [140418957993536] DEBUG: instantiate the client w/o a config file -  take config params from the CS
2026-09-02T12:51:32,660897Z WorkloadManagement/Optimizers/DIRAC.Resources.Catalog.RucioFileCatalogClient [140418957993536] ERROR: Cannot instantiate RucioFileCatalog interface Can't find proxy ( 1101 : )
2026-09-02T12:51:32,660977Z WorkloadManagement/Optimizers/FileCatalogFactory [140418957993536] DEBUG: Loaded module RucioFileCatalogClient
2026-09-02T12:51:32,662197Z WorkloadManagement/Optimizers/WorkloadManagement/InputData/InputData/JID_5 [140418957993536] VERBOSE: Catalog replicas lookup time 0.06 seconds
2026-09-02T12:51:32,662236Z WorkloadManagement/Optimizers/WorkloadManagement/InputData [140418957993536] WARN: FileCatalog error ( 1604 : Failed to perform getReplicas from any catalog)
2026-09-02T12:51:32,662286Z WorkloadManagement/Optimizers/WorkloadManagement/InputData/InputData/JID_5 [140418957993536] WARN: FileCatalog error ( 1604 : Failed to perform getReplicas from any catalog)
2026-09-02T12:51:32,662326Z WorkloadManagement/Optimizers/WorkloadManagement/InputData/InputData/JID_5 [140418957993536] INFO: Set to Failed/FileCatalog error ( 1604 : Failed to perform getReplicas from any catalog)

Manually checking though where the Optimizers run, the shifter proxy works:

>>> import DIRAC
>>> from DIRAC.Core.Utilities.Shifter import setupShifterProxyInEnv
>>> from pathlib import Path
>>> from rucio.client import Client
>>> config_files = [str(c) for c in Path("/opt/dirac/etc").glob("*.cfg")]
>>> DIRAC.initialize(extra_config_files=config_files, host_credentials=True)
>>> setupShifterProxyInEnv("DataManager")
2026-09-02T13:00:51,861293Z Framework INFO: Getting VOMS [/ctao.dpps.test/Role=user] proxy for shifter test-user@dpps_group (/CN=DPPS User)
{'OK': True, 'Value': {'DN': '/CN=DPPS User', 'username': 'test-user', 'group': 'dpps_group', 'chain': <X509Chain 5 certs [/CN=DPPS User/CN=5229753638/CN=7133985932/CN=1069567426/CN=1143134613][/CN=DPPS User/CN=5229753638/CN=7133985932/CN=1069567426][/CN=DPPS User/CN=5229753638/CN=7133985932][/CN=DPPS User/CN=5229753638][/CN=DPPS User] with key>, 'proxyFile': '/tmp/tmps36n4buu'}}
>>> import os
>>> os.environ["X509_USER_PROXY"]
'/tmp/tmps36n4buu'
>>> c = Client()
>>> c.list_scopes()
['root', 't_20260902_094605_afe7', 't_20260902_101150_47cc', 't_20260902_101355_a076', 't_20260902_101435_3cde', 't_20260902_101729_9c22', 't_20260902_125131_4693']

[maxnoe]

We put the shifterProxy section into the wrong section, needs to be in InputData:

    Systems
    {
      WorkloadManagement
      {
        Executors
        {
          Optimizers
          {
            Load = JobPath
            Load += JobSanity
            Load += InputData
            Load += JobScheduling
          }

          InputData {
            shifterProxy = DataManager
          }
        }
      }
    }

[maxnoe]

This works, thanks for the help @chaen

[chaen]

glad it worked ! :-)