Sentinel Data Lake Query Provider Support

[JPvRiel]

Is your feature request related to a problem? Please describe.

We have a subset of data stored as lake-only-mode tables in sentinel which avoids cost, as well as for archive, but I cannot find a documented query provider I can use for this with msticpy.

Describe the solution you'd like

I'd like a data driver/query provider implementation that supports sentinel data lake features.

Describe alternatives you've considered

As a workaround, an external table definition can be created via ADX. See: https://learn.microsoft.com/en-us/azure/sentinel/datalake/kql-queries#query-the-data-lake-with-azure-data-explorer

However, the way the existing ADX data driver / query provider wraps authentication does not work well and below is an example of vibe coding past the exceptions with monkey patches.

# FIXME: monkey patch the missmatch auth_types vs auth_methods as the Kusto driver calls az_connect with auth_types instead of auth_methods which is what the az_connect function expects?
import msticpy.data.drivers.azure_kusto_driver as akd
from msticpy.auth.azure_auth import az_connect as _orig_az_connect
def _az_connect_compat(*, auth_types=None, auth_methods=None, **kwargs):
    if auth_methods is None and auth_types is not None:
        auth_methods = auth_types
    return _orig_az_connect(auth_methods=auth_methods, **kwargs)

akd.az_connect = _az_connect_compat


# FIXME: monkey patch so that correct KustoServiceResourceId is obtained from metadata endpoint. Assuming <https://api.securityplatform.microsoft.com/.default> does not work and results in AADSTS500011.
import requests
def _scope_from_cluster_uri(resource_uri: str) -> str:
    try:
        meta = requests.get(
            resource_uri.rstrip("/") + "/v1/rest/auth/metadata",
            timeout=30
        ).json()
        resource_id = meta["AzureAD"]["KustoServiceResourceId"]
        return f"{resource_id}/.default"
    except Exception:
        # safe fallback for regular ADX clusters
        return "https://kusto.kusto.windows.net/.default"

akd.get_default_resource_name = _scope_from_cluster_uri


sentinel_data_lake_adx_endpoint = 'https://api.securityplatform.microsoft.com/lake/kql'
database = '<workspace name>-<workspace id>'
q_prov_data_lake_via_adx = mp.QueryProvider('Kusto')
q_prov_data_lake_via_adx.connect(
    cluster=sentinel_data_lake_adx_endpoint, database=database,
    # FIXME: msticpy chained credentials and vscode auth did not work so use other auth types for now.
    auth_types=["cli", "interactive"]
)

Additional context

As per https://learn.microsoft.com/en-us/azure/sentinel/datalake/notebooks-overview, there's a lot more to Sentinel Data Lake besides running KQL queries and a more "native driver" implementation instead of wrapping ADX could expose more features, e.g. pyspark.