Add a recognizer for provider-issued API keys, access keys and bearer tokens

[developer0hye]

Is your feature request related to a problem? Please describe.

Presidio has no recognizer for provider-issued credentials. predefined_recognizers/generic/ covers credit card, crypto, date, email, IBAN, IP, MAC, phone, URL and UUID, and docs/supported_entities.md has no entry for API keys, access keys or bearer tokens.

Credentials appear in exactly the places Presidio is already pointed at — logs, support tickets, chat transcripts, notebooks, config dumps pasted for debugging. A leaked AWS access key or GitHub token in one of those is at least as damaging as the PII beside it, and today Presidio passes it through untouched.

#1339 proposed an APIKeyRecognizer. @omri374 replied "Thanks, great addition!" and asked for tests in a separate file plus removal of a redundant analyze() override. The author did not follow up and it was closed as stale, so the idea was never rejected on its merits.

Its pattern cannot be reused as-is:

r"\b(?i)([A-Za-z0-9]{20,40}|[A-Za-z0-9]{6}-[A-Za-z0-9]{6}-[A-Za-z0-9]{6})\b"  # score 0.2

1. It does not compile on Python 3.11+. A non-leading inline global flag became a hard error in 3.11. Reproduced on 3.12.10:

   re.error: global flags not at the start of the expression at position 2
   

2. Unbounded false positives. [A-Za-z0-9]{20,40} matches git SHAs, hex digests, unhyphenated UUIDs and base64 fragments. The 0.2 base score does not contain this — LemmaContextAwareEnhancer adds 0.35 and floors the result at 0.4 when any of the recognizer's own context words (api, token, secret, ...) appears within the preceding 5 tokens, which is routine in log and config text.
3. It misses real credentials. \b anchors plus {20,40} mean a run of 41+ alphanumerics matches nothing, so longer tokens are skipped silently.

The problem is the strategy rather than the tuning: one high-entropy string pattern for every credential cannot be made precise.

Describe the solution you'd like

Detect only credential formats whose prefix — or, for the AWS secret access key, whose credential name — is stated in vendor documentation, and anchor every pattern on that marker. Length and entropy then act as supporting evidence rather than as the primary signal.

Credential	Marker	Source
AWS access key ID	AKIA, ASIA, ABIA, ACCA	IAM identifiers — unique ID prefixes
AWS secret access key	aws_secret_access_key / AWS_SECRET_ACCESS_KEY	AWS CLI environment variables
GitHub tokens	ghp_, gho_, ghu_, ghs_, ghr_ + 36 base62	Behind GitHub's new authentication token formats
Google API key	AIza + 35	Manage API keys
Slack tokens	xoxb-, xoxp-, xoxe-, xapp-	Slack token types
Stripe secret / restricted keys	sk_live_, rk_live_	Stripe API keys
JSON Web Token	eyJ header and payload, three dot-separated segments	RFC 7519 §3

Two implementation details matter for correctness:

• Match case-sensitively. PatternRecognizer defaults to re.DOTALL | re.MULTILINE | re.IGNORECASE. Every vendor prefix here is case-sensitive, and matching case-insensitively makes markers such as AKIA and eyJ fire on ordinary lowercase text.

  Passing global_regex_flags without IGNORECASE — as IbanRecognizer already does — fixes this. Correction (see feat(analyzer): add ApiKeyRecognizer for provider-issued credentials #2203): constructor flags are not sufficient. RecognizerListLoader.get assigns the registry's global_regex_flags to every PatternRecognizer after construction, so a recognizer's own flags are discarded once it is loaded from default_recognizers.yaml. Verified: IbanRecognizer loaded from the shipped registry reports re.IGNORECASE|re.MULTILINE|re.DOTALL, so the cited precedent is subject to the same override. Case sensitivity has to travel with the pattern instead, via a scoped (?-i:...) group.

• Exclude the IAM identifier prefixes. The IAM table lists AKIA, ASIA, ABIA and ACCA as credentials, while AIDA, AROA, ANPA, ANVA, AGPA, AIPA, APKA and ASCA identify users, roles, groups and policies. Only the first group belongs in a credential recognizer.

The AWS secret access key is the one credential with no structure of its own — 40 characters of base64, indistinguishable from a digest or an encoded blob. Anchoring it on the credential name AWS documents keeps it precise. PatternRecognizer matches with the regex module, whose variable-length lookbehind lets the anchor sit outside the reported span so only the secret is returned.

Describe alternatives you've considered

Entropy-gated generic matching, as in #1339. Rejected: on a 40-character base64 candidate, entropy separates random strings from English text but not credentials from digests, ids or encoded blobs, which are the actual false positives.

Deferring to a dedicated secret scanner (gitleaks, detect-secrets, TruffleHog). Reasonable for repository scanning, but it does not help the case Presidio is built for — anonymizing free text that happens to contain a credential, where the caller wants one pipeline and one set of RecognizerResults.

I cross-checked the proposed patterns against gitleaks and detect-secrets, which resolved two open questions and surfaced two divergences worth stating up front:

• AWS access key charset. gitleaks uses [A-Z2-7]{16} (base32); detect-secrets uses [0-9A-Z]{16}. AWS documents neither. The proposal follows detect-secrets, since the narrower charset risks false negatives for no meaningful false-positive reduction behind a 4-character prefix.
• AWS secret access key. gitleaks has no rule for it at all. detect-secrets anchors on a variable name (aws.{0,20}?(?:key|pwd|...).{0,20}?['"]([0-9a-zA-Z/+]{40})['"]), which is the same strategy proposed here — this is prior art, not a new idea. The proposal anchors on the exact documented credential name instead of a fuzzy window, and does not require surrounding quotes, so it also covers the environment-variable and INI forms.
• GitHub token body. detect-secrets allows _ in the 36-character body; the GitHub post specifies base62, so the proposal excludes it.
• Legacy A3T[A-Z0-9] AWS prefix. Both tools carry it; it does not appear in any current AWS document. Omitted to keep every pattern citable — easy to add if maintainers prefer coverage over provenance.

Additional context

Implementation and tests are ready (34 parametrized cases, including negatives for hex digests, lowercase prefixes, IAM identifier prefixes, Stripe publishable and test-mode keys, and wrong-length tokens). A couple of open decisions for maintainers:

1. One entity or several? The proposal uses a single API_KEY entity with the provider carried in the pattern name (surfaced via analysis_explanation). Per-provider entities (AWS_ACCESS_KEY, GITHUB_TOKEN, ...) would give finer-grained anonymization at the cost of a much larger supported_entities.md surface. Happy to split.
2. Enabled by default? The proposal enables it alongside the other generic recognizers. Since it introduces a new entity type into default output, shipping it disabled first may be preferable.
3. Scope. Is a credential recognizer in scope for Presidio given the PII framing? API Key recognizer #1339 suggests yes, but it is worth settling before more providers are added.

---

Update — where #2203 diverges from the proposal above

The proposal text is left as originally filed. Source-backed review during #2203 changed several of its specifics, so read the table and the "Additional context" counts above as the request, not as the shipped behavior:

• Stripe test-mode keys are now detected, not excluded. The table lists only sk_live_ / rk_live_, and "Additional context" lists test-mode keys among the negatives. Stripe documents sandbox keys as pk_test_ (publishable), rk_test_ (restricted) and sk_test_ (secret), and states that only publishable keys are safe to expose outside a backend. Test mode limits blast radius; it does not make a secret or restricted key public. sk_test_ and rk_test_ are therefore positives; only pk_ remains excluded.
• Slack coverage changed in both directions. Added: xwfp- (workflow) and the rotation forms xoxe.xoxb- / xoxe.xoxp-, which are separate patterns from the xoxe- refresh token. Removed: the legacy xoxa / xoxr / xoxs prefixes, which are absent from Slack's current token-types documentation.
• GitHub coverage split. ghs_ installation tokens are a separate pattern because GitHub also issues a stateless JWT-format installation token and recommends accepting [A-Za-z0-9.\-_] with a minimum of 36 characters — the fixed 36-character base62 rule in the table would miss it. github_pat_ fine-grained PATs were also added; the table omits them.
• JWT scope was narrowed explicitly. The pattern covers the common compact signed eyJ….eyJ….… subset rather than every serialization RFC 7519 permits.
• Test count. 34 parametrized cases at filing; 60 tests as of the latest commit, including span-exactness and registry-level regression coverage.