chore(xtest): Add audit & log assertions#376
chore(xtest): Add audit & log assertions#376dmihalcik-virtru wants to merge 17 commits intoopentdf:mainfrom
Conversation
Summary of ChangesHello @dmihalcik-virtru, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a robust framework for collecting and asserting on KAS audit logs within the xtest suite. The changes are well-structured, with a clear separation of concerns between the log collection logic, pytest fixtures, and local development tooling. The new audit_logs fixture and AuditLogAsserter provide a powerful and developer-friendly way to write tests that verify logging behavior. The addition of run-local.sh and LOCAL_DEVELOPMENT.md significantly improves the local testing experience. I've identified a potential bug in the log parsing logic and a couple of minor improvements for the developer tooling.
xtest/run-local.sh
Outdated
| ( | ||
| cd "${PLATFORM_DIR}" | ||
| LOG_LEVEL="${LOG_LEVEL}" LOG_TYPE="${LOG_TYPE}" \ | ||
| test/local/start-kas.sh "${kas_name}" "${port}" ${extra_args} >/dev/null 2>&1 |
There was a problem hiding this comment.
Redirecting both stdout and stderr to /dev/null completely hides any output from the start-kas.sh script. If that script fails for any reason before the KAS process starts (e.g., a problem with its own logic, permissions, etc.), the failure will be silent, and the main script will just time out waiting for the health check. It would be more developer-friendly to let the output be printed for easier debugging.
| test/local/start-kas.sh "${kas_name}" "${port}" ${extra_args} >/dev/null 2>&1 | |
| test/local/start-kas.sh "${kas_name}" "${port}" ${extra_args} |
dmihalcik-virtru
force-pushed
the
DSPX-2178-auditious
branch
from
9d09da9 to
91508e9
Compare
Introduce a comprehensive audit log testing framework for validating platform audit events: - Add audit_logs.py module with lazy JSON parsing, structured assertions, and flexible filtering for efficient audit log processing - Add fixtures/audit.py with pytest fixtures for audit log collection, validation, and assertion helpers - Add test_audit_logs.py with self-tests to validate the audit framework - Enhance test_abac.py with audit log assertions for access control tests - Update CI workflow to enable audit log collection and include framework self-tests in validation runs The framework supports querying audit logs by subject, action, object, and custom predicates, with detailed assertion failures for debugging. Co-Authored-By: Claude Sonnet 4.5 <[email protected]> Signed-off-by: David Mihalcik <[email protected]>
dmihalcik-virtru
force-pushed
the
DSPX-2178-auditious
branch
from
e835ce4 to
0fa84be
Compare
still busted for things that use basekey (or rely on basekey not being set)
Create new fixtures in fixtures/keys.py and fixtures/attributes.py to allow tests to explicitly specify which key type (RSA or EC curves) they expect. This enables tests to control the KAO wrapping type instead of relying on platform defaults. New managed key fixtures (session-scoped): - managed_key_rsa_2048_default - RSA 2048 key - managed_key_rsa_4096_default - RSA 4096 key - managed_key_ec_secp256r1_default - EC secp256r1 key - managed_key_ec_secp384r1_default - EC secp384r1 key - managed_key_ec_secp521r1_default - EC secp521r1 key New attribute fixture (session-scoped): - attribute_keytype - Attribute with values for each key type, allowing tests to request specific wrapping types via ABAC policies These fixtures solve the KAO wrapping type assertion failures by allowing tests to explicitly request the wrapping type instead of relying on platform configuration defaults. cleanups Co-Authored-By: Claude Haiku 4.5 <[email protected]>
This reverts commit ca6d664.
Implement structured audit log assertions and test coverage for all audit events in the OpenTDF platform: rewrap, policy CRUD, and authorization decisions. CHANGES: Enhanced Framework (audit_logs.py): - Add ParsedAuditEvent dataclass with 20+ typed properties for structured access to audit event fields (action_type, action_result, object_type, object_id, key_id, algorithm, attr_fqns, etc.) - Add audit constants: OBJECT_TYPES, ACTION_TYPES, ACTION_RESULTS - Add 5 primary structured assertion methods: * assert_rewrap(result, policy_uuid, key_id, algorithm, attr_fqns) * assert_policy_crud(result, action_type, object_type, object_id) * assert_decision_v2(result, entity_id, action_name) * parse_audit_log(entry) - Parse JSON logs to structured events * get_parsed_audit_logs(since_mark, timeout) - Retrieve parsed events - Add 10 convenience methods: assert_rewrap_success/failure/error/cancelled, assert_policy_create/update/delete, assert_decision_v2 Unit Tests (test_audit_logs.py): - Add TestAuditConstants (4 tests) - Validates audit event constants - Add TestParsedAuditEvent (8 tests) - Event parsing and matching with rewrap, policy CRUD, and decision event validation - Add TestAuditLogAsserterEnhanced (5 tests) - New structured assertion methods with field filtering and object ID validation - Total: +14 test classes, 25 tests all passing Integration Tests (test_audit_logs_integration.py - NEW): - Add TestRewrapAudit (3 tests) - Rewrap success fields, failure on access denied, multiple KAO audit coverage - Add TestPolicyCRUDAudit (3 tests) - Namespace, attribute, and subject mapping CRUD audit trails - Add TestDecisionAudit (1 test) - Authorization decision audit coverage - Add TestEdgeCases (2 tests) - Tampering and sequential load scenarios Enhanced Existing Tests: - test_tdfs.py: Add rewrap audit assertions to roundtrip and tamper tests (test_tdf_roundtrip, test_tdf_with_unbound_policy, test_tdf_with_altered_policy_binding) - test_abac.py: Enhance ABAC tests with attr_fqns validation and multiple KAO assertions (test_autoconfigure_one_attribute_standard, test_autoconfigure_two_kas_or_standard, test_autoconfigure_double_kas_and) - test_self.py: Add policy CRUD audit assertions to namespace, attribute, and SCS creation tests (test_namespace_create, test_attribute_create, test_scs_create) COVERAGE: Rewrap Events (3 locations in platform/service/kas/access/rewrap.go): - RewrapSuccess: line 986 - RewrapFailure: line 965, 975 - RewrapError: (implicit in error handling) - Validates: policy_uuid, key_id, algorithm, attr_fqns Decision Events (2 locations): - GetDecision v1: authorization/authorization.go:713 - GetDecision v2: internal/access/v2/just_in_time_pdp.go:456 - Validates: entity_id, action_name, result (permit/deny) PolicyCRUD Events (150+ locations across policy services): - Namespace: create/update/deactivate/delete - Attribute/AttributeValue: create/update/deactivate - ConditionSet: create/update/deactivate - SubjectMapping: create/update/delete - KAS Registry: register/unregister/update - And more... FEATURES: - Structured ParsedAuditEvent objects with typed field access - JSON parsing with graceful error handling for invalid/non-audit logs - Multiple filter criteria (all must match for event selection) - Attribute FQN validation for rewrap events - Configurable timeouts (default 5.0 seconds) - Graceful degradation when audit logs unavailable (tests still pass) - Comprehensive error messages with context and matching logs METRICS: - 1824 total insertions across 6 files - 25 unit tests (all passing) - 9 integration test methods - 5+ enhanced existing test methods - 0 test failures - Backward compatible with existing test framework Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Format test_audit_logs_integration.py with ruff for improved code style and consistency. Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Add noqa comment to suppress false positive F401 warning for ParsedAuditEvent import, which is semantically used by the TestParsedAuditEvent test class. Co-Authored-By: Claude Haiku 4.5 <[email protected]>
CI failures in rewrap audit assertions showed the expected log event in the "Recent context" output, suggesting a race condition where logs arrive just after the timeout expires. Changes: - Increase default timeout from 5s to 20s for all assertion methods - Enhance _raise_assertion_error to capture and display logs that arrive after the timeout expires (with 0.5s grace period) - Add clear timeout marker in error output to visually separate "before" and "after" logs - Show warning when late arrivals are detected to indicate race condition The enhanced error output now shows: - Logs before timeout (last 10) - Timeout marker with timestamp - Logs AFTER timeout (up to 10 late arrivals) - Warning if late arrivals suggest a race condition This helps diagnose whether test failures are due to timing issues vs. missing audit events. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Implement robust clock skew detection by comparing LogEntry.timestamp (when logs are read by test machine) with ParsedAuditEvent.timestamp (when events occurred per service clock). This addresses potential issues where clock differences between test machine and services could cause assertions to miss events or incorrectly order them. Key additions: - parse_rfc3339(): Parse RFC3339 timestamps from service logs - ClockSkewEstimate: Track per-service skew statistics (min/max/median) - ClockSkewEstimator: Aggregate skew samples across all services - ParsedAuditEvent.event_time/collection_time/observed_skew properties - AuditLogAsserter now auto-adjusts mark timestamps for clock skew - Enhanced error messages include skew diagnostics Co-Authored-By: Claude Opus 4.5 <[email protected]>
Fix linting issues found by ruff: - Remove unnecessary blank lines - Fix import ordering - Apply consistent formatting Co-Authored-By: Claude Haiku 4.5 <[email protected]>
Change audit log behavior to fail tests if log files are not available, instead of silently skipping assertions. This ensures comprehensive test coverage and catches audit logging regressions. Key changes: - Add DISABLE_AUDIT_ASSERTIONS environment variable support - Fail fixture setup with clear error if logs not found - Update --no-audit-logs to work with new env var - Add documentation about required audit assertions To disable audit assertions (when needed): DISABLE_AUDIT_ASSERTIONS=1 pytest ... OR pytest --no-audit-logs ... Error message now explains: - Why logs are required - Where to find/configure log files - How to disable if needed This complements the clock skew estimation feature by ensuring audit logs are present for proper validation. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
|
1 participant
No description provided.