Add gathered filtering for vPC and policy list modules - #552
Draft
deekpand-cisco wants to merge 16 commits into
Draft
deekpand-cisco wants to merge 16 commits into
deekpand-cisco wants to merge 16 commits into
Conversation
deekpand-cisco
force-pushed
the
feature/gathered-vpc-policy-list-filtering
branch
from
September 3, 2026 12:00
7b96acc to
cff6925
Compare
Add gathered_all state to the state machine for modules that retrieve all instances, mask_secrets helper for sensitive field redaction, and associated unit tests. Note: gathered-all framework cherry-picked from PR CiscoDevNet#312
Add server-side Lucene candidate filtering for gathered loopback interfaces with pagination, deduplication, and final local matching. Support gather-all and login-ID filtering for local users while rejecting unsupported gathered criteria. Add unit and integration coverage for filtering, validation, secrecy, and reusable output.
- Remove early return in filter_gathered_response() so deduplication runs unconditionally regardless of whether filters are provided - Add validate_gathered_filters() pre-flight check called before query_all() to reject invalid filters without wasted API calls
- Add gathered_filter_properties ClassVar to NDBaseModel for declarative filter whitelisting per module - Add pre-flight property validation (_extract_active_leaf_paths, _reject_unsupported_filter_properties) that rejects unsupported filter fields before any API call - Replace local_user custom normalize_gathered_filter validation with shared gathered_filter_properties tuple (login_id, email, first_name, last_name) - Add gathered_filter_properties to loopback model (switch_ip, interface_name, admin_state, ip, ipv6, vrf) - Pass supported_properties from model to validate_gathered_filters in state machine - Add gathered_transform support in NDOutput for modules with input/output shape differences - Fix loopback pagination: add max_pages safety cap, handle missing or invalid remaining metadata gracefully - Update nd_local_user DOCUMENTATION with supported filter properties - Update local_user unit tests to use shared validation path
- Apply black formatting to all gathered filtering module and test files - Add gathered state as no-op pass in manage_state() so modules can call manage_state unconditionally without raising InvalidState - Address review formatting feedback (trailing whitespace, blank lines)
- Extract gathered query logic into _query_existing() private helper - Move user-input validation outside try block for clean error messages - Build proposed before querying ND (fail-fast on bad config) - Declare get_argument_spec on NDBaseModel and gathered_transform on NDBaseOrchestrator; remove getattr probes for discoverability - Rename supports_gathered_lucene_filtering to supports_gathered_server_filtering (mechanism-neutral) - Normalize filters once in state machine; pass normalize_filter=None downstream to eliminate triple normalization - Return models from filter_gathered_response to avoid double Pydantic validation; use pre-built models for NDConfigCollection construction - Cap Lucene query fan-out at 3 expressions per switch; collapse to base expression beyond threshold - Raise ValueError for unknown switch_ip in gathered filters instead of silently returning empty results - Raise RuntimeError on pagination limit exhaustion instead of silently truncating gathered results - Update unit tests for new return types, renamed flags, and pre-normalized filter inputs
…e, import cleanup - Refactor _build_gathered_query_plan to classify filters as fabric-wide or switch-scoped before expansion, preventing O(switches × filters) REST fanout on large fabrics (addresses AKDRG review concern) - Add _MAX_TOTAL_REQUESTS budget cap in NDBaseInterfaceOrchestrator so fabric-wide expressions collapse to the base query before multiplying across switches when the projected total exceeds the threshold - Move _MAX_EXPRESSIONS_PER_SWITCH to NDBaseInterfaceOrchestrator so all interface orchestrators (loopback, vpc, ethernet) inherit it - Include before/after/diff as empty lists in gathered output for consistent output shape across all states - Switch nd_interface_loopback to format_with_verbosity so gathered queries appear in api_* debug keys at -vvv - Add unit test for large-fabric budget collapse scenario - Collapse single-name multiline imports to single line across all PR files
…g_values and remove leftover assertion from conflict resolution
… group modules Enable gathered state and Lucene filtering for: - nd_interface_ethernet_access - nd_interface_ethernet_trunk_host - nd_fabric_update_group Includes: - Model gathered_spec and gathered_transform definitions - Orchestrator gathered workflow with pagination - Integration test tasks for gathered state - Unit tests for gathered filtering round-trip
- Add TODO(4.2.1) interface-lucene-or-silently-empty workaround markers - Remove unreachable _switches_to_query override from ethernet_base - Update query_all docstring to cover gathered path - Consolidate loopback duplicated pagination/query-plan into base class via _configure_lucene_endpoint hook - Remove gathered early-return in trunk-host so _is_unconfigured_default filters on all states - Use modern annotations (set[str], dict[str, Any]) in new code - Remove Python 2 boilerplate from new test file - Fix trailing whitespace
state: gathered was running the same pre-flight checks as merged, replaced and deleted, so gathering failed on a frozen fabric even though it only reads. Deployment freeze is meant to stop config reaching the switches, not to stop you looking at what is already there. FabricContext now has validate_for_read, which checks that the fabric exists and belongs to the controller we are talking to. validate_for_mutation calls it and then adds the freeze check on top. The interface orchestrators pick between the two based on the module state. Only gathered is treated as read-only. Every other state, including an unknown or missing one, keeps the existing checks, so merged, replaced, overridden and deleted behave exactly as before. This affects all 10 interface modules that support gathered, since they share NDBaseInterfaceOrchestrator. Tests cover validate_for_read, gathered succeeding on a frozen fabric, and deleted still being blocked on one.
- Add gathered state support to eBGP, iBGP, External, AI eBGP, AI iBGP - Add bounded pagination with meta.counts.remaining - Add exact-name endpoint optimization for fabric_name-only filters - Fallback uses type-only paginated query instead of unfiltered scan - Reject config_actions.save/deploy with state=gathered - Update config_actions docstring to reflect rejection behavior - Add unit and integration tests for all fabric gathered paths
Add gathered state and filtering support for vPC access, vPC trunk-host, prefix list, community list, and extended community list. The change adds gathered filter validation, server-side query filtering, unit coverage, and integration gathered tasks. vPC modules use the shared gathered query path while keeping existing peer deduplication and mutation workflows unchanged.
deekpand-cisco
force-pushed
the
feature/gathered-vpc-policy-list-filtering
branch
from
September 9, 2026 11:14
cff6925 to
3c63fd7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue(s)
Related to #340.
This PR continues the gathered-state framework and filtering work from:
#391,
#507, and
#514.
Proposed Changes
Adds
state: gatheredand filtering support for:nd_interface_vpc_accessnd_interface_vpc_trunk_hostnd_manage_prefix_listnd_manage_community_listnd_manage_extended_community_listEach module now includes:
gatheredstate documentation and module wiringThe vPC modules use the shared gathered query path in
VpcInterfaceBaseOrchestrator. Existing vPC peer-deduplication behavior andexisting mutation workflows remain unchanged for
merged,replaced,overridden, anddeleted.Stacked PR
Since the stacked branches are in a fork and each PR targets
develop, GitHubshows inherited parent changes in this PR. The review scope for this PR is commit
7b96accand the five modules listed above.Test Notes
Unit Tests
4189 passed— full repository unit suite300 passed— focused model and orchestrator tests for the five modulesIntegration Tests
state: gathered; no mutationstate is used.
Cisco Nexus Dashboard Version
4.2.1.10
Related ND API Resource Category