[sonic-frr/sonic-yang-models]: Sync EVPN MACs between SONiC and FRR over the FPM channel - #29312
tahmed-dev wants to merge 7 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
2199158 to
2f1487b
Compare
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
| {% set _dep = use_dependent_startup | default(false) %} | ||
| {% set _wait = frr_wait_for | default("rsyslogd:running") %} | ||
| {# EVPN-MH does hardware MAC learning and aging, which zebra only supports in | ||
| external-learn mode. L3EvpnMH needs it too, since RT-2 still carries MACs; |
There was a problem hiding this comment.
comments about L3evpnMH is not required.
There was a problem hiding this comment.
The comment now only says why the flag is set: MAC sync over FPM needs zebra in external-learn mode. It names L3EvpnMH once, because that subtype always syncs over FPM.
| {# EVPN-MH does hardware MAC learning and aging, which zebra only supports in | ||
| external-learn mode. L3EvpnMH needs it too, since RT-2 still carries MACs; | ||
| non-MH devices keep kernel MAC aging and remote neighbour install. #} | ||
| {% set _mac_ext_learn = DEVICE_METADATA is defined and DEVICE_METADATA.localhost.subtype is defined and DEVICE_METADATA.localhost.subtype in ["L3EvpnMH", "EvpnMH"] %} |
There was a problem hiding this comment.
can we keep only evpnMH... and in theory, this need capability should also work for EVPN SH too.
There was a problem hiding this comment.
The flag now follows FDB_SYNC|global|mac_sync_mode fpm, so single-homed EVPN gets it too. L3EvpnMH remains only as the subtype that always syncs over FPM.
| ! | ||
| {% endfor %} | ||
| {% endif %} | ||
| {% if VRF is defined %} |
There was a problem hiding this comment.
Removed. This PR no longer touches zebra.interfaces.conf.j2; that was the VRF VNI binding, not MAC sync.
| ROUTE_TABLE | ||
| LABEL_ROUTE_TABLE | ||
| {% endif %} | ||
| {% if DEVICE_METADATA.localhost.orch_northbond_evpn_mh_zmq_enabled != "false" %} |
There was a problem hiding this comment.
should this be a different PR?
There was a problem hiding this comment.
Yes. Removed from this PR.
| FRR = frr_$(FRR_VERSION)-sonic-$(FRR_SUBVERSION)_$(CONFIGURED_ARCH).deb | ||
| $(FRR)_DEPENDS += $(LIBSNMP_DEV) $(LIBYANG3_DEV) $(LIBFIB_DEV) | ||
| $(FRR)_RDEPENDS += $(LIBYANG3) $(LIBFIB) | ||
| $(FRR)_DEPENDS += $(LIBSNMP_DEV) $(LIBYANG3_DEV) $(LIBFIB_DEV) $(LIBSWSSCOMMON_DEV) |
There was a problem hiding this comment.
what does FRR need from SWSS_common?
There was a problem hiding this comment.
Nothing now. It was for the APP_DB publisher, which is no longer in this PR, so rules/frr.mk is not touched any more.
| int ifindex; | ||
| uint8_t mac[ETH_ALEN]; | ||
| uint16_t vid; | ||
| uint32_t generation; |
There was a problem hiding this comment.
can you add comments describing each elements
There was a problem hiding this comment.
Done: struct fpm_local_mac documents the fields whose meaning is not obvious (ifindex, vid, del, sticky), and the comment above fpm_mac_decode() has a table of what fpmsyncd sends.
| uint32_t generation; | ||
| uint8_t protocol; | ||
| bool del; | ||
| bool sticky; |
There was a problem hiding this comment.
do we need to know the difference between sync and sticky?
There was a problem hiding this comment.
No. The struct has no sync field any more; only sticky remains, read from NTF_STICKY.
| /* | ||
| * Decode an AF_BRIDGE RTM_NEWNEIGH/RTM_DELNEIGH message. Returns false and | ||
| * leaves *out untouched when the message is not a usable local MAC. | ||
| */ |
There was a problem hiding this comment.
I presume this function is called for any message coming from fdborch
There was a problem hiding this comment.
Yes: fpmsyncd follows STATE_DB FDB_TABLE, which fdborch writes, and sends one message for each add or delete; this decodes each of them. The comment now says so.
| /* RMAC walk finished. */ | ||
| FNE_RMAC_FINISHED, | ||
| /* Remote MAC walk finished. */ | ||
| FNE_MAC_FINISHED, |
There was a problem hiding this comment.
why extending fpm_nl struct?
There was a problem hiding this comment.
It is not extended any more. The remote MAC walk event is no longer in this PR, which adds no fields to struct fpm_nl_ctx.
| } | ||
|
|
||
| /* Nothing is feeding local MACs while the channel is down, so zebra has | ||
| * to own them again until fpmsyncd replays and takes them back. |
There was a problem hiding this comment.
what do you mean by owning by zebra?
There was a problem hiding this comment.
That comment went with the ownership latch, which this PR no longer has.
|
|
||
| memcpy(mac.octet, flm->mac, ETH_ALEN); | ||
|
|
||
| ifp = if_lookup_by_index(flm->ifindex, VRF_DEFAULT); |
There was a problem hiding this comment.
what is the check about?
why are you excluding VRF and not default?
There was a problem hiding this comment.
The lookup no longer involves a VRF. fpm_mac_bridge_port() finds the port by ifindex in the namespace, as zebra's kernel FDB path does, since a bridge port is not tied to a VRF, and requires it to be a bridge slave.
| * Runs on zebra's main thread: the interface table and the EVPN MAC hashes are | ||
| * owned by it and are not safe to touch from the FPM thread. | ||
| */ | ||
| static void fpm_apply_local_mac(struct event *t) |
There was a problem hiding this comment.
is this downloading MAC to sonic?
There was a problem hiding this comment.
No, the other way: this is SONiC to FRR, a MAC the ASIC learnt or aged, from fpmsyncd. FRR to SONiC is fpm_nl_enqueue(), remote MACs only. Both are now labelled.
| if (flm->del) { | ||
| zebra_vxlan_local_mac_del(ifp, zif->brslave_info.br_if, | ||
| &mac, flm->vid); | ||
| if (kernel_del_local_mac(ifp, flm->vid, &mac) < 0) |
There was a problem hiding this comment.
why kernel delete? Where is the FSM per MAC maintaining the multi-sources?
There was a problem hiding this comment.
The kernel write is there because the ASIC learnt the MAC and the kernel never saw it. zebra's MAC is the per-MAC state machine: its flags hold both sources, the ASIC (ZEBRA_MAC_LOCAL) and an Ethernet Segment peer's route (ZEBRA_MAC_ES_PEER_ACTIVE/PROXY). When the ASIC loses the MAC, fpm_local_mac_gone() tells zebra first. If the peer still holds the MAC, zebra keeps it local-inactive and rewrites the entry itself, and the module leaves it; otherwise the module removes the entry it wrote.
| * to happen before zebra sees the MAC: a MAC on an Ethernet Segment | ||
| * makes zebra queue its own sync install marking the entry static, and | ||
| * that runs on the dplane thread. Seeding afterwards would race it and | ||
| * could downgrade a static entry back to extern_learn. |
There was a problem hiding this comment.
dont you need to maintain 2 sources per MAC? control plane and data plane?
There was a problem hiding this comment.
Yes, and zebra's MAC flags already hold both: ZEBRA_MAC_LOCAL for the data plane and ZEBRA_MAC_ES_PEER_ACTIVE/PROXY for the control plane. The module follows them rather than keeping a copy; see the reply on the delete above.
| * arrive instead leaves a window at boot where zebra's Ethernet Segment | ||
| * sync writes the kernel entry before the ASIC has learnt anything. | ||
| * | ||
| * Generation 0 is the opposite statement: fpmsyncd left fpm mode and no |
There was a problem hiding this comment.
where is that generation 0 or not logic explained? ;(
There was a problem hiding this comment.
The generation and the replay end marker that carried it are no longer in this PR.
| * @return Matching ZMQ producer table, or NULL when unsupported. | ||
| */ | ||
| static SWSSZmqProducerStateTable | ||
| sonic_frr_redis_zmq_table(struct sonic_frr_redis_ctx *ctx, const char *table) |
There was a problem hiding this comment.
what is this again? zmq on NB api? Why? separate PR again?
There was a problem hiding this comment.
Yes, the ZMQ northbound. Removed from this PR.
| #define RTPROT_EIGRP 192 /* EIGRP Routes */ | ||
| - | ||
| +#define RTPROT_HW 193 /* Hardware Routes */ | ||
| +#define RTPROT_HW 199 /* Hardware Routes */ |
There was a problem hiding this comment.
Done: this PR drops patch 0029.
| These message types are consumed by fpmsyncd in SONiC to populate | ||
| EVPN_SPLIT_HORIZON_TABLE, EVPN_DF_TABLE, and EVPN_ES_BACKUP_NHG_TABLE | ||
| in APPL_DB. | ||
| EVPN_SPLIT_HORIZON_TABLE and EVPN_DF_TABLE in APPL_DB. |
There was a problem hiding this comment.
keep this one EVPN_ES_BACKUP_NHG_TABLE. Why do you remove this?
There was a problem hiding this comment.
Kept. This PR no longer touches patch 0077.
| + RTM_FPM_DEL_EVPN_ES_BACKUP_NHG, | ||
| + RTM_FPM_LAST = RTM_FPM_DEL_EVPN_ES_BACKUP_NHG, | ||
| + /* | ||
| + * End of a MAC replay generation. nlmsg_seq carries the generation. |
There was a problem hiding this comment.
keep them! RTM_FPM_ADD_EVPN_ES_BACKUP_NHG, RTM_FPM_DEL_EVPN_ES_BACKUP_NHG,
There was a problem hiding this comment.
Kept; patch 0077 is unchanged.
| leaf subtype { | ||
| type string { | ||
| pattern "DualToR|SmartSwitch|Supervisor|UpstreamLC|DownstreamLC"; | ||
| pattern "DualToR|SmartSwitch|Supervisor|UpstreamLC|DownstreamLC|L3EvpnMH|EvpnMH"; |
There was a problem hiding this comment.
again, why L3MH or evpnMH reference?
There was a problem hiding this comment.
EvpnMH is dropped; L3EvpnMH is kept on purpose. An L3EvpnMH device always syncs MACs over FPM, so docker-fpm-frr, fdbsyncd and fpmsyncd select fpm from the subtype whatever FDB_SYNC says. subtype is pattern constrained, so the value has to be listed here or config reload rejects it. DualToR could not be reused: it also starts ycabled and the mux tunnel packet handler.
|
This PR is a huge bag of many different functionalities that should have been done in separate PRs. 1- MAC over fpm (bi-directional) Also, FRRouting is missing a FSM per MAC evaluating its ownership. Comments for each directional function must be clearly state. (FRR->SONIC or SONIC->FRR) This work is highly confusing and will be very difficult to maintain / progress without proper cleanup. |
|
[P1] FPM ownership is enabled too early [P1] Bridge-port deletes can target VlanX:unknown [P2] PortChannel system_mac changes are not propagated |
2f1487b to
54f52fa
Compare
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
54f52fa to
9e01580
Compare
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Re #29312 (comment): done. This PR now does only MAC sync over FPM in both directions (1), and the companion PRs are cut the same way. Not in this PR, unlike its earlier revision: the ZMQ northbound (2), L2 nexthop groups over FPM (3), resynchronization after the FPM connection drops (5) and Ethernet Segment parameters from CONFIG_DB (6). For 4, the ES backup NHG messages are no longer removed: patch 0077 is untouched. Per-MAC ownership: zebra's MAC already holds both sources, the ASIC ( Each MAC path in |
|
Re #29312 (comment): [P1] FPM ownership: the [P1] [P2] |
This taught the bridge command to emit and print NDA_PROTOCOL on an FDB entry. The kernel side of that is the sonic-linux-kernel protocol patch, which sonic-linux-kernel#619 removes. Without it the kernel discards the attribute on add and never returns it on dump. Nothing consumes it either. Its only user is fdbsyncd, and the support probe behind that use can never succeed because it tests "dev lo", and loopback is not an Ethernet device. sonic-swss#4857 removes that plumbing. Where a MAC came from travels over the FPM channel instead, so the origin marker no longer needs a kernel or iproute2 carrier. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
Patch 0029 adds RTPROT_HW (193) to FRR's copy of linux/rtnetlink.h, to tag the kernel bridge FDB entries of MACs the hardware learnt. Nothing in FRR uses it: no other patch and no source file refers to it. The tag relied on kernel and iproute2 changes that are not upstream, and SONiC drops its copies of both: sonic-linux-kernel#619, and the iproute2 patch in the previous commit. Bridge FDB entries carry no protocol, and where a MAC came from travels over the FPM channel instead. 193 is also the value FRR already uses for RTPROT_LDP (zebra/rt_netlink.h). Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
FDB_SYNC|global|mac_sync_mode selects how MAC state is synchronized between SONiC and FRR: kernel fdbsyncd synchronizes through the Linux bridge FDB, as today. fpm fpmsyncd exchanges MACs with zebra over the FPM channel. It defaults to kernel, so nothing changes unless fpm is configured. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
dh_auto_test runs "make test", which only builds the pceplib helpers and runs nothing, so the FRR unit test suite never executed during the SONiC package build. Override it with tests/tests.xml, which builds every check program and runs the suite. A plain "make check" is not usable here: pceplib's *_valgrind.sh wrappers reference a relative path that does not resolve in an out-of-tree build and always fail with exit 127. The C unit tests added by the following patches link against CUnit, so add libcunit1-dev to Build-Depends. It is restricted to builds without the nocheck profile, which cross builds use and which skips the tests. This is SONiC-only build plumbing. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
Add FRR patch 0127. In fpm mode the ASIC learns local MACs, dplane_fpm_sonic hands them to zebra over the FPM channel, and zebra writes their kernel bridge FDB entries itself, including the entry of a MAC only an Ethernet Segment peer has, which zebra holds local-inactive. The kernel reports each of those writes back. In mac-ext-learn mode zebra tells its own writes apart only by NDA_PROTOCOL, which the kernel does not keep for bridge FDB entries, so the report arrives as a local learn: the inactive MAC turns active, is advertised as learnt here, and stays after the peer withdraws it. The patch lets a dataplane module tell zebra that it, not the kernel, learns and ages local MACs. While it does, zebra ignores kernel FDB notifications for local bridge ports, adds and deletes alike; VXLAN ports are not affected. zebra also flushes a MAC whose peer-active hold timer expires only if it is local-inactive. An active MAC is still in the ASIC, so flushing it would drop the host's MAC on a leaf whose Ethernet Segment peer stopped seeing the host, or rebooted. Nothing sets it yet; dplane_fpm_sonic does in the next commit. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
dplane_fpm_sonic exchanges MACs with fpmsyncd over the FPM channel, in both directions, for FDB_SYNC|global|mac_sync_mode fpm. SONiC to FRR: fpmsyncd sends each MAC the ASIC learns or ages as an AF_BRIDGE RTM_NEWNEIGH or RTM_DELNEIGH. The module decodes it on the FPM thread and applies it on zebra's main thread, which owns the interface table and the EVPN MAC hashes, through zebra_vxlan_local_mac_add_update() and zebra_vxlan_local_mac_del(). Those are the entry points the kernel path uses, so the EVPN state machine and its multihoming handling are unchanged. FRR to SONiC: remote MACs, which zebra learnt from BGP EVPN, already reach fpmsyncd as DPLANE_OP_MAC_INSTALL and DPLANE_OP_MAC_DELETE. Local MACs are no longer echoed back. They originate from fpmsyncd, and the encoder would also emit a 16 byte NDA_DST from the never-initialised vtep of dplane_local_mac_add(), which fpmsyncd would read as a remote VTEP. The kernel never saw a MAC the ASIC learnt, so the module writes its bridge FDB entry. It does so through dplane_local_mac_add() and dplane_local_mac_del(), so that the write shares one queue with zebra's own, and queues its add before handing the MAC to zebra, so zebra's write lands last. zebra's MAC flags hold both sources of a local MAC, the ASIC and an Ethernet Segment peer's route. When the ASIC loses a MAC the peer still advertises, zebra keeps it local-inactive and rewrites the entry itself, so the module leaves the entry alone; it removes the entry only once zebra no longer holds the MAC. In mac-ext-learn mode the module tells zebra at startup that it learns local MACs (FRR patch 0127), so zebra does not take the kernel's report of these writes for a local learn. zebra also flushes a local MAC when it expects it to be relearnt, as when an Ethernet Segment is configured on or removed from its port, and holds a MAC local-inactive until the dataplane reports activity. Over FPM nothing reports a MAC the ASIC keeps again. The module keeps a record of the MACs fpmsyncd reports, and when zebra removes the kernel entry of a local MAC the record still holds, or writes the entry of one it holds local-inactive, the module reports the MAC to zebra again as learnt. A MAC a remote route took over is left to zebra. The decode lives in fpm_mac.c, which depends only on the kernel netlink headers, so a CUnit test (FRR patch 0128) exercises it directly. It checks the message family and the attribute lengths, and writes the caller's struct only once the message is fully validated. Resynchronizing MACs after the FPM connection drops is not part of this change. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
…g when MACs sync over FPM MAC synchronization over FPM relies on the hardware doing MAC learning and aging, which zebra only supports when it is started with --kernel-mac-ext-learn. Nothing in SONiC ever passed that flag, so zebra kept driving the kernel MAC learning and aging state machine itself. Pass the flag when FDB_SYNC|global selects mac_sync_mode fpm. This does not depend on multihoming: a single-homed EVPN device syncing MACs over FPM needs it as much as a multihomed one. Every device that syncs MACs through the kernel keeps kernel MAC aging and remote neighbour install as before. An L3EvpnMH device always syncs MACs over FPM, so it gets the flag whatever FDB_SYNC says. Add the L3EvpnMH device subtype for that. subtype is pattern constrained in YANG, so the new value has to be added there or config reload rejects it. The existing DualToR subtype could not be reused: it also starts ycabled and the mux tunnel packet handler, which have no meaning on an EVPN multihoming leaf. zebra reads the flag when the bgp container starts, so a mode change reaches zebra with the next restart of that container. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
9e01580 to
d84ac13
Compare
|
/azp run Azure.sonic-buildimage |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Why I did it
SONiC exchanges EVPN MACs with FRR through the Linux kernel bridge FDB: fdbsyncd writes the MACs the ASIC learns into the kernel for zebra to pick up, and copies zebra's remote MACs from the kernel into APPL_DB. That path relies on a kernel patch and an iproute2 patch that tag bridge FDB entries with a protocol, and neither is upstream.
This PR moves MAC synchronization onto the FPM channel, in both directions. It is part 1 of the split proposed in #29312 (comment).
Work item tracking
How I did it
FDB_SYNC|global|mac_sync_mode(kernelorfpm, defaultkernel) selects the path (sonic-yang-models).--kernel-mac-ext-learninfpmmode, and on a device with theL3EvpnMHsubtype, which always syncs MACs over FPM (docker-fpm-frr supervisord).RTM_NEWNEIGH/RTM_DELNEIGHthat fpmsyncd sends for each MAC the ASIC learns or ages, and applies it on zebra's main thread throughzebra_vxlan_local_mac_add_update()andzebra_vxlan_local_mac_del(), the entry points the kernel path uses.RTPROT_HW) and the iproute2 bridge FDB protocol patch.Call flow:
Companion PRs:
Related:
VXLAN_FDB_TABLE; this PR does not change that behaviour.Not in this PR, unlike its earlier revision: the ZMQ northbound (2), L2 nexthop groups over FPM (3), resynchronization after the FPM connection drops (5) and Ethernet Segment parameters from CONFIG_DB (6). The ES backup NHG messages (4) are not touched.
How to verify it
target/debs/trixie/frr_10.5.4-sonic-0_amd64.deb. The FRR unit tests run during the build, includingfpm_mac_test.FDB_SYNCand theL3EvpnMHsubtype.supervisord.conf: zebra gets--kernel-mac-ext-learnonly withFDB_SYNC|global|mac_sync_mode fpmor theL3EvpnMHsubtype.config fdb mac-sync-mode fpm,config save -y, restart bgp. MACs the ASIC learns show as local inshow evpn mac vni <vni>, and remote MACs appear in APPL_DBVXLAN_FDB_TABLE.Which release branch to backport (provide reason below if selected)
None.
Tested branch (Please provide the tested image version)
Description for the changelog
Sync EVPN MACs between SONiC and FRR over the FPM channel (FDB_SYNC mac_sync_mode fpm).
Link to config_db schema for YANG module changes
FDB_SYNC: sonic-net/SONiC#2533