Skip to content

[sonic-frr/sonic-yang-models]: Sync EVPN MACs between SONiC and FRR over the FPM channel - #29312

Draft
tahmed-dev wants to merge 7 commits into
sonic-net:masterfrom
tahmed-dev:tahmed/fpm-mac-learning-master
Draft

tahmed-dev wants to merge 7 commits into
sonic-net:masterfrom
tahmed-dev:tahmed/fpm-mac-learning-master

Conversation

@tahmed-dev

@tahmed-dev tahmed-dev commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Why I did it

SONiC exchanges EVPN MACs with FRR through the Linux kernel bridge FDB: fdbsyncd writes the MACs the ASIC learns into the kernel for zebra to pick up, and copies zebra's remote MACs from the kernel into APPL_DB. That path relies on a kernel patch and an iproute2 patch that tag bridge FDB entries with a protocol, and neither is upstream.

This PR moves MAC synchronization onto the FPM channel, in both directions. It is part 1 of the split proposed in #29312 (comment).

Work item tracking
  • Microsoft ADO (number only): 38937153

How I did it

  • FDB_SYNC|global|mac_sync_mode (kernel or fpm, default kernel) selects the path (sonic-yang-models).
  • zebra starts with --kernel-mac-ext-learn in fpm mode, and on a device with the L3EvpnMH subtype, which always syncs MACs over FPM (docker-fpm-frr supervisord).
  • dplane_fpm_sonic, SONiC to FRR: decodes the AF_BRIDGE RTM_NEWNEIGH/RTM_DELNEIGH that fpmsyncd sends for each MAC the ASIC learns or ages, and applies it on zebra's main thread through zebra_vxlan_local_mac_add_update() and zebra_vxlan_local_mac_del(), the entry points the kernel path uses.
    • Writes the kernel bridge FDB entry of such a MAC through the dataplane, so it is ordered with zebra's own writes, and leaves the entry to zebra while an Ethernet Segment peer still holds the MAC.
    • Reports a MAC the ASIC still has back to zebra when zebra flushes it or holds it local-inactive.
  • dplane_fpm_sonic, FRR to SONiC: remote MACs go to fpmsyncd as before; local MACs are no longer echoed back.
  • FRR patch 0127 lets a dataplane module tell zebra that it learns local MACs. zebra then ignores kernel FDB notifications for local bridge ports, and a peer-active hold timer expiry flushes only a local-inactive MAC.
  • FRR patches 0118 and 0128 build the decode into the module, unit test it with CUnit, and run the FRR unit tests during the package build. 0118 is byte-identical to the one in [sonic-frr]: Fix EVPN multihoming MACs that are never withdrawn after a host goes quiet #29593.
  • Drops FRR patch 0029 (RTPROT_HW) and the iproute2 bridge FDB protocol patch.

Call flow:

SONiC to FRR, local MACs
  ASIC -> fdborch -> STATE_DB FDB_TABLE -> fpmsyncd
    -> FPM: AF_BRIDGE RTM_NEWNEIGH / RTM_DELNEIGH
    -> fpm_read() -> fpm_mac_decode()                          [FPM thread]
    -> fpm_apply_local_mac()                                   [zebra main thread]
         add: dplane_local_mac_add(), then zebra_vxlan_local_mac_add_update()
         del: zebra_vxlan_local_mac_del(), then dplane_local_mac_del()
              unless an Ethernet Segment peer still holds the MAC

FRR to SONiC, remote MACs
  BGP EVPN -> zebra -> DPLANE_OP_MAC_INSTALL / DELETE (DPLANE_MAC_REMOTE)
    -> fpm_nl_enqueue() -> FPM -> fpmsyncd -> APPL_DB VXLAN_FDB_TABLE -> fdborch

A MAC the ASIC still has, given back to zebra
  zebra writes or removes a local MAC's kernel entry
    -> fpm_nl_process() -> fpm_local_mac_recheck()             [zebra main thread]
    -> fpm_local_mac_learnt() if zebra flushed the MAC or holds it local-inactive

Companion PRs:

Related:

Not in this PR, unlike its earlier revision: the ZMQ northbound (2), L2 nexthop groups over FPM (3), resynchronization after the FPM connection drops (5) and Ethernet Segment parameters from CONFIG_DB (6). The ES backup NHG messages (4) are not touched.

How to verify it

  • Build target/debs/trixie/frr_10.5.4-sonic-0_amd64.deb. The FRR unit tests run during the build, including fpm_mac_test.
  • Build the sonic-yang-models wheel; its tests cover FDB_SYNC and the L3EvpnMH subtype.
  • Render docker-fpm-frr supervisord.conf: zebra gets --kernel-mac-ext-learn only with FDB_SYNC|global|mac_sync_mode fpm or the L3EvpnMH subtype.
  • With the companion PRs on an EVPN VXLAN device: config fdb mac-sync-mode fpm, config save -y, restart bgp. MACs the ASIC learns show as local in show evpn mac vni <vni>, and remote MACs appear in APPL_DB VXLAN_FDB_TABLE.

Which release branch to backport (provide reason below if selected)

None.

Tested branch (Please provide the tested image version)

  • master 37273ef: package and wheel builds with their unit tests (FRR 445 passed, sonic-yang-models 930 passed).
  • An image of master with this PR and the companion PRs, with the Ethernet Segments configured in FRR, on a two-leaf EVPN multihoming setup with dual-homed hosts: MACs the ASIC learns, MACs only the Ethernet Segment peer has, ageing, a move between Ethernet Segments and the peer losing a MAC all behave as expected on both leaves.

Description for the changelog

Sync EVPN MACs between SONiC and FRR over the FPM channel (FDB_SYNC mac_sync_mode fpm).

Link to config_db schema for YANG module changes

FDB_SYNC: sonic-net/SONiC#2533

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@tahmed-dev
tahmed-dev force-pushed the tahmed/fpm-mac-learning-master branch from 2199158 to 2f1487b Compare September 9, 2026 19:34
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

{% set _dep = use_dependent_startup | default(false) %}
{% set _wait = frr_wait_for | default("rsyslogd:running") %}
{# EVPN-MH does hardware MAC learning and aging, which zebra only supports in
external-learn mode. L3EvpnMH needs it too, since RT-2 still carries MACs;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comments about L3evpnMH is not required.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment now only says why the flag is set: MAC sync over FPM needs zebra in external-learn mode. It names L3EvpnMH once, because that subtype always syncs over FPM.

{# EVPN-MH does hardware MAC learning and aging, which zebra only supports in
external-learn mode. L3EvpnMH needs it too, since RT-2 still carries MACs;
non-MH devices keep kernel MAC aging and remote neighbour install. #}
{% set _mac_ext_learn = DEVICE_METADATA is defined and DEVICE_METADATA.localhost.subtype is defined and DEVICE_METADATA.localhost.subtype in ["L3EvpnMH", "EvpnMH"] %}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we keep only evpnMH... and in theory, this need capability should also work for EVPN SH too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flag now follows FDB_SYNC|global|mac_sync_mode fpm, so single-homed EVPN gets it too. L3EvpnMH remains only as the subtype that always syncs over FPM.

!
{% endfor %}
{% endif %}
{% if VRF is defined %}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why did you added this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed. This PR no longer touches zebra.interfaces.conf.j2; that was the VRF VNI binding, not MAC sync.

ROUTE_TABLE
LABEL_ROUTE_TABLE
{% endif %}
{% if DEVICE_METADATA.localhost.orch_northbond_evpn_mh_zmq_enabled != "false" %}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be a different PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Removed from this PR.

Comment thread rules/frr.mk Outdated
FRR = frr_$(FRR_VERSION)-sonic-$(FRR_SUBVERSION)_$(CONFIGURED_ARCH).deb
$(FRR)_DEPENDS += $(LIBSNMP_DEV) $(LIBYANG3_DEV) $(LIBFIB_DEV)
$(FRR)_RDEPENDS += $(LIBYANG3) $(LIBFIB)
$(FRR)_DEPENDS += $(LIBSNMP_DEV) $(LIBYANG3_DEV) $(LIBFIB_DEV) $(LIBSWSSCOMMON_DEV)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does FRR need from SWSS_common?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing now. It was for the APP_DB publisher, which is no longer in this PR, so rules/frr.mk is not touched any more.

int ifindex;
uint8_t mac[ETH_ALEN];
uint16_t vid;
uint32_t generation;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add comments describing each elements

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: struct fpm_local_mac documents the fields whose meaning is not obvious (ifindex, vid, del, sticky), and the comment above fpm_mac_decode() has a table of what fpmsyncd sends.

uint32_t generation;
uint8_t protocol;
bool del;
bool sticky;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to know the difference between sync and sticky?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. The struct has no sync field any more; only sticky remains, read from NTF_STICKY.

/*
* Decode an AF_BRIDGE RTM_NEWNEIGH/RTM_DELNEIGH message. Returns false and
* leaves *out untouched when the message is not a usable local MAC.
*/

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume this function is called for any message coming from fdborch

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes: fpmsyncd follows STATE_DB FDB_TABLE, which fdborch writes, and sends one message for each add or delete; this decodes each of them. The comment now says so.

/* RMAC walk finished. */
FNE_RMAC_FINISHED,
/* Remote MAC walk finished. */
FNE_MAC_FINISHED,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why extending fpm_nl struct?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not extended any more. The remote MAC walk event is no longer in this PR, which adds no fields to struct fpm_nl_ctx.

}

/* Nothing is feeding local MACs while the channel is down, so zebra has
* to own them again until fpmsyncd replays and takes them back.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you mean by owning by zebra?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That comment went with the ownership latch, which this PR no longer has.


memcpy(mac.octet, flm->mac, ETH_ALEN);

ifp = if_lookup_by_index(flm->ifindex, VRF_DEFAULT);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the check about?
why are you excluding VRF and not default?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lookup no longer involves a VRF. fpm_mac_bridge_port() finds the port by ifindex in the namespace, as zebra's kernel FDB path does, since a bridge port is not tied to a VRF, and requires it to be a bridge slave.

* Runs on zebra's main thread: the interface table and the EVPN MAC hashes are
* owned by it and are not safe to touch from the FPM thread.
*/
static void fpm_apply_local_mac(struct event *t)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this downloading MAC to sonic?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the other way: this is SONiC to FRR, a MAC the ASIC learnt or aged, from fpmsyncd. FRR to SONiC is fpm_nl_enqueue(), remote MACs only. Both are now labelled.

if (flm->del) {
zebra_vxlan_local_mac_del(ifp, zif->brslave_info.br_if,
&mac, flm->vid);
if (kernel_del_local_mac(ifp, flm->vid, &mac) < 0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why kernel delete? Where is the FSM per MAC maintaining the multi-sources?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kernel write is there because the ASIC learnt the MAC and the kernel never saw it. zebra's MAC is the per-MAC state machine: its flags hold both sources, the ASIC (ZEBRA_MAC_LOCAL) and an Ethernet Segment peer's route (ZEBRA_MAC_ES_PEER_ACTIVE/PROXY). When the ASIC loses the MAC, fpm_local_mac_gone() tells zebra first. If the peer still holds the MAC, zebra keeps it local-inactive and rewrites the entry itself, and the module leaves it; otherwise the module removes the entry it wrote.

* to happen before zebra sees the MAC: a MAC on an Ethernet Segment
* makes zebra queue its own sync install marking the entry static, and
* that runs on the dplane thread. Seeding afterwards would race it and
* could downgrade a static entry back to extern_learn.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dont you need to maintain 2 sources per MAC? control plane and data plane?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and zebra's MAC flags already hold both: ZEBRA_MAC_LOCAL for the data plane and ZEBRA_MAC_ES_PEER_ACTIVE/PROXY for the control plane. The module follows them rather than keeping a copy; see the reply on the delete above.

* arrive instead leaves a window at boot where zebra's Ethernet Segment
* sync writes the kernel entry before the ASIC has learnt anything.
*
* Generation 0 is the opposite statement: fpmsyncd left fpm mode and no

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is that generation 0 or not logic explained? ;(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generation and the replay end marker that carried it are no longer in this PR.

* @return Matching ZMQ producer table, or NULL when unsupported.
*/
static SWSSZmqProducerStateTable
sonic_frr_redis_zmq_table(struct sonic_frr_redis_ctx *ctx, const char *table)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this again? zmq on NB api? Why? separate PR again?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the ZMQ northbound. Removed from this PR.

#define RTPROT_EIGRP 192 /* EIGRP Routes */
-
+#define RTPROT_HW 193 /* Hardware Routes */
+#define RTPROT_HW 199 /* Hardware Routes */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: this PR drops patch 0029.

These message types are consumed by fpmsyncd in SONiC to populate
EVPN_SPLIT_HORIZON_TABLE, EVPN_DF_TABLE, and EVPN_ES_BACKUP_NHG_TABLE
in APPL_DB.
EVPN_SPLIT_HORIZON_TABLE and EVPN_DF_TABLE in APPL_DB.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keep this one EVPN_ES_BACKUP_NHG_TABLE. Why do you remove this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept. This PR no longer touches patch 0077.

+ RTM_FPM_DEL_EVPN_ES_BACKUP_NHG,
+ RTM_FPM_LAST = RTM_FPM_DEL_EVPN_ES_BACKUP_NHG,
+ /*
+ * End of a MAC replay generation. nlmsg_seq carries the generation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keep them! RTM_FPM_ADD_EVPN_ES_BACKUP_NHG, RTM_FPM_DEL_EVPN_ES_BACKUP_NHG,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept; patch 0077 is unchanged.

leaf subtype {
type string {
pattern "DualToR|SmartSwitch|Supervisor|UpstreamLC|DownstreamLC";
pattern "DualToR|SmartSwitch|Supervisor|UpstreamLC|DownstreamLC|L3EvpnMH|EvpnMH";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, why L3MH or evpnMH reference?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EvpnMH is dropped; L3EvpnMH is kept on purpose. An L3EvpnMH device always syncs MACs over FPM, so docker-fpm-frr, fdbsyncd and fpmsyncd select fpm from the subtype whatever FDB_SYNC says. subtype is pattern constrained, so the value has to be listed here or config reload rejects it. DualToR could not be reused: it also starts ycabled and the mux tunnel packet handler.

@pbrisset

Copy link
Copy Markdown
Contributor

This PR is a huge bag of many different functionalities that should have been done in separate PRs.

1- MAC over fpm (bi-directional)
2- ZMQ NB
3- L2_NEXTHOP_GROUP over fpm
4- fpm ES_BACKUP msg removal (disagreeing with that)
5- reconcilation between FRRouting and SONiC
6- ES parameters config db (push from SONIC)

Also, FRRouting is missing a FSM per MAC evaluating its ownership.

Comments for each directional function must be clearly state. (FRR->SONIC or SONIC->FRR)

This work is highly confusing and will be very difficult to maintain / progress without proper cleanup.

@pbrisset

Copy link
Copy Markdown
Contributor

[P1] FPM ownership is enabled too early
fpm_apply_local_mac() sets fpm_owns_local_macs = true on the first received MAC, while the replay-end handler correctly treats the marker as the ownership boundary. Once the first MAC arrives, the pre-kernel provider suppresses all subsequent local-MAC installs/deletes, including entries not yet replayed. A MAC added or removed during replay can therefore be missing or stale in the kernel FDB.
Keep ownership disabled until the replay marker, or track ownership per MAC.

[P1] Bridge-port deletes can target VlanX:unknown
sonic_frr_redis_br_port_key() reconstructs the interface name using if_lookup_by_index() and substitutes "unknown" when lookup fails. During interface/port teardown, the interface may already be gone when the queued delete is processed, so the code deletes Vlan100:unknown instead of the original key. The SHL/DF entries then remain stale in APP_DB.
Use the interface name already snapshot in the dataplane context rather than looking it up again.

[P2] PortChannel system_mac changes are not propagated
After an EVPN segment is configured, self.es_cfg caches the generated commands. Later PORTCHANNEL updates only retry set_queue; already-configured segments are not reprocessed. Changing system_mac therefore leaves FRR using the old evpn mh es-sys-mac.
Reapply affected configured segments whenever their PortChannel dependency changes.

@tahmed-dev
tahmed-dev force-pushed the tahmed/fpm-mac-learning-master branch from 2f1487b to 54f52fa Compare September 30, 2026 11:50
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@tahmed-dev
tahmed-dev force-pushed the tahmed/fpm-mac-learning-master branch from 54f52fa to 9e01580 Compare October 1, 2026 20:46
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@tahmed-dev tahmed-dev changed the title [sonic-frr/bgpcfgd]: Add EVPN MAC synchronization over the FPM channel [sonic-frr/sonic-yang-models]: Sync EVPN MACs between SONiC and FRR over the FPM channel Oct 1, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@tahmed-dev

Copy link
Copy Markdown
Contributor Author

Re #29312 (comment): done. This PR now does only MAC sync over FPM in both directions (1), and the companion PRs are cut the same way. Not in this PR, unlike its earlier revision: the ZMQ northbound (2), L2 nexthop groups over FPM (3), resynchronization after the FPM connection drops (5) and Ethernet Segment parameters from CONFIG_DB (6).

For 4, the ES backup NHG messages are no longer removed: patch 0077 is untouched.

Per-MAC ownership: zebra's MAC already holds both sources, the ASIC (ZEBRA_MAC_LOCAL) and an Ethernet Segment peer's route (ZEBRA_MAC_ES_PEER_ACTIVE/PROXY). The module follows those flags instead of keeping a second state machine; it only remembers which MACs the ASIC reported.

Each MAC path in dplane_fpm_sonic.c now names its direction, and the description has the call flow for both.

@tahmed-dev

Copy link
Copy Markdown
Contributor Author

Re #29312 (comment):

[P1] FPM ownership: the fpm_owns_local_macs latch and the pre-kernel provider are gone, so nothing depends on when the first MAC arrives. For each MAC the ASIC learns, the module writes the kernel entry through dplane_local_mac_add(), queued ahead of any write zebra makes for that MAC. When the ASIC loses the MAC, the module removes the entry unless an Ethernet Segment peer still holds the MAC.

[P1] VlanX:unknown: agreed. sonic_frr_redis_interface.c is no longer touched by this PR.

[P2] system_mac: agreed. The Ethernet Segment manager is no longer in this PR.

This taught the bridge command to emit and print NDA_PROTOCOL on an FDB entry.
The kernel side of that is the sonic-linux-kernel protocol patch, which
sonic-linux-kernel#619 removes. Without it the kernel discards the attribute
on add and never returns it on dump.

Nothing consumes it either. Its only user is fdbsyncd, and the support probe
behind that use can never succeed because it tests "dev lo", and loopback is
not an Ethernet device. sonic-swss#4857 removes that plumbing.

Where a MAC came from travels over the FPM channel instead, so the origin
marker no longer needs a kernel or iproute2 carrier.

Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
Patch 0029 adds RTPROT_HW (193) to FRR's copy of linux/rtnetlink.h, to tag the
kernel bridge FDB entries of MACs the hardware learnt. Nothing in FRR uses it:
no other patch and no source file refers to it.

The tag relied on kernel and iproute2 changes that are not upstream, and SONiC
drops its copies of both: sonic-linux-kernel#619, and the iproute2 patch in the
previous commit. Bridge FDB entries carry no protocol, and where a MAC came
from travels over the FPM channel instead.

193 is also the value FRR already uses for RTPROT_LDP (zebra/rt_netlink.h).

Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
FDB_SYNC|global|mac_sync_mode selects how MAC state is synchronized between
SONiC and FRR:

  kernel  fdbsyncd synchronizes through the Linux bridge FDB, as today.
  fpm     fpmsyncd exchanges MACs with zebra over the FPM channel.

It defaults to kernel, so nothing changes unless fpm is configured.

Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
dh_auto_test runs "make test", which only builds the pceplib helpers and runs
nothing, so the FRR unit test suite never executed during the SONiC package
build. Override it with tests/tests.xml, which builds every check program and
runs the suite. A plain "make check" is not usable here: pceplib's
*_valgrind.sh wrappers reference a relative path that does not resolve in an
out-of-tree build and always fail with exit 127.

The C unit tests added by the following patches link against CUnit, so add
libcunit1-dev to Build-Depends. It is restricted to builds without the
nocheck profile, which cross builds use and which skips the tests.

This is SONiC-only build plumbing.

Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
Add FRR patch 0127.

In fpm mode the ASIC learns local MACs, dplane_fpm_sonic hands them to zebra
over the FPM channel, and zebra writes their kernel bridge FDB entries itself,
including the entry of a MAC only an Ethernet Segment peer has, which zebra
holds local-inactive.

The kernel reports each of those writes back. In mac-ext-learn mode zebra
tells its own writes apart only by NDA_PROTOCOL, which the kernel does not keep
for bridge FDB entries, so the report arrives as a local learn: the inactive
MAC turns active, is advertised as learnt here, and stays after the peer
withdraws it.

The patch lets a dataplane module tell zebra that it, not the kernel, learns
and ages local MACs. While it does, zebra ignores kernel FDB notifications for
local bridge ports, adds and deletes alike; VXLAN ports are not affected. zebra
also flushes a MAC whose peer-active hold timer expires only if it is
local-inactive. An active MAC is still in the ASIC, so flushing it would drop
the host's MAC on a leaf whose Ethernet Segment peer stopped seeing the host,
or rebooted.

Nothing sets it yet; dplane_fpm_sonic does in the next commit.

Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
dplane_fpm_sonic exchanges MACs with fpmsyncd over the FPM channel, in both
directions, for FDB_SYNC|global|mac_sync_mode fpm.

SONiC to FRR: fpmsyncd sends each MAC the ASIC learns or ages as an AF_BRIDGE
RTM_NEWNEIGH or RTM_DELNEIGH. The module decodes it on the FPM thread and
applies it on zebra's main thread, which owns the interface table and the EVPN
MAC hashes, through zebra_vxlan_local_mac_add_update() and
zebra_vxlan_local_mac_del(). Those are the entry points the kernel path uses,
so the EVPN state machine and its multihoming handling are unchanged.

FRR to SONiC: remote MACs, which zebra learnt from BGP EVPN, already reach
fpmsyncd as DPLANE_OP_MAC_INSTALL and DPLANE_OP_MAC_DELETE. Local MACs are no
longer echoed back. They originate from fpmsyncd, and the encoder would also
emit a 16 byte NDA_DST from the never-initialised vtep of
dplane_local_mac_add(), which fpmsyncd would read as a remote VTEP.

The kernel never saw a MAC the ASIC learnt, so the module writes its bridge
FDB entry. It does so through dplane_local_mac_add() and
dplane_local_mac_del(), so that the write shares one queue with zebra's own,
and queues its add before handing the MAC to zebra, so zebra's write lands
last. zebra's MAC flags hold both sources of a local MAC, the ASIC and an
Ethernet Segment peer's route. When the ASIC loses a MAC the peer still
advertises, zebra keeps it local-inactive and rewrites the entry itself, so
the module leaves the entry alone; it removes the entry only once zebra no
longer holds the MAC. In mac-ext-learn mode the module tells zebra at startup
that it learns local MACs (FRR patch 0127), so zebra does not take the
kernel's report of these writes for a local learn.

zebra also flushes a local MAC when it expects it to be relearnt, as when an
Ethernet Segment is configured on or removed from its port, and holds a MAC
local-inactive until the dataplane reports activity. Over FPM nothing reports
a MAC the ASIC keeps again. The module keeps a record of the MACs fpmsyncd
reports, and when zebra removes the kernel entry of a local MAC the record
still holds, or writes the entry of one it holds local-inactive, the module
reports the MAC to zebra again as learnt. A MAC a remote route took over is
left to zebra.

The decode lives in fpm_mac.c, which depends only on the kernel netlink
headers, so a CUnit test (FRR patch 0128) exercises it directly. It checks the
message family and the attribute lengths, and writes the caller's struct only
once the message is fully validated.

Resynchronizing MACs after the FPM connection drops is not part of this
change.

Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
…g when MACs sync over FPM

MAC synchronization over FPM relies on the hardware doing MAC learning and
aging, which zebra only supports when it is started with
--kernel-mac-ext-learn. Nothing in SONiC ever passed that flag, so zebra kept
driving the kernel MAC learning and aging state machine itself.

Pass the flag when FDB_SYNC|global selects mac_sync_mode fpm. This does not
depend on multihoming: a single-homed EVPN device syncing MACs over FPM needs
it as much as a multihomed one. Every device that syncs MACs through the
kernel keeps kernel MAC aging and remote neighbour install as before.

An L3EvpnMH device always syncs MACs over FPM, so it gets the flag whatever
FDB_SYNC says. Add the L3EvpnMH device subtype for that. subtype is pattern
constrained in YANG, so the new value has to be added there or config reload
rejects it. The existing DualToR subtype could not be reused: it also starts
ycabled and the mux tunnel packet handler, which have no meaning on an EVPN
multihoming leaf.

zebra reads the flag when the bgp container starts, so a mode change reaches
zebra with the next restart of that container.

Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>
@tahmed-dev
tahmed-dev force-pushed the tahmed/fpm-mac-learning-master branch from 9e01580 to d84ac13 Compare October 2, 2026 00:47
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run Azure.sonic-buildimage

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants