[fpmsyncd/fdbsyncd]: Sync EVPN MACs with FRR over the FPM channel by tahmed-dev · Pull Request #4857 · sonic-net/sonic-swss

tahmed-dev · 2026-09-01T20:58:35Z

Description of PR

Summary:
SONiC exchanges EVPN MACs with FRR through the Linux kernel bridge FDB: fdbsyncd writes the MACs the ASIC learns into the kernel for zebra to pick up, and copies zebra's remote MACs from the kernel into APPL_DB. With FDB_SYNC|global|mac_sync_mode fpm, fpmsyncd exchanges them with zebra over the FPM channel instead, in both directions. The default stays kernel, so nothing changes unless fpm is configured.

This is the SONiC side of part 1 of the split proposed in sonic-net/sonic-buildimage#29312 (comment).

Companion PRs:

[schema]: Add the FDB_SYNC CONFIG_DB table sonic-swss-common#1243 (merges first; adds CFG_FDB_SYNC_TABLE_NAME)
[sonic-frr/sonic-yang-models]: Sync EVPN MACs between SONiC and FRR over the FPM channel sonic-buildimage#29312 (the FRR side and the FDB_SYNC YANG model)
[fdb]: Add CLI to select the MAC synchronization mode sonic-utilities#4833 (config fdb mac-sync-mode)
[patches-sonic]: Drop the bridge FDB protocol field patch sonic-linux-kernel#619 (merges after this PR)

[portsorch]: Do not let an optional DF attribute fail object creation #4841 proposes the portsorch change below on its own.

Type of change

Approach

What is the motivation for this PR?

The kernel path relies on a kernel patch and an iproute2 patch that tag bridge FDB entries with a protocol, and neither is upstream. Over FPM, fpmsyncd and zebra exchange MACs directly.

How did you do it?

fpmsyncd, SONiC to FRR: MacSync follows STATE_DB FDB_TABLE, which fdborch writes, and sends each add or delete to zebra as an AF_BRIDGE RTM_NEWNEIGH/RTM_DELNEIGH, encoded the way zebra encodes its own FDB writes. A delete still reaches zebra when the port is already gone, on the ifindex the MAC was added with.
fpmsyncd, FRR to SONiC: zebra's remote MACs are written to APPL_DB VXLAN_FDB_TABLE with the fields fdbsyncd writes (remote_vtep, nexthop_group, ifname, type, vni). Attribute lengths are checked, and a truncated attribute drops the message.
fdbsyncd steps back in fpm mode: it no longer programs local MACs into the kernel bridge or writes remote MACs to APPL_DB. IMET routes and L2 nexthop groups still come from the kernel.
fdborch stops asking nbrmgrd to resolve the neighbour of an aged-out MAC in fpm mode; zebra owns that decision once the ASIC ages the MAC.
fdbsyncd drops its kernel FDB protocol plumbing. It never worked: the support probe used dev lo, which the kernel rejects.
portsorch sets the optional EVPN-MH DF attribute after the bridge port and VLAN member are created, tolerating NOT_IMPLEMENTED/NOT_SUPPORTED, so a platform without it no longer loses the port from the data plane.
An L3EvpnMH device always syncs MACs over FPM.

Not in this PR, unlike its earlier revision: resynchronization after the FPM connection drops and L2 nexthop groups over FPM.

How did you verify/test it?

make target/debs/trixie/swss_1.0.0_amd64.deb      # mock tests: 0 failures

On an EVPN VXLAN device with the companion PRs:

config fdb mac-sync-mode fpm
config save -y
systemctl restart bgp
vtysh -c "show evpn mac vni <vni>"           # MACs the ASIC learns are local
redis-cli -n 0 keys 'VXLAN_FDB_TABLE:*'      # remote MACs
pgrep -c -f 'bridge fdb'                     # 0 in fpm mode

Tested on:

master 111ab52: package build with the mock tests.
An image of master with this PR and the companion PRs, on a two-leaf EVPN multihoming setup with dual-homed hosts: MACs the ASIC learns, MACs only the Ethernet Segment peer has, ageing, a move between Ethernet Segments and the peer losing a MAC all behave as expected on both leaves.

Any platform specific information?

No. The portsorch change only matters where the platform does not implement the optional DF attributes.

Documentation

MAC over FPM HLD: sonic-net/SONiC#2533

azure-pipelines · 2026-09-01T20:58:45Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

mssonicbld · 2026-09-01T20:58:54Z

/azp run

azure-pipelines · 2026-09-01T20:59:07Z

Azure Pipelines: Successfully started running 1 pipeline(s).

pbrisset · 2026-09-02T17:10:17Z

+ * tell the two apart. */
+#ifndef RTPROT_HW
+#define RTPROT_HW 193


PROT_HW should not longer be used.

Removed. Nothing in this PR uses a protocol, and fdbsyncd's kernel FDB protocol plumbing is dropped as well.

pbrisset · 2026-09-02T17:42:19Z

+    }
+}
+
+void MacSync::sendLocalMac(const string& vlanName, const string& mac, const string& port,


Local MAC delete can be lost if the port disappears before the STATE_FDB_TABLE delete is processed.

For deletes, processStateFdb() takes the cached port name, erases the local MAC from m_localMacs, and then sendLocalMac() resolves the port with if_nametoindex(). If the Linux interface is already gone, if_nametoindex() fails and zebra never receives the local MAC delete. That can leave stale local MAC state in zebra until a later full replay.

Can we cache the ifindex with the local MAC when the add is accepted, or avoid erasing the local cache entry until the delete is successfully sent?

Fixed: MacSync keeps the ifindex each MAC was added with and sends the delete on it when the port no longer resolves (sendLocalMac()).

pbrisset · 2026-09-02T17:43:00Z

+ * here instead. Returns true once NHA_FDB identifies the message as ours, even
+ * when nothing is published, because the route path must never see it.
+ */
+bool MacSync::onFdbNhgMsg(struct nlmsghdr *h, int len)


FDB nexthop groups can be dropped permanently if the group arrives before all members.

When an NHA_GROUP references an unknown member, the group is skipped and not remembered. If FPM replay/order delivers the group before one of its member NHA_FDB nexthops, the L2 NHG will not be published unless zebra later resends the group.

Can we cache pending FDB nexthop groups and rederive/publish them when missing members arrive?

FDB nexthop groups over FPM are no longer in this PR; fdbsyncd keeps publishing L2_NEXTHOP_GROUP_TABLE from the kernel in both modes.

pbrisset · 2026-09-02T17:43:55Z

+    return m_cfgEvpnEsTable.get(ifname, values);
+}
+
+void MacSync::setMacSyncMode(const string& mode)


Remote replay state is armed even when FPM MAC mode is disabled.

onFpmConnected() sets m_remoteReplayPending=true regardless of m_fpmMode. If the FPM connection comes up while mac_sync_mode=kernel and the config later changes to fpm, the remote replay boundary may be stale or associated with the wrong mode transition.

Can we either arm remote replay tracking only when FPM MAC mode is active, or explicitly reset/re-arm the remote replay state when entering fpm mode?

The remote replay is no longer in this PR.

pbrisset · 2026-09-02T17:44:42Z

+ * when nothing is published, because the route path must never see it.
+ */
+bool MacSync::onFdbNhgMsg(struct nlmsghdr *h, int len)
+{


Raw FPM netlink attribute parsing should validate payload sizes before dereferencing.

The new raw MAC/NHG path directly reads attributes such as NHA_ID, NHA_GATEWAY, NDA_VLAN, NDA_NH_ID, NDA_SRC_VNI, and NDA_VNI without first checking RTA_PAYLOAD() is large enough for the expected type. A malformed or mismatched FPM message could crash fpmsyncd.

Can we add small helpers for typed attribute reads and drop/log malformed messages instead of dereferencing them directly?

Done: fixed-size attributes are read through getAttr(), which rejects a short payload, and a truncated attribute or an NDA_LLADDR that is not 6 bytes drops the message. The NHA_* parsing is no longer in this PR.

pbrisset · 2026-09-02T17:46:00Z

+ *
+ * The cases here are deliberately shaped around defects that a build cannot
+ * catch: every one of them produces a plausible-looking but wrong
+ * APP_VXLAN_FDB_TABLE entry rather than a crash or a link error.


Please add coverage for FPM connect in kernel mode followed by runtime switch to fpm mode.

This should verify that remote replay state, adopted remote MACs, local replay, and the replay-end marker are initialized from the fpm-mode transition rather than stale state from a connection that was established while mac_sync_mode=kernel.

In this PR fpmsyncd keeps no replay state across a mode change. zebra reads --kernel-mac-ext-learn when the bgp container starts, so a mode change takes effect with the next bgp restart: fpmsyncd starts in the new mode and sends the MACs already in STATE_DB through the subscriber's initial read.

pbrisset · 2026-09-02T17:46:13Z

+ * The cases here are deliberately shaped around defects that a build cannot
+ * catch: every one of them produces a plausible-looking but wrong
+ * APP_VXLAN_FDB_TABLE entry rather than a crash or a link error.
+ */


Please add malformed/truncated FPM message tests.

The raw parser should drop/log short attributes without crashing or writing DB state. Useful cases include short NDA_LLADDR, NDA_VLAN, NDA_NH_ID, NDA_SRC_VNI/NDA_VNI, NHA_ID, and NHA_GATEWAY payloads.

Added for the remote path: RemoteMacWithTruncatedAttributeIsDropped, RemoteMacWithShortLladdrIsDropped, RemoteMacWithoutVlanIsRejected and RemoteMacWithoutVtepIsRejected. The NHA_* parsing is no longer in this PR.

pbrisset · 2026-09-02T17:51:36Z

+    /* A MAC behind an Ethernet Segment reaches several VTEPs, so zebra resolves
+     * it to an L2 nexthop group and sends NDA_NH_ID with no NDA_DST. The group
+     * itself is published to L2_NEXTHOP_GROUP_TABLE by fdbsyncd, which keeps
+     * reading it from the kernel in either mac_sync_mode. */


The comment says fdbsyncd publishes L2_NEXTHOP_GROUP_TABLE in either mac_sync_mode, but this PR makes fdbsyncd skip onMsgNhg() in fpm mode and MacSync publishes the table instead.

Can we update the comment here to avoid misleading future readers? In fpm mode, fpmsyncd/MacSync owns FDB nexthop publication; in kernel mode, fdbsyncd owns it.

In this PR the comment holds: fdbsyncd keeps publishing L2_NEXTHOP_GROUP_TABLE in both modes, since the nexthop groups over FPM are no longer in this PR.

pbrisset · 2026-09-02T17:52:28Z

+    }
+}
+
+void MacSync::onFpmConnected(FpmInterface& fpm)


[P3] onFpmConnected() allocates/persists a new generation even when mac_sync_mode is not fpm.

The send paths are gated later, but nextGeneration() still updates STATE_DB on every FPM reconnect in kernel mode. It would be cleaner to allocate a generation only when fpmsyncd is actually taking local MAC ownership over FPM, either on connect while m_fpmMode is true or when transitioning into fpm mode on an existing connection.

The generation is no longer in this PR.

pbrisset · 2026-09-02T17:53:06Z

 }

-bool FdbSync::checkFdbProtoSupport()
+void FdbSync::setMacSyncMode(const string& mode)


Runtime transition into fpm mode reuses whatever m_generation was last set to.

If the FPM connection was already established while mac_sync_mode=kernel, setMacSyncMode("fpm") replays local MACs using the generation allocated at connection time. That may be fine, but it makes the generation lifecycle less obvious. Can we either allocate the generation when entering fpm mode, or add a comment/test documenting that the connection-time generation is intentionally reused?

fdbsyncd has no generation in this PR.

mssonicbld · 2026-09-09T19:34:37Z

/azp run

azure-pipelines · 2026-09-09T19:34:49Z

Azure Pipelines: Successfully started running 1 pipeline(s).

EVPN multihoming marks the DF role by appending SAI_BRIDGE_PORT_ATTR_EGRESS_FILTERING to create_bridge_port() and SAI_VLAN_MEMBER_ATTR_TUNNEL_TERM_BUM_TX_DROP to create_vlan_member(). Both stand in for an attribute SAI does not define yet, and on a platform that does not implement them the whole create fails with SAI_STATUS_ATTR_NOT_IMPLEMENTED. That leaves an Ethernet Segment port without a bridge port, so it never becomes a VLAN member, hardware learning never starts and the FDB stays empty. Configuring an Ethernet Segment silently removes the port from the data plane. Create the object with the attributes every platform implements, then set the DF role separately, tolerating NOT_IMPLEMENTED and NOT_SUPPORTED as vlanMembersApplyNonDF() already does. Where the attribute is unavailable the port forwards normally and only non-DF BUM suppression is lost. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>

FDB_SYNC|global|mac_sync_mode selects how MACs are synchronized with FRR. In fpm mode fpmsyncd exchanges them with zebra over the FPM channel, so fdbsyncd steps back from both directions: - Local MACs: STATE_DB FDB entries are no longer programmed into the kernel bridge, so no bridge command is run for them (updateLocalMac(), addLocalMac(), macRefreshStateDB()). - Remote MACs: kernel notifications for unicast remote MACs are no longer written to APPL_DB VXLAN_FDB_TABLE. IMET routes and L2 nexthop groups still come from the kernel. - Warm restart: VXLAN_FDB_TABLE is not registered for reconciliation, which deletes every entry left stale, because fpmsyncd produces the table in this mode. fdbsyncd reads the mode at startup and follows it through a CONFIG_DB subscriber. An L3EvpnMH device always syncs MACs over FPM. FdbOrch also stops asking nbrmgrd to resolve the neighbour of an aged-out MAC in fpm mode. zebra owns that decision once the ASIC ages the MAC, and the resolve can make the ASIC learn the MAC again from the still-programmed neighbour without any host traffic, which keeps a departed host alive in EVPN. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>

The protocol tag on a kernel bridge FDB entry is going away: sonic-linux-kernel#619 drops the kernel patch that stored it, sonic-buildimage#29312 drops the iproute2 patch that spelled it, and where a MAC came from travels over the FPM channel instead. None of this could work anyway. checkFdbProtoSupport() probed with "dev lo", which the kernel rejects outright because loopback is not an Ethernet device, so m_isFdbProtoSupported was always false and every proto_string was always empty. Its RTPROT_HW was also 193, the value FRR uses for LDP. The "protocol" field fdbsyncd added to VXLAN_FDB_TABLE goes too; FdbOrch never read it. The MCLAG paths lose their protocol arguments as well. That plumbing was added to them by the EVPN-MH integration, not by MCLAG, so removing it restores their original behaviour rather than changing it. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>

In FDB_SYNC|global|mac_sync_mode fpm, fpmsyncd replaces fdbsyncd as the path that tells zebra about the MACs the ASIC learns. MacSync follows STATE_DB FDB_TABLE, which FdbOrch writes, and sends each add or delete to zebra as an AF_BRIDGE RTM_NEWNEIGH or RTM_DELNEIGH, encoded the way zebra encodes its own FDB writes: ndm_ifindex bridge port ndm_flags NTF_MASTER | NTF_EXT_LEARNED, plus NTF_STICKY for a static MAC ndm_state NUD_REACHABLE, plus NUD_NOARP for a static MAC NDA_LLADDR MAC NDA_VLAN VLAN Only a static MAC, which configuration pins to a port, is sticky; a learnt MAC stays mobile. A delete still reaches zebra when the port is already gone, on the ifindex the MAC was added with. MacSync reads the mode at startup and follows it through a CONFIG_DB subscriber; an L3EvpnMH device always runs in fpm mode. In kernel mode it does nothing. MACs already in STATE_DB when fpmsyncd starts reach zebra through the subscriber's initial read. Resynchronizing after zebra reconnects is not part of this change. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>

In fpm mode zebra's remote MACs reach fpmsyncd over the FPM channel, instead of fdbsyncd reading them from the kernel. FpmLink hands AF_BRIDGE RTM_NEWNEIGH and RTM_DELNEIGH to RouteSync raw, and RouteSync passes them to MacSync, which writes APPL_DB VXLAN_FDB_TABLE with the fields fdbsyncd writes: remote_vtep MAC behind a remote VTEP (NDA_DST) nexthop_group MAC behind a remote Ethernet Segment (NDA_NH_ID); fdbsyncd still publishes the group itself ifname MAC on a local Ethernet Segment port, which zebra sends with neither type static for NUD_NOARP, dynamic otherwise vni NDA_SRC_VNI zebra sends each remote MAC twice, and the bridge-side copy, which has no destination, is skipped. The VLAN comes from the VxLAN device name, as in fdbsyncd. An add carrying NUD_INCOMPLETE or NUD_FAILED is a removal, and only MACs MacSync wrote are removed. A truncated attribute drops the message. Signed-off-by: Tamer Ahmed <tamerahmed@microsoft.com>

mssonicbld · 2026-10-02T00:51:54Z

/azp run

azure-pipelines · 2026-10-02T00:52:06Z

Azure Pipelines: Successfully started running 1 pipeline(s).

pbrisset suggested changes Sep 2, 2026

View reviewed changes

tahmed-dev force-pushed the tahmed/fpm-mac-learning-master branch from 4e6ed17 to fe4745b Compare September 9, 2026 19:34

tahmed-dev mentioned this pull request Oct 1, 2026

[sonic-frr/sonic-yang-models]: Sync EVPN MACs between SONiC and FRR over the FPM channel sonic-net/sonic-buildimage#29312

Draft

tahmed-dev added 4 commits October 1, 2026 14:38

tahmed-dev force-pushed the tahmed/fpm-mac-learning-master branch from fe4745b to b8b3435 Compare October 2, 2026 00:51

tahmed-dev changed the title ~~[fpmsyncd/fdbsyncd]: Add FDB synchronization over the FPM channel~~ [fpmsyncd/fdbsyncd]: Sync EVPN MACs with FRR over the FPM channel Oct 2, 2026

Conversation

tahmed-dev commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Documentation

Uh oh!

azure-pipelines Bot commented Sep 1, 2026

Uh oh!

mssonicbld commented Sep 1, 2026

Uh oh!

azure-pipelines Bot commented Sep 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Sep 9, 2026

Uh oh!

azure-pipelines Bot commented Sep 9, 2026

Uh oh!

mssonicbld commented Oct 2, 2026

Uh oh!

azure-pipelines Bot commented Oct 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tahmed-dev commented Sep 1, 2026 •

edited

Loading