accept/accept4 events may report the tuple of an unrelated socket

[leogr]

Describe the bug

Related: falcosecurity/falco#3620

That issue has been open for a while without a root cause, so I've spent some time narrowing it down. I'm opening this here since the evidence points to the driver.

TL;DR: the tuple is not reversed. The packing is correct in both drivers. The problem is that the socket we read to build the tuple is not the socket accept() returned.

Every sample in the original report has a well-known service port on the client side and an ephemeral port on the server side. For instance:

"evt.type": "accept",
"evt.rawres": 8,
"fd.is_server": true,
"fd.name": "172.30.222.222:9090->172.30.216.244:37610",
"fd.cip": "172.30.222.222", "fd.cport": 9090,
"fd.sip": "172.30.216.244", "fd.sport": 37610

where 172.30.216.244 is the prometheus-1 pod and 172.30.222.222 is the zookeeper-4 pod.

Now, for INBOUND we push remote first and local second 👇

https://github.com/falcosecurity/libs/blob/0.25.4/driver/modern_bpf/helpers/store/auxmap_store_params.h#L925-L935

and parse_accept_exit() maps the first pair to fd.cip/fd.cport and the second one to fd.sip/fd.sport 👇

• https://github.com/falcosecurity/libs/blob/0.25.4/userspace/libsinsp/parsers.cpp#L2666-L2674
• https://github.com/falcosecurity/libs/blob/0.25.4/userspace/libsinsp/sinsp_filtercheck_fd.cpp#L600
• https://github.com/falcosecurity/libs/blob/0.25.4/userspace/libsinsp/sinsp_filtercheck_fd.cpp#L877

So the socket we read had local = 172.30.216.244:37610 and remote = 172.30.222.222:9090, which is a client socket of prometheus-1 scraping zookeeper-4:9090, not an accepted one. The same holds for all the other samples in that issue (kubelet included).

In other words, the packing is fine. The question is which fd we resolve.

Both drivers resolve the fd at syscall exit, against the live fd table, when the syscall has already returned:

• modern probe: extract__file_struct_from_fd() reads current->files->fdt->fd[fd] via bpf_probe_read_kernel(), without taking a reference 👉 https://github.com/falcosecurity/libs/blob/0.25.4/driver/modern_bpf/helpers/extract/extract_from_kernel.h#L233-L258
• kmod: fd_to_socktuple() does a sockfd_lookup() 👉 https://github.com/falcosecurity/libs/blob/0.25.4/driver/ppm_events.c#L1007

Since the fd table is shared among all threads of the process, I believe there is a window where that fd may no longer point to the accepted socket. Both prometheus and kubelet are heavily threaded with a lot of fd churn, and that would fit the "periodically" nature of the report.

However, I could not verify the mechanism, so this is just a hypothesis atm.

A second observation, possibly related

One accept4 sample in the original report shows:

fd=55834574866(<4t>172.30.12.69:9195->172.30.35.222:34134) tuple=172.30.12.69:9195->172.30.35.222:34134

55834574866 is 0xD00000012, which cannot be a valid fd.

What is interesting is that, in the modern probe, the two params of the same event are derived from two different values:

• Parameter 1 (fd) pushes the raw 64-bit ret 👉 https://github.com/falcosecurity/libs/blob/0.25.4/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/accept.bpf.c#L26
• Parameter 2 (tuple) resolves the socket from (int32_t)ret 👉 https://github.com/falcosecurity/libs/blob/0.25.4/driver/modern_bpf/programs/tail_called/events/syscall_dispatched_events/accept.bpf.c#L44

So, whenever the upper 32 bits are not clean, we publish one fd and read the socket of another one. Userspace takes param 0 as-is 👉 https://github.com/falcosecurity/libs/blob/0.25.4/userspace/libsinsp/event.h#L733 so the fdinfo gets keyed by the wrong fd too.

N.B. The kmod does not have this asymmetry, since it truncates before pushing 👉 https://github.com/falcosecurity/libs/blob/0.25.4/driver/ppm_fillers.c#L2014
In the modern probe, only dup, dup2, dup3, and signalfd4 do the same (see 0932ac0), all the other programs push the raw ret.

I have no explanation for where those upper bits come from, so I may be wrong here. Still, the two params of the same event disagreeing looks worth a check.

How to reproduce it

No reliable reproducer atm 😞

The original reporter observed it on a busy Kubernetes node running prometheus, kubelet, and other Go services, both with Falco (modern probe) and with a sysdig build patched to use the very same libs version.

I believe a synthetic reproducer would need a multi-threaded process doing accept4() and outbound connect() on the same fd table at a high rate.

Expected behaviour

The tuple of an accept/accept4 exit event always describes the socket that syscall returned.

Environment

• Falco version: 0.41.0 and 0.41.2 (libs 0.21.0), as originally reported
• OS: Ubuntu 24.04.2 LTS (Noble Numbat)
• Kernel: 6.8.0-60-generic
• Installation method: from source

Additional context

This is still reproducible in principle on master. I diffed the whole path between 0.21.0 and 0.25.4 (so Falco 0.41 up to 0.44.1) and nothing relevant changed:

• accept.bpf.c / accept4.bpf.c: only the enter events removal, the get_sock_from_file() 👉 extract__socket_from_file() rename (94813a6), and the new flags param on accept4 (ee97d16)
• auxmap__store_socktuple_param(): only the same rename
• parse_accept_exit(): same semantics, it still copies the driver tuple as-is and marks the fd as server unconditionally

N.B. #2817 tackles a similar class of problem for dirfd (an fd resolved later than the syscall, so it may point to something else in the meanwhile), even if there the resolution happens in userspace.

cc @falcosecurity/libs-maintainers
cc @dnwe (original reporter)

/kind bug
/area driver-modern-bpf
/area driver-kmod