Add Tier 1 fields to systemd journal event data#5139
Merged
joachimmetz merged 1 commit intoJul 8, 2026
Conversation
Surface additional systemd journal fields on systemd:journal event data. Most are trusted (underscore-prefixed) fields that journald adds itself and the logging process cannot forge: user_identifier (_UID), group_identifier (_GID), process_name (_COMM), executable (_EXE), command_line (_CMDLINE), systemd_unit (_SYSTEMD_UNIT), boot_identifier (_BOOT_ID), machine_identifier (_MACHINE_ID), transport (_TRANSPORT), audit_login_identifier (_AUDIT_LOGINUID) and selinux_context (_SELINUX_CONTEXT) as plain passthroughs. severity (PRIORITY) and facility (SYSLOG_FACILITY) are mapped to the syslog labels. In the journal these are already plain decimal values (0-7 and 0-23) that index the label tables directly; the RFC 3164 priority-byte decomposition used for text syslog does not apply. They are unvalidated user fields, so the values are bounds-checked before use. recorded_time (_SOURCE_REALTIME_TIMESTAMP) is added as a second timestamp: the earliest trusted event time on the originating host, distinct from written_time (journald's reception/collection time). It gets a timeliner.yaml mapping. reporter now falls back to the trusted _COMM when SYSLOG_IDENTIFIER is absent, and the pid docstring is corrected from int to str. New attributes default to None and are not serialized by acstore, so entries that do not populate them are byte-for-byte unchanged. Expected test values are derived from journalctl --output=json ground truth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5139 +/- ##
==========================================
+ Coverage 85.06% 85.07% +0.01%
==========================================
Files 455 455
Lines 40425 40468 +43
==========================================
+ Hits 34387 34430 +43
Misses 6038 6038 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
|
Thanks for the proposed changes. Any indication about the average increase of number of events? Never mind I see it in #5118 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the Tier 1 field set from the design proposal in #5118, which you
approved ("Please feel free to go ahead ... Let's start with tier 1 and assess
what the volume of additional events is"). Built on the now-merged #5113
binary-safe field decode.
The
systemd_journalparser previously surfaced only 5 attributes and discardedthe rest. This adds 14 attributes, most of them trusted (
_-prefixed) fieldsthat journald adds itself and that the logging process cannot forge.
New attributes
user_identifier_UIDgroup_identifier_GIDprocess_name_COMMexecutable_EXEcommand_line_CMDLINEsystemd_unit_SYSTEMD_UNITboot_identifier_BOOT_IDmachine_identifier_MACHINE_IDtransport_TRANSPORTaudit_login_identifier_AUDIT_LOGINUIDselinux_context_SELINUX_CONTEXTseverityPRIORITY(→ syslog label)facilitySYSLOG_FACILITY(→ syslog label)recorded_time_SOURCE_REALTIME_TIMESTAMPAlso:
reporterfalls back to the trusted_COMMwhenSYSLOG_IDENTIFIERisabsent, and the
piddocstring is corrected (int→str).Volume of additional events
The only new attribute that generates timeline events is
recorded_time. Acrossthe four test journals (4,398 entries) it adds 3,177 events on top of the
4,398
written_timeevents → 7,575 total, +72.2% — only for entries thatcarry
_SOURCE_REALTIME_TIMESTAMP(measured with log2timeline + pinfo).Notes / decisions
syslog:line, for cross-parserparity. Per
systemd.journal-fields(7)PRIORITYis 0–7 andSYSLOG_FACILITY0–23, indexed directly (no RFC 3164
>>3/&7). These are unvalidated userfields — the samples contain e.g.
SYSLOG_FACILITY=DHCP4(NetworkManager) — sovalues are bounds-checked and non-numeric ones yield
Nonesilently ratherthan flooding extraction warnings.
text_plugins/syslog.py; glad to switch to importing them instead of the localcopies if you prefer reuse.
written_time(reception/collection time); forsystemd-journal-remoterelayed journals the two diverge.
(generated on an AppArmor host), so it ships without a value assertion — happy
to add one if a SELinux sample is contributed.
Backward compatibility
New attributes default to
None; acstore does not serializeNone, so entriesthat do not populate a field are byte-for-byte unchanged (no event-identity /
dedup-hash change, no storage bloat). No attribute is renamed or removed.
Testing
Expected values are derived from
journalctl --output=jsonground truth over theexisting test journals — no new test data. Verified locally:
tests.parsers.systemd_journal,tests.engine.timeliner+tests.engine.yaml_timeliner_file, pylint / black / docformatter / yamllint,tests.cli.psteal_tool, and the end-to-end suite (clean — no over-claim, nopsort crash).