Added Ivanti Connect/Pulse Secure VC0 logs parser#5070
Conversation
295f007 to
2283e09
Compare
|
Rebased with HEAD |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5070 +/- ##
==========================================
+ Coverage 85.07% 85.09% +0.02%
==========================================
Files 454 455 +1
Lines 40221 40393 +172
==========================================
+ Hits 34217 34374 +157
- Misses 6004 6019 +15 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
2283e09 to
22419b8
Compare
|
@julianghill thanks for the proposed changes, overall they look good. I left some question/comments. |
|
@julianghill can you rebase with HEAD there are merge conflicts |
2009ec8 to
26b4f68
Compare
|
Rebased with HEAD |
231182d to
2bfe079
Compare
joachimmetz
left a comment
There was a problem hiding this comment.
Heads up, I've made some changes to address some of these already
| hex_timestamp.hex_line_identifier. | ||
| record_values (str): original tab-separated values after the realm field. | ||
| recorded_time (dfdatetime.DateTimeValues): record timestamp. | ||
| source_filename (str): name of the source log file. |
There was a problem hiding this comment.
Note that Plaso already track this using the event_data_source attribute container
| self._ExtractIPAddress(fields[5]) if len(fields) > 5 else None | ||
| ) | ||
| event_data.line_identifier = line_identifier | ||
| event_data.log_file_type = self._ExtractLogFileType(source_filename) |
There was a problem hiding this comment.
Given this is dependent on the filename it can be moved to the main parsing loop and only extracted once
| if not file_entry: | ||
| return False | ||
|
|
||
| return bool(self._FILENAME_RE.match(file_entry.name.lower())) |
There was a problem hiding this comment.
lower() can be replaced by adding flags=re.IGNORECASE to re.compile()
| _NON_PRINTABLE_CHARACTER_TRANSLATION_TABLE.pop(ord("\n"), None) | ||
| _NON_PRINTABLE_CHARACTER_TRANSLATION_TABLE.pop(ord("\t"), None) | ||
|
|
||
| _RECORD_SEPARATOR_RE = re.compile(r"[\x17\x15\x13\x12\x05\x04\x03\x02\x01\x00]") |
There was a problem hiding this comment.
this can be simplified to r"[\x00-\x05\x12\x13\15\x17]"
| """ | ||
| file_object.seek(self._HEADER_SIZE) | ||
|
|
||
| decoder = codecs.getincrementaldecoder("utf-8")(errors="replace") |
There was a problem hiding this comment.
why the replace? is this because incremental decoding?
| if record_buffer: | ||
| text = "".join([record_buffer, text]) | ||
|
|
||
| records = text.split("\n") |
There was a problem hiding this comment.
why not use self._RECORD_SEPARATOR_RE.split() instead of translating the separators to newlines first?
|
|
||
| record_values = fields[5:] | ||
| event_data.record_values = "\t".join(record_values) or None | ||
| event_data.body = self._CreateBody( |
There was a problem hiding this comment.
why is the body not fields[7] or fields[7:] ?
|
|
||
| match = self._LOG_FILENAME_RE.match(source_filename) | ||
| if match: | ||
| return match.group("log_file_type") |
There was a problem hiding this comment.
missing lower()? in case the filename is in upper case
|
@julianghill PTAL I made some changes. Some more example files would be beneficial, the current parser does a lot of text scraping, which makes it fragile |
| event_data.record_identifier = record_identifier | ||
|
|
||
| if number_of_fields > 4: | ||
| event_data.realm = fields[4] or None |
There was a problem hiding this comment.
How certain are you about these values, given other implementations seem to refer to these in a more generic way. Or is this something specific to the "events" log type?
7c94f54 to
451d70c
Compare
| f"as UTF-8" | ||
| ) | ||
| return | ||
| record = record_data.decode("utf-8", errors="replace") |
There was a problem hiding this comment.
IMHO this is not forensically sound. You are changing data without any notice, better to err and debug what is wrong
|
Adjustment made. For field [5], the samples checked show it is either empty or an IP address. The The fields after that vary by log type and message code, so they remain in I also added small test logs. They are based on the observed layouts, including empty field [5]/[6] seen in the real logs. |
|
@julianghill thanks for the updates, left one comment already, but will take a closer look tomorrow. |
| ) | ||
| _NON_PRINTABLE_CHARACTER_TRANSLATION_TABLE.pop(ord("\t"), None) | ||
|
|
||
| _RECORD_SEPARATORS_RE = re.compile(b"[\\x00-\\x05\\x0a\\x12\\x13\\x15\\x17]") |
There was a problem hiding this comment.
the test files only use 0x05 as a record separator. Any test files from which can be observed that other separator values are also used?
| self.CheckEventData(event_data, expected_event_values) | ||
|
|
||
| def testParseWithEmptyVC0File(self): | ||
| """Tests the Parse function with an empty .vc0 file.""" |
There was a problem hiding this comment.
Note that log.policytrace.vc0 is also empty, which makes this test redundant
| hostname (str): appliance hostname. | ||
| ip_address (str): IP address found in the record values. | ||
| line_number (str): line number. | ||
| log_context (str): context value found before the record values. |
There was a problem hiding this comment.
Isn't this value related to https://help.ivanti.com/ps/help/en_US/ICS/21.9R1/ag/auth_realms.htm ?
There was a problem hiding this comment.
One line description of pull request
Add parser for Ivanti Connect Secure VC0 runtime logs.
Description:
Adds a parser for Ivanti Connect Secure / Pulse Secure .vc0 log files.
The parser extracts fields including:
Some .vc0 files only contain the file header and no log records yet. The parser treats those as valid empty log files.
The tests use small fake .vc0 records created in the test code, so no real Ivanti appliance logs or case data are included. Did test it on real data, but obvisously cant include it here.
Related issue (if applicable): N/A
Notes:
All contributions to Plaso undergo code review.
This makes sure that the code has appropriate test coverage and conforms to the
Plaso style guide.
One of the maintainers will examine your code, and may request changes. Check off the items below in
order, and then a maintainer will review your code.
Checklist: