Sometimes nanoseconds are added in the date time column

[kkettinger]

First, really nice app, use it regularly for all kinds of projects. Very reliable.

I've found that sometimes, nanoseconds are added in the .txt file for the date time column, e.g.

type,date time,latitude,longitude,accuracy(m),altitude(m),geoid_height(m),speed(m/s),bearing(deg),sat_used,sat_inview,name,desc
...
T,2025-05-19 13:00:52,48.93052728,7.94226081,3.83,200.193,,4.870,159,20,21,,
T,2025-05-19 13:00:53.329,48.93046868,7.94225131,3.79,199.988,,5.070,180,11,23,,
T,2025-05-19 13:00:55,48.93038298,7.94225280,3.79,199.573,,6.520,185,23,34,,
...

In this example: 2025-05-19 13:00:53.329

I noticed it that most of my logs were parsed with python just fine, just sometimes it threw an error because of unexpected nano seconds in time field.

[JLJu]

Hi @kkettinger,
Did you try to use the brutal Python script here?
https://github.com/BasicAirData/GuTs/tree/main/Scripts/GuTs
If the script is not working with your file, can I ask to share the problematic txt file with me?

[kkettinger]

Hi @JLJu,
i'm using my own script using pandas: pd.to_datetime(gps_df['date time'])
This is the error for timestamps with nanoseconds:

ValueError: unconverted data remains when parsing with format "%Y-%m-%d %H:%M:%S": ".329", at position 3595. You might want to try:
    - passing `format` if your strings have a consistent format;
    - passing `format='ISO8601'` if your strings are all ISO8601 but not necessarily in exactly the same format;
    - passing `format='mixed'`, and the format will be inferred for each element individually. You might want to use `dayfirst` alongside this.

When looking at the timestamps, i do not expect nanoseconds, as they are missing for all but one or two items. I think the best would be to always include nanoseconds / fractional seconds, or to not include them. If they pop up randomly, it makes it hard to parse the data.

[JLJu]

@kkettinger I understand that a fixed format is gold.:-) However, if you want to play around, look within the file Ctime.py. At line 62, you will find a way to deal with different formats.

try:
OldDate = datetime.strptime(row[puntaData],"%Y-%m-%d %H:%M:%S")
except:
OldDate = datetime.strptime(row[puntaData],"%Y-%m-%d %H:%M:%S.%f")
finally:
NewDate = OldDate + timedelta(hours = offsetHours)
try:
riga[puntaData]= NewDate.strftime("%Y-%m-%d %H:%M:%S")
except:
riga[puntaData]= NewDate.strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
finally:

The idea is to try to parse the data with a format; if an error is thrown, you try with another format.

[kkettinger]

Thank you, but the issue is not handling multiple date formats, i already fixed that in my code. This goal of this ticket was to fix the issue at the core. As you said, fixed format is gold, and i agree!

[GrazianoCapelli]

Hi,
first of all thanks for opening this issue.
I write here because we are planning to make a new maintenance release of the app soon, and we can take the opportunity to fix some small issues, so I want to dig a bit into this one.

We could evaluate to use the fixed format (with three decimal digits) for TXT files, that are often used for automatic parsing.
I am a bit skeptic for GPX files, because that files are often used for direct editing and I though that they could be less clean to read, but I'm open to change my mind and I think that I'll do some test in order to evaluate it.
Maybe we should also switch to fixed format for all the <time> tags into the head of the GPX file.

In this case do you think that something like 2025-05-19 13:00:52.0 could be enough, or we should use always 3 decimal digits (2025-05-19 13:00:52.000) when the decimal part is zero?