In the implementation of checkSpatialTemporal (once #71 is merged), we determine a column's spatial/temporal-ness (semantic type) based on the first ProfilerConfig.NUM_RECORD_READ data. If all these data are null, the column's semantic type will be considered NONE, even though there may exist non-null values later on in the table.
We could either drop all non-null values before passing data into the PreAnalyzer or modify the estimateSemanticType function to retry the determination of a column's semantic type if all read values were null.
In the implementation of
checkSpatialTemporal(once #71 is merged), we determine a column's spatial/temporal-ness (semantic type) based on the firstProfilerConfig.NUM_RECORD_READdata. If all these data are null, the column's semantic type will be consideredNONE, even though there may exist non-null values later on in the table.We could either drop all non-null values before passing data into the PreAnalyzer or modify the
estimateSemanticTypefunction to retry the determination of a column's semantic type if all read values were null.