Usage of obs_to_check parameter for the buddy_check test #39
Description
Hello,
I was trying to apply the buddy_check test to a dataset of relative humidity observations (the 2D Numpy array values_insitu_rel_hum), and I was doing a parallelised loop over the dataset rows (i.e., over the time steps) in this way:
def run_buddy(row):
obs_to_check = (~np.isnan(row)).astype(int)
return titanlib.buddy_check(
points,
row,
radius,
num_min,
threshold,
max_elev_diff,
elev_gradient,
min_std,
num_iterations,
obs_to_check=obs_to_check # explicitly pass it
)
radius = [40000]
num_min = [5]
threshold = 2
elev_gradient = -0.0065
max_elev_diff = 1000
min_std = 0.01
num_iterations = 2
flags_buddy_check = Parallel(n_jobs=4, backend="threading")(
delayed(run_buddy)(row)
for row in values_insitu_rel_hum
)so that, passing the obs_to_check parameter to the buddy_check function, I can avoid the check of missing data in my observations, because it would not be reasonable to flag them either as "good" (flag=0) or "bad" (flag=1) data; anyway, this is just how I was planning to use the obs_to_check parameter, I do not state this is the correct way to interpret it.
According to the buddy_check wiki, and also to the API reference, obs_to_check is an optional parameter; nevertheless, when running my code I received the following error message:
joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/_utils.py", line 109, in __call__
return self.func(**kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/parallel.py", line 607, in __call__
return [func(*args, **kwargs) for func, args, kwargs in self.items]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/parallel.py", line 607, in <listcomp>
return [func(*args, **kwargs) for func, args, kwargs in self.items]
^^^^^^^^^^^^^^^^^^^^^
File "/home/diego.bandera/PhD/DATA_QualityCheck/QC_rel_hum.py", line 168, in run_buddy
return titanlib.buddy_check(
^^^^^^^^^^^^^^^^^^^^^
TypeError: buddy_check() got an unexpected keyword argument 'obs_to_check'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/diego.bandera/PhD/DATA_QualityCheck/QC_rel_hum.py", line 344, in <module>
flags_buddy_check = Parallel(n_jobs=4, backend="threading")(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/parallel.py", line 2072, in __call__
return output if self.return_generator else list(output)
^^^^^^^^^^^^
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/parallel.py", line 1682, in _get_outputs
yield from self._retrieve()
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/parallel.py", line 1784, in _retrieve
self._raise_error_fast()
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/parallel.py", line 1859, in _raise_error_fast
error_job.get_result(self.timeout)
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/parallel.py", line 758, in get_result
return self._return_or_raise()
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/diego.bandera/micromamba/envs/titan/lib/python3.11/site-packages/joblib/parallel.py", line 773, in _return_or_raise
raise self._result
TypeError: buddy_check() got an unexpected keyword argument 'obs_to_check'
Hence, apparently the buddy_check function did not accept obs_to_check as additional parameter.
I wonder if others have encountered the same problem, or already came out with a solution to this.
Regards,
Diego