Assimilation is the process of parsing the output of the validated results and collecting them into some file or data structure specific to the project, or possibly creating more tasks for a second stage.
To facilitate this the server needs to store, for each task, whether it has already been assimilated, to prevent it from being assimilated twice. I suggest we add an enum task_state with variants init, validated, and assimilated. Note that the validated state is not strictly required, as it can be computed by checking if any of the results for that task have the valid state, but it will simplify many of the queries.
The server also needs routes to fetch tasks that are pending assimilation. There's a few options for this:
- We have only an
assimilate_fetch route, this route immediately sets the task state to assimilated. The issue with this option is that if assimilation failed for some reason, we might lose the results from that task.
- We have only an
assimilate_submit route, the assimilator can use the already existing /tasks?state=validated route to fetch tasks that need to be assimilated. The issue with this option is that the server cannot control when assimilation happens, and we cannot support parallel assimilation.
- We have both an
assimilate_fetch and assimilate_submit route. This is the most complex option but I cannot think of any issues.
I'm leaning towards option 3.
Assimilation is the process of parsing the output of the validated results and collecting them into some file or data structure specific to the project, or possibly creating more tasks for a second stage.
To facilitate this the server needs to store, for each task, whether it has already been assimilated, to prevent it from being assimilated twice. I suggest we add an enum
task_statewith variantsinit,validated, andassimilated. Note that thevalidatedstate is not strictly required, as it can be computed by checking if any of the results for that task have thevalidstate, but it will simplify many of the queries.The server also needs routes to fetch tasks that are pending assimilation. There's a few options for this:
assimilate_fetchroute, this route immediately sets the task state toassimilated. The issue with this option is that if assimilation failed for some reason, we might lose the results from that task.assimilate_submitroute, the assimilator can use the already existing/tasks?state=validatedroute to fetch tasks that need to be assimilated. The issue with this option is that the server cannot control when assimilation happens, and we cannot support parallel assimilation.assimilate_fetchandassimilate_submitroute. This is the most complex option but I cannot think of any issues.I'm leaning towards option 3.