Default time frame for data collection

[canasdiaz]

Most of the CHAOSS-related metrics need data that is recent in time. If the platform collects all the data available for all the analyzed projects, it may add a lot of information that 1) is not useful and 2) may add extra overload to the platform. From my point of view, the platform should allow the user to obtain only the most recent data.

My proposal is to allow the operator/user to set a default time frame, so that the platform collects the events for the monitored projects after that date and not before.

[sduenas]

The initial idea in this new version of GrimoireLab is that retrieval tasks are shared by all the ecosystems/projects. This means if two ecosystems want to analyze data from the same datasource, data will be only retrieved once and shared for both of them. If a new ecosystem wants to analyze the same datasource, data will already be ready.

What we were thinking is to limit the time frame of the analysis. This means, data will be fully retrieved from the datasource and stored but only the data between the specific time frame will be available for the ecosystem that limits it.

Your concerns about retrieving all the data are perfectly valid. One option could be that every ecosystem has a separate domain/sandbox where tasks are not shared. My concern is that datasources with large amount of data are normally analyzed and the time frame of analysis can be different between ecosystems.