bottomless: validate snapshots before queuing them to be sent

[psarna]

Snapshots sent to bottomless are just regular db files. Let's make sure that before they get sent to S3, we know that they are correct. The best way to do that that comes to my mind is to create a local connection to the database and run pragma integrity_check;.

[psarna]

ah and since we probably compress the snapshots, we should also check if our compression libraries support additional validation/checksumming.

[Horusiath]

My idea was to: currently the snapshot upload comes in two parts:

1. Gzip db file into local disk.
2. Upload gzipped db file as /db.gz.
3. Upload change counter (which keeps [u8;4] used to compare local and remote db versions) as /.changecounter.

My idea was to change snapshotting process to:

1. Gzip db file - to be sure that we have enough disk space to do so.
2. While gzipping, compute a checksum (any one will do).
3. Upload gzipped db file as /db.gz.
4. Upload change counter AND checksum as /.changecounter. This is basically our finaliser.

Now during restore, first thing we do is always downloading remote .changecounter first and comparing it against local. With this change we can also download checksum. When we download db file, compute its checksum and verify it again against the one from .changecounter file.

Now if the checksum will fail, let's try to see if we have a .dep file which describes previous generation. If there is any let's try to recover from it instead and then potentially try to apply the WAL from current generation, skipping the damaged snapshot.