pass3

Notes, Scripts, etc. regarding Pass3

Step 1 at TACC

Building the Container

Step 1 runs in an apptainer container. Mostly because we want a CVMFS-like environment, but we don't have ARM builds right now. We couldn't use the main py4.4.2 cvmfs/spack install because HTCondor pybindings versions get deleted from pypi. Trying to build a container from scratch will require changing the HTCondor pybindings version to one that is available. This is needed because of limited space on pypi. There is a branch pass3 in the cvmfs repo that allows us to compensate for this.

Getting Data from Tape

The workflow is roughly the same on both Stampede3 and Vista. The main difference is the size of the scratch disk (located at $SCRATCH) and the location where the $WORK variable points.

On Ranch, the data is located at /scoutfs/projects/TG-PHY150040/data/exp/IceCube/<YYYY>/unbiased/PFRaw/

for 2011-2014 data you will want to use the

/scoutfs/projects/TG-PHY150040/OldRanchData/data/exp/IceCube/<YYYY>/unbiased/PFRaw/

until the transition to LTO-10 tapes has been made. This is scheduled for later in 2026. The files in /scoutfs/projects/TG-PHY150040/data/exp/IceCube/<2011-2014>/ are spread across several tapes (usually number of tapes = number of tape drives available at the time). Pulling from several tapes to get the files is painfully slow compared to using OldRanchData.

When the code was originally written the tape archive was fast enough and the bundles small enough that one could transfer the data at the start of the job. With larger bundles in earlier years (pre-2019) and a transition to a "new" Ranch (marked by the transition to scoutfs), the transfer speeds degraded to the point that pre-staging the data is required. A large part of the decrease in speed on Ranch was that "new" Ranch only has 12 tape drives. This is supposed to be upped to 24 (or even 40) drives down the road. For right now though, pre-staging is best.

To transfer data from Ranch to either scratch disk there are two methods:

1. rsync

Using

rsync -aviP $ARCHIVER:/scoutfs/projects/TG-PHY150040/data/exp/IceCube/YYYY/unbiased/PFRaw/MM* $SCRATCH/<your_data_dir>/YYYY/

This is how this looks for me:

rsync -aviP $ARCHIVER:/scoutfs/projects/TG-PHY150040/data/exp/IceCube/2024/unbiased/PFRaw/09* /scratch/04799/tg840985/tmp.v2/2024/

$ARCHIVER is ranch.tacc.utexas.edu

2. Globus

To set up your account for Globus at TACC go to https://docs.tacc.utexas.edu/datatransfer/globus/

Using Globus (web interface, CLI, python API) you can transfer from Ranch collection (TACC Ranch3 GCS v5.4 Tape Archive, 57c4032a-2b50-47f0-adf8-13fff3a7d77d) to the Stampede3 (TACC Stampede3 GCS v5.4 Filesystems, 1e9ddd41-fe4b-406f-95ff-f3d79f9cb523) or Vista (TACC Vista GCS v5.4 Filesystems, fcb0b578-dcb3-4043-a841-8bd0974d6af1) collection.

I would recommend rsync because you can run 4-6 transfers in parallel. Globus only allows 3 transfers in parallel at the moment

Generating submit files

submit_stampede3.py (located in pass3/scripts/submit/step1/) is setup to generate a SLURM submit file that than uses srun multiprog files. Multiprog files is a list of commands that should be run on the node that srun assigns it to. The one advantage of multiprog files is that srun does the per node assignment , so you don't have to manually assign things to different nodes, i.e. makes things more legible and understandable.

The scripts are setup to work in month data increments. Once you have transferred a month worth data to your temporary location

YEAR=2024; MONTH=2; python3 $HOME/pass3/scripts/submit/step1/submit_stampede3.py --container /scratch/04799/tg840985/pass3_cvmfs_v4.4.2_icetray_v1.17.0_arm.sif --scratchdir /scratch/04799/tg840985/tmp_scratch/ --submitfile /home1/04799/tg840985/pass3/submit_files/${YEAR}/${YEAR}_${MONTH}_new_gcd.slurm --checksum-file /home1/04799/tg840985/pass3/data/checksums.sha512sum --year ${YEAR} --month ${MONTH} --gcddir /work/04799/tg840985/stampede3/GCD.v2/ --multiprogfile /home1/04799/tg840985/pass3/submit_files/${YEAR}/${YEAR}_${MONTH}_new_gcd.multiprog --slurmqueue gg --numcores 100 --allocation PHY20012 --grl /home1/04799/tg840985/pass3/data/grl.pass3 --badfiles /home1/04799/tg840985/pass3/data/known_bad_files --cpuarch aarch64 --bundlesready --bundledir $SCRATCH/tmp.v2/ --outdir $SCRATCH/testout.v2/${YEAR} --duplicate-skip-dir /home1/04799/tg840985/pass3/submit_files/${YEAR}/skip/

An option to note is

--duplicate-skip-dir /home1/04799/tg840985/pass3/submit_files/${YEAR}/skip/

this will look for duplicates across the contents of different bundles. There are bundles that contain the same file. This internal algorithm picks one of the two duplicates to process and ignores the other.

On Vista, we need to limit the number of cores. There are total of 144 cores and 237 GB per Grace-Grace node. The code uses ~2.2 GB RAM per instance.

A sample of a SLURM submit file is:

#!/bin/bash
#SBATCH -t 24:00:00
#SBATCH -A PHY20012
#SBATCH -p gg
#SBATCH -J /home1/04799/tg840985/pass3/submit_files/2022/2022_1_new_gcd.slurm
#SBATCH -N 32
#SBATCH -n 32
#SBATCH -o /home1/04799/tg840985/pass3/submit_files/2022/2022_1_new_gcd.slurm.o.%j
#SBATCH -e /home1/04799/tg840985/pass3/submit_files/2022/2022_1_new_gcd.slurm.e.%j

echo `date`

LD_PRELOAD=

if [ ! -e /home1/04799/tg840985/pass3/submit_files/2022/2022_1_new_gcd.multiprog0.done ]; then
echo Starting /home1/04799/tg840985/pass3/submit_files/2022/2022_1_new_gcd.multiprog0
echo `date`
srun --nodes=32 --ntasks-per-node=1 --exclusive --cpus-per-task=$SLURM_CPUS_ON_NODE --multi-prog /home1/04799/tg840985/pass3/submit_files/2022/2022_1_new_gcd.multiprog0 && touch /home1/04799/tg840985/pass3/submit_files/2022/2022_1_new_gcd.multiprog0.done || touch /home1/04799/tg840985/pass3/submit_files/2022/2022_1_new_gcd.multiprog0.failed
fi

We have LD_PRELOAD= to avoid some errors with apptainer. The *.done and *.failed files are empty. They are just indicators whether the srun command has succeed or not. Failure of a "program" from the multiprog file will also make srun fail.

The smallest good processing increment is a bundle. Each "program" will take a bundle (and necessary information for said bundle) and process it.

An example "program" line for multiprog file:

0  /opt/apps/tacc-apptainer/1.3.3/bin/apptainer exec -B /home1/04799/tg840985/pass3:/opt/pass3 -B /work/04799/tg840985/vista/splines/splines:/cvmfs/icecube.opensciencegrid.org/data/photon-tables/splines -B /work2 -B /scratch /scratch/04799/tg840985/pass3_cvmfs_v4.4.2_icetray_v1.17.0_arm.sif /cvmfs/icecube.opensciencegrid.org/py3-v4.4.2/RHEL_9_aarch64/metaprojects/icetray/v1.17.0/bin/icetray-shell python3 /opt/pass3/scripts/icetray/step1/run_step1.py --bundle /scratch/04799/tg840985/tmp.v2/2022/0101/24a633c4f11c11ecb5b232208fe3aaeb.zip --gcddir /work2/04799/tg840985/stampede3/GCD.v2 --outdir /scratch/04799/tg840985/testout.v2/2022/0101 --checksum 3e08159c72bb6e7a37aaeb6ae0389ba8aa6d98c14b90e9296c6e77f8544ebb4f9bb241a479443b4707e5f53db4bbe9a542774a98ad197907f0dad2b645e40254 --scratchdir /scratch/04799/tg840985/tmp.v2/2022 --grl /home1/04799/tg840985/pass3/data/grl.pass3 --badfiles /home1/04799/tg840985/pass3/data/known_bad_files  --duplicate-skip-json /home1/04799/tg840985/pass3/submit_files/2022/skip/24a633c4f11c11ecb5b232208fe3aaeb.zip.duplicate_skip.json --maxnumcpus 100

0: is a label for the program in the multiprog file. /opt/apps/tacc-apptainer/1.3.3/bin/apptainer: Apptainer is a Go binary, so you can just grab it without loading the module -B /home1/04799/tg840985/pass3:/opt/pass3: Gets the pass3 code pieces into the container -B /work/04799/tg840985/vista/splines/splines:/cvmfs/icecube.opensciencegrid.org/data/photon-tables/spline: the necessary splines for the realtime filter:

One key aspect is that number of nodes passed to srun needs to match the number of "programs" in the multiprog file, so for some years and months there are padding. We waste some resources but in the grant scheme it is okay.

Submitting Jobs

Now just sbatch <submit_file> you generated with submit_stampede3.py.

What does the job/"program" do?

On a high level, the job itself runs run_step1.py (located in pass3/scripts/icetray/step1/) on each node. run_step1.py takes the files in the bundle and applies pass3_reprocess_PFRaw.py to the PFRaw files in the bundle. The bundles only contain a manifest file (earlier years it is a *.json file with a single json object for all file, in later years it is *.ndjson with each file being on a new line as a json object). The PFRaw files end in *.tar.gz. The PFRaw filenames frequently esp in the earliest years.

The concurrent.futures.ProcessPoolExecutor allows a mapping list of inputs to N workers (each being a Python subprocess) and monitors the state of each worker. This bypasses the GIL for "real" multi-processing. For each runner, we prepare the required inputs (directory to find GCD files, path to the archival bundle, PFRaw file, directory to place outputs, expected SHA512 checksum for the PFRaw file from the archive bundle). The input preparation includes making sure that the bundle SHA512 is correct. It will also check whether some or all of the PFRaw files in a bundle should be skipped because they appear in a different bundle. There are some bundles that are complete duplicates of others, so there will be an error that there are no inputs.

After the inputs are checked and the list of inputs is passed to ProcessPoolExecutor. The runner will:

1. Create a temporary working directory
2. Find the GCD file for the run from the PFRaw file name - Issues a warning and returns if it can't be found
3. Ensure the GCD file has the expected quantities (pass3_check_gcd.py) - FADC gains are what we expect (Each GCD file contains a map of DOM to original FADC value. ). mean ATWD, FADC charge = 1, relative DOM efficiency is not NAN.
4. Extract the PFRaw file
5. Check the checksum for the PFRaw file - Issues a warning and returns it isn't what we expect
6. Check if the outfile already exsits - Issues a warning and returns if it does. To replace an output file you need to rename or delete it.
7. Run pass3_reprocess_PFRaw.py
8. Run monitoring scripts (pass3_calc_filter_rates.py and pass3_check_charge_filter.py)
9. Calculate the SHA512 checksum for the output file
10. Moves all outputs from the temporary working directory to the output dir

Steps 6 through 8 log into separate files that will be colocated with the output files.

The runner will generally not raise Exceptions. This is to allow for better handling of errors and logging. The ProcessPoolExecutor fails when one of the workers fails. This causes unprocessed/subsequent PFRaw files to not be processed. Checking the return values allows us to summarize what has failed.

Transferring Output

The output is best tranferred using Globus. Transferring a whole year is best given the directory structure

Open Issues

Missing GCD Files

Runs that have missing GCD files for Step1

All of calendar year 2024 and 2025

128826
131264
131243
133284
133408
129212
128826
125068
124703

Missing files 2014

These files are missing / not processed, and are not in the pass2 missing files list.

They need to be hunted down and processed again.

Found 1473 missing files across 40 unique Run IDs:

Run 123615 (Started: 2014-01-01 00:46:16, Good: True) is missing files! Missing file numbers: [1, 3, 5, 7, 9, 10, 12, 14, 16]
Run 123616 (Started: 2014-01-01 01:25:24, Good: True) is missing files! Missing file numbers: [2, 4, 6, 8, 11, 13, 15, 17, 19, 20, 22, 24, 26, 28, 31, 33, 35, 37, 39, 40, 42, 44]
Run 123618 (Started: 2014-01-01 03:17:24, Good: True) is missing files! Missing file numbers: [2, 4, 6, 8, 11]
Run 123619 (Started: 2014-01-01 03:46:05, Good: True) is missing files! Missing file numbers: [1, 3, 5, 7, 9, 10, 12, 14, 16, 18, 21, 23, 25, 27, 29, 30, 32, 34, 36, 38]
Run 123620 (Started: 2014-01-01 05:12:20, Good: True) is missing files! Missing file numbers: [1, 3, 5, 7, 9, 10, 12, 14, 16, 18, 21, 23, 25, 27, 29, 30, 32, 34, 36, 38, 41, 43, 45, 47, 49, 50, 52, 54, 56, 58, 61, 63, 65, 67, 69, 70, 72, 74, 76, 78, 81, 83, 85, 87, 89, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 111, 113, 115, 117, 119, 120, 122, 124, 126, 128, 131, 133, 135, 137, 139, 140, 142, 144, 146, 148, 151, 153, 155, 157, 159, 160, 162, 164, 166, 168, 171, 173, 175, 177, 179, 180, 182, 184, 186, 188, 191, 193, 195, 197, 199, 201, 203, 205, 207, 209, 210, 212, 214, 216, 218, 221, 223, 225, 227]
Run 123621 (Started: 2014-01-01 13:13:39, Good: True) is missing files! Missing file numbers: [2, 4, 6, 8, 11, 13, 15, 17, 19, 20, 22, 24, 26, 28, 31, 33, 35, 37, 39, 40, 42, 44, 46, 48, 51, 53, 55, 57, 59, 60, 62, 64, 66, 68, 71, 73, 75, 77, 79, 80, 82, 84, 86, 88, 91, 93, 95, 97, 99, 101, 103, 105, 107, 109, 110, 112, 114, 116, 118, 121, 123, 125, 127, 129, 130, 132, 134, 136, 138, 141, 143, 145, 147, 149, 150, 152, 154, 156, 158, 161, 163, 165, 167, 169, 170, 172, 174, 176, 178, 181, 183, 185, 187, 189, 190, 192, 194]
Run 123622 (Started: 2014-01-01 20:03:23, Good: True) is missing files! Missing file numbers: [1, 3, 5, 7, 9, 10, 12, 14, 16, 18, 21, 23, 25, 27, 29, 30, 32, 34, 36, 38, 41]
Run 123627 (Started: 2014-01-01 22:22:35, Good: True) is missing files! Missing file numbers: [2, 4, 6, 8, 11, 13]
Run 123628 (Started: 2014-01-01 23:01:07, Good: True) is missing files! Missing file numbers: [1, 3, 5, 7, 9, 10, 12, 14, 16, 18, 21, 23, 25, 27, 29, 30, 32, 34, 36, 38]
Run 123629 (Started: 2014-01-02 00:28:55, Good: True) is missing files! Missing file numbers: [2, 4, 6, 8, 11, 13, 15, 17, 19, 20, 22]
Run 123631 (Started: 2014-01-02 01:27:44, Good: True) is missing files! Missing file numbers: [1, 3, 5, 7, 9, 10, 12, 14, 16, 18, 21, 23, 25, 27, 29, 30, 32, 34, 36, 38, 41, 43, 45, 47, 49, 50, 52, 54, 56, 58, 61, 63, 65, 67, 69, 70, 72, 74, 76, 78, 81, 83, 85, 87, 89, 90, 92, 94, 96, 98, 100, 102, 104, 106, 108, 111, 113, 115, 117, 119, 120, 122, 124, 126, 128, 131, 133, 135, 137, 139, 140, 142, 144, 146, 148, 151, 153, 155, 157, 159, 160, 162, 164, 166, 168, 171, 173, 175, 177, 179, 180, 182, 184, 186, 188, 191, 193, 195, 197, 199, 201, 203, 205, 207, 209, 210, 212, 214, 216, 218, 221, 223, 225, 227]
Run 123632 (Started: 2014-01-02 09:29:14, Good: True) is missing files! Missing file numbers: [2, 4, 6, 8, 11, 13, 15, 17, 19, 20, 22, 24, 26, 28, 31, 33, 35, 37, 39, 40, 42, 44, 46, 48, 51, 53, 55, 57, 59, 60, 62, 64, 66, 68, 71, 73, 75, 77, 79, 80, 82, 84, 86, 88, 91, 93, 95, 97, 99, 101, 103, 105, 107, 109, 110, 112, 114, 116, 118, 121, 123, 125, 127, 129, 130, 132, 134, 136, 138, 141, 143, 145, 147, 149, 150, 152, 154, 156]
Run 123639 (Started: 2014-01-02 15:10:38, Good: True) is missing files! Missing file numbers: [1, 3, 5]
Run 123640 (Started: 2014-01-02 15:46:59, Good: True) is missing files! Missing file numbers: [1, 3, 5, 7, 9, 10]
Run 124724 (Started: 2014-05-13 16:57:22, Good: True) is missing files! Missing file numbers: [115, 116, 117, 118]
Run 124755 (Started: 2014-05-16 17:56:15, Good: True) is missing files! Missing file numbers: [104]
Run 124982 (Started: 2014-07-01 12:06:06, Good: True) is missing files! Missing file numbers: [91, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157]
Run 125145 (Started: 2014-08-07 03:18:41, Good: True) is missing files! Missing file numbers: [48, 50, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 134, 135]
Run 125175 (Started: 2014-08-13 21:54:42, Good: True) is missing files! Missing file numbers: [44, 46, 47]
Run 125214 (Started: 2014-08-21 23:21:42, Good: True) is missing files! Missing file numbers: [192, 193]
Run 125215 (Started: 2014-08-22 07:22:49, Good: True) is missing files! Missing file numbers: [73]
Run 125216 (Started: 2014-08-22 15:23:52, Good: True) is missing files! Missing file numbers: [129]
Run 125217 (Started: 2014-08-22 23:24:52, Good: True) is missing files! Missing file numbers: [31]
Run 125238 (Started: 2014-08-29 23:45:26, Good: True) is missing files! Missing file numbers: [142, 144]
Run 125242 (Started: 2014-08-31 07:49:43, Good: True) is missing files! Missing file numbers: [63]
Run 125330 (Started: 2014-09-21 01:54:35, Good: True) is missing files! Missing file numbers: [191]
Run 125349 (Started: 2014-09-27 02:34:13, Good: True) is missing files! Missing file numbers: [190]
Run 125380 (Started: 2014-10-04 02:38:54, Good: True) is missing files! Missing file numbers: [128, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 175, 176, 177]
Run 125414 (Started: 2014-10-10 15:23:21, Good: True) is missing files! Missing file numbers: [26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39, 40]
Run 125461 (Started: 2014-10-21 22:15:58, Good: True) is missing files! Missing file numbers: [57, 68, 73, 86, 88, 92, 93, 94, 95, 98, 99, 101, 103, 106, 112, 116, 118]
Run 125552 (Started: 2014-11-12 04:11:02, Good: True) is missing files! Missing file numbers: [70, 80, 85]
Run 125597 (Started: 2014-11-22 20:39:02, Good: True) is missing files! Missing file numbers: [108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175]
Run 125598 (Started: 2014-11-23 03:03:15, Good: True) is missing files! Missing file numbers: [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 98, 99, 100, 101, 102, 103, 104, 105, 107, 108, 110, 111, 112, 113, 114, 115, 116, 117, 118, 120]
Run 125599 (Started: 2014-11-23 07:32:16, Good: True) is missing files! Missing file numbers: [4, 5, 6, 7, 8, 9, 10, 11, 12, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 49, 50, 51, 53, 54, 56, 57, 62, 64, 66, 67, 69, 70, 71, 73, 74, 76, 77, 78, 79, 80, 82, 83, 85, 86, 87, 89, 90, 91, 92, 93, 94, 95, 96, 98, 99, 100, 103, 105, 106, 107, 109, 112, 114, 116, 117, 118, 119, 122, 124, 127, 128, 129, 135, 137, 138, 139, 141, 143, 145, 146, 147, 151, 152, 154, 156, 157, 159, 161, 163, 164, 165, 167, 168, 170, 173, 177, 178, 179, 180, 183, 184, 185, 186, 187, 188, 190, 191, 192, 193, 194, 195, 196, 197, 198, 200, 201, 202, 203, 204, 205, 206, 207, 209, 210, 211, 212, 213, 214, 215, 217]
Run 125600 (Started: 2014-11-23 15:33:23, Good: True) is missing files! Missing file numbers: [2, 4, 5, 7, 8, 10, 11, 12, 14, 15, 16, 18, 19, 21, 22, 23, 25, 29, 33, 35, 36, 37, 38, 39, 40, 41, 42, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53, 55, 56, 57, 58, 60, 62, 63, 64, 65, 66, 67, 69, 71, 72, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 89, 90, 91, 92, 93, 95, 96, 97, 98, 99, 100, 101, 103, 104, 106, 107, 108, 109, 112, 113, 115, 116, 118, 120, 121, 122, 123, 124, 125, 126, 128, 129, 130, 131, 132, 133, 135, 136, 137, 138, 139, 140, 141, 142, 143, 145, 146, 147, 148, 149, 150, 152, 153, 155, 157, 158, 159, 160, 161, 163, 164, 167, 168, 169, 171, 172, 174, 175, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 189, 190, 191, 192, 193, 195, 196, 199, 202, 203, 204, 205, 207, 208, 209, 210, 211, 212, 213, 214, 216, 220, 221, 223, 224, 226, 227]
Run 125601 (Started: 2014-11-23 23:34:21, Good: False) is missing files! Missing file numbers: [3]
Run 125606 (Started: 2014-11-24 00:39:52, Good: True) is missing files! Missing file numbers: [6, 7, 9, 10, 11, 13, 14, 15, 16, 19, 20, 22, 24, 25, 27, 28, 30, 31, 33, 34, 35, 36, 37, 38, 39, 41, 43, 44, 46, 47, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 70, 71, 72, 73, 74, 75, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 97, 98, 99, 100, 101, 103, 104, 105, 106, 107, 108, 109, 110, 112, 113, 115, 118, 120, 122, 124, 125, 126, 127, 128, 129, 130, 132, 133, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 146, 147, 148, 149, 151, 153, 155, 158, 159, 160, 161, 163, 164, 166, 168, 169, 170, 171, 172, 174, 175, 176, 177, 178, 180, 181, 183, 184, 185, 186, 187, 188, 190, 191, 193, 195, 196, 197, 198, 201, 202, 203, 204, 205, 207, 208, 209, 210, 211, 212, 213, 214, 217, 218, 219, 220, 221, 223]
Run 125608 (Started: 2014-11-24 08:52:08, Good: True) is missing files! Missing file numbers: [4, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 27, 28, 29, 30, 31, 33, 34, 35, 36, 51, 78, 81]
Run 125661 (Started: 2014-12-01 14:28:51, Good: True) is missing files! Missing file numbers: [158, 169, 175, 176, 187, 189]
Run 125662 (Started: 2014-12-01 22:29:44, Good: True) is missing files! Missing file numbers: [2, 3, 4, 5, 6, 7, 9]

Missing files 2018

Oct 16 2018 missing data in archive at TACC. Reprocess whole day, and check the day before/after for missing.

Broken files

2011:

./2011/2011_11_new_gcd.slurm.e.742041:Exception: Did not finish ['PFRaw_PhysicsTrig_PhysicsFiltering_Run00118918_Subrun00000000_00000157.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2011/1113/d17a34d6-6615-4b83-9963-e4cf9a9c291f.zip

2012:

./2012/2012_8_new_gcd.slurm.e.736828:Exception: Did not finish ['PFRaw_PhysicsTrig_PhysicsFiltering_Run00120558_Subrun00000000_00000132.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120558_Subrun00000000_00000070.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120558_Subrun00000000_00000101.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2012/0814/41571adf-5614-4a76-ac4a-bcf49a138fff.zip
./2012/2012_8_new_gcd.slurm.e.736828:Exception: Did not finish ['PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000141.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2012/0815/40152ec5-f468-4006-9d70-d7187514d274.zip
./2012/2012_8_new_gcd.slurm.e.736828:Exception: Did not finish ['PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000171.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000203.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120562_Subrun00000000_00000001.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000196.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000179.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2012/0816/a2de4101-47e4-45e6-aaa0-f1618774973d.zip
./2012/2012_8_new_gcd.slurm.e.736828:Exception: Did not finish ['PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000127.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120559_Subrun00000000_00000096.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000113.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000078.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120561_Subrun00000000_00000149.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120558_Subrun00000000_00000201.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120558_Subrun00000000_00000181.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120559_Subrun00000000_00000017.tar.gz', 'PFRaw_PhysicsTrig_PhysicsFiltering_Run00120559_Subrun00000000_00000155.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2012/0815/85064db4-7465-440a-b3c8-8404f904c0f8.zip
./2012/2012_11_new_gcd.slurm.e.736851:Exception: Did not finish ['PFRaw_PhysicsTrig_PhysicsFiltering_Run00121106_Subrun00000000_00000000.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2012/1126/bf57331c-4ca9-4555-8a3e-35f4044b062b.zip

2013:

./2013/2013_10_new_gcd.slurm.e.727158:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123097_Subrun00000000_00000039.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1011/748fce05-2c77-45c0-a924-ab70cbe89ed6.zip
./2013/2013_10_new_gcd.slurm.e.727158:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123111_Subrun00000000_00000062.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1016/31154d33-dc6a-45b6-89c2-19aa3685f444.zip
./2013/2013_10_new_gcd.slurm.e.727158:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123061_Subrun00000000_00000022.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1001/deef9c99-553e-4436-ad4b-331f8a8900d6.zip
./2013/2013_10_new_gcd.slurm.e.727158:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123101_Subrun00000000_00000193.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1013/98fb0e53-f046-4be5-935d-690262ce1d3b.zip
./2013/2013_10_new_gcd.slurm.e.727158:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123185_Subrun00000000_00000131.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1031/e9e9f077-f194-4908-8493-5a23afd3ef43.zip
./2013/2013_10_new_gcd.slurm.e.727158:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123179_Subrun00000000_00000099.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1029/51d0c3cb-0978-4fbe-91c5-8e005a1f5162.zip
./2013/2013_5_new_gcd.slurm.e.725427:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122357_Subrun00000000_00000048.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0518/ab545d3f-4ae6-484d-b781-e887b0f9a212.zip
./2013/2013_5_new_gcd.slurm.e.725427:Exception: Did not finish ['PFRaw_PhysicsTrig_PhysicsFiltering_Run00122275_Subrun00000000_00000026.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0502/e0e7b969-41fe-4ee8-8866-d0d5f55365e8.zip
./2013/2013_5_new_gcd.slurm.e.725427:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122361_Subrun00000000_00000117.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0519/06693187-2839-49bb-917f-7fdc238a902e.zip
./2013/2013_5_new_gcd.slurm.e.725427:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122439_Subrun00000000_00000131.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0525/6bce5bf7-fb54-4cce-ab46-70906b734fd5.zip
./2013/2013_9_new_gcd.slurm.e.725430:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122972_Subrun00000000_00000143.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0907/ac68515a-969a-4c25-9b1a-3b830cb0e510.zip
./2013/2013_9_new_gcd.slurm.e.725430:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123015_Subrun00000000_00000201.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0917/f125b77c-5a45-4696-bec3-c0d3da6c37d2.zip
./2013/2013_8_new_gcd.slurm.e.725429:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122788_Subrun00000000_00000166.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0803/270d3baa-17c0-4f14-90fd-75df7e448b74.zip
./2013/2013_8_new_gcd.slurm.e.725429:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122829_Subrun00000000_00000097.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0812/ae602067-1ee6-4b5f-9ff8-02ed3927b000.zip
./2013/2013_8_new_gcd.slurm.e.725429:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122804_Subrun00000000_00000120.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0808/256b3150-1821-4bf2-871b-e86b6328a03e.zip
./2013/2013_7_new_gcd.slurm.e.719312:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122672_Subrun00000000_00000043.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0714/718d7bce-2c80-4bef-bf54-cb172dbd9285.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123385_Subrun00000000_00000173.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1218/e247490c-4cef-4b8d-809f-52f313f1c819.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123323_Subrun00000000_00000048.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1201/5f5289d4-5df0-466d-98c8-7f8ae9ecb08c.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123330_Subrun00000000_00000004.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1203/6580da1a-aaca-4fc9-ad6b-6ae2b0ceec85.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123361_Subrun00000000_00000142.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1211/3c2412c7-52dc-4bf1-9d6a-75ae38f66148.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123466_Subrun00000000_00000025.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1223/7a47f83a-0b5c-4dcf-92b4-758739108082.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123432_Subrun00000000_00000001.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1222/cad531a8-4dc0-491f-a603-17845125b2da.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123388_Subrun00000000_00000053.tar.gz', 'PFRaw_PhysicsFiltering_Run00123386_Subrun00000000_00000106.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1219/f4da39a7-5816-4335-bbc1-ba6e09e53a1c.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123501_Subrun00000000_00000029.tar.gz', 'PFRaw_PhysicsFiltering_Run00123501_Subrun00000000_00000022.tar.gz', 'PFRaw_PhysicsFiltering_Run00123501_Subrun00000000_00000024.tar.gz', 'PFRaw_PhysicsFiltering_Run00123501_Subrun00000000_00000050.tar.gz', 'PFRaw_PhysicsFiltering_Run00123501_Subrun00000000_00000046.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1227/f21b238c-a9ad-4aca-9197-9275211fae43.zip
./2013/2013_12_new_gcd.slurm.e.725426:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123501_Subrun00000000_00000003.tar.gz', 'PFRaw_PhysicsFiltering_Run00123501_Subrun00000000_00000002.tar.gz', 'PFRaw_PhysicsFiltering_Run00123501_Subrun00000000_00000006.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1226/1b51e952-fd9b-4cb9-8734-15adb2d94f04.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123240_Subrun00000000_00000080.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1111/0e25bb5f-ad67-4efa-b19b-71daca494d77.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123245_Subrun00000000_00000064.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1113/cd00b320-2d72-48e6-9244-97da7183b2ec.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123249_Subrun00000000_00000154.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1114/3aa3e9f7-963e-4df6-8ebc-91e5b4d07829.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123236_Subrun00000000_00000023.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1110/8db7ce7e-e613-49a8-9fd2-3f06e726b737.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123195_Subrun00000000_00000030.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1103/c2c4b34e-894a-47df-ae10-30a003bf134a.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123217_Subrun00000000_00000180.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1107/b9de2cc8-62a3-47c1-9832-a30eec26d014.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123190_Subrun00000000_00000059.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1101/3924a664-ffea-4c91-9d96-c9899190a565.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123281_Subrun00000000_00000145.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1124/35797648-176a-4b18-a6c3-030d75ede83f.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123285_Subrun00000000_00000089.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1125/2d6de6a1-5d89-47b8-8557-5915013abe2d.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123290_Subrun00000000_00000001.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1126/fcdfbc37-298f-4ce4-a596-843d98af6234.zip
./2013/2013_11_new_gcd.slurm.e.725425:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123254_Subrun00000000_00000221.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/1116/83f6faad-ecd7-403c-bc68-c37746724d94.zip
./2013/2013_6_new_gcd.slurm.e.725428:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122493_Subrun00000000_00000118.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0605/4e16915b-2c20-4bb5-8164-e6b2b0e48d16.zip
./2013/2013_6_new_gcd.slurm.e.725428:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00122610_Subrun00000000_00000148.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2013/0629/5ae70d32-9509-4cad-91f1-f238c4b10aaf.zip

2014:

./2014/2014_7_new_gcd.slurm.e.718358:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000079.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000098.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000179.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000121.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000024.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000119.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000029.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000161.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000194.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000037.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000038.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000149.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000158.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000054.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000092.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000147.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000115.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000095.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000162.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000036.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000144.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000143.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000069.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000193.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000180.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000160.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000041.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000181.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000022.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000004.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000051.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000072.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000074.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000057.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000159.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000122.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000093.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000163.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000123.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000044.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000101.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000014.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000084.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000028.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000183.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000040.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000177.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000088.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000178.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000110.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000168.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000192.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000104.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000148.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000078.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000071.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000043.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000176.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000050.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000141.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000132.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000140.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000185.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000117.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000103.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000189.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000188.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000125.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000150.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000166.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000165.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000107.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000058.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000128.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000112.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000134.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000045.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000009.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000089.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000027.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000137.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000053.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000083.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000016.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000062.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000035.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000017.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000056.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000085.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000012.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000030.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000052.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000118.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000068.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000090.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000091.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000145.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000127.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000000.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000049.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000130.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000082.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000111.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000153.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000023.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000086.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000174.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000047.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000186.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000003.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000008.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000065.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000124.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000067.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000059.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000139.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000061.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000019.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000033.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000097.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000039.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000021.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000080.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000002.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000152.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000151.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000135.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000182.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000154.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000018.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000046.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000063.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000105.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000020.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000187.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000138.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000126.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000031.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000157.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000073.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000015.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000005.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000042.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000136.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000156.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000196.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000026.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000146.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000076.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000081.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000032.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000114.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000099.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000190.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000170.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000100.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000142.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000013.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000195.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000173.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000116.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000096.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000094.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000167.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000133.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000070.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000108.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000055.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000064.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000164.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000155.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000011.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000075.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000007.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000172.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000060.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000025.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000106.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000184.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000048.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000120.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000131.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000001.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000087.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000129.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000175.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000077.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000191.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000010.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000169.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000113.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000109.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000102.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000034.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000006.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000171.tar.gz', 'PFRaw_PhysicsFiltering_Run00125068_Subrun00000000_00000066.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0720/8c6f58ba-f7da-40b4-b028-161338c4140c.zip
./2014/2014_7_new_gcd.slurm.e.718358:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00124982_Subrun00000000_00000092.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0701/606206f6-16d3-453d-8404-7ba7dc687d82.zip
./2014/2014_11_new_gcd.slurm.e.718361:Exception: Did not finish ['key_803125_PFRaw_PhysicsFiltering_Run00125553_Subrun00000000_00000113.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/1112/b555e0e0-47d8-4d62-8418-b3595add3f55.zip
./2014/2014_1_new_gcd.slurm.e.712512:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00123701_Subrun00000000_00000117.tar.gz', 'PFRaw_PhysicsFiltering_Run00123701_Subrun00000000_00000116.tar.gz', 'PFRaw_PhysicsFiltering_Run00123701_Subrun00000000_00000120.tar.gz', 'PFRaw_PhysicsFiltering_Run00123701_Subrun00000000_00000122.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0106/f9356653-1f8a-4037-b887-d847a088709e.zip
./2014/2014_6_new_gcd.slurm.e.718357:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00124960_Subrun00000000_00000039.tar.gz', 'PFRaw_PhysicsFiltering_Run00124962_Subrun00000000_00000004.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0625/2f1d7f86-c60c-4371-a048-5afb1223a717.zip
./2014/2014_5_new_gcd.slurm.e.725356:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000005.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000032.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000023.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000000.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000028.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000011.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000007.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000002.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000020.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000001.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000035.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000008.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000037.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000034.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000018.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000017.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000014.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000004.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000040.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000026.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000025.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000036.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000013.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000024.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000012.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000030.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000015.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000038.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000042.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000031.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000019.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000033.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000009.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000043.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000022.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000041.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000016.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000021.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000010.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000029.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000006.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000003.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000039.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000027.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0506/0d7b56ed-b84a-45e0-914b-3e4864174e96.zip
./2014/2014_5_new_gcd.slurm.e.725356:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000063.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000105.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000050.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000093.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000176.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000074.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000075.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000148.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000119.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000122.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000115.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000071.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000113.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000129.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000155.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000094.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000062.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000195.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000151.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000057.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000064.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000189.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000156.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000139.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000181.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000162.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000080.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000149.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000127.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000154.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000135.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000172.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000179.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000111.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000092.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000067.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000055.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000204.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000186.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000144.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000060.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000182.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000191.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000201.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000174.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000132.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000194.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000145.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000116.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000085.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000045.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000083.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000051.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000086.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000159.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000110.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000065.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000142.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000131.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000160.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000137.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000108.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000133.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000180.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000044.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000091.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000170.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000141.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000153.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000136.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000109.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000134.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000059.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000106.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000152.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000205.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000130.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000187.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000056.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000190.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000143.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000164.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000193.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000087.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000099.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000150.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000185.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000066.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000053.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000048.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000098.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000202.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000076.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000184.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000052.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000054.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000206.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000104.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000198.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000197.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000177.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000188.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000167.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000138.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000173.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000203.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000114.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000101.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000183.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000146.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000112.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000196.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000124.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000073.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000049.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000079.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000089.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000171.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000126.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000046.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000207.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000090.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000077.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000161.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000178.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000047.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000192.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000070.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000175.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000088.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000118.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000140.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000084.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000072.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000147.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000097.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000103.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000095.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000208.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000121.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000168.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000061.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000068.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000165.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000117.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000158.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000058.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000128.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000169.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000081.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000102.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000123.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000199.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000107.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000100.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000163.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000125.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000082.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000166.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000096.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000120.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000078.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000200.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000157.tar.gz', 'PFRaw_PhysicsFiltering_Run00124703_Subrun00000000_00000069.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0507/e925df19-a6a8-43fe-b966-3ba8be1c347b.zip
./2014/2014_5_new_gcd.slurm.e.725356:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00124754_Subrun00000000_00000114.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0516/86abca59-36d0-4c47-9f65-6dba3e870fe5.zip
./2014/2014_10_new_gcd.slurm.e.724617:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125453_Subrun00000000_00000114.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/1019/176ff40b-d882-4967-9f7b-c91fd14708a3.zip
./2014/2014_4_new_gcd.slurm.e.724618:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000089.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000056.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000103.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000037.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000101.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000058.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000047.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000203.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000174.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000173.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000175.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000068.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000187.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000007.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000013.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000193.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000126.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000166.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000184.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000002.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000022.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000135.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000150.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000127.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000160.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000009.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000161.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000205.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000172.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000151.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000140.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000171.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000114.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000005.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000133.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000180.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000148.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000129.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000176.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000186.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000049.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000097.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000061.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000006.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000111.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000138.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000115.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000017.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000185.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000199.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000071.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000192.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000110.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000095.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000035.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000183.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000081.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000165.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000154.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000032.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000143.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000091.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000078.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000043.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000158.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000077.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000001.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000018.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000112.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000082.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000211.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000099.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000094.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000164.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000010.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000132.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000163.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000038.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000059.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000012.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000057.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000042.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000182.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000041.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000208.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000030.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000169.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000060.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000181.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000107.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000021.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000015.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000027.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000085.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000093.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000053.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000040.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000090.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000137.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000083.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000152.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000170.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000052.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000177.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000034.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000128.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000100.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000106.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000075.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000045.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000036.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000213.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000167.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000050.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000210.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000116.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000023.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000212.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000086.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000194.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000074.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000033.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000207.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000113.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000048.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000055.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000080.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000142.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000124.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000204.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000216.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000155.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000145.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000003.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000088.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000200.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000070.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000149.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000201.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000008.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000117.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000179.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000067.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000141.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000105.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000215.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000026.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000202.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000020.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000191.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000054.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000162.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000069.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000108.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000016.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000197.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000188.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000096.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000024.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000189.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000104.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000156.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000168.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000118.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000139.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000029.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000076.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000120.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000198.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000072.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000144.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000153.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000064.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000062.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000044.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000131.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000119.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000063.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000066.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000092.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000209.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000109.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000000.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000178.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000102.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000206.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000046.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000190.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000051.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000014.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000214.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000039.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000079.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000157.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000025.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000195.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000065.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000196.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000087.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000028.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000011.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000019.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000098.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000146.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000147.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000159.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000130.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000136.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000125.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000031.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000123.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000084.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000134.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000073.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000004.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000121.tar.gz', 'PFRaw_PhysicsFiltering_Run00124567_Subrun00000000_00000122.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0411/751788fc-448e-4aa8-8e46-8bf181c547f4.zip
./2014/2014_9_new_gcd.slurm.e.725408:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125329_Subrun00000000_00000133.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0920/67340a07-d083-4235-9330-d050407893b0.zip
./2014/2014_9_new_gcd.slurm.e.725408:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125348_Subrun00000000_00000120.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0926/b003e953-44f1-4206-9422-a818ae53e79f.zip
./2014/2014_8_new_gcd.slurm.e.718359:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125174_Subrun00000000_00000091.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0813/5442602d-51c6-4acb-9d3f-f404e30f78c1.zip
./2014/2014_8_new_gcd.slurm.e.718359:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125216_Subrun00000000_00000131.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0822/2666ae38-a76e-42ed-a200-70d3a08744a5.zip
./2014/2014_8_new_gcd.slurm.e.718359:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125175_Subrun00000000_00000063.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0814/e68ea53c-1fba-450a-988c-2ab25da14dae.zip
./2014/2014_8_new_gcd.slurm.e.718359:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125237_Subrun00000000_00000001.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0829/a078bc69-5c87-452d-93c6-6f6048e32308.zip
./2014/2014_8_new_gcd.slurm.e.718359:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125240_Subrun00000000_00000126.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0830/f3945da7-532c-499a-8a76-f04d8dddcc56.zip
./2014/2014_8_new_gcd.slurm.e.718359:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125145_Subrun00000000_00000051.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0807/d7859abd-bd2e-4f8b-bde0-50781eb88b02.zip
./2014/2014_8_new_gcd.slurm.e.718359:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125214_Subrun00000000_00000194.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/0822/2f46939f-d329-4f2a-9a17-18edcef57d50.zip
./2014/2014_12_new_gcd.slurm.e.718360:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125737_Subrun00000000_00000038.tar.gz', 'PFRaw_PhysicsFiltering_Run00125737_Subrun00000000_00000126.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/1216/195c0087-c6a4-4624-80f3-3735cffb5569.zip
./2014/2014_12_new_gcd.slurm.e.718360:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125760_Subrun00000000_00000181.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/1222/fa0d1e0e-36d4-4ef3-831f-cae8edd8a253.zip
./2014/2014_12_new_gcd.slurm.e.718360:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125756_Subrun00000000_00000223.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2014/1221/739a140e-0577-40ae-aedf-a65c9b7c6bfe.zip

2015:

./2015/2015_3_new_gcd.slurm.e.704374:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00126158_Subrun00000000_00000114.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/0316/a04c459e-8d69-4e2f-beea-06242e765f9c.zip
./2015/2015_12_new_gcd.slurm.e.701413:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00127230_Subrun00000000_00000039.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/1211/713acf8a-a617-4b9b-9736-ff0dd4e75b27.zip
./2015/2015_12_new_gcd.slurm.e.701413:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00127238_Subrun00000000_00000092.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/1213/f1a7c422-9395-49ca-be2c-fa66a3ac2c7c.zip
./2015/2015_7_new_gcd.slurm.e.714073:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00126627_Subrun00000000_00000199.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/0717/84fe1f3a-d458-4d12-93f7-2b8d3e227929.zip
./2015/2015_7_new_gcd.slurm.e.714073:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00126555_Subrun00000000_00000125.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/0702/37516e38-dace-42a2-9c62-0f2803dde0b0.zip
./2015/2015_11_new_gcd.slurm.e.701401:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00127130_Subrun00000000_00000060.tar.gz', 'PFRaw_PhysicsFiltering_Run00127129_Subrun00000000_00000080.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/1118/2bec33fd-2973-4668-91fc-aa34784d5ec3.zip
./2015/2015_1_new_gcd.slurm.e.710710:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125792_Subrun00000000_00000078.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/0101/66d504b7-5b77-4425-8a1f-0ba1e9379700.zip
./2015/2015_1_new_gcd.slurm.e.710710:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125797_Subrun00000000_00000160.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/0103/e19d5461-de7a-4fb3-9ccd-9004e0908f99.zip
./2015/2015_1_new_gcd.slurm.e.710710:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00125801_Subrun00000000_00000080.tar.gz', 'PFRaw_PhysicsFiltering_Run00125802_Subrun00000000_00000106.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/0104/a7871eaf-9e76-4273-b49a-df921535d8b9.zip
./2015/2015_10_new_gcd.slurm.e.700600:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00127000_Subrun00000000_00000164.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2015/1020/74e03311-279d-4866-a925-a75833ebd8d5.zip

2016:

./2016/2016_1_new_gcd.slurm.e.703350:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00127356_Subrun00000000_00000150.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2016/0104/e2b7f4bf-7194-4dfd-93e1-29ee1c47807f.zip
./2016/2016_6_new_gcd.slurm.e.710840:Exception: Did not finish ['key_31596557_PFRaw_PhysicsFiltering_Run00128175_Subrun00000000_00000065.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2016/0626/67e99bd0-a307-4af4-97fe-6737bbb670c1.zip

2018:

./2018/2018_7_new_gcd.slurm.e.707167:Exception: Did not finish ['ukey_72b39c36-611a-41e2-9468-8aca17073251_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000062.tar.gz', 'ukey_a8cd8000-74f7-4d30-97c7-2733fdae44ff_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000128.tar.gz', 'ukey_9ca7474b-8236-4cef-a47c-6a7fb8e9b395_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000161.tar.gz', 'ukey_fb3c98cb-3506-487b-809a-830d785351e9_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000016.tar.gz', 'ukey_e9a676b3-0d14-4e26-819f-46b0d673de25_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000112.tar.gz', 'ukey_c1ec5743-8f12-4855-a977-01929990ed1a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000009.tar.gz', 'ukey_dc1562ef-aefa-426c-bc47-5af815c459cd_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000108.tar.gz', 'ukey_97fc67e0-3b0a-4d70-82fa-1d495ff10b96_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000199.tar.gz', 'ukey_1f30b99b-11a5-42ea-a4b0-e47bc6690267_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000084.tar.gz', 'ukey_bb5a0fcf-a20f-479d-b7d5-7bd9ff1d3217_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000166.tar.gz', 'ukey_ed9a181e-4ad4-4431-a8f9-693efe59aa1b_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000194.tar.gz', 'ukey_91d56675-a1d8-4b5f-aebf-5366c3f5bdda_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000169.tar.gz', 'ukey_e9e4791d-3fd0-4e3d-beeb-204118d9ebea_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000191.tar.gz', 'ukey_7d947f63-be8f-4270-8daa-c91e347013c7_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000162.tar.gz', 'ukey_46b62e7a-ada8-418e-a213-647deb27c372_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000048.tar.gz', 'ukey_f0257a92-30ab-4207-a773-17a04db0f2b7_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000136.tar.gz', 'ukey_74f233c2-e705-4efd-8209-22207a602c48_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000171.tar.gz', 'ukey_4ab3f263-dfe2-474c-84a7-a8965213131e_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000029.tar.gz', 'ukey_a181269c-e00c-422a-8dd8-1fce288a142e_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000087.tar.gz', 'ukey_1935cd3d-bf31-4bb8-bf18-96ccf6ecbc74_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000145.tar.gz', 'ukey_dea89463-6ffb-453d-8158-806358cc94e8_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000122.tar.gz', 'ukey_6efb88dc-92a4-46f3-809f-1a440b0d9515_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000033.tar.gz', 'ukey_8bd248a7-5d6d-452b-8364-ac7eb1adc86e_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000034.tar.gz', 'ukey_fde591a2-cba9-488a-bce2-a7845e5070c5_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000196.tar.gz', 'ukey_2e3ac484-35f6-48aa-b8bf-d411cbcb0eb7_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000042.tar.gz', 'ukey_4debf80c-a327-4bec-9342-24892710ec56_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000055.tar.gz', 'ukey_05690c9d-1612-438f-96b2-e1a37072eb59_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000142.tar.gz', 'ukey_997f7f98-45b7-44c7-81cd-8f6fb5fc079f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000070.tar.gz', 'ukey_2e069fb4-f113-4ef4-9583-28241b5963bb_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000014.tar.gz', 'ukey_0c66dfa2-81e5-4193-8c3a-02d9828912d4_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000071.tar.gz', 'ukey_f05f3436-6edc-4a6b-a42d-6f89f408b36f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000035.tar.gz', 'ukey_c6589018-fadb-4a89-af0c-58d2bb38728f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000065.tar.gz', 'ukey_a353c053-d649-4fe5-a98e-39ab5191c985_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000019.tar.gz', 'ukey_0aee29eb-8803-4a6f-ac25-d31d567915c0_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000140.tar.gz', 'ukey_9ac9c38e-c133-444b-83c9-80dd83e65c41_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000130.tar.gz', 'ukey_e7d6308f-378a-498c-a149-0e81a9269b7d_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000002.tar.gz', 'ukey_4ba7b93d-a9b0-4065-9316-9dec845d39dd_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000185.tar.gz', 'ukey_6baac85f-a1a1-4381-b48a-bf7e15e9c2a1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000001.tar.gz', 'ukey_a57b18c7-3ce5-4a78-af2e-9889afdf3f94_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000132.tar.gz', 'ukey_13c95419-7a43-40f0-8db3-d62d0d8fb1d0_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000149.tar.gz', 'ukey_82235a0d-3ef4-4afb-a924-276c8fabc65d_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000015.tar.gz', 'ukey_8943fcfa-bd82-4f06-a9a1-0549dd8dabe4_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000043.tar.gz', 'ukey_706a4567-4ca2-45b5-96c9-fe505cc353bc_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000124.tar.gz', 'ukey_038d393b-b37a-4529-893e-e826140ba470_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000177.tar.gz', 'ukey_30b8d5ad-a347-4350-ab0e-736fc4400eef_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000011.tar.gz', 'ukey_ee0eea4b-8d7a-4c86-9e3d-9e575991a8d1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000125.tar.gz', 'ukey_956f4754-9d6f-4125-9821-60104774ea6c_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000075.tar.gz', 'ukey_e6f68975-9ec1-47e8-accf-d7ccd10ec8d1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000010.tar.gz', 'ukey_68a75d54-def9-48a3-a7c3-254baa6cd9de_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000126.tar.gz', 'ukey_d609698b-9e0b-44b3-8fd5-0e4911df2050_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000036.tar.gz', 'ukey_0f01b26b-d6c0-4d44-9e76-201bb5b04552_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000183.tar.gz', 'ukey_b3da319a-fa92-471c-b99b-5e3001a605a1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000051.tar.gz', 'ukey_dd278403-a1ce-4372-a90c-afffc3d48610_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000195.tar.gz', 'ukey_3f59139f-50ff-4e12-b772-a5bf901bc856_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000186.tar.gz', 'ukey_fdcd7e53-c51a-4ca4-ad89-5d9e91bca718_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000167.tar.gz', 'ukey_ce45106a-9fa1-4360-97ac-391ecf37881f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000114.tar.gz', 'ukey_2ee4870d-12dc-47ac-a345-efd962e6cbff_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000054.tar.gz', 'ukey_ad0b97e2-cb01-46f1-8867-8dde55fe4f83_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000049.tar.gz', 'ukey_b03ec143-83b7-453a-a91c-a973aa5e9caa_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000024.tar.gz', 'ukey_ce374ead-6303-449a-a771-e16c922211d1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000037.tar.gz', 'ukey_f2684668-dcc8-41cd-8c6c-1f3299941ff2_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000080.tar.gz', 'ukey_32558731-5bff-4923-a3a5-5dd7da753cae_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000175.tar.gz', 'ukey_d95c0b3b-b7ff-4033-be0c-6a49d625cb1c_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000004.tar.gz', 'ukey_8dfaea07-40bc-48d4-b499-39c1a5942e99_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000121.tar.gz', 'ukey_0233c6e3-f25e-46e6-9e5f-9da339b39f78_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000102.tar.gz', 'ukey_c3261912-a849-4cef-a7fb-0ebfc9b71375_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000093.tar.gz', 'ukey_18caf041-e693-4a73-a510-7f85c5efb684_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000066.tar.gz', 'ukey_0a61304e-f76a-4283-9fa3-40ea67191d09_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000096.tar.gz', 'ukey_082ca3c7-71a9-472a-883c-bba1b3e2c1b0_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000178.tar.gz', 'ukey_e937fbf0-6132-4844-a4d0-1fa36fcb59e5_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000090.tar.gz', 'ukey_52b3295d-c299-44be-867c-1405c162e7db_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000170.tar.gz', 'ukey_58586c45-7a33-4fb6-98eb-6b77a1a77083_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000023.tar.gz', 'ukey_09cef292-3fa0-41bd-a790-7a43c7a3c567_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000131.tar.gz', 'ukey_ec829db4-9b0e-4123-a75d-c7ae6afa2524_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000157.tar.gz', 'ukey_4044cff5-7889-4443-bb32-fa595bd76d7a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000104.tar.gz', 'ukey_bda19153-4018-41e2-8327-8e3076609791_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000030.tar.gz', 'ukey_129b4127-3af6-415a-9b9e-484a18e36ec2_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000046.tar.gz', 'ukey_374a0beb-76c2-4451-94f3-f58c28391bf5_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000123.tar.gz', 'ukey_e92b3ce2-03b3-4d33-b9cb-825e48d5d8db_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000017.tar.gz', 'ukey_90f72986-dbcb-4a75-90ce-4d2c8367995a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000176.tar.gz', 'ukey_f65e37d3-2b0c-42ae-b730-cd96b8bddcc0_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000022.tar.gz', 'ukey_26ed411d-bc19-49da-95bf-a3c4478b394d_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000094.tar.gz', 'ukey_6fc262b4-5d9b-437d-9828-b834afcf24ab_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000007.tar.gz', 'ukey_3e48b5c3-a349-46cf-8219-d005bbb63b45_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000059.tar.gz', 'ukey_fea99f9f-dcb4-4aed-ac86-6e4efee0895a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000135.tar.gz', 'ukey_87d86d86-7636-4b2c-bebc-4103501bd1b6_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000058.tar.gz', 'ukey_631cafa3-40d5-4c18-8449-a04ce73f42f1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000032.tar.gz', 'ukey_e27675ca-ef23-4075-8a5c-caa0c599ce78_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000044.tar.gz', 'ukey_cd7f569c-6123-4660-af66-c64182065abd_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000003.tar.gz', 'ukey_077a4ddf-d8e8-47a3-9091-b88712cb0222_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000083.tar.gz', 'ukey_4b0004a9-9e12-48b0-a40f-8a0db9d4f357_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000067.tar.gz', 'ukey_0d57958b-5986-436d-bda8-5bfc12836056_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000173.tar.gz', 'ukey_2ef71282-f566-4f9f-a442-6ef25a7a0263_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000052.tar.gz', 'ukey_0023903b-e26c-4fd5-94de-73fc82adef30_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000088.tar.gz', 'ukey_6e8604ba-24ee-4fec-afb0-9ed10a4dd59f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000134.tar.gz', 'ukey_6ff6c244-c5bd-45e4-9e6e-aa839edfbd22_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000021.tar.gz', 'ukey_b3d3081c-1043-4202-a37f-dd5cb453c110_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000100.tar.gz', 'ukey_8c7a3e9e-b266-4460-b0d1-b2cbdabf7e60_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000174.tar.gz', 'ukey_905573d0-4375-42a9-867e-a78f82775ecf_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000190.tar.gz', 'ukey_359bec29-139c-4bbd-aa7d-4b9b20a667f5_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000008.tar.gz', 'ukey_a4890baa-2fdd-437e-8bc6-bf2190b8152e_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000147.tar.gz', 'ukey_870a0656-8102-4e74-910f-65219133e3c4_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000138.tar.gz', 'ukey_2bdbc905-8348-41cb-9b50-13d759f63642_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000074.tar.gz', 'ukey_e34da7df-ac3a-42f4-a335-79b66226d10a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000120.tar.gz', 'ukey_230145e5-47ae-4fda-bebf-c434380ec8c8_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000110.tar.gz', 'ukey_9e05ed36-ceaa-4f20-9629-ea939c44d882_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000006.tar.gz', 'ukey_ae21546b-ba4c-4fe1-ac72-0bd320ab4185_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000116.tar.gz', 'ukey_fd94bd12-feaa-465f-9e9e-d273d71912b7_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000085.tar.gz', 'ukey_78698f05-09c5-4caa-aac9-fc738ea30e1d_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000097.tar.gz', 'ukey_2bcd655f-5cbd-4693-93d1-1d92e781b2c1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000137.tar.gz', 'ukey_ee0c8124-38ae-4be0-814b-ef4adc065255_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000139.tar.gz', 'ukey_2f9d4065-48ea-4d27-9358-2c31e58caf38_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000168.tar.gz', 'ukey_32e407ee-7455-48ce-a26c-6fde885bdbeb_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000146.tar.gz', 'ukey_2b254da6-e464-461c-9aec-34627b56694d_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000040.tar.gz', 'ukey_24ac1562-0f41-4735-b33e-d59a49e14772_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000129.tar.gz', 'ukey_42756ff6-61ad-49e9-88c9-ce57342dfea2_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000050.tar.gz', 'ukey_8f895fa2-cbc9-49ce-9bad-869e31d65284_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000091.tar.gz', 'ukey_eeaed1bb-298c-4e57-9592-02c05d67077b_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000188.tar.gz', 'ukey_a569f753-7901-4796-8054-58b015b6c48c_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000181.tar.gz', 'ukey_01989310-b457-4185-8ae5-d2aebae458b6_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000193.tar.gz', 'ukey_7b0de838-b923-4f40-86e9-029a8c63340a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000182.tar.gz', 'ukey_3d386a80-aac5-4d7f-b5b4-00629c1eae1d_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000187.tar.gz', 'ukey_959dd41e-21ec-4104-88b2-2453ccc5fa4f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000082.tar.gz', 'ukey_71d33835-bf60-43ba-aaf2-94c41faecb85_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000061.tar.gz', 'ukey_778d9b6f-8fea-4710-bc28-28f9fd180c28_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000154.tar.gz', 'ukey_c95a16f5-758d-4535-92bd-367d642fe299_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000101.tar.gz', 'ukey_fd2c1b14-4d18-4bcf-a3e9-81decbb5bfbb_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000047.tar.gz', 'ukey_1044eea3-34f1-45af-ae89-abf73fb8a953_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000099.tar.gz', 'ukey_3dd8e203-7d04-4029-8bbf-c1331d95b937_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000045.tar.gz', 'ukey_fcf94a95-b786-4193-9a5d-ea7efd7ec487_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000079.tar.gz', 'ukey_a3dcacf2-25db-4fd4-9ff7-48cc898de54c_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000143.tar.gz', 'ukey_fb0ec57a-3dc0-4358-b003-969b25085556_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000200.tar.gz', 'ukey_617f1ea3-bfbe-4d99-879e-15fcadfc00b7_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000092.tar.gz', 'ukey_e58a49ad-1d7c-4c38-b37d-a4dbb2d200e0_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000202.tar.gz', 'ukey_474544ee-69ed-4b0b-a6d5-17314751ac1f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000150.tar.gz', 'ukey_fbc26733-6572-481d-9f39-ecaa9c860dca_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000113.tar.gz', 'ukey_86a55e7d-b24e-410b-a394-2786ff6f6062_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000089.tar.gz', 'ukey_7a5d5044-eb4f-4b5d-b7d6-6c459634eda7_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000159.tar.gz', 'ukey_8a18ff56-e456-44db-bb8a-01f7183716d8_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000000.tar.gz', 'ukey_d60aa9e2-078c-4cce-9844-120bc22d8ae9_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000072.tar.gz', 'ukey_5aa37c2c-cbf1-4bf6-9d21-0f3a27e10879_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000115.tar.gz', 'ukey_8d71adcf-dee3-4d1f-b0bd-fd59143877d4_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000109.tar.gz', 'ukey_99095bc2-f472-443a-9667-30b8cd8b6447_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000148.tar.gz', 'ukey_c16eb529-c6bd-422d-b412-db85e664efd1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000041.tar.gz', 'ukey_dc424dfc-cf40-4f7f-a2a3-8fbf92653d35_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000060.tar.gz', 'ukey_251e1db9-35e3-4c4c-b053-cc63373e11ab_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000005.tar.gz', 'ukey_c03a3f0d-7b0d-4e24-a5da-81cd396787c0_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000064.tar.gz', 'ukey_ec545206-b780-4b99-8ff2-26c8ec091fed_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000141.tar.gz', 'ukey_9aa6da0a-9fc0-4411-b72b-bfbbc16f46a1_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000103.tar.gz', 'ukey_c3d9b5ee-9c72-4186-b3e0-15563cda8704_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000164.tar.gz', 'ukey_079bc66c-066e-4b2f-8973-11718acb9b88_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000119.tar.gz', 'ukey_783b4d4a-0ee3-4f8e-b35d-cbe4bc84fb7c_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000155.tar.gz', 'ukey_9bc33790-6ffc-435f-8816-0a4027eb1e97_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000027.tar.gz', 'ukey_47075de3-c905-4cae-aa9f-621f74b27c58_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000038.tar.gz', 'ukey_fa8e13e9-1b0b-4a63-b536-4dcf7bf728f4_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000039.tar.gz', 'ukey_fc7377c1-112c-4c56-be41-32ad84cc8ca2_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000189.tar.gz', 'ukey_86c93faa-f926-42d9-94bf-6caf199614ae_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000106.tar.gz', 'ukey_bec70211-75ac-45ad-b6d8-bacc79e4b98e_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000156.tar.gz', 'ukey_ab69dc8e-0a39-435d-b7b8-8fe4de99263b_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000160.tar.gz', 'ukey_deca1944-8860-46e4-9ca8-47d4b5735994_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000179.tar.gz', 'ukey_d249ddec-80e1-417f-a0de-52f2eda0032a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000201.tar.gz', 'ukey_47b22ff5-f3f3-4471-b37d-8ddcb74dda7a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000111.tar.gz', 'ukey_12a29674-33bc-4833-8be1-21fbbd413a3a_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000086.tar.gz', 'ukey_388bfaff-4cda-40c3-9a28-ae03000a63c2_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000144.tar.gz', 'ukey_1727062c-74de-4813-b333-a9c3819abcaa_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000152.tar.gz', 'ukey_bf689c18-0057-4c5b-956c-77dcad9a7ef9_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000151.tar.gz', 'ukey_e75d347c-d964-404c-a113-248d2a827625_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000081.tar.gz', 'ukey_fd7919e5-7cba-466d-963b-139417ba850f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000192.tar.gz', 'ukey_dda95544-22ac-4ff9-824a-5a0c051da09c_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000031.tar.gz', 'ukey_464b7616-ba61-4c3f-a7a1-a4e7c2340d92_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000020.tar.gz', 'ukey_9ed94f0e-8ac3-480d-a4a0-4cbc5cd22da9_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000028.tar.gz', 'ukey_c851573a-d735-43ff-b780-e94f8885b5ea_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000158.tar.gz', 'ukey_679feb73-8485-48bd-bb8d-d8ab5a1f9044_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000095.tar.gz', 'ukey_0d997687-f8a6-420a-8c11-3c445eec3c04_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000012.tar.gz', 'ukey_15ee795d-e68e-43fe-87d2-f68ee06848ff_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000184.tar.gz', 'ukey_68bc42d2-27cc-4668-b4bb-50dc7686e1f9_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000163.tar.gz', 'ukey_947a0305-a203-467d-8f22-bf6abdf0e017_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000078.tar.gz', 'ukey_d97257f8-ae3d-4092-a0d7-ea62363f2361_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000073.tar.gz', 'ukey_692940b1-c67d-4ed6-aca3-f9fc7f560ca4_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000069.tar.gz', 'ukey_f397a8bd-5ea2-400b-925a-959d5d54d894_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000133.tar.gz', 'ukey_968d6e70-d416-4c3b-8eef-42dfab1efdba_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000197.tar.gz', 'ukey_16ffe06c-41fc-4417-966c-007de79a4774_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000117.tar.gz', 'ukey_25c23c32-ae3b-457d-95a7-77e12c667115_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000105.tar.gz', 'ukey_1e9c31be-daa8-4b36-9f81-c8ee34f01c36_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000172.tar.gz', 'ukey_0cdcf2bb-f484-4ef0-9101-e0a8ed3a0bf6_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000056.tar.gz', 'ukey_df28b992-811a-4394-9970-acd183020367_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000165.tar.gz', 'ukey_e5a8af57-3051-4b27-a10a-f1728fdd4733_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000026.tar.gz', 'ukey_4d3731b5-ceb0-4667-8838-1d34ff161268_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000025.tar.gz', 'ukey_dfd7b125-e876-4f74-9187-cfb6f4fe8cce_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000127.tar.gz', 'ukey_778b46b8-55f1-488f-b5d8-3de1d61f7560_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000018.tar.gz', 'ukey_699aaacd-14b4-4deb-85a5-f84cd0977dc4_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000098.tar.gz', 'ukey_d2014ad2-72d4-49ed-a52c-cbc27d7136d3_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000068.tar.gz', 'ukey_8c8ec1f2-47f8-4363-8291-233238aabeff_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000118.tar.gz', 'ukey_9ba296bd-1c55-4865-abbf-c5db35f046ea_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000107.tar.gz', 'ukey_3c66195f-076f-4e53-b084-e18813dfe5b2_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000153.tar.gz', 'ukey_b344aab3-11e2-4c7a-ab46-ac1ed11cceae_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000180.tar.gz', 'ukey_a0b430d4-6675-4b8f-9b85-06594f3b188d_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000057.tar.gz', 'ukey_0a13312d-76e3-4c19-8f50-3033c18f6050_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000053.tar.gz', 'ukey_bcc0d90a-8fbd-4e48-8c45-87bc472f00e4_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000077.tar.gz', 'ukey_2c7e3b1a-f80c-4b7b-acdb-684374a73a2f_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000198.tar.gz', 'ukey_1e6e976e-1dbf-452b-8a65-e089e9b9617b_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000076.tar.gz', 'ukey_6db69b50-d381-4826-a8a1-56ded9232b21_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000013.tar.gz', 'ukey_8c67d5ae-ee6c-4a5a-bc72-b2e2dd8b604c_PFRaw_PhysicsFiltering_Run00131263_Subrun00000000_00000063.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2018/0710/1fe5b76f-7dff-4e40-8e92-093b0e38aec4.zip

2019:

./2019/2019_9_new_gcd.slurm.e.559034:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00133046_Subrun00000000_00000113.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2019/0909/a6a39f10e5a311ea84e646aae246590a.zip

2021:

./2021/potential_final/2021_7_new_gcd.slurm.e.538594:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00135518_Subrun00000000_00000103.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2021/0722/18da3014e22f11ecbeffdab892edaf5f.zip

2023:

./2023/2023_8_new_gcd.slurm.e.782755:Exception: Did not finish ['PFRaw_PhysicsFiltering_Run00138217_Subrun00000000_00000205.tar.gz'] from bundle /scratch/04799/tg840985/tmp.v2/2023/0803/ec3f91b2019d11efa4a01eb515fa4a53.zip

these were extracted using this command:

find . -type f -name "*2023*.slurm.e.*" | awk -F. '
{
    # Save the full original path
    full_path = $0

    # Get the trailing number (the very last field)
    num = $NF

    # Reconstruct the stub by removing the last number
    # (sub replaces the last match of .followed by numbers with nothing)
    stub = full_path
    sub(/\.[0-9]+$/, "", stub)

    # If this is the first time seeing this stub, or if the current
    # number is higher than what we have saved, update it.
    if (!(stub in max_num) || num > max_num[stub]) {
        max_num[stub] = num
        max_file[stub] = full_path
    }
}
END {
    # Print out the winning file for each stub
    for (stub in max_file) {
        print max_file[stub]
    }
}' | xargs grep -E "Exception: Did not finish \[.*\] from bundle /scratch/04799/tg840985/tmp\.v2/.*"

Finding the missing files

There are gaps in the bundles, i.e. for run 123456 you don't have file number 10, 74, 200. Not sure where these files went, at least they are not in the bundles or the manifest for the bundle. This script will look through the manifests and look for these missing file numbers

import json
import re
from collections import defaultdict
from pathlib import Path

def map_and_check_runs(directory_path, file_ending=".json"):
    if not file_ending.startswith('.'):
        file_ending = f".{file_ending}"

    run_mapping = defaultdict(set)
    regex_pattern = re.compile(r'Run(\d+)_Subrun\d+_(\d+)\.')
    target_dir = Path(directory_path)
    
    if not target_dir.is_dir():
        print(f"Error: The directory '{directory_path}' does not exist.")
        return None, None

    print(f"Searching for '*{file_ending}' files in: {target_dir.resolve()}...\n")

    # 1. Parse files and build the mapping
    for file_path in target_dir.rglob(f"*{file_ending}"):
        with open(file_path, 'r') as f:
            try:
                data = json.load(f)
            except json.JSONDecodeError:
                print(f"Warning: Invalid JSON in -> {file_path}")
                continue

            file_entries = data.get("files", [])
            for entry in file_entries:
                file_name = entry.get("fileName", "")
                match = regex_pattern.search(file_name)
                
                if match:
                    run_num = int(match.group(1))
                    file_num = int(match.group(2))
                    run_mapping[run_num].add(file_num)

    final_mapping = {run: sorted(list(files)) for run, files in run_mapping.items()}
    
    # 2. Check for continuity
    discontinuous_runs = {}
    
    for run, files in final_mapping.items():
        if not files:
            continue
            
        # If the difference between the max and min file numbers doesn't match 
        # the total number of files minus one, there is a gap.
        if max(files) - min(files) != len(files) - 1:
            # Generate the full expected sequence from min to max
            expected_sequence = set(range(min(files), max(files) + 1))
            # Find the difference between the expected sequence and actual files
            missing_files = sorted(list(expected_sequence - set(files)))
            
            discontinuous_runs[run] = {
                "present_files": files,
                "missing_files": missing_files
            }

    return final_mapping, discontinuous_runs

# ==========================================
# Example Usage
# ==========================================
if __name__ == "__main__":
    my_directory = "./my_data_folder" 
    my_file_ending = ".json" 
    
    # Unpack the two returned dictionaries
    all_runs, bad_runs = map_and_check_runs(my_directory, my_file_ending)
    
    if all_runs is not None:
        print("--- Discontinuous Runs Found ---")
        if bad_runs:
            for run, details in bad_runs.items():
                print(f"\nRun {run} is missing files!")
                print(f"Missing file numbers: {details['missing_files']}")
        else:
            print("All runs have perfectly continuous file numbers!")
            
        # If you still want to see the full mapping of all files, you can print it here:
        # print("\n--- Full Run Mapping ---")
        # print(json.dumps(all_runs, indent=4))