Difference between revisions of "Spring 2016 Tape Notes"
From GlueXWiki
| Line 9: | Line 9: | ||
Managing our cache disk usage ourselves during production running will not be practical, since we will be running three separate workflows from three accounts simultaneously: | Managing our cache disk usage ourselves during production running will not be practical, since we will be running three separate workflows from three accounts simultaneously: | ||
| − | + | # Monitoring of data as it hits the tape | |
| − | + | # Calibration of data | |
| − | + | # Initial reconstruction of full runs | |
Revision as of 15:04, 3 March 2016
2016/3/3
We are seeing the following behavior from the first two calibration runs over "production" data: - If we cache all files from tape beforehand using jcache, we get reasonable performance. We have to be sure to only cache enough data that will fit on the disk - If we only cache the first file from each run, then processing is much slower, and we get hit by a "long tail" of jobs which seem to take forever to load the files from tape (multiple days).
Note that we are also running monitoring jobs over the data as it comes in.
Managing our cache disk usage ourselves during production running will not be practical, since we will be running three separate workflows from three accounts simultaneously:
- Monitoring of data as it hits the tape
- Calibration of data
- Initial reconstruction of full runs
