Grid-compatible MIPP Software

With Open Science Grid being "the way of the future" computing, it is advantageous to have MIPP software that can run on the grid. This page primarily focuses on documenting scripts in BatchProc/grid.

Grid environment

One can assume a reasonable flavor of Linux, but beyond that one cannot make any assumptions about what is available on a node! Therefore, one needs a release that packages all the necessary libraries. However faulty, we take the approach to bring in all libraries on which our software depends.

make-grid-rel

Once a MIPP release is compiled, one can create a release to be used on the grid with make-grid-rel script. Most recent releases will be kept here. Frozen releases should also be put into Enstore (/pnfs/e907/releases).
By default, symbols will be stripped from release libraries -- this shrinks the file size by a factor of 2. This will make it impossible to debug software, so if you want useful core dumps, run make-grid-rel --no-strip.

How portable is "grid" release?

The answer is not portable at all!
Despite the fact that most system libraries are packaged into release, it does not port very well between different flavors of Linux. Release created on FermiLinux 4.2 (kernel 2.6.9) could not run on FermiLinux 3.x (kernel 2.4.21) or on Fedora 5 (kernel 2.6.17). Most tools depend on glibc, so it's tough to port code. Therefore, it is highly recommended to compile release on the same flavor of Linux that it will be running on.
Note: if you know how to make release more portable, please let me know or update make-grid-rel.

Environment variables

One can count on the following evironment variables being set on a grid node:

OSG_APP - application area, where release is to be unpacked. This area is typically shared by nodes on the same grid site, so release needs to be unpacked only once.
OSG_WN_TMP - temp working area. This area is typically locally mounted, so data files should be brought here and results saved in this area until they are shipped to Enstore. User is responsible for cleaning the area up.
OSG_GRID - grid software utilities. One would typically source $OSG_GRID/setup.[c]sh to pick up binaries like srmcp, globus-url-copy, etc

Grid scripts

grid-ana

Design of MIPP grid scripts is such: most jobs will be started through grid-ana. This script does the following:

Creates temporary directory $OSG_WN_TMP/<job name>.<run>
Sources grid setup script ($OSG_GRID/setup.sh)
Sets up environment to access mass storage

E907_SRM_BASE=srm://fndca1.fnal.gov:8443//pnfs/fnal.gov/usr/e907
E907_IN_DIR -- (relative to e907 Enstore directory) passed by job submitter; input files are taken from here
E907_OUT_DIR -- passed by job submitter; output files are saved there

Imports release to be run into $OSG_APP/mipp from mass storage releases. If release is not there, tries wget from MIPP page.

development release is an exception: it is brought into temporary directory created in step 1 and always fetched through wget

Sources release setup script
Verifies that it can connect to the MIPP database
Executes a binary (typically another script) requested by user
Removes temporary directory created in step 1.

gridPass

gridPass is one of the scripts that is designed to be called by grid-ana. It can run passes through the data that do not require ROOT output to be saved, but do require that histograms and calib DB constants be saved, and a log made in corresponding batchproc DB table.
Currently, pass1, calign, and pass2 are to be executed this way. The script does the following:

Verifies that environment is correctly set up

E907_IN_DIR and E907_OUT_DIR are set
XML file name is recognized

Queries the number of subruns for that run from DB
Copies subruns from mass storage to local folder
Runs anamipp
Collects statistics about how long the job took, size of histogram file and logs to batchproc DB
Copies histogram file to mass storage if anamipp didn't crash

gridSubmitPass

The script is installed as a binary, but it is intended to be run from BatchProc/grid directory, where gridPass.cndr is found.
It requires 3 arguments

Software release name (eg R07.01.11_Linux2.4_gcc3.2.3)
Pass name (pass1, calign, or pass2)
Run number or file with run numbers

It is recommened to run one short job to make sure that everything runs correctly before submitting everything

gridSubmitPass will redirect job log files to ~/scratch/log/<pass>-<month><day> directory. It will refuse to submit jobs if that directory does not exist.
Two more options are found in the body of the script, since they don't change that frequently:

Input directory in Enstore (set to reco3/FNAL/root)
Output directory (set to reco3/FNAL/$pass)

Note that run/1000 is appended to directory name by gridPass, so make sure the right directory structure exists before you submit a pass. Otherwise, given the fact that files are not overwritten by srmcp, you will not have any histogram files from the pass.

gridUserJob / gridSubmitUserJob

Similar to gridPass and gridSubmitPass, except no log in batchproc DB will be made, and by default histogram files are not saved. Any xml file in the release can be run.
Ideally, histogram files should go to volatile dCache area

srm://fndca1.fnal.gov:8443//pnfs/fnal.gov/usr/fermigrid/volatile/mipp

Choosing grid sites

Selecting which site the job goes to is done by setting GlobusScheduler in condor job description (eg gridPass.cndr).

fngp-osg.fnal.gov/jobmanager-condor will send the job to GP farm
fermigrid1.fnal.gov/jobmanager-cemon will send the job to GP farm, CDF, D0, or CMS -- whichever has free slots

Other sites can be identified by going here. At this time, I don't know how to find out the jobmanager name. Sending to atlas.iu.edu/jobmanager worked.
Before you send your jobs to a job manager, find out how many slots you may get. On GP farm, MIPP quota is 100 as of Jan, 2007. Other sites may limit the number of jobs that VO's can run simultaneously.

Andre Lebedev - January 13, 2006