Processing Data on FNAL Farm

Overview

Enstore & dCache

A job running on the farm can pull data from Enstore and put files back through dCache. MIPP jobs are using dccp for this (alternatives are gridftp, srmcp). Authentication is achieved through x509 DOE certificates.
[Link on how to get and install a certificate]
A proxy can be created through

voms-proxy-init -hours 7000 -voms fermilab:/fermilab/mipp

then grid jobs can be submitted and they will use that proxy to authenticate to dCache.

Condor

Since FNAL has condor that interacts with grid, jobs are submitted as user e907, but are then executed as user mipp. If you want to find out status of your jobs, use the following command

condor_q e907 [-direct schedd]

Add direct schedd if you do not get meaningful results without it, which happens frequently as of Jul 2006.

Before you begin a Round

Courtesy e-mail before each pass

Send a courtesy e-mail to enstore-admin@fnal.gov, dcache-admin@fnal.gov, and farm-admin@fnal.gov notifying them of plans to process data. They will appreciate estimates of data being pulled from and pushed back onto Enstore.

Enstore Directory structure

Convention is that processed MIPP data lives in /pnfs/e907/recoN, where N is the processing round (a round is the entire process from raw2root to DST's). Before starting a round, execute the following sequence of commands from e907anaX machine:

mkdir /pnfs/e907/recoN
mkdir /pnfs/e907/recoN/{FNAL,LLNL}
mkdir /pnfs/e907/recoN/{FNAL,LLNL}/{root,histo1,histo2,root3,histo3}
cd /pnfs/e907/recoN/FNAL
cd root; enstore pnfs --file-family root; cd -
cd root3; enstore pnfs --file-family root3; cd -
cd histo1; enstore pnfs --file-family mipphisto; cd -
cd histo2; enstore pnfs --file-family mipphisto; cd -
cd histo3; enstore pnfs --file-family mipphisto; cd -
cd /pnfs/e907/recoN/LLNL
cd root; enstore pnfs --file-family root; cd -
cd root3; enstore pnfs --file-family root3; cd -
cd histo1; enstore pnfs --file-family mipphisto; cd -
cd histo2; enstore pnfs --file-family mipphisto; cd -
cd histo3; enstore pnfs --file-family mipphisto; cd -
cd /pnfs/e907/recoN
mkdir {FNAL,LLNL}/{root,root3,histo1,histo2,histo3}/{11,12,13,14,15,16,17}

(Repeat the process for /pnfs/e907/recoN/LLNL). This will create all necessary directories and label the with different Enstore file families. File family makes sure that different files go to different tapes, making access to files more efficient when files are read back from tape. In my opinion, separating root files, histo files and pass3 root files should be sufficient. To find out file family of a given directory, use the following command inside PNFS directory

enstore pnfs --file-family

By default subdirectories inherit file family of directory they are created in.

Andre Lebedev - Mar 27, 2006