Retrieving MIPP data from Enstore and dCache

Enstore is the tape storage system at Fermilab. It can be accessed directly or through the dCache system. dCache provides disk cache for frequently used tape files as well as a \emph{volatile} (not tape-backed) disk area.
Getting data from enstore is quite simple. The MIPP DAQ system is writing data to /data/0 or /data/1 partitions on e907daq.fnal.gov. From there files get moved to Enstore regularly, and files that have been moved are deleted in order to make room for new data.
Reconstructed data is also stored in Enstore. Some intermediate test jobs write data to volatile dCache.

On e907ana cluster

If you are looking for a recent run (a few days old), then the file should still be in either /data/0 or /data/1. If you intend to keep the file on the cluster for a while, then copy it to one of the /disks/X directories.
Please do not copy files to /home partition!

If the run you are looking for is not in /data/0 or /data/1, then you will have to extract the file from Enstore. Make sure that /usr/local/bin is in your PATH environment variable

You have several options. The first is prefered onsite: \emph{dccp} leaves a copy of the file in dCache. \emph{ENSgetRun.pl} is a Perl script that retrieves all sub runs for a given run. \emph{encp} allows you to use wildcards, etc. For all intents and purposes, encp works just like cp when operated on PNFS file system.
Again, please do not copy files to /home partition! If /home fills up, neither you nor anyone else can continue to work.

Through kerberized FTP

If you would rather get files directly, bypassing e907ana cluster (suppose you want to do processing off-site!), then kerberized FTP is the only way as of this writing to get files.

MIPP users' kerberos principals (user names) should be added to Enstore configuration to allow access to our data. If your principal is not added, file a helpdesk ticket. Contact Holger Meyer or Jon Paley if you continue to have trouble.

Once you obtain a kerberos ticket (kinit -f <user name>), you should be able to get read only access by executing
> ftp fndca1.fnal.gov 24127
Connected to stkendca3a.fnal.gov.
220 Kerberos FTP Door ready
334 ADAT must follow
GSSAPI accepted as authentication type
GSSAPI authentication succeeded
Name (fndca1.fnal.gov:alebedev): alebedev
200 User alebedev logged in
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
You should use your kerberos principal (user name) in order to get access. Once you are in, use the following sequence of commands:
Fndca FTP server used to accepts Java style wildcards, so to get all sub runs of a particular run, execute
mget mipp00013200.000..raw
As of Nov, 2005, shell-style wildcards, i.e. '*' works (at least for me). Go figure.

Useful hint

If you would like for ftp to log you on automatically, add this line to ~/.netrc file (create the file if you don't have one):
machine fndca1.fnal.gov login your-kerberos-principal

Directory structure

It will probably be valuable to get files processed through the farm. This data can be obtained the same way. Directory structure is the following:
reco<round>/<site>/root/NN/mipp000NNNNN.SSSS.root
contains output of raw2root (MIPP ROOT file) from a given processing round (use the latest-greatest), from a given site (FNAL or LLNL). 'NN' refers to the first 2 digits of the run. For example, run 12345, subrun 0 processed at LLNL during round 2 will live in
reco2/LLNL/root/12/mipp00012345.0000.root
Similarly, 
reco<round>/<site>/histo/...
contains the histgorams from pass 1,
reco<round>/<site>/histo2/...
contains the histograms from pass 2, etc.

FYI, typically odd files will be processed at LLNL, even files at FNAL. LLNL files will appear in Enstore with a (potentially significant) delay. All new passes are processed at FNAL only.

volatile dCache

Full details on dCache and its usage in MIPP can be found in MIPP-DocDB document 318 (pdf). You may not need to copy the data! You may be able to access it directly in dCache.

Created by Andre Lebedev Nov 10, 2005
Updated by Holger Meyer June 2, 2008