Near-line archive (NLA)
Sometimes we keep data on tape only. This is used for very voluminous data sets such as Sentinel.
Whilst much of the CEDA archive is stored on disk we do not have the storage capacity to keep it all online. We therefore migrate many data sets on to tape. If the data you require is held on tape then you can use the Near-line archive (NLA) tool to request the files are restored to disk.
Accessing the tape archive from JASMIN
1. Requesting a tape quota
Firstly, you need to request a tape quota by emailing the CEDA Helpdesk.
2. Downloading the tool (on JASMIN)
On a JASMIN server, download the command-line interface from the GitHub NLA, by typing:
git clone https://github.com/cedadev/nla_client
As well as the command line tool, this code contains a python library for use in scripting use of the NLA system. The code has inline documentation.
3. Using the command line tool
Run the command-line tool with:
cd nla_client/ python2.7 nla.py
This opens an interactive tool. You can type “help” to get a list of help topics / commands. For example:
=========================== CEDA Near line tape utility. NLA>>> help Documented commands (type help <topic>): ======================================== EOF listing_request notify_first quit requested_files expire ls notify_last quota requests label notify pattern_request req retain
Type "help <command>" to get help using a particular command.
The first stage is to determine the names of the files in the NLA that you want to restore. Use the “ls” command to do this, which can also take a sub-string to search for, e.g. “ls sentinel1a”.
4. Requesting files to be restored to disk
Once you know which files you want to restore to disk you can issue a request. There are two ways to make a request:
- Listing request. Here you supply a list of file names to restore in a file. E.g. “
listing_request request_1.txt”. (you can supply any path to the listing file, here it is in the same directory as nla.py, which is probably not how you would typically invoke it).
- Pattern request. Here you supply a sub-string that must appear in the filename. E.g. “pattern_request 2015” will recover all files with “2015” in the filename. This particular request ("2015") is not recommend as it will restore a lot of files! Something more specific like “S2A_OPER_PRD_MSIL1C_PDMC_20161007” would be better.
5. Additional commands
You can view your requests using the “requests” command. This will show you how much of your quota you have used.
You can view the details of a request by “req <request_number>”.
You can check which files are in a request using: “requested_files <request_number>”.
Requests have a retention date. After this date the restored files will be removed. You can extend this retention date by using “retain request-number yyyy-mm-dd”.
You can expire a request early using “expire request-number”. This will remove your restored files within 24 hours and free up some of your quota.
You can label a request by using “label label-name”. The default label name is either the first file in a Listing Request or the pattern in a Pattern Request.
You can be informed via email when your files are ready using the “notify” command. The system knows your email address so simply doing “notify” will inform you when the first files arrive and the last files arrive. To notify someone else use “notify <email_address>”. To notify different people when the first and last arrive use “notify_first <email_address>” and “notify_last <email_address>".
You can check your quota via “quota”.