FTP with Python and PERL

Most programming languages have FTP client libraries. These can be used to grab data from the CEDA archives programmatically rather than interactively. Two examples are given below. The first uses Python to extract ECMWF ERA40 data and the second uses Perl to grab radiosonde data.

Python example

Python is available on most platforms and can be downloaded from www.python.org. The script below opens an FTP connection then loops over a number of years, months, days, hours and variables. It constructs each file name and then retrieves is to a local directory.

#!/usr/bin/env python

# Import required python modules
import ftplib
import os

# Define the local directory name to put data in
ddir="C:\\datadir"

# If directory doesn't exist make it
if not os.path.isdir(ddir):
    os.mkdir(ddir)

# Change the local directory to where you want to put the data
os.chdir(ddir)

# login to FTP
f=ftplib.FTP("ftp.ceda.ac.uk", "", "")

# loop through years
for year in range(1990,2001):

    # loop through months
    for month in range(1,13):

        # get number of days in the month
        if year%4==0 and month==2:
            ndays=29
        else:
            ndays=int("dummy 31 28 31 30 31 30 31 31 30 31 30 31".split()[month])

        # loop through days
        for day in range(1, ndays+1):

            # loop through hours
            for hour in range(0, 19, 6):

                # loop through variables
                for var in ("10u", "10v"):

                    # change the remote directory
                    f.cwd("/badc/ecmwf-e40/data/gg/as/%.4d/%.2d/%.2d" % (year, month, day))
                    # define filename
                    file="ggas%.4d%.2d%.2d%.2d%s.grb" % (year, month, day, hour, var)
                    # get the remote file to the local directory
                    f.retrbinary("RETR %s" % file, open(file, "wb").write)

# Close FTP connection
f.close()

Perl example

Perl is available on most platforms and can be downloaded from www.perl.org. The script below opens an FTP connection then downloads all the camborne radiosonde files for the current year.

# libraries for FTP and time strings
use Net::FTP;
use POSIX qw(strftime);

# Get this year 
$thisyear = strftime("%Y", localtime);

# radiosonde directory with thisyears data in
$dir = "/badc/ukmo-rad/data/united_kingdom/camborne/$thisyear";

# Connect to BADC ftp server
$ftp = Net::FTP->new("ftp.ceda.ac.uk", Debug => 0 ) or die;

# Login 
$ftp->login("",'') or die;

# change the remote directory
$ftp->cwd($dir) or die;

# list dir
@files = $ftp->ls($dir);

# get the remote file to the local directory
for $file  (@files) {
  print "Getting $file\n";
  $ftp->get($file) or die;
}

# Close FTP connection
$ftp->quit;