Python Virtual Environments
This article describes how to install Python packages in your personal file space. You might wish to do this if you want to use different package versions from those installed on the system, or if you have requested for a package to be installed system-wide but wish to start using it before this request can be acted upon.
Creating a virtual environment
Python allows you to create a directory containing a private virtual environment, into which you can install your packages of choice. This is done using the
virtualenv command. The basic syntax to create a virtual environment is:
The path can be an absolute or relative path, but it should not already exist. In the JASMIN Analysis Platform (JAP), the
virtualenv command will (by default) use the Python 2.7 installation used for the scientific software (rather than the Python 2.6 installation used by the operating system).
Note that if you create a virtual environment using the above syntax, the packages initially installed in it will be only those in the standard Python library. This means, for example, that the
numpy package (not in the standard library, but installed as part of the JAP) will be unavailable unless you install it yourself. If you would prefer as a starting point to have all the add-on packages which have already been installed in the JAP, then use instead:
virtualenv --system-site-packages /path/to/my_virtual_env
Activating a virtual environment
Before the virtual environment can be used, it needs to be "activated". This is done by running the
(If you prefer, you can use
. instead of
After you run the activate script, some environment variables will be set so that the
python2.7 command will point to the one in the virtual environment, allowing installation and use of packages in that environment.
python will also point to this same
python2.7 executable, but it is suggested that you continue to use the full
python2.7 command in order to avoid confusion when not using a virtual environment, because in that case plain
python points to the python 2.6 installation.)
When the virtual environment is activated, this is also reflected in the shell prompt. For example:
[my_username@jasmin-sci1 ~]$ source /path/to/my_virtual_env/bin/activate (my_virtual_env) [my_username@jasmin-sci1 ~]$
And you can see that
python2.7 points to the virtual environment:
(my_virtual_env) [my_username@jasmin-sci1 ~]$ which python2.7 /home/users/my_username/my_virtual_env/bin/python2.7
Note that you have to source the
activate script in every shell (login session) in which you intend to use the virtual environment. If there is a particular virtual environment which you want to use consistently, you might consider putting the command to source the
activate script in your
If you wish to deactivate the currently active virtual environment in a particular shell, just type
deactivate. The environment variable changes will be undone, and you will again be using the system default set of packages. This is also reflected in the shell prompt.
Installing packages into a virtual environment
One you have activated a virtual environment, the
pip utility will be available. This allows package installation into the environment using the command:
pip install your_package
pip is quite flexible what you can use for
your_package. It can include:
- a package name in the Python Package Index (PyPI)
- the local path of a
.zipfile containing the package source
- the local path of a directory containing the extracted package source
- the download URL of a
If the package requires other packages that are not already installed into the virtual environment, then
pip will use the package's requirements file to install them automatically from PyPI.
To upgrade an existing package, use:
pip install --upgrade your_package
If your Python package cannot be installed with
pip for any reason, it can also be installed directly from the
setup.py file after activating the virtual environment.
python2.7 setup.py install
Inspecting the virtual environment
To list the packages installed into the virtual environment, with their version numbers, type:
Using the virtual environment
After you have activated the virtual environment in your shell, any packages that you have installed into it can be imported into an interactive python session.
(my_virtual_env) [my_username@jasmin-sci1 ~]$ python2.7 >>> import my_package
If a script is run using the
python2.7 command on the command line in a similar way to when starting an interactive Python session, this will use any virtual environment that has been activated in the calling shell.
(my_virtual_env) [my_username@jasmin-sci1 ~]$ python2.7 my_script.py
If an executable script is run using the
#! mechanism, and the first line of the script has the hard-coded path to the executable in the virtual environment, then it is not necessary to activate the virtual environment in the calling shell.
[my_username@jasmin-sci1 ~]$ head -n 1 myscript.py # show the first line #!/path/to/my_virtual_env/bin/python2.7 [my_username@jasmin-sci1 ~]$ chmod u+x myscript.py # ensure that it is executable [my_username@jasmin-sci1 ~]$ ./myscript.py # run it
As an alternative to hard-coding the path of the virtual environment, it is possible to use the
/usr/bin/env trick to ensure that the script is run using whichever python executable is found via
$PATH. The script will then run using any virtual environment that has been activated in the calling shell. This makes the script more portable, although at the expense of having to source the activate script.
(my_virtual_env) [my_username@jasmin-sci1 ~]$ head -n 1 myscript.py #!/usr/bin/env python2.7 (my_virtual_env) [my_username@jasmin-sci1 ~]$ chmod u+x myscript.py (my_virtual_env) [my_username@jasmin-sci1 ~]$ ./myscript.py
Note that if your executable script starts with
#!/usr/bin/python2.7 and you run it by typing its filename as above, this will not run in the virtual environment.