Python Virtual Environments

th  pip This article describes how to install Python packages in your personal file space. You might wish to do this if you want to use different package versions from those installed on the system, or if you have requested for a package to be installed system-wide but wish to start using it before this request can be acted upon.

Creating a virtual environment

Python allows you to create a directory containing a private virtual environment, into which you can install your packages of choice. This is done using the virtualenv command. The basic syntax to create a virtual environment is:

virtualenv /path/to/my_virtual_env

The path can be an absolute or relative path, but it should not already exist. In the JASMIN Analysis Platform (JAP), the virtualenv command will (by default) use the Python 2.7 installation used for the scientific software (rather than the Python 2.6 installation used by the operating system).

Using the system "site-packages" with your virtual environment

Note that if you create a virtual environment using the above syntax, the packages initially installed in it will be only those in the standard Python library. This means, for example, that the numpy package (not in the standard library, but installed as part of the JAP) will be unavailable unless you install it yourself. If you would prefer as a starting point to have all the add-on packages which have already been installed in the JAP, then use instead:

virtualenv --system-site-packages /path/to/my_virtual_env

Activating a virtual environment

Before the virtual environment can be used, it needs to be "activated". This is done by running the activate script:

source /path/to/my_virtual_env/bin/activate

(If you prefer, you can use . instead of source.)

After you run the activate script, some environment variables will be set so that the python2.7 command will point to the one in the virtual environment, allowing installation and use of packages in that environment.

(Typing simply python will also point to this same python2.7 executable, but it is suggested that you continue to use the full python2.7 command in order to avoid confusion when not using a virtual environment, because in that case plain python points to the python 2.6 installation.)

When the virtual environment is activated, this is also reflected in the shell prompt. For example:

[my_username@jasmin-sci1 ~]$ source /path/to/my_virtual_env/bin/activate

(my_virtual_env) [my_username@jasmin-sci1 ~]$

And you can see that python2.7 points to the virtual environment:

(my_virtual_env) [my_username@jasmin-sci1 ~]$ which python2.7
/home/users/my_username/my_virtual_env/bin/python2.7

Note that you have to source the activate script in every shell (login session) in which you intend to use the virtual environment. If there is a particular virtual environment which you want to use consistently, you might consider putting the command to source the activate script in your $HOME/.bashrc file.

If you wish to deactivate the currently active virtual environment in a particular shell, just type deactivate. The environment variable changes will be undone, and you will again be using the system default set of packages. This is also reflected in the shell prompt.

Installing packages into a virtual environment

One you have activated a virtual environment, the pip utility will be available. This allows package installation into the environment using the command:

pip install your_package

pip is quite flexible what you can use for your_package. It can include:

  • a package name in the Python Package Index (PyPI)
  • the local path of a .tar.gz or .zip file containing the package source
  • the local path of a directory containing the extracted package source
  • the download URL of a .tar.gz or .zip file

If the package requires other packages that are not already installed into the virtual environment, then pip will use the package's requirements file to install them automatically from PyPI.

To upgrade an existing package, use:

pip install --upgrade your_package

If your Python package cannot be installed with pip for any reason, it can also be installed directly from the setup.py file after activating the virtual environment.

python2.7 setup.py install

To install a specific version of a package, this can be specified with:

pip install your_package==1.2.3

Inspecting the virtual environment

To list the packages installed into the virtual environment, with their version numbers, type:

pip freeze

Using the virtual environment

Interactive use

After you have activated the virtual environment in your shell, any packages that you have installed into it can be imported into an interactive python session.

(my_virtual_env) [my_username@jasmin-sci1 ~]$ python2.7

>>> import my_package

Scripts

If a script is run using the python2.7 command on the command line in a similar way to when starting an interactive Python session, this will use any virtual environment that has been activated in the calling shell.

(my_virtual_env) [my_username@jasmin-sci1 ~]$ python2.7 my_script.py

If an executable script is run using the #! mechanism, and the first line of the script has the hard-coded path to the executable in the virtual environment, then it is not necessary to activate the virtual environment in the calling shell.

[my_username@jasmin-sci1 ~]$  head -n 1 myscript.py  # show the first line
#!/path/to/my_virtual_env/bin/python2.7

[my_username@jasmin-sci1 ~]$  chmod u+x myscript.py  # ensure that it is executable

[my_username@jasmin-sci1 ~]$  ./myscript.py  # run it

As an alternative to hard-coding the path of the virtual environment, it is possible to use the /usr/bin/env trick to ensure that the script is run using whichever python executable is found via $PATH. The script will then run using any virtual environment that has been activated in the calling shell.  This makes the script more portable, although at the expense of having to source the activate script.

(my_virtual_env) [my_username@jasmin-sci1 ~]$  head -n 1 myscript.py
#!/usr/bin/env python2.7

(my_virtual_env) [my_username@jasmin-sci1 ~]$  chmod u+x myscript.py

(my_virtual_env) [my_username@jasmin-sci1 ~]$  ./myscript.py

Note that if your executable script starts with #!/usr/bin/python2.7 and you run it by typing its filename as above, this will not run in the virtual environment.

Still need help? Contact Us Contact Us