Conda (anaconda) usage

‘Conda’ (or ‘Anaconda’) is the ‘de-facto’ standard to encapsulate and manage a set of Python packages needed for a task.

The description from the official site is: “Conda is an open source package management system and environment management system for installing multiple versions of software packages and their dependencies and switching easily between them. It works on Linux, OS X and Windows, and was created for Python programs but can package and distribute any software.”

This instructions covers the very basic steps needed to run a job in the computing cluster of the Department of Mathematics ‘Tullio Levi-Civita’.

Conda is installed in “/conf/shared-software/anaconda”. To verify that you are using the correct conda installation you can give the command in boldface (on labsrv8 or labsrv7) and compare your output:

The next step will create an ‘environment‘ e.g. a set of a python interpreter with the needed packages. We will use the name gpuenv for the environment:

labsrv8:~$ conda create --name gpuenv
Collecting package metadata (current_repodata.json): done
Solving environment: done
# Package Plan ##

   environment location: /home/USERNAME/.conda/envs/gpuenv

Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate gpuenv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Retrieving notices: ...working... done

labsrv8:~$

Before activate the new environment we need modify the ‘.bashrc’ and ‘.profile’ files that contain the initialization commands for the shell. The conda command is:

This modifies the .bashrc file in your home directory. Usually this is enough for the normal use of conda but in a Cluster/SLURM enviroment we need add the conda initialization commands even to the .profile file. You can edit the two files and copy the conda commands (very easy to recognize) from .bashrc to .profile or use the following command:

At this point we need to logout and re-login to enable the enviroment modification. After this we ‘activate‘ the (empty) environment using this command:

Please, note that the prompt has changed adding the string ‘(gpuenv)‘ before the usual prompt value.

Now we are operating inside the new environment and the next step will install the packages. In this example we install the package ‘pytorch-gpu‘ from the repository ‘conda-forge’:

labsrv8:~$ conda install -c conda-forge pytorch-gpu
## Package Plan ##

  environment location: /home/USERNAME/.conda/envs/gpuenv

  added / updated specs:
    - pytorch-gpu


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    brotli-python-1.1.0        |  py311hb755f60_0         343 KB  conda-forge
    certifi-2023.7.22          |     pyhd8ed1ab_0         150 KB  conda-forge
    charset-normalizer-3.2.0   |     pyhd8ed1ab_0          45 KB  conda-forge
    freetype-2.12.1            |       hca18f0e_1         611 KB  conda-forge
   ...(continue)

The following NEW packages will be INSTALLED:

    brotli-python      conda-forge/linux-64::brotli-python-1.1.0-py311hb755f60_0 None
    certifi            conda-forge/noarch::certifi-2023.7.22-pyhd8ed1ab_0 None
    charset-normalizer conda-forge/noarch::charset-normalizer-3.2.0-pyhd8ed1ab_0 None
   ...(continue)

Proceed ([y]/n)? y

Downloading and Extracting Packages
idna-3.4             | 55 KB     | ######################################### | 100% 
libwebp-base-1.3.1   | 391 KB    | ######################################### | 100% 
   ...(continue)

At this point we can test our environment. Create a python script file with the following content and save it ‘condatest.py’:

#!/usr/bin/env python

import torch

if torch.cuda.is_available():
    print ('GPU is available')
    dev = "cuda:0" 
else:
    print ('GPU is *NOT* available')
    dev = "cpu" 
device = torch.device(dev) 
a = torch.zeros(4,3) 
a = a.to(device)

Now you can run the test provided that the ‘gpuenv‘ is active:

N.B.If you run the program on labsrv7 (after the activation command for the environment) you will get the result ‘GPU is *NOT* available‘ as labsrv7 lacks of a GPU.

Now you can get the conda example in the Basic/Examples page, adapt it and run this code over the cluster