Python in VRLab

Python in VRLab
last updated	2024-03-11
relates to	VRLab

Set up conda environment and use it in your slurm job

For VRLab, it is ideal to install your own instance of miniconda in your directory. Then it doesn’t have to interfere with any of the existing versions on the cluster. Here’s how you can do it step by step.

1. Download the relevant miniconda distribution. You can do this with wget. For example:

wget "https://repo.anaconda.com/miniconda/Miniconda3-py310_23.11.0-1-Linux-x86_64.sh%20-O%20miniconda.sh")

2. Install the downloaded distribution of miniconda with a specified path in your directory. This specified path must be one where you have rights to install it. Assuming your pwd is where the distribution is present, enter:

bash miniconda.sh -b -p /net/labdata/geet/miniconda

Don't forget to change your username ;)

3. Add the miniconda path to your PATH:

export PATH="/net/labdata/geet/miniconda/bin:$PATH"

4. Check if conda is installed properly by runnning conda -V or by checking the list of environments with conda info –envs

5. If step-4 works with no issues, initiate conda with

conda init

6. Now restart your shell, e.g. with

ssh shell-1

All of these were one-time steps! Once you have your own conda installation, you can create your environment and then modify it to your needs. One way to do this is, if you have a YAML file with a list of libraries, you can create and populate an environment with

conda env create -f environment.yml

Many more details of managing conda environments can be found in conda's documentation.

For your slurm job to use your environment, it must first initialize conda and then activate the environment. To do so, you simply need to add two lines in your .sbatch file.

At the point in your job, where you want your environment to be activated (this must be after all #SBATCH parameters are provided), add the following lines:

source ~/.bashrc
conda activate my_conda_environment