DataHub workflow instructions and tutorials

The following pages offer practical instructions on how to use the DataHub services for preparing, uploading, computing, downloading, and sharing your data. Written instructions as well as tutorial videos are provided.

However, this Manual is meant to be used after you consulted the local Data Stewards and already received an introduction by e.g., a workshop to all services. The following pages can only give general instructions but the workflow might differ for your specific case. For comprehensive background information on all services, see services pages.

You can get a first impression of the DataHUb workflows from our demonstration videos.

Access to the DataHub

Please see the section Access and Setup for information on how to get permission for our services.

Required Installations and Configuration

Please follow the instructions in Git and TAM GitLab. Make sure that Git is installed on your local computer (or on MaRC3 if you plan to complete this tutorial there), your local Git setup is connected to TAM GitLab, and Git LFS is enabled for handling large files.

Working with virtual environments

We recommend to first set up an environment using the environment and package manager conda. Creating environments basically means that you separate the work on your local computer into different environments. In these single environments you only have installed the software you actually need for this particular work and also only in the versions you need for this particular work. This is very useful as sometimes updates of a software can affect your previous work. See below for ressources and tutorials on virtual environments.

If you want to work in an environment on your local computer, please skip the steps 1.-4 and start with step 5. Step 1.-4. only apply if you want to work on the MaRC3a cluster.

By the nature of the MaRC3a cluster, you're provided only with a simple Linux environment at the beginning and you need to first activate miniconda before you can use it. You don't need to do this if you're working on your local machine.

1. Go to the JupyterHub of Marburg university and open a terminal. Alternatively, you can connect directly via SSH to the MaRC3 cluster.
2. Make sure to be in your home folder: pwd should give you the output /home/your-username
3. Activate the module Miniconda for environment activation: module load miniconda (In front of your username you should now see (base))
4. Activate environment: source $CONDA_ROOT/bin/activate
5. Create a new empty environment for your project: conda create --name my-project
6. Activate this new empty enviroment: conda activate my-project (In front of your username you should now see (my-project))
7. Install the packages you need in this environment, like so: conda install -c conda-forge pandas==2.2.0
hint: you can create an requirements.txt file that lists all the packages you need and then load this file instead of loading every single packages one after another.

!!NOTE!! Environment reload:
MaRC3a setup: Every time you restart JupyterLab / reconnect to MaRC3, you need to go through steps 3., 4., and 6. again.
Local setup: Every time after you closed the terminal with the environment and want to continue working in this environment, you have to execute step 6. again.

In our FAQs you can find instructions on how to automate the installations described above.

Ressources and tutorials on virtual environments

Everything set up? Great! Let's prepare your data in a next step!