TAM-DataHub Workshop: JupyterHub & MaRC3¶

This presentation will give an overview about JupyterHub and MaRC3 for TAM users.

Dr. Christian Berger, Philipps-Universität Marburg

Jupyter¶

Interactive computing, data processing and programming for beginners and experts: Jupyter is a versatile tool and offers an introduction to the world of programming.

Content¶

  • Jupyter Project - What is Jupyter?
  • Live Demo
    • JupyterLab with Python
      • Simple Math and Variables
      • Importing Packages
      • Loading Data
      • Printing Outputs
      • Plotting Data
    • JupyterHub
      • Accessing the Base System
      • Customizing the System

Link List¶

  • Website of Project Jupyter
  • Project Jupyter Documentation
  • Jupyter Service at Philipps-Universität Marburg
  • Information on Jupyter Service of Philipps-Universität Marburg

What is Jupyter?¶

Jupyter is a project (https://jupyter.org) for

Free software, open standards, and web services for interactive computing across all programming languages

The project was initiated as interactive web-based tool for running Python, based on iPython. (https://ipython.org)

Subprojects of Jupyter¶

All subprojects are related to interactive computing and provide software to run a web service. You can install and run such a service on your own or it can be run centrally. Even if Python is the original programming language, Jupyter projects are usually not limited to Python.

  • JupyterLab: A Next-Generation Notebook Interface
  • Jupyter Notebook: The Classic Notebook Interface
    • Not to be confused with the file ending .ipynb - These files are nowadays commonly refered to as Jupyter Notebooks. They do not only store the code itself, but also the results which were evaluated when running the notebook.
  • JupyterHub: A multi-user version of the notebook designed for companies, classrooms and research labs

What are the advantadges of joining the Jupyter ecosystem?¶

  • The interactive graphical web interface provides direct feedback to your programming/scripting
    • Output of results, graphics, and errors/warnings
  • All skill levels can benefit from the low initial threshold to get started
    • Learning, Extension of existing scripts, Development, ...
  • Flexible in changing kernels (programming language) and environments (versions)
  • JupyterHub: Run your code on big systems which usually are harder to access, e.g. bypass batch processing of (super) computing clusters
  • Flexibility: While you can run your code on huge systems, everything remains portable and you can run it locally as well

Live Demo (Python)¶

Start: Code and commands in cells. Cells can be Code, Markdown, or Raw. You can evaluate cells using Shift+Enter.

In [1]:
# Simple Math (implicit output)
1 + 1
Out[1]:
2
In [2]:
# Using Variables (explicit output)
a = 1
b = 1
c = a + b
print(c)

2
In [3]:
# Printing multiple Results
print('Declaration:')
print('a = 1')
print('b = 1')

print('Calculation:')
print('a + b =', c)
print('a * b =', a*b)

Declaration:
a = 1
b = 1
Calculation:
a + b = 2
a * b = 1

Pay attention to order of execution of cells

In [4]:
result = a + b
print('a + b =', result)

a + b = 2
In [5]:
a = 2
In [ ]:

More complex Example¶

In [6]:
# Importing Python Packages
# They need to be installed in advance (Don't forget to restart your Kernel after installation)
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
import csv
In [7]:
# Defining Variables
state = 6

temp = []
year = []

Download data and save it in the same folder as this file:

https://opendata.dwd.de/climate_environment/CDC/regional_averages_DE/annual/air_temperature_mean/regional_averages_tm_year.txt

In [8]:
!wget https://opendata.dwd.de/climate_environment/CDC/regional_averages_DE/annual/air_temperature_mean/regional_averages_tm_year.txt

--2024-10-15 14:31:51--  https://opendata.dwd.de/climate_environment/CDC/regional_averages_DE/annual/air_temperature_mean/regional_averages_tm_year.txt
Resolving www-proxy1.hrz.uni-marburg.de (www-proxy1.hrz.uni-marburg.de)... 137.248.1.6
Connecting to www-proxy1.hrz.uni-marburg.de (www-proxy1.hrz.uni-marburg.de)|137.248.1.6|:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: 26413 (26K) [text/plain]
Saving to: 'regional_averages_tm_year.txt.2'

regional_averages_t 100%[===================>]  25.79K  --.-KB/s    in 0.004s  

2024-10-15 14:31:51 (5.69 MB/s) - 'regional_averages_tm_year.txt.2' saved [26413/26413]
In [9]:
# Loading and Processing Data
with open("regional_averages_tm_year.txt", "r") as read_file:
    data = csv.reader(read_file, delimiter=';')
    next(data)
    next(data)
    for row in data:
        temp.append(float(row[state]))
        year.append(float(row[0]))
In [10]:
plt.plot(year, temp)
Out[10]:
[<matplotlib.lines.Line2D at 0x14f377d01550>]
No description has been provided for this image
In [11]:
# Plotting processed Data
fig, ax = plt.subplots(figsize=(8, 2))

ax.bar(year, 1.0, width=1.0, color=cm.get_cmap('RdBu')(1-(temp-np.amin(temp))/(np.amax(temp)-np.amin(temp))))
ax.axis('off')

plt.title('Hesse')
plt.tight_layout()
No description has been provided for this image

Code Source: https://www.s4f-freiburg.de/temperaturstreifen/

Features of JupyterLab & JupyterHub¶

  • Autocomplete for Code (Variables and Functions)
  • Instant Access to Documentation of Functions
  • Direct Presentation of Error Messages
  • Visual Options and Extensions
  • Access to Files and Folders on System - Uploading and Downloading Files
  • Command Line Usage via JupyterLab
  • Kernels (Choosing and Restarting)
  • Accessing Compute Resources via JupyterHub

Additional Information¶

Overview about complete project scope of Jupyter with all its subprojects: https://jupyter.org.

How to get started using Jupyter...¶

Jupyter @ TAM¶

Philipps-Universität Marburg offers a research JupyterHub, running of our high-perfomance computing cluster (MaRC3) which can be used within the TAM-DataHub. It acts as a graphical user interface to the cluster and enables an easy access for results evaluation, testing and development.

Direct Access to JupyterHub on MaRC3: https://jupyter-hpc.uni-marburg.de

Information on using JupyterHub on MaRC3, including custom kernels and using R: https://share.uni-marburg.de/de/hrz/it-mitarbeiter/hpc/hpc-user/marc3/using-jupyterhub-on-marc3a

All information can be found on the service website: https://jupyter.uni-marburg.de

For any kind of support contact: jupyter@uni-marburg.de

...on my own? (JupyterLab)¶

Short Sketch:

  • Install JupyterLab on your machine
  • Start the JupyterLab
  • Connect to it via your Web Browser

Installation is well documented, e.g. in official project documentation: https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html.

In [ ]:

MaRC3¶

MaRC3 offers access to the world of high-performance computing (HPC) by providing compute power which is distributed between all researchers via SLURM. Compute jobs shall be run mainly as batch processing tasks.

Content¶

  • HPC & Batch Processing - What is different to running software on my machine?
  • Live Demo
    • General
      • Accessing MaRC3
      • Running Jobs
    • Python
      • Custom environment
    • R (RStudio)

Link List¶

  • MaRC3a General Information
  • Information on High Performance Computing Service of Philipps-Universität Marburg
In [ ]:

HPC & Batch Processing - What is different to running software on my machine?¶

  • Typical you are running software interactively on your local machine - Batch processing is encouraged as main usage on MaRC3
  • HPC systems mostly don't provide a GUI - Exception web browser based GUIs like JupyterHub
  • Super computers are large and thus run way more jobs in parallel
  • Runtime of computational tasks can be as large as weeks on HPC systems
  • Resources have to be reserved suiting the needs of your job - Else your job may get killed before finishing his task
In [ ]:

General¶

Login to MaRC3 from inside the university network of Philipps-Universität Marburg (VPN) via SSH (you have to change your shell before the first login via https://admin.staff.uni-marburg.de/aendshell.html):

image.png

Submit jobs using the command srun or sbatch (Demo). Monitor progress of job with squeue. Depeding on the requested resources, choosen partition, and load on the cluster the time your job is pending before starting to run may vary.

image.png

In [ ]:

Programming Language Specifics¶

Not all installed software on MaRC3 is active at all times. Some tools or versions have to be activated by yourself if you need to use them. All information on how to use, configure, and customize the MaRC3 cluster can be found in the user manual: https://share.uni-marburg.de/de/hrz/it-mitarbeiter/hpc/hpc-user/marc3a-masc

Python¶

In case you require specific Python environments, this can be done using (Ana)conda: https://share.uni-marburg.de/de/hrz/it-mitarbeiter/hpc/hpc-user/marc3/using-ana-conda-envrionments-on-marc3

R¶

Besides running R code directly in batch mode you can also use RStudio as IDE: https://share.uni-marburg.de/de/hrz/it-mitarbeiter/hpc/hpc-user/marc3/using-the-rstudio-ide-on-marc3

In [ ]:

Additional Information¶

Overview about the HPC service at Philipps-Universität Marburg which can be used in TAM: https://www.uni-marburg.de/en/hrz/services/high-performance-computing

In case of a problem - How do I get support?¶

Philipps-Universität Marburg is running of the high-perfomance computing cluster (MaRC3) which can be used within the TAM-DataHub. Thus, you can contact our colleagues in case you need support while using the cluster: marc3@hrz.uni-marburg.de

Courses and Tutorials¶

The Hessian Competence Center for High Perfomance Computing (HKHLR) offers courses on a regular basis for different topics: https://www.hkhlr.de/en/events/courses-and-tutorials

In [ ]: