The command line

Controlling local and remote machines

Lukas Kluft and Florian Ziemen

Shell

• A shell exposes an operating system to a human user or other programs
• On most Linux systems the default shell is Bash (PowerShell on Windows; Zsh on MacOs)
• Different shells come with different syntax and characteristics

Built-in commands

• Allow to operate a computer in a text-only way
• Built-in commands like cd, ls, cp, mv, and rm provide all features of a graphical file manager (e.g. Finder)
• Type info <command> for more information

Coreutils - GNU core utilities

• Basic file, shell and text manipulation utilities that extend the shell-builtins

• Some useful commands are sed, grep, awk, head, tail, …

• Standard output can be piped (|) to another command

  ps aux | grep $USER

• Type man <command> to read the manual for a command

Text editors

• Editors (e.g. vim, emacs, or nano) allow you to modify text files from the command line

  vim test.txt

• Basic usage: hit i to activate insert mode. Type something and hit <Esc> to fall back into normal mode. Save and close the file using :wq.

Scripts

• You can bundle commands in a shell script, e.g.:

  script.sh

  hostname
  date

  Running the script will execute the commands sequentially:

  bash script.sh

  levante4.lvt.dkrz.de
  Mon Mar 25 16:49:06 CET 2024

Take home messages

• The command line is a text interface to interact with a computer
• Built-in commands and core utilities are available on (almost) all machines
• Sequences of commands can be stored and executed as shell scripts

Hands-on session

1. Open the command line (Terminal)
2. Create a shell script using an editor of your choice
3. Run the script and check the output

Levante

Levante is the current supercomputer at DKRZ (since 2022)

Configuration

Supercomputers consist of various components

High-performance computers consist of various components

• Login (login nodes)
• Computing (CPU & GPU partitions)
• Storage (disks, tape, object store)
• Services (GitLab, JupyterHub, …)

Working on Levante

Different ways to work on a cluster depending on the task:

• Housekeeping/compiling (login nodes)
• Interactive sessions/scripts on a compute node (salloc/sbatch)
• External services (e.g. Jupyterhub)

SSH

• Connect to the login nodes via secure shell (ssh)

  ssh a123456@levante.dkrz.de

  (use your user id instead of a123456)

• Now you have access to the command line on levante

Public keys

Generating SSH keys

• Generate a private / public ssh key pair for authentication on your local system

  ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_levante

• Upload the public (.pub) key to your DKRZ profile (instructions), so levante can check your connection attempts

• Use your private ssh key instead of your password when connecting to levante

  ssh -i ~/.ssh/id_ed25519_levante a123456@levante.dkrz.de

Configuring SSH

• Create a config file for convenience

  ~/.ssh/config

  Host levante # the name you use in the shell
     Hostname levante.dkrz.de # the official name of the system
     User a123456 # your user id
     IdentityFile ~/.ssh/id_ed25519_levante # your private key

  (call man ssh_config for more info)

• The connection to levante simplifies to

  ssh levante

• The configuration is used by all tools using an ssh connection (more than you think)

Remote file transfer

• There are different tools to transfer between your local machine and a remote server

• For single files, a simple scp is sufficient

  scp a123456@levante.dkrz.de:file_in_home .

• rsync is a more powerful alternative

  rsync a123456@levante.dkrz.de:file_in_home .

Remote file transfer

• There are different tools to transfer between your local machine and a remote server

• For single files, a simple scp is sufficient

  scp levante:file_in_home .

• rsync is a more powerful alternative with more options

  rsync levante:file_in_home .

Three parts of the file system

/home	/work	/scratch
keep scripts	store output	temporary stuff
small	big	big
SSD	HDD	HDD
backuped	no backup	deleted after 2 weeks

We all lost data to /scratch, many killed something on /work

Compute nodes

• Jobs that require more resources or run longer can be submitted to the job scheduler (SLURM)

  sbatch --account=<PROJ_NUMBER> --partition=compute script.sh

• There are dedicated partitions for different use cases (e.g., compute, shared, interactive, …)

• Try to use the smallest amount of resources (e.g. shared with --mem=50G)

Take home messages

• High-performance computers like Levante are technically a cluster of numerous normal computers that share the same hard disks
• Using a secure shell (ssh) one can login to Levante and use it as any other machine
• Computationally demanding or long-lasting jobs have to be submitted to the job queue

Hands-on session

1. Create an SSH key pair on your local machine
2. Upload the public key to luv.dkrz.de
3. Set up an entry for levante in your SSH config file
4. Copy & run your shell script to levante (using bash and sbatch)

Shotgun buffet

I am just gonna throw a bunch of stuff at you. Take what you might find interesting. — Scott Chacon

IDEs

• Integrated Development Environments are text editors on steroids
• Specific functionality for your programming language (tab completion, jump to definition, scope-aware renaming, debugger, auto documentation, …)
• VSCode, PyCharm, Jupyterhub (in a way), vim + emacs (with plugins), …

tmux

tmux enables to create and (re)attach to terminal sessions

• Crate a new session

  tmux

• Attach to to running sessions

  tmux attach -t 0

• You need to login to the same login node (e.g., levante3¹)

1. SSH config

X-forwarding

You can use X-forwarding to forward graphical user interfaces (GUIs) from the server

• Install an X-server on your local machine (e.g. XQuartz)

• Pass the -X option to your ssh command¹

  ssh -X levante

1. or check the SSH config option

YubiKey

• SSH keys at DKRZ expire after a month
• When using a hardware authenticators (e.g. YubiKey) the key is valid for a year

References

• Software Carpentry: The Unix Shell
• Software Carpentry Incubator: Unix Shell Extras (incl. ssh)