Perseus, Slayer of Medusa, by jovial_cynic, 2008. Image freely copyable under a Creative Commons license. Some rights reserved.

Tips and Tricks for Running PERSEUS on Zeus, Hermes, and Athena – the High Performance Servers at Cornell's LPS Logging In with Secure Shell (ssh)       -  X11 Forwarding via ssh       -  Shell Navigation and Terminal Multiplexing       -  Shell Customization       -  Editors Storing and Accessing Files       -  Backups Compiling and Linking Codes       -  Using the Intel Fortran Compiler       -  Linking MPI and Other Libraries       -  FLASH Visualization and Analysis       -  Opening a Remote Desktop with Xrdp       -  Opening a Remote Desktop with X2Go       -  Running ParaView and VisIt       -  Running Mathematica       -  Setting Up Python 3 and Conda Environments Monitoring the Load

Zeus, Hermes, and Athena are high-performance workstations at the Laboratory of Plasma Studies at Cornell University. They are primarily intended for running the simulation code PERSEUS. For technical details on these resources, see the system configuration pages for the 36-core Hermes server and the 56-core Athena server. (You can also view a similar page for the now-defunct 64-core Perseus server, which was a nice resource in its time.) Athena is reserved for Prof. Seyler's use. This page is meant to be a basic user guide to all three workstations.

These are Linux-based machines: Zeus runs Ubuntu 24.04 LTS for its operating system, Hermes runs CentOS 7.9, and Athena runs Ubuntu 20.04 LTS.

Logging In with Secure Shell (ssh)

Once you have been assigned a username, the easiest way to access Zeus, Hermes, or Athena from your local machine is to use ssh. This well-known Unix/Linux client for making network connections is included with macOS and Windows, too; simply open the command-line Terminal application in these OSes. The LPS servers have the network names zeus.lps.cornell.edu, hermes.lps.cornell.edu, and athena.lps.cornell.edu. To open a connection, enter (e.g.)

ssh <your_username>@hermes.lps.cornell.edu

For security reasons, Zeus, Hermes, and Athena are on an ECE subnet that restricts incoming connections to those that appear to originate from an allowed Cornell subdomain. This means you must use VPN if you want your ssh client to connect from a network location outside of Cornell, or perhaps even from the public Wi-Fi on campus. See CU VPN to learn how to install and use Cornell's VPN software.

• Use ssh for your very first login, as it is good for testing your ability to connect. Also, ssh is fast, if command-line access is all that is needed for a given task.
• Set up X11 forwarding with ssh, if all you need to do in addition is display simple graphics. This technique is described next.
• But start up a remote desktop, if you intend to run ParaView or any complex GUI application. Full remote desktop solutions for Linux are covered later.

X11 Forwarding via ssh

A way to view simple graphical displays from the Linux servers is to have an X11 "X server" installed locally. Like ssh, X11 is native to Unix/Linux, but server implementations exist for macOS and Windows. In addition, you need to enable "X forwarding" in ssh so that individual windows will be forwarded to your local display.

But be aware that significant lag may occur when running GUI applications in separate X windows over the network, especially from off campus. Furthermore, OpenGL/GLX applications like ParaView may not work very well, or at all. Nevertheless, X11 forwarding can come in handy in some situations. Here are the recommended packages to install:

• macOS: Install XQuartz.
• Windows: Install WSL 2, the Windows Subsystem for Linux, which brings in a whole suite of Ubuntu tools.
  (This Microsoft software is supported in up-to-date versions of Windows 10, 11, and Server 2022.)
• A Windows alternative: Install VcXsrv, an open-source X server for Windows that does not require WSL.

The "-X" or "-Y" option to ssh must be present before any remotely generated "X windows" can appear on your local desktop, e.g.,

ssh -X <your_username>@hermes.lps.cornell.edu

Notes: if text labels are badly proportioned in ParaView's GUI, try checking the Override Font box in its Settings dialog. On Windows, if the text looks too big in any GUI, go to Start Menu > Preferences > Look and Feel > Appearance > Fonts > Details > Resolution, and change 192 to 96, e.g.

In the past, Xming was the go-to X server for Windows. Unfortunately, though, its public domain version hasn't been updated in 10 years. It was often paired with PuTTY, a free, open-source ssh client. In case you are feeling nostalgic for these still-available tools, here is the sequence for starting an ssh session using Xming and PuTTY, after installing them:

• Start Xming by clicking its Quick Launch icon or by selecting it from the Start menu
• Start PuTTY via the shortcut on the desktop or via the Start menu
• Enter the address of your destination (or just click a name in your saved list and "Load" it)
• Include your username in the address line if you like, e.g., my_username@hermes.lps.cornell.edu
• Verify that SSH | X11 | Enable X11 forwarding is selected
• Select SSH | Enable compression, too, for better X11 performance
• Return to Session, give it a name if needed, and click "Save" so you can re-use it later
• Click "Open" to log in

After logging out from PuTTY, you can choose to quit Xming by right-clicking its icon in the System Tray and selecting "Exit".

Shell Navigation and Terminal Multiplexing

On Zeus, Hermes, and Athena, the default Linux shell (=OS command processor) is bash. Becoming competent in bash is a whole learning experience in itself. Here are some links to get you started:

• UNIX Tutorial for Beginners
• Shell stuff: job control and screen
• Introduction to GNU Screen
• GNU Screen and tmux, which should you choose

The last three links give you advice on (1) how to get a long-running code like PERSEUS out of your way while it is running, and (2) how to log out without killing your PERSEUS run. All by itself, screen will be adequate for both (1) and (2). The last link compares screen to tmux (terminal multiplexer), an alternative that is preferred by many. For screen, just do this:

screen
mpiexec -n 16 ./perseus
<Ctrl>-a d

The initial screen command places you into a new, detachable session. The final key sequence is what detaches the session and drops you back to your original shell. You can safely log out at this point. After you log back in, you can restore the previous session with "screen -r". To terminate the screen, first restore it, then kill all its processes and exit the screen. If you want to be totally safe when invoking screen (probably to the point of overkill), add nohup and put everything in the background (&):

screen; nohup mpiexec -n 16 ./perseus > outputfile &

Screen is per-session; it allows you to disconnect from a terminal session without killing it, then come back to it later. Nohup (no-hangup) is per-process; it keeps the process running even if the parent shell goes away. Probably using just one or the other of these methods is sufficient.

Shell Customization

Why would you want to customize your shell environment in Linux? The main reasons would be to put MPI tools such as mpif90, mpiifx, and mpiexec (mpirun) into your PATH, and to set up your environment to favor the Intel compilers. Zeus, Hermes, and Athena are already set up to do these things for you automatically every time you log in or start up a new shell. Therefore, the following information is presented just for your general enlightenment.

In bash, login shells are initialized by the file ~/.bash_profile (in Ubuntu, ~/.profile), while non-login interactive shells are initialized by the file ~/.bashrc. (Note, ~ and $HOME both refer to your home directory.) To ensure consistency between both types of interactive shells, the default ~/.bash_profile (or ~/.profile) merely calls the ~/.bashrc at login, before adding a few locations to the PATH. So why not just put tweaks to the PATH or other environment variables in ~/.bashrc? The problem is, a parent shell will export its environment variables to its child shells. So if one interactive shell starts another, which starts another... and ~/.bashrc is executed repeatedly... then PATH (e.g.) grows and grows as a consequence. That is why your default .bash_profile and .bashrc look like the following in CentOS 7, prior to any customization:

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
	. ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH
-- PUT YOUR ENVIRONMENT ADDITIONS HERE --

# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
-- PUT YOUR ALIAS ADDITIONS HERE --

You still have to be careful with such customizations because in Linux, shells you open on a remote desktop or in a screen session can have unexpected behavior. As a result, you may want to put if-then constructs into your .bashrc; testing is definitely a good idea. Note too that macOS behaves a bit differently from Linux; interactive shells started by the Terminal app are all login shells.

Editors

The usual Linux command-line editors are available: vi (really vim), emacs, and nano. If you prefer working with a basic GUI editor, gedit is available as well.

One advantage of learning emacs is that its commands also work in Linux shells, so you can edit as you type. It's the "readline" interface in Linux that allows you to do this; it can be configured to work with vi commands, but vi is not the default.

If you prefer emacs—meaning, you have already learned emacs well enough not to be intimidated by it—then you don't necessarily have to start it in its own window via the "emacs" command. Instead, you might want to start it with "emacs -nw" so it occupies your current terminal window. Why? Generally it starts up faster that way in an "ssh -X" session. However, if you would rather run emacs in a separate X window, that should be fine too (since the emacs window is no longer unstable due to a bug in GTK).

Storing and Accessing Files

The main locations to store your files are (1) in your $HOME, i.e., in the directory /home/$USER, or (2) in a personal data directory, /dataN/$USER, which is found in the large-capacity data partition N. Hermes has 4 such data partitions, Athena has 2, and Zeus has just one extra-large partition on a RAID array (thus your data directory on Zeus would be /data/$USER). Handy commands to see how much disk space is used/available at your current location are "du -s ." and "df ."

Backups

Data on Hermes and Athena are not backed up. To reduce the risk of data loss, keep copies of your important files on more than one of the above devices (or on external storage media). Here is a command that is useful for doing such a backup:

rsync -avz /dataP/$USER/ /dataQ/$USER

The trailing slash on the name of the first directory is significant—it means "get the contents of this directory". The interpretation of the flags, -avz, is as follows: archive the files with original properties, do it verbosely, and use compression while copying.

Compiling and Linking Codes

For everyone's convenience, certain commands are AUTOMATICALLY run in each new shell that is started on Zeus, Hermes, and Athena. These commands are the ones that must be run in your shell before you can compile and link your MPI codes. Zeus has Intel MPI with ifx, the Intel oneAPI Fortran compiler; Hermes and Athena have Open MPI with ifort, the older Intel Fortran compiler. For the record, here are the commands that run automatically to set up your environment properly on each system:

if [ -z "`echo $PATH | grep intel`" ]; then
  . /opt/intel/oneapi/compiler/latest/env/vars.sh > /dev/null
  . /opt/intel/oneapi/mpi/latest/env/vars.sh > /dev/null
fi
# The next line enables snap apps to run in a remote desktop
[ "$DISPLAY" != ":1" ] && export XAUTHORITY=$HOME/.Xauthority

if [ -z "`echo $PATH | grep intel`" ]; then
  . /opt/intel/bin/ifortvars.sh intel64
fi
PATH=/usr/bin:$PATH
LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH 
export OMPI_FC=ifort

if [ -z "`echo $PATH | grep intel`" ]; then
   . /opt/intel/oneapi/compiler/latest/env/vars.sh
fi
export PATH=$PATH:/opt/ParaView-5.10/bin/
export HWLOC_COMPONENTS=-gl
export OMPI_FC=ifort

On less-tailored systems, you might have to place similar commands in your .bash_profile or .profile as noted above.

Using the Intel Fortran Compiler

Click the link for a recommended Makefile that you can use for compiling PERSEUS in any shell. It assumes that the above commands were auto-run for you when you opened the shell. Note that the command for compiling MPI codes is mpiifx on Zeus, and mpif90 on Hermes and Athena.

Linking MPI and Other Libraries

To link an MPI library, just be sure to use the command mpiifx (on Zeus) or mpif90 when you compile, and the MPI linking should be done correctly for you. Remember, every MPI implementation comes with its own "mpif.h" file; therefore, that file is not part of your code, and you should never copy it from system to system!

For other libraries, the Unix convention is that the option "-lmylib" refers to libmylib.so or libmylib.a. If both libraries are present in the same directory, the .so file (shared-object or dynamic library) will be chosen over the .a file (static library). If the library cannot be found at link time, you can specify the path to it via the -L option to the compiler. (The compiler just passes this option to the linker.)

FLASH

The FLASH code should be set up and built separately for each particular user application. Therefore, to use FLASH on Hermes, you should follow the instructions in /data3/scratch/FLASH/README to produce your own custom build. The compiling and linking commands, along with all the appropriate options, are encapsulated in /data3/scratch/FLASH/Makefile.h. Note that FLASH is an MPI application, so it should be run with mpirun as described in the README.

Visualization and Analysis

Windowing in Linux is built around X11, a system for displaying individual windows locally or remotely. But the performance of X11 can be draggy if your remote window is running a rich, graphics-heavy application like ParaView or Mathematica. For doing interactive visualization, it is preferable to set up an entire remote desktop. Several solutions exist for this, notably VNC and OpenNX. However, for the LPS servers, the chosen solutions are Xrpd for Zeus, and X2Go for Hermes and Athena.

If you need to terminate a desktop session that has become unreachable—or if you just want to make sure you've cleaned up all your stray processes—the easiest way to do so is to log in with ssh and enter the following, which will kill every single process that you are running (including the shell you just started!):

pkill -9 -u $USER

Opening a Remote Desktop with Xrdp

The remote desktop server installed on Zeus is Xrdp, which is the Linux implementation of Microsoft's Remote Desktop Protocol (RDP). Therefore, Microsoft is the primary souce of the client software you will need for connecting to a remote desktop on Zeus.

• Microsoft's Windows client will eventually get the (dumb) new name "Windows App", but for now, it still goes by the (reasonable) old name, Remote Desktop Connection. This client comes preinstalled as an accessory in all current versions of Windows.
• Microsoft's Mac client has already acquired the (dumb) new name "Windows App" in the Apple App Store. Formerly, it had the (reasonable) name Microsoft Remote Desktop. Amazingly, if you search for "Windows App" in the App Store, you will find it. It is a free download.
• Microsoft does not provide a client for Linux, but the best one out there is probably Remmina. Another option is the older rdesktop client. Either of them can be installed through your usual Linux package manager.

There are two Linux desktop environments available over RDP: Xfce and GNOME. The default is Xfce, as it is lightweight and performs well from off-campus locations. While GNOME is beautiful to look at, it tends to be laggy over Wi-Fi or comparable networks. (This is not caused by the presence of a few extensions that make GNOME resemble an older version of itself.) But if you have a strong preference for GNOME—or if you are connecting to an already-running GNOME desktop at the machine's console—simply move a single file out of the way:

mv ~/.xsession ~/.not-xsession

Do this in reverse to restore Xfce for your next session.

Here are the steps to follow for connecting to Zeus. They are tailored to the newer "Windows App" client, but they should translate well enough to the others.

1. If you're off campus, set up a VPN as you would for an ssh connection.
2. Install (if necessary) and open the RDP client that is appropriate for your system.
3. Enter settings for the computer you intend to connect to, by clicking on the plus (+) so you can "Add PC", or on the "Show Options" drop-down.

PC name: zeus.lps.cornell.edu
Credentials: "Ask when required" or "Add Credentials..."
<save your login info under its own "Friendly name">
Friendly name: Zeus (no other changes needed)

Update the session resolution on resize -
  (clear check box)
Start session in full screen -
  (clear check box, unless you want this)

Printers -
  (clear check box)
Smart cards -
  (clear check box)

Redirect folders -
  (check and click "+")
  <choose local path>
  (click "Open")

4. Click "Add". A new "Zeus" tile appears in the client window. Double-click it. Enter credentials if needed. (You can edit the tile later to add credentials.)
5. Click "Continue" or "Yes" if you see any complaints about certificates.
6. After a short wait, a new window should appear with a remote desktop on Zeus! Resize it manually if you like.
7. If you designated one or more shared folders, they will appear under "thinclient_drives" in your home directory.
8. KEYBOARD SHORTCUTS: copy is Ctrl-SHIFT-C, paste is Ctrl-SHIFT-V. The remote clipboard is shared; your usual keyboard shortcuts will work locally.
9. If you right-click the tile in the app and choose "Disconnect", or just close the local window, your remote session continues to run disconnected!
10. To reconnect, double-click the tile and authenticate as before. Your desktop should appear exactly as you left it (even from a different machine!).
11. To terminate the session completely, click the upper right-hand corner of the remote desktop and find "Log Out..." (maybe under your name, or 0/1).

Xfce tip: The top title bar for Firefox, which normally displays the name of the active tab, does not appear by default. To enable it, right-click the toolbar, choose "Customize Toolbar...", and put a check by "Title Bar". The setting will be remembered.

GNOME tip: You can type Ctrl-Alt-arrow to switch between workspaces, and Ctrl-Alt-shift-arrow to move a window to the next workspace (Alt = option on Macs). If you hit the Windows key (Cmd key on Macs), you can simply swipe between workspaces.

Opening a Remote Desktop with X2Go

Here are the steps to follow for Hermes and Athena:

1. First, download and install the appropriate X2Go client on your laptop or workstation.
2. Once that's done, set up a VPN as you would for an ssh connection.
3. Start the X2Go client. From the Session menu, choose "New Session...", then enter the following (for Athena, modify this in obvious ways):

Session name: hermes
Host: hermes.lps.cornell.edu
Login: <your username>
Session type - MATE (in drop-down)

Connection speed -
WAN (move slider)

Display - Custom -
Width: 1600, Height: 900

Enable sound support -
  (clear check box)
Client side printing support -
  (clear check box)

Path: <choose local path>
  (click "Add")
  (put checks in all 3 boxes)

4. OK the settings. A new "hermes" tile appears in the client window. Click it. Enter your password, then "Ok".
5. After a short wait, a new window should appear with a remote desktop on Hermes! Resize it manually if you like.
6. If you designated a shared folder, a file icon may appear on your desktop with a name ending in (sshfs-disk-mate); ignore it.
   Visit the X2Go website for complete information on how to access your shared folders.
7. If you click the Pause button, or close the local window, your remote session continues to run! You have merely disconnected it.
8. To log back in, enter your password and hit "Ok" again. Your desktop should appear exactly as you left it (even from a different machine!).
9. To terminate the session completely, don't log out. Go back to the initial X2Go window and click the 0/1 button.

In macOS, the X2Go client does not always have the correct key mappings. The familiar "Cmd+V" shortcut for pasting the Clipboard may not work in xterm and other remote windows. Try the following modification to .Xdefaults on your Mac to restore the Cmd+V shortcut (a restart of XQuartz will be required):

echo "*VT100.translations: #override Meta <KeyPress> V: insert-selection(PRIMARY, CUT_BUFFER0) \n" >> ~/.Xdefaults

If that doesn't work, simply right-click and choose "Paste" from the contextual menu, or use the X11 mouse shortcut for paste instead. (The latter trick may require you to enable 3-button-mouse emulation in XQuartz Preferences.)

Finally, if you see the wrong characters showing up while you are typing on your Mac, disconnect your session and reconnect to it. Often this is enough to correct the problem.

Running ParaView and VisIt

As mentioned previously, visualization applications like ParaView and Visit are likely to be more responsive if they are run in a remote desktop.

On Zeus, open an Xrdp desktop, and in the Xfce desktop environment, you can simply click on the ParaView icon in the bottom panel to launch version 5.11.2. ParaView can also be launched from the "Applications > Science >" menu, or from the command line with "paraview &".

On Hermes and Athena, open an X2Go desktop. In the MATE desktop environment on Hermes, the "ParaView 5.8.1" application can be launched from the "Applications > Graphics >" menu; on Athena, look for "ParaView 5.10 (Mesa)" under "Menu > Other". An even handier shortcut is available on the top menu bar.

• On Hermes, you have the choice of starting ParaView 5.8.1 or 4.3.1 from the command line, as shown below. If you like, the command can be shortened by adjusting your PATH to include the appropriate ParaView directory. Note that with ParaView 5.8.1, paraview must be started through the paraview-mesa wrapper in order to get OpenGL to work properly. If the application crashes, setting the indicated variable may prevent trouble.

VTK_DEBUG_SKIP_VISRTX_CHECK=1 /opt/ParaView-5.8.1-MPI-Linux-Python2.7-64bit/bin/paraview-mesa paraview &
/opt/ParaView-4.3.1-Linux-64bit/bin/paraview &

• On Athena, ParaView 5.10 can be easily started from the command line, as the application should already be in your path:

paraview --mesa &

• On either system, if the ParaView icon doesn't appear on the top panel, try entering the following commands. The second one is needed only if the ParaView icon doesn't appear after you enter the first. The third command will completely reset the appearance of the desktop.

mate-panel --reset
mate-panel --reset --layout rhel-pv   # replace rhel-pv with familiar-pv on Athena
rm -rf ~/.config   # undoes all your desktop customizations and reverts to system defaults

VisIt is installed only on Hermes. You can start it with the following command:

/opt/VisIt-3.3.3/bin/visit -gui

Running Mathematica

Mathematica is installed only on Hermes. If you are logged into Hermes with ssh -X and have an X server running locally, just type "mathematica" in your shell, and this will start up the GUI for version 13.3.

This also works in an X2Go remote desktop session, though your cursor may appear annoying offset from its actual position when you are typing commands into the GUI for version 13.3.

Version 13.1, which does not have the issue with the cursor, is also available as "mathematica.old". If you want to run the older Mathematica in an X2Go remote desktop, the following startup command is necessary to avoid a Qt5 bug (or quirk):

QT_XCB_GL_INTEGRATION=none mathematica &

You might need an activation key if want to use Mathematica on Hermes. To get one, send an email to the author of this page.

Setting Up Python 3 and Conda Environments

The best way to access Python is by customizing one or more conda environments so it contains your preferred set(s) of libraries and version(s) of Python. A minimally-configured conda package manager is already installed for you in /opt/miniforge3 (Zeus) or /usr/local/miniconda3 (Hermes, Athena). The pip package manager is included in teh base environment for conda, so you can get packages that are only available through pip. To initialize conda and access the base environment, run the following commands:

/usr/local/miniconda3/bin/conda init
conda activate base

The above command modifies your .bashrc so Anaconda Python and its tools will be found ahead of others. This in turn enables you to use conda environments to install one or more collections of Python libraries to suit your own needs:

conda create -n <name-of-your-environment> python=<your-preferred-version>
conda activate <name-of-your-environment>

You may want to edit your .bashrc to add the above "conda activate" command. Insert it after the lines with conda's other modifications. That way, your custom environment, including your preferred version of python, will be automatically activated every time you start a new shell. Conversely, you may want to deactivate all conda environments (including conda's base environment) by default when you log in. To do that, run the following command:

conda config --set auto_activate_base false

This produces a ~/.condarc file that enforces the desired configuration change. It prevents conda's environments from altering your $PATH and other environment variables in in ways that might interfere with your non-Python applications.

Note that the NumPy and SciPy libraries have been preinstalled for you in the base environment, i.e., the one that gets activated if you don't create an environment of your own and specify it by name in the "conda activate" command. If you need NumPy and SciPy in your own environment, simply install them in that one too.

One of the packages you can install in a conda environment is the Spyder IDE (Integrated Development Environment). Spyder is a popular choice for working with Python code. To install it in your currently activated environment, simply type

conda install spyder

QT_XCB_GL_INTEGRATION=none spyder &

If you don't want to bother with conda at all, you don't have to, as Python 3 is available in all standard Linux distributions (e.g, Python 3.6 was introduced to CentOS 7 between versions 7.6 and 7.8; previously, Python 3 had to be enabled from the Red Hat Software Collections, RHSCL). Thus, the command "python3" is sufficient to start a Python 3 interpreter, independent of conda.

The pip package manager is also installed on Hermes and available through the command "pip3".

Monitoring the Load

Here are some helpful commands to see what's happening on the machine. You may want to run them if you're about to start a new MPI job, or if your current MPI job seems to be running more slowly than you expected.

w
uptime
top
ps aux | grep -v root

A fully loaded 36-core machine will have a load average of 36 in "w" and "uptime". Anything over that is likely to be using the cores inefficiently. The "top" command lets you monitor the machine's activity continuously and in much more detail. It lets you see which processes are the heavy CPU users (type "q" to quit). The one-shot "ps" command, when combined with "grep", allows you to filter out things you don't want to see in "top".

If you run "ps aux | grep $USER" and notice processes that shouldn't be there, do yourself and everyone else a favor by running "kill -9 <pid>" on each of them, where <pid> is the process id of the offending process. This is especially important to do for rogue processes that are taking up appreciable CPU time.