Hermes and Athena are high-performance workstations at the
Laboratory of Plasma Studies at Cornell University.
They are primarily intended for running the simulation code
PERSEUS.
For technical details on these resources, see the system configuration pages for the
36-core Hermes server and the
56-core Athena server.
(You can also view a similar page for the now-defunct
64-core Perseus server, which was a nice resource in its time.)
Athena is reserved for Prof. Seyler's use.
This page is meant to be a basic user guide to Hermes and Athena.
These are Linux-based machines: Hermes runs CentOS 7.9 for its operating system, while Athena runs
Ubuntu 20.04 LTS.
Logging In
Once you have been assigned a username, the easiest way to access Hermes or Athena from your local
machine is to use ssh. The well-known Unix/Linux client for making network connections is included
with macOS and Windows,
too, through the Terminal application in these OSes. The LPS servers have the network names
hermes.lps.cornell.edu and athena.lps.cornell.edu. To open a connection, enter:
ssh <your_username>@hermes.lps.cornell.edu
For security reasons, Hermes and Athena are on an ECE subnet that restricts incoming connections to
those that appear to originate from an allowed Cornell subdomain. This means you must use VPN if you want
your ssh client to connect from a network location outside of Cornell, or perhaps even from the public
Wi-Fi on campus. See CU VPN to learn how to install and use
Cornell's VPN software.
For viewing graphical displays or using GUI applications from Hermes and Athena, you must also have an
X11 "X server" installed locally. X11 is again native to Unix/Linux, but implementations exist for macOS
and Windows. (Additionally, there is a full remote desktop solution called X2Go; this is covered
later.)
- macOS: Install XQuartz.
- Windows: Install WSL 2,
the Windows Subsystem for Linux, which brings in a whole suite of Linux tools.
(This Microsoft software is supported in up-to-date versions of Windows 10, 11, and Server 2022.)
One additional ssh option is needed before any remotely generated "X windows" can appear on your local
desktop, e.g.,
ssh -X <your_username>@hermes.lps.cornell.edu
Note, if text labels are badly proportioned in ParaView's GUI, try checking
the Override Font box in its Settings dialog. On Windows, if the text looks too big in any GUI, go to
Start Menu > Preferences > Look and Feel > Appearance > Fonts > Details > Resolution,
and change 192 to 96, e.g.
Connecting from Older Versions of Windows
PuTTY is a free, secure ssh client for connecting to Linux or Unix systems. To enable graphical
display from your remote systems, PuTTY must be coupled with an X window server. VcXsrv and
Xming (both free and open-source) are good choices for this purpose. Here is the sequence for
starting an ssh session with Xming and X11 forwarding enabled:
- Start Xming by clicking its Quick Launch icon or by selecting it from the Start menu
- Start PuTTY via the shortcut on the desktop or via the Start menu
- Enter the address of your destination (or just click a name in your saved list and "Load" it)
- Include your username in the address line if you like, e.g., my_username@hermes.lps.cornell.edu
- Verify that SSH | X11 | Enable X11 forwarding is selected
- Select SSH | Enable compression, too, for better X11 performance
- Return to Session, give it a name if needed, and click "Save" so you can re-use it later
- Click "Open" to log in
After logging out from PuTTY, you can choose to quit Xming by right-clicking its icon in the System
Tray and selecting "Exit".
Shell Navigation and the Screen Command
On Hermes and Athena, the default Linux shell (=OS command processor) is bash. Becoming competent
in bash is a whole learning experience in itself. Here are some links to get you started:
The last two links give you advice on (1) how to get a long-running code like PERSEUS out of your way
while it is running, and (2) how to log out without killing your PERSEUS run. All by itself,
screen
will be adequate for both (1) and (2). Just do this:
screen
mpiexec -n 16 ./perseus
<Ctrl>-a d
The initial screen command places you into a new, detachable session. The final key sequence is what
detaches the session and drops you back to your original shell. You can safely log out at this point.
After you log back in, you can restore the previous session with "screen -R". To terminate the screen,
first restore it, then kill all its processes and exit the screen. If you want to be totally safe when
invoking screen (perhaps to the point of overkill), add nohup:
screen nohup mpiexec -n 16 ./perseus > outputfile &
Screen is per-session; it allows you to disconnect from a terminal session without killing it. Nohup
(no-hangup) is per-process; it keeps the process running even if the parent shell goes away.
Shell Customization
Why would you want to customize your shell environment in Linux? The main reasons would be to put
Open MPI tools such as mpif90 and mpiexec (mpirun) into your PATH, and to set up your environment to
favor the Intel compilers. However, Hermes and Athena are already set up to do these things for you
automatically every time you log in or start up a new shell. Therefore, the following information is
presented just for your general enlightenment.
In bash, login shells are initialized by the file ~/.bash_profile (in Ubuntu, ~/.profile),
while non-login interactive shells are initialized by the file ~/.bashrc. (Note, ~ and $HOME both
refer to your home directory.) To ensure consistency between both types of interactive shells, the
default ~/.bash_profile (or ~/.profile) merely calls the ~/.bashrc at login, before adding a few
locations to the PATH. So why not just put tweaks to the PATH or other environment variables in
~/.bashrc? The problem is, a parent shell will export its environment variables to its child shells. So
if one interactive shell starts another, which starts another... and ~/.bashrc is executed repeatedly...
then PATH (e.g.) grows and grows as a consequence. That is why your default .bash_profile and .bashrc
look like the following in CentOS 7, prior to any customization:
| .bash_profile |
.bashrc |
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/.local/bin:$HOME/bin
export PATH
-- PUT YOUR ENVIRONMENT ADDITIONS HERE --< |
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
-- PUT YOUR ALIAS ADDITIONS HERE -- |
You still have to be careful with such customizations because in Linux, shells you open on a
remote desktop or in a screen session can have unexpected behavior. As a result, you may
want to put if-then constructs into your .bashrc; testing is definitely a good idea. Note too that macOS
behaves a bit differently from Linux; interactive shells started by the Terminal app are all
login shells.
Editors
The usual Linux command-line editors are available: vi (really vim), emacs, and nano. If you prefer
working with a basic GUI editor, gedit is available as well.
If you prefer emacs—meaning you have already learned emacs well enough not to be intimidated by
it—then you don't necessarily have to start it in its own window via the "emacs" command. Instead,
you might want to start it with "emacs -nw" so it occupies your current terminal window. Why? Generally
it starts up faster that way in an "ssh -X" session. However, if you prefer to run emacs in a separate X
window, that should be fine too (since the emacs window is no longer unstable due to a bug in GTK).
Storing and Accessing Files
The main locations to store your files are (1) in your $HOME, i.e., in the directory /home/$USER, and
(2) on Hermes only, in a personal data directory, /dataN/$USER, which is on one of the two large-capacity
hard drives on that system. Handy commands to see how much disk space is used/available at your current
location are "du -s ." and "df ."
Backups
Data on Hermes and Athena are not backed up. To reduce the risk of data loss, keep copies of your
important files on more than one of the above devices (or on external storage media). Here is a command
that is useful for doing such a backup:
rsync -avz /dataP/$USER/ /dataQ/$USER
The trailing slash on the name of the first directory is significant—it means "get the contents
of this directory". The interpretation of the flags, -avz, is as follows: archive the files with original
properties, do it verbosely, and use compression while copying.
Compiling and Linking Codes
For everyone's convenience, certain commands are AUTOMATICALLY run in each new shell that is started
on Hermes and Athena. These commands are the ones that must be run in your shell before you can compile
and link your MPI codes using Open MPI and ifort, the Intel Fortran compiler. For the record, here are
the commands that run automatically on Hermes:
if [ -z "`echo $PATH | grep intel`" ]; then
. /opt/intel/bin/ifortvars.sh intel64
fi
PATH=/usr/bin:$PATH
LD_LIBRARY_PATH=/usr/lib64/openmpi/lib:$LD_LIBRARY_PATH
export OMPI_FC=ifort
And here are the ones that run automatically on Athena:
if [ -z "`echo $PATH | grep intel`" ]; then
source /opt/intel/oneapi/compiler/latest/env/vars.sh
fi
export PATH=$PATH:/opt/ParaView-5.10/bin/
export HWLOC_COMPONENTS=-gl
export OMPI_FC=ifort
On less-tailored systems, you might have to place similar commands in your .bash_profile or .profile
as noted above.
Using the Intel Fortran Compiler
Here is a recommended Makefile that you can use for compiling PERSEUS in
any shell. (It assumes that the above commands were auto-run for you when you opened the shell.)
Linking an MPI Library
Just make sure you use the "mpifort" (or mpif90) command, and the MPI linking should be done
correctly. Remember, every MPI implementation comes with its own "mpif.h" file; therefore, that file
is not part of your code, and you should never copy it from system to system!
Linking Other Libraries
The Unix convention is that the option "-lmylib" refers to libmylib.so or libmylib.a. If both
libraries are present in the same directory, the .so file (shared-object or dynamic library) will be
chosen over the .a file (static library). If the library cannot be found at link time, you can specify
the path to it via the -L option to the compiler. (The compiler just passes this option to the
linker.)
FLASH
The FLASH code should be set up and built
separately for each particular user application. Therefore, to use FLASH, you should follow the
instructions in /data3/scratch/FLASH/README to produce your own custom build. The compiling and linking
commands, along with all the appropriate options, are encapsulated in /data3/scratch/FLASH/Makefile.h.
Note that FLASH is an MPI application, so it should be run with mpirun as described in the README.
Visualization and Analysis
Windowing in Linux is built around X11, a system for displaying individual windows locally or
remotely. But the performance of X11 can be draggy if your remote window is running a rich,
graphics-heavy application like ParaView or Mathematica. For doing interactive visualization, it is
preferable to set up an entire remote desktop. Several solutions exist for this, notably VNC and OpenNX.
The solution chosen for Hermes and Athena is X2Go.
Opening a Remote Desktop with X2Go
- First, download and install the appropriate
X2Go client on your laptop or
workstation.
- Once that's done, set up a VPN as you would for an ssh connection.
- Start the X2Go client. From the Session menu, choose "New Session...", then enter the
following (for Athena, modify this in obvious ways):
| Session tab |
Connection tab |
Input/Output tab |
Media tab |
Shared folders tab |
Session name: hermes
Host: hermes.lps.cornell.edu
Login: <your username>
Session type - MATE (in drop-down) |
Connection speed -
WAN (move slider) |
Display - Custom -
Width: 1600, Height: 900 |
Enable sound support -
(clear check box)
Client side printing support -
(clear check box) |
(optional)
Path: <choose local path>
(click "Add")
(put checks in all 3 boxes) |
*Obviously these settings are just the minimal recommendations; you are welcome to
customize them further to suit your own preferences.
|
- OK the settings. A new "hermes" tile appears in the client window. Click it. Enter your password,
then "Ok".
- After a short wait, a new window should appear with a remote desktop on Hermes! Resize it
manually if you like.
- If you designated a shared folder, a file icon may appear on your desktop with a name ending in
(sshfs-disk-mate); ignore it.
Visit the X2Go website for complete information on
how to access your shared
folders.
- If you click the Pause button, or close the local window, your remote session continues to run!
You have merely disconnected it.
- To log back in, enter your password and hit "Ok" again. Your desktop should appear exactly as you
left it (even from a different machine!).
- To terminate the session completely, don't log out. Go back to the initial X2Go window and click the
0/1 button.
If you need to terminate an X2Go session on Hermes that has become unreachable, the easiest way to do
so is to log in with ssh and enter the following, which will kill every single process that you are
running (including the shell you just started!):
pkill -9 -u $USER
In macOS, the X2Go client does not always have the correct key mappings. The familiar "Cmd+V" shortcut
for pasting the Clipboard may not work in xterm and other remote windows. Try the following
modification to .Xdefaults on your Mac to
restore the Cmd+V shortcut (a restart of XQuartz will be required):
echo "*VT100.translations: #override Meta <KeyPress> V: insert-selection(PRIMARY, CUT_BUFFER0) \n" >> ~/.Xdefaults
If that doesn't work, simply right-click and choose "Paste" from the contextual menu, or use the
X11 mouse shortcut
for paste instead. (The latter trick may require you to enable 3-button-mouse emulation in XQuartz
Preferences.)
Finally, if you see the wrong characters showing up while you are typing on your Mac, disconnect your
session and reconnect to it. Often this is enough to correct the problem.
Running ParaView and VisIt
As mentioned previously, visualization applications like ParaView and Visit are likely to be more
responsive if they are started up within an X2Go desktop.
From the MATE desktop on Hermes, the "ParaView 5.8.1" application can be started from the
"Applications > Graphics >" menu; on Athena, look for "ParaView 5.10 (Mesa)" under
"Menu > Other". An even handier shortcut is available on the top menu bar.
On Hermes, ParaView 5.8.1 or 4.3.1 can also be started from the command line. If you like, the command
can be shortened by adjusting your PATH to include the ParaView directory.
VTK_DEBUG_SKIP_VISRTX_CHECK=1 /opt/ParaView-5.8.1-MPI-Linux-Python2.7-64bit/bin/paraview-mesa paraview &
/opt/ParaView-4.3.1-Linux-64bit/bin/paraview &
Note that with ParaView 5.8.1, paraview must be started through the paraview-mesa wrapper in order to
get OpenGL to work properly. If the application crashes, setting the indicated variable may prevent
trouble.
On Athena, ParaView 5.10 can be easily started from the command line, as the application should
already be in your path:
paraview --mesa
On either system, if the ParaView icon doesn't appear on the top panel, try entering the following
commands. The second one is needed only if the ParaView icon doesn't appear after you enter the first.
The third command will completely reset the appearance of the desktop.
mate-panel --reset
mate-panel --reset --layout rhel-pv # replace rhel-pv with familiar-pv on Athena
rm -rf ~/.config # undoes all your desktop customizations and reverts to system defaults
Only Hermes has VisIt, and you can start it with the following command:
/opt/VisIt-3.3.3/bin/visit -gui
Running Mathematica
If you are logged into Hermes with ssh -X and have an X server running locally, just type "mathematica"
in your shell, and this will start up the GUI for version 13.3.
This also works in an X2Go remote desktop session, though your cursor may appear annoying offset from
its actual position when you are typing commands into the GUI for version 13.3.
Version 13.1, which does not have the issue with the cursor, is also available as "mathematica.old".
If you want to run the older Mathematica in an X2Go remote desktop, the following startup command is
necessary to avoid a Qt5 bug (or quirk):
QT_XCB_GL_INTEGRATION=none mathematica &
You might need an activation key if want to use Mathematica on Hermes. To get one, send an email to
the author of this page.
Setting Up Python 3 and Conda Environments
Your best Python alternative is to use the minimal Anaconda Python distribution available in
/usr/local/miniconda3. The installation includes Python 3.9, as well as the handy conda and pip package
managers. The latter are very useful for installing your preferred set of Python libraries in a customized
environment. To set up your (permanent) default Python environment through conda, run the following
command:
/usr/local/miniconda3/bin/conda init
The above command modifies your .bashrc so Anaconda Python and its tools will be found ahead of
others. This in turn enables you to use conda environments to install one or more collections of Python
libraries to suit your own needs:
conda create -n <name-of-your-environment> python=<your-preferred-version>
conda activate <name-of-your-environment>
You may want to edit your .bashrc to add the above "conda activate" command. Insert it after the lines
with conda's other modifications. That way, your custom environment, including your preferred version of
python, will be automatically activated every time you start a new shell. Conversely, you may want to
deactivate all conda environments (including conda's base environment) by default when you log in. To do
that, run the following command:
conda config --set auto_activate_base false
This produces a ~/.condarc file that enforces the desired configuration change. It prevents conda's
environments from altering your $PATH and other environment variables in in ways that might interfere
with your non-Python applications.
Note that the NumPy and SciPy libraries have been preinstalled for you in the base environment,
i.e., the one that gets activated if you don't create an environment of your own and specify it by
name in the "conda activate" command. If you need NumPy and SciPy in your own environment, simply
install them in that one too.
One of the packages you can install in a conda environment is the Spyder IDE (Integrated Development
Environment). Spyder is a popular choice for working with Python code. To install it in your currently
activated environment, simply type
conda install spyder
The Spyder app requires X windows. Therefore, either connect to Hermes via "ssh -X" and type "spyder"
at the command line; or, better, connect via X2Go and type the following command in a terminal window,
in order to avoid an X2Go-related bug (or quirk) with Qt5:
QT_XCB_GL_INTEGRATION=none spyder &
If you don't want to bother with conda at all, you don't have to: it turns out Python 3.6 is part of
standard CentOS 7—it was introduced between versions 7.6 and 7.8. Thus, the command "python3" is
sufficient to start a Python 3.6 interpreter (previously, it was necessary to enable Python 3 from the
Red Hat Software Collections, RHSCL). To make CentOS's Python your default interpreter, simply add this
to your .bashrc:
alias python=python3
The pip package manager is also installed on the system and available through the command "pip3".
Monitoring the Load
Here are some helpful commands to see what's happening on the machine. You may want to run them if
you're about to start a new MPI job, or if your current MPI job seems to be running more slowly than
you expected.
w
uptime
top
ps aux | grep -v root
A fully loaded 36-core machine will have a load average of 36 in "w" and "uptime". Anything over that
is likely to be using the cores inefficiently. The "top" command lets you monitor the machine's activity
continuously and in much more detail. It lets you see which processes are the heavy CPU users (type "q"
to quit). The one-shot "ps" command allows you to filter out things you don't want to see in "top".
Last updated on 11/11/24 by Steve Lantz (slantz ~at~ cac.cornell.edu)
