https://www.cac.cornell.edu/wiki/api.php?action=feedcontributions&user=Pzv2&feedformat=atomCAC Documentation wiki - User contributions [en]2024-03-29T07:03:04ZUser contributionsMediaWiki 1.35.5https://www.cac.cornell.edu/wiki/index.php?title=Docker&diff=3590Docker2021-10-06T16:56:50Z<p>Pzv2: Added an introduction section</p>
<hr />
<div>'''This page is under construction'''<br />
<br />
'''"Docker"''' is a word used to refer to the company [https://www.docker.com/ Docker Inc.], the container runtime application they developed, and many related technologies. In our usage here, we are typically referring to the ''Docker runtime'' for containers, and will try to otherwise specify with more description.<br />
<br />
[https://en.wikipedia.org/wiki/Docker_(software) Docker] is a [https://en.wikipedia.org/wiki/Containerization containerization] technology, developed in 2013 based on [https://en.wikipedia.org/wiki/LXC Linux Containers (LXC)]. The Docker runtime has become a very popular technology for software applications and cyberinfrastructure implementations in a broad range of areas. Here we will focus on scientific software applications running in Docker on [[Red Cloud]] virtual machines (VMs) or [[OpenStack#Instances|instances]]. In addition to this page, you may want to explore the [https://docs.docker.com/ official documentation] or the Docker guide to [https://www.docker.com/get-started Get Started] with their technologies.<br />
<br />
Please note that for CAC resources, Docker is only available in Red Cloud, not on [[Private Clusters]]. Some Private Clusters, however, may have [[Singularity]] installed, which can run Docker images after they have been converted. For more information, please see the documentation for your cluster.<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
To get started with using Docker on Red Cloud VMs, you will need to use a <br />
<br />
=== Installation ===<br />
<br />
For a complete list of instructions, see the Docker [https://docs.docker.com/engine/install/ installation guide]. Docker provides instructions for both types of [[Red Cloud Linux Instances|Linux instance]] types available in Red Cloud: [https://docs.docker.com/engine/install/ubuntu/ Ubuntu] and [https://docs.docker.com/engine/install/centos/ CentOS]. While there are instructions for Windows, we do not recommend running Docker on Windows in Red Cloud. For this step-by-step guide, we will use a Ubuntu VM.<br />
<br />
# First, [[OpenStack#Launch_an_Instance|launch a new instance]], specifically a new [[Red_Cloud_Linux_Instances#Steps|Linux instance]] with the Ubuntu 18.04 LTS [[Images|image]]. [[Red_Cloud_Linux_Instances#Secure_Shell_.28SSH.29|Access]] the instance via <code>ssh</code>.<br />
# Make sure the system is up-to-date:<br />
#* <code>sudo apt update</code> and <code>sudo apt upgrade</code><br />
#* You may need to run <code>sudo reboot</code> and reconnect once the instance has finished restarting.<br />
# <br />
<br />
Once installed, see the post installation instructions (https://docs.docker.com/engine/install/linux-postinstall/), specifically the first step on user groups. Next, their getting started page (https://docs.docker.com/get-started/) provides a good base for using Docker.<br />
<br />
=== Using A Docker Image ===<br />
<br />
<br />
=== Useful Commands ===<br />
<br />
<br />
<br />
== Creating a Dockerfile ==</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Docker&diff=3589Docker2021-10-06T16:45:07Z<p>Pzv2: </p>
<hr />
<div>'''This page is under construction'''<br />
<br />
'''"Docker"''' is a word used to refer to the company [https://www.docker.com/ Docker Inc.], the container runtime application they developed, and many related technologies. In our usage here, we are typically referring to the ''Docker runtime'' for containers, and will try to otherwise specify with more description.<br />
<br />
The Docker runtime has become a very popular technology for software applications and cyberinfrastructure implementations in a broad range of areas. Here we will focus on scientific software applications running in Docker on Red Cloud. Please not that Docker is only available in [[Red Cloud]], not available on CAC Private Clusters. Some Private Clusters, however, may have [[Singularity]] installed, which can run Docker images after they have been converted. For more information, please see the documentation for your cluster.<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Installation ===<br />
<br />
For a complete list of instructions, see the Docker [https://docs.docker.com/engine/install/ installation guide]. Docker provides instructions for both types of [[Red Cloud Linux Instances|Linux instance]] types available in Red Cloud: [https://docs.docker.com/engine/install/ubuntu/ Ubuntu] and [https://docs.docker.com/engine/install/centos/ CentOS]. While there are instructions for Windows, we do not recommend running Docker on Windows in Red Cloud. For this step-by-step guide, we will use a Ubuntu VM.<br />
<br />
# First, [[OpenStack#Launch_an_Instance|launch a new instance]], specifically a new [[Red_Cloud_Linux_Instances#Steps|Linux instance]] with the Ubuntu 18.04 LTS [[Images|image]]. [[Red_Cloud_Linux_Instances#Secure_Shell_.28SSH.29|Access]] the instance via <code>ssh</code>.<br />
# Make sure the system is up-to-date:<br />
#* <code>sudo apt update</code> and <code>sudo apt upgrade</code><br />
#* You may need to run <code>sudo reboot</code> and reconnect once the instance has finished restarting.<br />
# <br />
<br />
Once installed, see the post installation instructions (https://docs.docker.com/engine/install/linux-postinstall/), specifically the first step on user groups. Next, their getting started page (https://docs.docker.com/get-started/) provides a good base for using Docker.<br />
<br />
=== Using A Docker Image ===<br />
<br />
<br />
=== Useful Commands ===<br />
<br />
<br />
<br />
== Creating a Dockerfile ==</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Resizing_an_Instance&diff=3587Resizing an Instance2021-10-01T02:20:25Z<p>Pzv2: Added a warning about instance reboot</p>
<hr />
<div>'''WARNING:''' Resizing an instance in the "Active" state will '''''reboot''''' the instance, so be sure to save any active work before attempting.<br />
<br />
<br />
<br />
A variety of [[OpenStack#Instances|instance sizes]] are available on [[Red Cloud]]. The instance size (or type) defines how much memory (RAM) is available, the amount of per-instance storage (typically available through /dev/vdb in [[Red Cloud Linux Instances | Linux instances]]), and the number of CPU cores available.<br />
<br />
Resizing an instance allows you to dynamically control your work process. During periods of heavy development, you may only want a small instance type to develop on, but during periods of heavy computational activity, a large instance (or multiple large instances) may be desirable.<br />
<br />
The options can be found by clicking on the Resize Instance option in the menu for the instance on the right side of the instances listing page:<br />
<br />
[[File:White_square.png|100px|frameless]][[File:Resize Instance Menu.png|150px|frameless|border]]<br />
<br />
This option is only available for instances whose [[OpenStack#Instance States|state]] is either "Active" or "Stopped". The dialog for resizing the instance type is shown below:<br />
<br />
[[File:Resize Instance Dialog.png|500px|frameless|border]]<br />
<br />
Simply select the new flavor you would like and then select "Resize".</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Share_An_Image_In_Openstack&diff=3551Share An Image In Openstack2021-06-11T14:57:23Z<p>Pzv2: Added some more details to the process of sharing an image</p>
<hr />
<div>You can share an image from your project with a collaborator in a different project in OpenStack cloud so you can both launch instances using the same image. As the owner of the image, you can revoke the sharing privilege at any time. You can also use these methods to share an image with yourself in other projects, just think of yourself as the collaborator.<br />
<br />
'''Prerequisites:'''<br />
# Both you and your collaborator need to use the [[OpenStack CLI]].<br />
#* You will need to set up the CLI to work with the project where the image to be shared is located.<br />
#* If you are sharing an image with yourself, you will need to download the [[OpenStack_CLI#Download_OpenStack_RC_File|RC File]] for both projects (the one you are sharing from and to).<br />
# You will need to know the project name or ID of your collaborator.<br />
<br />
=== Share an Image With Another Project ===<br />
<br />
'''1. Find the project ID of your collaborator first:'''<br />
:The project ID is the string of the format <code>xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx</code> composed of lowercase letters and numbers. The project name in the typical format will not work in the below commands. You will need to have sourced the [[OpenStack_CLI#Download_OpenStack_RC_File|RC File]] of the project where the image is located before running the [[OpenStack CLI]] command to get the project ID:<br />
openstack project list<br />
:and the first field of the output is the ID of your collaborator's project ID. If you know the name of your collaborator's project, you can find the ID by adding <code>| grep <collaborator's project name></code> to the above command.<br />
'''2. Find the ID of the image you want to share:'''<br />
openstack image show <image name><br />
:Similar to project ID, this will be a long string of letters and numbers next to the name of your image. You will use the image ID, rather than the name, in the commands below.<br />
'''3. Share the image with your collaborator's project:'''<br />
openstack image set --shared <image ID><br />
openstack image add project <image ID> <collaborator's project ID><br />
'''4. Verify the image is now shared:'''<br />
openstack image member list <image ID><br />
:The status field should say <code>pending</code> until your collaborator accepts the image.<br />
'''5. Give the ID of the shared image to your collaborator''' so they can follow the steps in the [[#Accept a Shared Image|Accept a Shared Image]] section to add the image to their project. If you are sharing the image with yourself, you will need to source the [[OpenStack_CLI#Download_OpenStack_RC_File|RC File]] of the second project and then perform the acceptance yourself.<br />
<br />
=== Share an Image with All Projects ===<br />
<br />
* Currently this action can only be performed by cloud administrators.<br />
* This generally shouldn't be done unless creating something that is a public resource that is intended to be used with any project.<br />
* Additionally, don't use such an image from a third party unless you trust the source.<br />
openstack image set --public <image ID><br />
<br />
=== Accept a Shared Image ===<br />
'''1. Get the ID of the shared image from the owner'''. Make sure the owner has performed the steps in the [[#Share_an_Image | Share an Image]] section above, and source the [[OpenStack_CLI#Download_OpenStack_RC_File|RC File]] of your project.<br />
'''2. Accept the shared image:'''<br />
openstack image set --accept <image ID><br />
'''3. Verify the image is now available to your project:'''<br />
openstack image list<br />
:If the image is listed in the output, it should also appear in OpenStack web console (Horizon).<br />
<br />
=== Unshare an Image ===<br />
* An image's owner can see which projects have access to the image:<br />
openstack image member list <image ID><br />
<br />
* The owner can unshare an image like this:<br />
openstack image remove project <image ID> <project ID></div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Linux_Tutorial_for_Red_Cloud_Test&diff=3444Linux Tutorial for Red Cloud Test2021-02-08T21:53:36Z<p>Pzv2: Edit got missed when page title changed</p>
<hr />
<div>This tutorial is intended as a basic introduction to [https://en.wikipedia.org/wiki/Linux Linux] for users of CAC's Linux resources, especially those who are managing [[Red_Cloud|Red Cloud]] services and are creating or using a [[Red Cloud Linux Instances|Linux Instance]]. There are two [https://en.wikipedia.org/wiki/Linux_distribution Linux distributions] (AKA distros) available for [[Images|images]] on Red Cloud: [https://en.wikipedia.org/wiki/Ubuntu_(operating_system) Ubuntu] and [https://en.wikipedia.org/wiki/CentOS CentOS]. In this tutorial, you will learn how to add a user, install software using the distribution's [https://en.wikipedia.org/wiki/Package_manager package manager], and enable remote password logins, along with a number of related tips. The tutorial begins with [[Linux_Tutorial#Basic_Useful_Commands|common commands]] between both distributions, followed by sections for the specific commands you are likely to need on [[Linux_Tutorial#Ubuntu|Ubuntu]] and [[Linux_Tutorial#CentOS|CentOS]]. While many commands are identical across Linux systems, there are some areas that will differ from distribution to distribution, such as package management, service control, and (to a lesser extent) user management.<br />
<br />
If you want more help with Linux, you may find the [https://cvw.cac.cornell.edu/Linux/ Introduction to Linux] topic on the [https://cvw.cac.cornell.edu/topics Cornell Virtual Workshop] useful. For a really thorough treatment, including guidance on how to write scripts in bash, you can check out "Learning the Bash Shell" from the [https://www.library.cornell.edu Cornell Library] (available as an e-book).<br />
<br />
__TOC__<br />
<br />
== Definitions ==<br />
<br />
This section contains some basic working definitions to help you through this tutorial if you have never used Linux before. This list and the definitions in it should not be considered authoritative.<br />
<br />
'''directory''' - folder<br />
<br />
'''path''' - the sequence of directories leading to a particular subdirectory or file<br />
<br />
'''shell''' (A.K.A. '''console''' or '''terminal''') - a text-only user interface for interacting with an operating system's programs and services. This is where commands are entered.<br />
<br />
'''command''' - a task for the computer to execute that is entered via the shell<br />
<br />
'''environment''' - the set of all variables defined in the current shell. The special environment variable <tt>PATH</tt> shows the sequence of paths that will be searched to find the commands that you enter.<br />
<br />
'''package''' - an archive of software and metadata that can be downloaded, installed, and removed via a package manager<br />
<br />
'''root''' - the system administrative account with all the highest privileges, also known as the superuser. By default, most Linux distros have a single root account when installed, and no user accounts.<br />
<br />
'''sudo''' - a program that allows a user to run commands with the privileges of another user, most often the root or superuser account. This is typically used by typing sudo before a command.<br />
<br />
'''root directory''' - the top-level directory of the system, denoted <code>/</code> (forward slash). It is the start of most paths. This is not the same as the root user.<br />
<br />
'''home directory''' - the top-level directory of the user, denoted <code>~</code> (tilde). It is the start of most paths where the user will store work.<br />
<br />
== Basic Useful Commands ==<br />
<br />
This list is not comprehensive, but rather a starting point.<br />
<br />
; <code>pwd</code> : print working directory - specifically, print the full path to the current working directory<br />
; <code>ls</code> : list directory contents<br />
; <code>cd</code> : change directory <p>Example: <code>cd ~</code> will take you to your home directory</p><br />
; <code>mkdir <name></code> : make a directory with the specified name<br />
; <code>man <command></code> : display a manual pages for the specified command<br />
; <code>which <command></code> : show the full path to the given command, as found from the paths in the PATH environment variable<br />
; <code>history</code> : display a list of commands that have been executed via the terminal<br />
; <code>cat <file></code> : output (concatenate) the contents of a file to the terminal, with many other options available (check out <code>man cat</code> for more info)<br />
; <code>less <file></code> : output the contents of a file one screenful at a time, with page-advance (spacebar) and search (/) functions. The <code>more</code> command works similarly.<br />
; <code>grep <pattern> <file></code> : print lines from the file that match the specified pattern. To search multiple files, use the <code>*</code> wildcard. To search a directory tree, use the <code>-r</code> option. The <code>-x</code> option prints lines that DON'T match the pattern.<br />
; <code><command> | less</code><br />
; <code><command> | grep <pattern></code> : join commands with a <code>|</code> in order to "pipe" the output from the first command into the second, e.g., into <code>less</code> to paginate it, or into <code>grep</code> to search it. <p>Example: <code>history | grep mkdir</code> would search the history output for each time the <code>mkdir</code> command was executed, thus determining all the directories you had created.</p><br />
; <code>export VAR=value</code> : set an environment variable (VAR in this example) to have a certain value<br />
; <code>ssh</code> : [[Connect_to_Linux#Using_Secure_Shell|Secure Shell (ssh)]] is great for getting a quick command-line interface where you can enter bash shell commands. It also allows you to do X11 forwarding, which enables you to interact graphically with your instance using [[Connect_to_Linux#Using_X-Windows|X Windows]]. If you have not already, it would also be good to familiarize yourself with how to [[Connect_to_Linux|connect to Linux machines remotely]].<br />
<br />
=== Text Editors === <br />
<br />
Since the default interaction with a [[Red Cloud Linux Instances|Linux Instance]] is through a terminal, it may be useful to familiarize yourself with at least one text editor that can be used in the terminal. Here are a few, with links to get more information about them, but there are more.<br />
<br />
; <code>vim</code> : [https://en.wikipedia.org/wiki/Vim_(text_editor) Vim] is often already installed with many Linux distros, and is therefore useful to learn. There are many online tutorials, but you can also simply type <code>vimtutor</code> in the terminal to learn how to use vim.<br />
; <code>emacs</code> : [https://en.wikipedia.org/wiki/Emacs Emacs] is a family of text editors including the very popular [https://en.wikipedia.org/wiki/GNU_Emacs GNU Emacs]. If you want to use it, it may be helpful to take a [https://www.gnu.org/software/emacs/tour/ guided tour] or to consult the [https://www.gnu.org/software/emacs/manual/html_node/emacs/index.html manual].<br />
; <code>nano</code> : [https://en.wikipedia.org/wiki/GNU_nano GNU nano] is a simpler text editor than something like vim because it doesn't have modes, you simply type when it opens. If you'd like more information, consult the [https://www.nano-editor.org/docs.php documentation].<br />
<br />
== Ubuntu ==<br />
<br />
This section has specific instructions for Ubuntu images on how to create your [[#Initial User Setup| first user with sudo privileges]], [[#Additional Users|create additional users]], and [[#Installing Software|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup|Initial User Setup]] steps. If you do want to create [[#Additional Users|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used Ubuntu before, please read this whole section.<br />
<br />
=== The "ubuntu" User ===<br />
<br />
Since the Ubuntu distribution of Linux [https://help.ubuntu.com/community/RootSudo locks the root account] by default, you cannot use that account to ssh when you first setup a new image. Instead, there is a default account with the username <code>ubuntu</code>, with a blank password, that has sudo privileges. Unless you are the sole user of your machine, it is still recommended that you create a new [https://help.ubuntu.com/lts/serverguide/user-management.html user account], for which the steps are detailed below.<br />
<br />
=== Initial User Setup ===<br />
<br />
These steps create a new sudo user, and must all be '''completed in order''':<br />
# <code>ssh -i <keyname>.pem ubuntu@<ip of instance></code><br />
#* Connects to the instance via ssh as the [[Linux_Tutorial#The_.22ubuntu.22_user|ubuntu account]].<br />
# <code>sudo adduser <username></code><br />
#* You will be prompted to enter & verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one. You will also be prompted for some information (i.e. name, phone number, etc.) which is optional. If you do not wish to add this information, simply hit "enter".<br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members.<br />
#* This adds a new user with the name <username>.<br />
# <code>sudo adduser <username> sudo</code><br />
#* This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>sudo mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh.<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>sudo cp ~/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>sudo chown -hR <username>:<username> /home/<username></code><br />
#* Changes the ownership of the user's home directory and subdirectories (including .ssh) to the user.<br />
# <code>sudo chmod 700 -R ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the folder and all files contained within.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* Verify the line that says <code>PasswordAuthentication</code> has a <code>no</code> next to it (this should be the default).<br />
#* '''Change this to <code>yes</code> only if''' you intend to have multiple users and wish to allow them to connect via ssh with a password, without requiring a [[OpenStack#Key Pairs| key pair]].<br />
#* You could also use your preferred text editor<br />
# <code>sudo systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to connect via ssh.<br />
# <code>sudo apt update</code> and <code>sudo apt upgrade</code><br />
#* The first 'sudo apt update' command is what updates the package manager's list of available packages, upgrades, etc.<br />
#* The second 'sudo apt upgrade' command is what actually performs the upgrades to packages that have updates.<br />
#* You can now begin [[Linux_Tutorial#Installing_Software|Installing Software]].<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo chown -hR <username> /home/<username></code> <br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo adduser <username> sudo</code><br />
# <code>sudo chown -hR <username> /home/<username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code> <p>'''Or''' you will be prompted for a password when you initially create the user with <code>sudo adduser <username></code></p><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to edit the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and change the line that says <code>PasswordAuthentication no</code> to <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for Ubuntu is called [https://en.wikipedia.org/wiki/APT_(Debian) apt] (also see the Ubuntu docs on [https://help.ubuntu.com/lts/serverguide/apt.html apt] and [https://help.ubuntu.com/lts/serverguide/aptitude.html aptitude]). Here are some basic commands worth making sure you understand (again, <code>man apt</code> will help here):<br />
<br />
:* <code>sudo apt update</code><br />
:* <code>sudo apt upgrade</code><br />
:* <code>apt search <package></code><br />
:* <code>sudo apt install <package></code><br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance. <br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
To find available packages (from currently installed repositories), the following command may be used: <code>apt search <package></code>. For instance, here are the first 6 results for <code>apt search python</code>:<br />
<br />
p bpython - fancy interface to the Python interpreter<br />
p bpython-gtk - fancy interface to the Python interpreter<br />
p bpython-urwid - fancy interface to the Python interpreter<br />
p bpython3 - fancy interface to the Python3 interpreter<br />
p cairo-dock-plug-ins-dbus-interf - Python interface to interact with Cairo-Do<br />
p cantor-backend-python - Python backend for Cantor<br />
<br />
Note that the ‘p’ in the first column means that no trace of package exists on the system (run <code>man apt</code> for more details).<br />
<br />
== CentOS ==<br />
<br />
This section has specific instructions for CentOS images on how to create your [[#Initial User Setup_2| first user with sudo privileges]], [[#Additional Users_2|create additional users]], and [[#Installing Software_2|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup_2|Initial User Setup]] steps. If you do want to create [[#Additional Users_2|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used CentOS before, please read this whole section.<br />
<br />
=== Initial User Setup ===<br />
Once you have started a [[Red Cloud Linux Instances|Linux Instance]], you will want to connect as the [http://en.wikipedia.org/wiki/Superuser#Unix_and_Unix-like root user] and set up a new user account that you can use for your day-to-day work. The way to do this depends on the CentOS base image: on some you can log in directly as ‘root’, but on others you first have to log in as user ‘centos’, then switch to ‘root’ as shown below. In the latter case you could choose to make ‘centos’ the account you use each time you want to connect, but it is recommended that you follow the steps below so you can pick a separate username of your own choosing. <br />
<br />
# <code>ssh -i <keyname>.pem centos@<ip of instance></code><br />
#* Connects to the instance via ssh as the ‘centos’ account (note, by default in Red Cloud, direct login by root is disabled )<br />
# <code>sudo su -</code><br />
#* Subsequent steps are easier if you become root, so you don't have to type ‘sudo’ ahead of each command.<br />
# <code>adduser <username></code><br />
#* Adds a new user with the name <username><br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members<br />
#* Multiple users may be added at the instance owner’s discretion (see below). <br />
# <code>passwd <username></code><br />
#* This will prompt you to set and verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one.<br />
#* '''Note''': if you do not run this command, a password will not be set for the user!<br />
# <code>usermod -aG wheel <username></code><br />
#*This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>cp ~centos/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>chown -hR <username>:<username> /home/<username>/.ssh</code><br />
#* Changes the ownership of the user's .ssh directory and all the files and subdirectories in it to the user.<br />
# <code>chmod 700 ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the .ssh folder to the recommended level.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* '''Skip this step if''' you never intend for users to connect via ssh with a password and instead want them to use a [[OpenStack#Key Pairs| key pair]].<br />
#* Uncomment the line that says <code>PasswordAuthentication yes</code>.<br />
#* Comment out the line that says <code>PasswordAuthentication no</code>.<br />
#* Note: you may need to install vim first by running <code>yum install vim</code>. See [[#Installing_Software_2|Installing Software]] for more information.<br />
#* You could also use your preferred text editor<br />
# <code>systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to ssh<br />
# <code>sudo yum update</code><br />
#* This makes sure the system is up-to-date<br />
#* You may notice that certain packages do not get updated; don't be alarmed, as this is expected<br />
#* It's a good idea to restart the instance after the update completes<br />
#* You can now begin [[#Installing_Software_2|Installing Software]]<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
# <code>usermod -aG wheel <username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to check the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and ensure the line that says <code>PasswordAuthentication</code> says <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for CentOS is called [https://en.wikipedia.org/wiki/Yum_(software) yum]. Here are some basic commands worth making sure you understand (again, <code>man yum</code> will help here):<br />
<br />
:* <code>yum check-update</code><br />
:* <code>sudo yum update</code><br />
:* <code>yum search <package></code><br />
:* <code>sudo yum install <package></code><br />
<br />
You may notice that certain packages do not get updated with <code>sudo yum update</code>; don't be alarmed, as this is expected. It's a good idea to restart the instance after the update completes.<br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance.<br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
=== SSH Security ===<br />
<br />
Once you have set up a user with sudo privileges and ensured that you can indeed login and perform sudo commands successfully (it would be good to test this to be sure), you may want to secure the root login by disabling it.<br />
<br />
'''Disable root login:'''<br />
This must be done while logged in either as root or your user with sudo privileges.<br />
<br />
# <code>vim /etc/ssh/sshd_config</code><br />
# Change the the line <code>PermitRootLogin yes</code> to <code>PermitRootLogin no</code><br />
# '''Note''': if this line is commented out (with a <code>#</code> character in the front), you will need to uncomment it.<br />
# <code>systemctl restart sshd</code><br />
<br />
When you exit, you should verify that you cannot login as root, but that you can still login as your user.<br />
<br />
For more information on SSH Security, see the [https://wiki.centos.org/HowTos/Network/SecuringSSH CentOS guide on Securing OpenSSH].</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Security_Practices&diff=3443Security Practices2021-02-08T21:05:26Z<p>Pzv2: Added links</p>
<hr />
<div>There are different ways to provide security to your [[Red Cloud]] [[OpenStack#Instances|instances]]. The two main steps that go into creating an instance are [[OpenStack Key Pairs|key pair]] and [[OpenStack Security Groups|security group]] creation. By modifying these two steps you can provide greater security to your instance. On this page, we will go over recommended security practices for your purposes.<br />
<br />
__TOC__<br />
<br />
== Recommended Key Pair Security Practices ==<br />
<br />
The recommended way to use key pairs is through a password-protected key pair. The instructions to do so are [[OpenStack_Key_Pairs_Test#Creating_a_Password-Protected_Key_Pair_(Recommended)|here]]. <br />
<br />
You can also create a key pair using the OpenStack GUI, and the instructions are found [[OpenStack_Key_Pairs#Creating_a_Key_Pair_(using_OpenStack)|here]].<br />
<br />
== Recommended Security Group Practices ==<br />
<br />
== Recommended CIDR Specifications ==<br />
<br />
== Other Security Options ==</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Docker&diff=3407Docker2021-01-11T20:01:58Z<p>Pzv2: Started installation instructions</p>
<hr />
<div>'''This page is under construction'''<br />
<br />
Intro<br />
<br />
Only available in [[Red Cloud]], not available on CAC Private Clusters.<br />
<br />
__TOC__<br />
<br />
<br />
== Getting Started ==<br />
<br />
=== Installation ===<br />
<br />
For a complete list of instructions, see the Docker [https://docs.docker.com/engine/install/ installation guide]. Docker provides instructions for both types of [[Red Cloud Linux Instances|Linux instance]] types available in Red Cloud: [https://docs.docker.com/engine/install/ubuntu/ Ubuntu] and [https://docs.docker.com/engine/install/centos/ CentOS]. While there are instructions for Windows, we do not recommend running Docker on Windows in Red Cloud. For this step-by-step guide, we will use a Ubuntu VM.<br />
<br />
# First, [[OpenStack#Launch_an_Instance|launch a new instance]], specifically a new [[Red_Cloud_Linux_Instances#Steps|Linux instance]] with the Ubuntu 18.04 LTS [[Images|image]]. [[Red_Cloud_Linux_Instances#Secure_Shell_.28SSH.29|Access]] the instance via <code>ssh</code>.<br />
# Make sure the system is up-to-date:<br />
#* <code>sudo apt update</code> and <code>sudo apt upgrade</code><br />
#* You may need to run <code>sudo reboot</code> and reconnect once the instance has finished restarting.<br />
# <br />
<br />
Once installed, see the post installation instructions (https://docs.docker.com/engine/install/linux-postinstall/), specifically the first step on user groups. Next, their getting started page (https://docs.docker.com/get-started/) provides a good base for using Docker.<br />
<br />
=== Using A Docker Image ===<br />
<br />
<br />
=== Useful Commands ===<br />
<br />
<br />
<br />
== Creating a Dockerfile ==</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Linux_Tutorial_Test&diff=3405Linux Tutorial Test2021-01-11T19:43:40Z<p>Pzv2: /* Initial User Setup */</p>
<hr />
<div>This tutorial is intended as a basic introduction to [https://en.wikipedia.org/wiki/Linux Linux] for users of CAC's Linux resources, especially those who are managing [[Red_Cloud|Red Cloud]] services and are creating or using a [[Red Cloud Linux Instances|Linux Instance]]. There are two [https://en.wikipedia.org/wiki/Linux_distribution Linux distributions] (AKA distros) available for [[Images|images]] on Red Cloud: [https://en.wikipedia.org/wiki/Ubuntu_(operating_system) Ubuntu] and [https://en.wikipedia.org/wiki/CentOS CentOS]. In this tutorial, you will learn how to add a user, install software using the distribution's [https://en.wikipedia.org/wiki/Package_manager package manager], and enable remote password logins, along with a number of related tips. The tutorial begins with [[Linux_Tutorial#Basic_Useful_Commands|common commands]] between both distributions, followed by sections for the specific commands you are likely to need on [[Linux_Tutorial#Ubuntu|Ubuntu]] and [[Linux_Tutorial#CentOS|CentOS]]. While many commands are identical across Linux systems, there are some areas that will differ from distribution to distribution, such as package management, service control, and (to a lesser extent) user management.<br />
<br />
If you want more help with Linux, you may find the [https://cvw.cac.cornell.edu/Linux/ Introduction to Linux] topic on the [https://cvw.cac.cornell.edu/topics Cornell Virtual Workshop] useful. For a really thorough treatment, including guidance on how to write scripts in bash, you can check out "Learning the Bash Shell" from the [https://www.library.cornell.edu Cornell Library] (available as an e-book).<br />
<br />
__TOC__<br />
<br />
== Definitions ==<br />
<br />
This section contains some basic working definitions to help you through this tutorial if you have never used Linux before. This list and the definitions in it should not be considered authoritative.<br />
<br />
'''directory''' - folder<br />
<br />
'''path''' - the sequence of directories leading to a particular subdirectory or file<br />
<br />
'''shell''' (A.K.A. '''console''' or '''terminal''') - a text-only user interface for interacting with an operating system's programs and services. This is where commands are entered.<br />
<br />
'''command''' - a task for the computer to execute that is entered via the shell<br />
<br />
'''environment''' - the set of all variables defined in the current shell. The special environment variable <tt>PATH</tt> shows the sequence of paths that will be searched to find the commands that you enter.<br />
<br />
'''package''' - an archive of software and metadata that can be downloaded, installed, and removed via a package manager<br />
<br />
'''root''' - the system administrative account with all the highest privileges, also known as the superuser. By default, most Linux distros have a single root account when installed, and no user accounts.<br />
<br />
'''sudo''' - a program that allows a user to run commands with the privileges of another user, most often the root or superuser account. This is typically used by typing sudo before a command.<br />
<br />
'''root directory''' - the top-level directory of the system, denoted <code>/</code> (forward slash). It is the start of most paths. This is not the same as the root user.<br />
<br />
'''home directory''' - the top-level directory of the user, denoted <code>~</code> (tilde). It is the start of most paths where the user will store work.<br />
<br />
== Basic Useful Commands ==<br />
<br />
This list is not comprehensive, but rather a starting point.<br />
<br />
; <code>pwd</code> : print working directory - specifically, print the full path to the current working directory<br />
; <code>ls</code> : list directory contents<br />
; <code>cd</code> : change directory <p>Example: <code>cd ~</code> will take you to your home directory</p><br />
; <code>mkdir <name></code> : make a directory with the specified name<br />
; <code>man <command></code> : display a manual pages for the specified command<br />
; <code>which <command></code> : show the full path to the given command, as found from the paths in the PATH environment variable<br />
; <code>history</code> : display a list of commands that have been executed via the terminal<br />
; <code>cat <file></code> : output (concatenate) the contents of a file to the terminal, with many other options available (check out <code>man cat</code> for more info)<br />
; <code>less <file></code> : output the contents of a file one screenful at a time, with page-advance (spacebar) and search (/) functions. The <code>more</code> command works similarly.<br />
; <code>grep <pattern> <file></code> : print lines from the file that match the specified pattern. To search multiple files, use the <code>*</code> wildcard. To search a directory tree, use the <code>-r</code> option. The <code>-x</code> option prints lines that DON'T match the pattern.<br />
; <code><command> | less</code><br />
; <code><command> | grep <pattern></code> : join commands with a <code>|</code> in order to "pipe" the output from the first command into the second, e.g., into <code>less</code> to paginate it, or into <code>grep</code> to search it. <p>Example: <code>history | grep mkdir</code> would search the history output for each time the <code>mkdir</code> command was executed, thus determining all the directories you had created.</p><br />
; <code>export VAR=value</code> : set an environment variable (VAR in this example) to have a certain value<br />
; <code>ssh</code> : [[Connect_to_Linux#Using_Secure_Shell|Secure Shell (ssh)]] is great for getting a quick command-line interface where you can enter bash shell commands. It also allows you to do X11 forwarding, which enables you to interact graphically with your instance using [[Connect_to_Linux#Using_X-Windows|X Windows]]. If you have not already, it would also be good to familiarize yourself with how to [[Connect_to_Linux|connect to Linux machines remotely]].<br />
<br />
=== Text Editors === <br />
<br />
Since the default interaction with a [[Red Cloud Linux Instances|Linux Instance]] is through a terminal, it may be useful to familiarize yourself with at least one text editor that can be used in the terminal. Here are a few, with links to get more information about them, but there are more.<br />
<br />
; <code>vim</code> : [https://en.wikipedia.org/wiki/Vim_(text_editor) Vim] is often already installed with many Linux distros, and is therefore useful to learn. There are many online tutorials, but you can also simply type <code>vimtutor</code> in the terminal to learn how to use vim.<br />
; <code>emacs</code> : [https://en.wikipedia.org/wiki/Emacs Emacs] is a family of text editors including the very popular [https://en.wikipedia.org/wiki/GNU_Emacs GNU Emacs]. If you want to use it, it may be helpful to take a [https://www.gnu.org/software/emacs/tour/ guided tour] or to consult the [https://www.gnu.org/software/emacs/manual/html_node/emacs/index.html manual].<br />
; <code>nano</code> : [https://en.wikipedia.org/wiki/GNU_nano GNU nano] is a simpler text editor than something like vim because it doesn't have modes, you simply type when it opens. If you'd like more information, consult the [https://www.nano-editor.org/docs.php documentation].<br />
<br />
== Ubuntu ==<br />
<br />
This section has specific instructions for Ubuntu images on how to create your [[#Initial User Setup| first user with sudo privileges]], [[#Additional Users|create additional users]], and [[#Installing Software|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup|Initial User Setup]] steps. If you do want to create [[#Additional Users|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used Ubuntu before, please read this whole section.<br />
<br />
=== The "ubuntu" User ===<br />
<br />
Since the Ubuntu distribution of Linux [https://help.ubuntu.com/community/RootSudo locks the root account] by default, you cannot use that account to ssh when you first setup a new image. Instead, there is a default account with the username <code>ubuntu</code>, with a blank password, that has sudo privileges. Unless you are the sole user of your machine, it is still recommended that you create a new [https://help.ubuntu.com/lts/serverguide/user-management.html user account], for which the steps are detailed below.<br />
<br />
=== Initial User Setup ===<br />
<br />
These steps create a new sudo user, and must all be '''completed in order''':<br />
# <code>ssh -i <keyname>.pem ubuntu@<ip of instance></code><br />
#* Connects to the instance via ssh as the [[Linux_Tutorial#The_.22ubuntu.22_user|ubuntu account]].<br />
# <code>sudo adduser <username></code><br />
#* You will be prompted to enter & verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one. You will also be prompted for some information (i.e. name, phone number, etc.) which is optional. If you do not wish to add this information, simply hit "enter".<br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members.<br />
#* This adds a new user with the name <username>.<br />
# <code>sudo adduser <username> sudo</code><br />
#* This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>sudo mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh.<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>sudo cp ~/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>sudo chown -hR <username>:<username> /home/<username></code><br />
#* Changes the ownership of the user's home directory and subdirectories (including .ssh) to the user.<br />
# <code>sudo chmod 700 -R ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the folder and all files contained within.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* Verify the line that says <code>PasswordAuthentication</code> has a <code>no</code> next to it (this should be the default).<br />
#* '''Change this to <code>yes</code> only if''' you intend to have multiple users and wish to allow them to connect via ssh with a password, without requiring a [[OpenStack#Key Pairs| key pair]].<br />
#* You could also use your preferred text editor<br />
# <code>sudo systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to connect via ssh.<br />
# <code>sudo apt update</code> and <code>sudo apt upgrade</code><br />
#* The first 'sudo apt update' command is what updates the package manager's list of available packages, upgrades, etc.<br />
#* The second 'sudo apt upgrade' command is what actually performs the upgrades to packages that have updates.<br />
#* You can now begin [[Linux_Tutorial#Installing_Software|Installing Software]].<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo chown -hR <username> /home/<username></code> <br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo adduser <username> sudo</code><br />
# <code>sudo chown -hR <username> /home/<username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code> <p>'''Or''' you will be prompted for a password when you initially create the user with <code>sudo adduser <username></code></p><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to edit the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and change the line that says <code>PasswordAuthentication no</code> to <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for Ubuntu is called [https://en.wikipedia.org/wiki/APT_(Debian) apt] (also see the Ubuntu docs on [https://help.ubuntu.com/lts/serverguide/apt.html apt] and [https://help.ubuntu.com/lts/serverguide/aptitude.html aptitude]). Here are some basic commands worth making sure you understand (again, <code>man apt</code> will help here):<br />
<br />
:* <code>sudo apt update</code><br />
:* <code>sudo apt upgrade</code><br />
:* <code>apt search <package></code><br />
:* <code>sudo apt install <package></code><br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance. <br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
To find available packages (from currently installed repositories), the following command may be used: <code>apt search <package></code>. For instance, here are the first 6 results for <code>apt search python</code>:<br />
<br />
p bpython - fancy interface to the Python interpreter<br />
p bpython-gtk - fancy interface to the Python interpreter<br />
p bpython-urwid - fancy interface to the Python interpreter<br />
p bpython3 - fancy interface to the Python3 interpreter<br />
p cairo-dock-plug-ins-dbus-interf - Python interface to interact with Cairo-Do<br />
p cantor-backend-python - Python backend for Cantor<br />
<br />
Note that the ‘p’ in the first column means that no trace of package exists on the system (run <code>man apt</code> for more details).<br />
<br />
== CentOS ==<br />
<br />
This section has specific instructions for CentOS images on how to create your [[#Initial User Setup_2| first user with sudo privileges]], [[#Additional Users_2|create additional users]], and [[#Installing Software_2|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup_2|Initial User Setup]] steps. If you do want to create [[#Additional Users_2|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used CentOS before, please read this whole section.<br />
<br />
=== Initial User Setup ===<br />
Once you have started a [[Red Cloud Linux Instances|Linux Instance]], you will want to connect as the [http://en.wikipedia.org/wiki/Superuser#Unix_and_Unix-like root user] and set up a new user account that you can use for your day-to-day work. The way to do this depends on the CentOS base image: on some you can log in directly as ‘root’, but on others you first have to log in as user ‘centos’, then switch to ‘root’ as shown below. In the latter case you could choose to make ‘centos’ the account you use each time you want to connect, but it is recommended that you follow the steps below so you can pick a separate username of your own choosing. <br />
<br />
# <code>ssh -i <keyname>.pem centos@<ip of instance></code><br />
#* Connects to the instance via ssh as the ‘centos’ account (note, by default in Red Cloud, direct login by root is disabled )<br />
# <code>sudo su -</code><br />
#* Subsequent steps are easier if you become root, so you don't have to type ‘sudo’ ahead of each command.<br />
# <code>adduser <username></code><br />
#* Adds a new user with the name <username><br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members<br />
#* Multiple users may be added at the instance owner’s discretion (see below). <br />
# <code>passwd <username></code><br />
#* This will prompt you to set and verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one.<br />
#* '''Note''': if you do not run this command, a password will not be set for the user!<br />
# <code>usermod -aG wheel <username></code><br />
#*This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>cp ~centos/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>chown -hR <username>:<username> /home/<username>/.ssh</code><br />
#* Changes the ownership of the user's .ssh directory and all the files and subdirectories in it to the user.<br />
# <code>chmod 700 ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the .ssh folder to the recommended level.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* '''Skip this step if''' you never intend for users to connect via ssh with a password and instead want them to use a [[OpenStack#Key Pairs| key pair]].<br />
#* Uncomment the line that says <code>PasswordAuthentication yes</code>.<br />
#* Comment out the line that says <code>PasswordAuthentication no</code>.<br />
#* Note: you may need to install vim first by running <code>yum install vim</code>. See [[#Installing_Software_2|Installing Software]] for more information.<br />
#* You could also use your preferred text editor<br />
# <code>systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to ssh<br />
# <code>sudo yum update</code><br />
#* This makes sure the system is up-to-date<br />
#* You may notice that certain packages do not get updated; don't be alarmed, as this is expected<br />
#* It's a good idea to restart the instance after the update completes<br />
#* You can now begin [[#Installing_Software_2|Installing Software]]<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
# <code>usermod -aG wheel <username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to check the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and ensure the line that says <code>PasswordAuthentication</code> says <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for CentOS is called [https://en.wikipedia.org/wiki/Yum_(software) yum]. Here are some basic commands worth making sure you understand (again, <code>man yum</code> will help here):<br />
<br />
:* <code>yum check-update</code><br />
:* <code>sudo yum update</code><br />
:* <code>yum search <package></code><br />
:* <code>sudo yum install <package></code><br />
<br />
You may notice that certain packages do not get updated with <code>sudo yum update</code>; don't be alarmed, as this is expected. It's a good idea to restart the instance after the update completes.<br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance.<br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
=== SSH Security ===<br />
<br />
Once you have set up a user with sudo privileges and ensured that you can indeed login and perform sudo commands successfully (it would be good to test this to be sure), you may want to secure the root login by disabling it.<br />
<br />
'''Disable root login:'''<br />
This must be done while logged in either as root or your user with sudo privileges.<br />
<br />
# <code>vim /etc/ssh/sshd_config</code><br />
# Change the the line <code>PermitRootLogin yes</code> to <code>PermitRootLogin no</code><br />
# '''Note''': if this line is commented out (with a <code>#</code> character in the front), you will need to uncomment it.<br />
# <code>systemctl restart sshd</code><br />
<br />
When you exit, you should verify that you cannot login as root, but that you can still login as your user.<br />
<br />
For more information on SSH Security, see the [https://wiki.centos.org/HowTos/Network/SecuringSSH CentOS guide on Securing OpenSSH].</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud_Linux_Instances_Test&diff=3395Red Cloud Linux Instances Test2021-01-06T18:15:10Z<p>Pzv2: /* Troubleshooting */ Added a link</p>
<hr />
<div>Linux Instances can be created and maintained on [[Red_Cloud|Red Cloud]] using the [https://redcloud.cac.cornell.edu OpenStack Web Interface]. This documentation assumes a basic understanding of [[OpenStack]], so please review that page as needed. If you are '''new to Linux''', you may want to read the [[Linux Tutorial]] first. It is also a good idea to be familiar with the [[Linux Tutorial]] if you have not previously done '''Linux system administration''', which is an assumed prerequisite to managing Linux Instances. Additionally, you may find the [https://cvw.cac.cornell.edu/Linux/ Introduction to Linux] topic on the [https://cvw.cac.cornell.edu/topics Cornell Virtual Workshop] helpful.<br />
<br />
__TOC__<br />
<br />
== Creating a New Linux Instance ==<br />
<br />
You can boot an instance with most modern Linux distributions. Currently Red Cloud offers pre-made VM images running the following Linux distributions:<br />
<br />
:* CentOS<br />
:* Ubuntu (including [https://wiki.ubuntu.com/LTS LTS])<br />
<br />
=== Steps ===<br />
# Log in to the [https://redcloud.cac.cornell.edu OpenStack Web Interface] (check out [[OpenStack#Logging_In|how to log in]] if you need to)<br />
# If you have not already, [[OpenStack Key Pairs#Creating_a_Key_Pair|create a key pair]]<br />
# If you have not already, [[OpenStack Security Groups#Creating a Security Group|create a security group]]. Note that your security group should include the inbound SSH rule so you can connect to it.<br />
# '''Optional:''' [[Networks#Private Networks|Set up a Private Network]]<br />
# Select <tt>Launch Instance</tt> from the [https://redcloud.cac.cornell.edu/dashboard/project/instances/ Instances] page<br />
# Follow the instructions about [[OpenStack#Launch an Instance|launching a new instance]], and select one of the a CentOS or Ubuntu [[Images|images]] under the <tt>Source</tt> tab<br />
# '''Optional:''' [[Volumes#Create and Attach a Volume|Create and attach a Volume]]<br />
# '''Optional:''' [[Networks#Floating IP Addresses|Associate a Floating IP address]]<br />
<br />
Now that you have created and launched an instance, your next steps will be to [[#Accessing_Instances|connect to it]] and set up a new user account. See the [[Linux_Tutorial#Initial_User_Setup_2|CentOS steps]] or [[Linux_Tutorial#Initial_User_Setup|Ubuntu steps]] for more information on how to set up a new user, update, and install software for each distribution.<br />
<br />
== Accessing Instances ==<br />
<br />
First, establish access to your instance using [[Connect_to_Linux#Using_Secure_Shell|Secure Shell (ssh)]], possibly including [[Connect_to_Linux#Using_X-Windows|X Windows]] for remote graphical display. If you are at all unfamiliar with Linux, we recommend following the [[Linux Tutorial]]. If you would like to have a desktop environment (not available by default for Linux instances), then you may want to follow the steps for [[XFCE Desktop on CentOS]]. Establishing an SSH connection is a prerequisite for creating a Linux desktop environment such as XFCE.<br />
<br />
=== Secure Shell (SSH) ===<br />
<br />
The main requirements for logging in to your instance using secure shell are:<br />
:* The [[OpenStack Security Groups|security group]] for your instance must permit SSH connections (TCP port 22) from your current IP address.<br />
<br />
:* You must use the private key that matches the public key in the [[OpenStack Key Pairs|key pair]] you specified when launching the instance.<br />
<br />
:* You must log in to your instance using the correct initial username:<br />
:** For CentOS 6 and 7, the username is <tt>centos</tt>,<br />
:** For CentOS 8, the username is <tt>cloud-user</tt>, and<br />
:** For Ubuntu, it is <tt>ubuntu</tt>.<br />
<br />
To log in through SSH, specify the key pair file (for example <tt>my_key.pem</tt>),<br />
username and IP address as follows:<br />
<br />
ssh -i my_key.pem <username>@<IP address of your instance><br />
<br />
For more information on how to use SSH, see the [[Connect to Linux]] page.<br />
<br />
Note: Transferring files can also be done over <code>ssh</code>. See the <code>scp</code> and <code>sftp</code> commands, or programs like [//winscp.net/eng/index.php WinSCP] and [//apple.stackexchange.com/questions/25661/whats-a-good-graphical-sftp-utility-for-os-x similar software for Mac OS X].<br />
<br />
=== Troubleshooting ===<br />
<br />
There are several common error messages you may see when trying to access your Linux instance using SSH.<br />
These are the most likely causes for each common message.<br />
<br />
:* '''"Connection timed out"''' means that your SSH command can't communicate at all with the instance.<br />
:** Note: It may take a while for the connection time out to occur, so it may seem that the system is not responding at all initially. However, this is still likely a "Connection timed out" error.<br />
:** Has the instance been started? Check the instance's console in the [[OpenStack|OpenStack Web Interface]]. Did the instance boot successfully?<br />
:** Do the [[OpenStack Security Groups|security group's]] rules allow incoming ssh connections (TCP port 22) from ''the IP address of your computer''? Double check that you followed the [[OpenStack Security Groups|instructions for security groups]].<br />
:** Alternatively, if you are not on campus but the security group is configured for ingress from 10.0.0.0/8, you would need to be connected to the [https://it.cornell.edu/cuvpn Cornell VPN] in order to SSH into the instance.<br />
:** Is your instance on the [[Networks#Public Network|public network]]? If not, does it have an associated [[Networks#Floating IP Addresses|floating IP address]] and are you using the correct address?<br />
<br />
:* An error like '''"Permissions <4-digits> for <key-file-name> are too open"''' means you can reach the instance, but your private key file has improper permissions.<br />
:** Make sure your private key file is saved as a ".pem" extension and that it has the proper permissions: <p><code>chmod 600 <key name>.pem</code></p><br />
<br />
:* '''"Permission denied (<some details>)"''' means that the combination of the username and SSH key you are providing are not correct for this instance.<br />
:** Make sure you are using the correct username:<br />
:*** If you are using Ubuntu, did you login as the '''ubuntu''' user? For more information on that, see the [[Linux_Tutorial#The_.22ubuntu.22_User|Linux Tutorial]].<br />
:*** If you are using CentOS 6 or 7, did you login as the '''centos''' user?<br />
:*** If you are using CentOS 8, did you login as the '''cloud-user''' user?<br />
:*** If you are using the CentOS MATLAB instance, did you login as the '''root''' user?<br />
:** Did you [[OpenStack_Key_Pairs#Creating_a_Key_Pair|create a key pair]] and make sure to [[OpenStack_Key_Pairs#Selecting_a_Key_Pair_When_Creating_an_Instance|select it when creating the instance]]?<br />
:** Are you supplying your key pair in the command? See the [[#Secure_Shell_.28SSH.29|SSH]] instructions above for an example.<br />
:* If you get an '''unexpected password prompt''':<br />
:** Did you use the correct user name? See suggestions above.<br />
:** Make sure your private key matches the public key of the [[OpenStack Key Pairs|key pair]]. <br />
:**# On your computer, run: <code>ssh-keygen -y -f <private key file></code><br />
:**# Confirm that the output matches the public key on Red Cloud at [https://redcloud.cac.cornell.edu/dashboard/project/key_pairs https://redcloud.cac.cornell.edu/dashboard/project/key_pairs]/<your keypair name>.<br />
<br />
=== XFCE Desktop on CentOS ===<br />
<br />
In case one would prefer a desktop environment over a command-line, this section details the steps to setting up an [https://xfce.org/ XFCE Desktop Environment] on a CentOS 7.4 instance. Other typical Linux desktop environments, such as Gnome, are also available but XFCE is used as an example here. Additionally, setting up a desktop environment may work on Ubuntu instances as well, with some differences. Once the environment is set up, you can run a VNC server and connect to it through an ssh tunnel.<br />
<br />
==== XFCE Setup ====<br />
<br />
# Log in as root via ssh as [[#Secure_Shell_.28ssh.29 | described above]].<br />
# Install needed packages:<br />
#* <code>yum groupinstall xfce</code><br />
#* <code>yum install tigervnc-server</code><br />
# Install some additional software that most users will want. These are only suggestions, and this is not a comprehensive list<br />
#* <code>yum install gedit</code><br />
#* <code>yum install firefox</code><br />
#* [https://www.tecmint.com/install-libreoffice-on-rhel-centos-fedora-debian-ubuntu-linux-mint/ LibreOffice]<br />
<br />
==== VNC User Setup ====<br />
<br />
For each user that will want to use the XFCE Desktop, you will need to set up VNC (Virtual Network Computing) cability. To do this, follow the directions below. Alternatively, there is also an [https://linuxtechlab.com/secure-vnc-server-tls-encryption/ Easy guide to secure VNC server with TLS encryption].<br />
<br />
# Open a shell as that user<br />
# <code>vncpasswd</code><br />
#* Sets the user's VNC password<br />
#* This step is '''not necessary''' for read-only VNC<br />
#* This creates a ~/.vnc folder<br />
# <code>vim ~/.vnc/xstartup</code><br />
#* Paste this text into the file: <br /><tt>#!/bin/bash<br />xrdb $HOME/.Xresources<br />startxfce4 &<br /></tt><br />
# <code>chmod 775 ~/.vnc/xstartup</code><br />
# <code>mkdir ~/bin</code><br />
# <code>vim ~/bin/start_vncserver.sh</code><br />
#* Paste this text into the file: <br /><tt>#!/bin/bash<br />vncserver -geometry 1680x1050<br /></tt><br />
# <code>chmod 775 ~/bin/start_vncserver.sh</code><br />
<br />
==== Using VNC ====<br />
<br />
A brief overview for users is provided here, and for more information please see the [[Getting_Started#Using_VNC|Using VNC section on our Getting Started page]].<br />
<br />
===== Manage the VNC Server =====<br />
<br />
Whenever an instance gets rebooted, you can '''restart''' the VNC server by doing the following<br />
# ssh into the instance<br />
# run <code>~/bin/start_vncserver.sh</code><br />
<br />
To find the port, you can run e.g. <code>ps gxu | grep vnc</code>.<br />
<br />
You can '''stop''' the NVC server by running <code>vncserver -kill :1</code>. If this doesn't work,<br />
you can try <code>pkill Xvnc</code>.<br />
<br />
To '''change or reset the VNC password''', you can simply run <code>vncpassword</code><br />
<br />
To '''change the screen resolution''':<br />
:* Permanently: edit the script <code>~/.vnc/xstartup</code><br />
:* For this session only: use Applications->Display<br />
<br />
===== View the Desktop =====<br />
<br />
# Open a local shell and ssh to establish the tunnel (recommended local port 10000):<br />
#* <code>ssh -L 10000:localhost:5901 <remote ip></code><br />
#* '''Note:''' When this shell is closed, the VNC viewer will have to close, too, though VNC is still running.<br />
# In a VNC viewer app, connect to the VNC server <code>localhost:10000</code> (or whichever port you chose above).<br />
#* This should open a window showing the desktop<br />
#* The '''first time''' you do this:<br />
#** For the remote computer, you may have to dismiss a warning dialog<br />
#** You will need to initialize a "panel". Click "Use default config"<br />
<br />
'''Note for Windows users:''' You can also find useful instructions on the [[VNC Tunnel Windows]] page. You may also want to look into [https://fossbytes.com/enable-built-windows-10-openssh-client/ OpenSSH] or [https://www.windowscentral.com/how-install-bash-shell-command-line-windows-10 Bash on Ubuntu on Windows].<br />
<br />
== Instance Maintenance ==<br />
<br />
All self-managed desktops, laptops, servers, and Red Cloud instances, both Windows and Linux, should be updated with Operating System, Adobe Flash, Acrobat, and Reader critical and security updates on a '''''monthly''''' basis. For Linux instances:<br />
# Check for updates<br />
#* Ubuntu: <code>sudo apt update</code><br />
#* CentOS: <code>yum check-update</code><br />
# Install updates<br />
#* Ubuntu: <code>sudo apt upgrade</code><br />
#* CentOS: <code>sudo yum update</code><br />
# Reboot the instance with <code>sudo reboot</code> on both Ubuntu and CentOS<br />
<br />
Before rebooting make sure to save all active work. Rebooting will disconnect you from the instance. Wait a minute or two to allow the instance to restart before reconnecting. When you reconnect, you should verify that the updates were applied by repeating step 1.<br />
<br />
== Initialize and Mount a Volume ==<br />
<br />
The instructions here are for formatting and mounting [[Volumes|attached volumes]], though steps like these can only be performed if you have [[Volumes#Create_and_Attach_a_Volume|allocated and attached the volume]] through OpenStack, which can be done while the instance is running.<br />
<br />
'''Note:''' These instructions assume you are the [[Linux_Tutorial#Definitions|root user]]. If you are not (such as on [[Linux_Tutorial#The_.22ubuntu.22_User|Ubuntu]]), then you may need to prepend <code>sudo</code> where appropriate.<br />
<br />
# Set up file system:<br />
#* <code>mkfs.ext4 /dev/vdb</code><br />
# Make a directory where device will be mounted, for example "<tt>scratch</tt>":<br />
#* <code>mkdir /scratch</code><br />
# Mount the device:<br />
#* <code>mount /dev/vdb /scratch</code><br />
# To have the mount automatically renewed after reboot, add an <code>fstab</code> entry (this is a little dangerous)<br />
#* <code>vim /etc/fstab</code><br />
#* Add a line with tab separations between four fields: disk device, mounted location, "ext4", "defaults":<br />
#** <code>/dev/vdb /scratch ext4 defaults</code></div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=OpenStack_Test&diff=3394OpenStack Test2021-01-06T18:11:14Z<p>Pzv2: Suggested rewording and moved location of statement about flavor size</p>
<hr />
<div>[https://en.wikipedia.org/wiki/OpenStack OpenStack] is an [https://en.wikipedia.org/wiki/Open-source_model open-source] [https://en.wikipedia.org/wiki/Cloud_computing#Service_models cloud stack] that is currently running on [[Red_Cloud|Red Cloud]]. Also, for more information, see the [https://docs.openstack.org/ Official Documentation for OpenStack].<br />
<br />
This page is intended as a quick walk-through of the most-used features of OpenStack, so it is not comprehensive, but links to a lot of supporting documentation for more thorough explanations and advanced topics.<br />
<br />
__TOC__<br />
<br />
== Using the OpenStack Web Interface (Horizon) ==<br />
<br />
There are two ways to manage [[Red Cloud]] resources:<br />
# [https://redcloud.cac.cornell.edu OpenStack Web Interface]<br />
# [[OpenStack CLI]]<br />
<br />
Most users will use the OpenStack Web Interface (called [https://docs.openstack.org/horizon/latest/ Horizon]). This web-based interface can be used to manage [[#Instances|instances]] and [[Volumes|volumes]]. For [[Red Cloud Linux Instances|Linux Instances]], however, some users may choose to use the OpenStack CLI. This section focuses on the OpenStack Web Interface.<br />
<br />
=== Logging into OpenStack===<br />
<br />
Log in to the [https://redcloud.cac.cornell.edu OpenStack Web Interface] to create and manage Red Cloud resources. There are two ways to login: <br />
<br />
[[File:RedCloudCACLogin.png|300px|frameless|border]][[File:White_square.png|100px|frameless]][[File:RedCloudGlobusAuthLogin.png|300px|frameless|border]]<br />
<br />
# [https://www.cac.cornell.edu/services/myacct.aspx CAC Account] - Enter '''cac''' as the "<tt>Domain</tt>" and your [https://www.cac.cornell.edu/services/myacct.aspx CAC username] and password, not your Cornell NetID. If your CAC password has expired, you will need to [https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started#Managing_your_password reset it] before you will be able to login to the OpenStack Web Interface.<br />
# [https://www.globus.org/tags/globus-auth Globus Auth] - Log in through Globus<br />
#* Currently, this feature is '''only available to Aristotle users'''. This feature will be enabled for all users in the future.<br />
#* You must ''link your Cornell account'', or any accounts attached to the projects you are on, in order to have access to them when using Globus Auth.<br />
#* If you can't log in with Globus Auth, it may be that you have not linked your account yet.<br />
<br />
You can use the "<tt>Authenticate using</tt>" drop-down to switch between the two options. Neither option requires you to enter a project ID; you can switch between the projects you are on once logged in.<br />
<br />
=== Overview Page ===<br />
<br />
The Overview page is the first place you will be taken upon logging into Red Cloud.<br />
<br />
:* Provides useful metrics on currently selected project<br />
:* '''Before creating an instance''', you will need to:<br />
:** Select the correct project from the "<tt>Project</tt>" drop-down at the top right of the page (if you are on multiple projects)<br />
:** [[#Key_Pairs|Create a key pair]] - for authentication when you log in the first time<br />
:** [[#Security_Groups|Create a security group]] - defines allowable types of port access for an instance<br />
:** Optional: [[Networks#Private_Networks|Set up a private network]] - if you do not want your instance to be available on the [[Networks#Public Network|public network]]<br />
:* You may also want to:<br />
:** [[Volumes#Create and Attach a Volume|Create and Attach a Volume]] (can also be done when launching an instance)<br />
:** [[Networks#Floating IP Addresses|Associate a Floating IP address]] - a fixed IP address that can be assigned to an instance<br />
<br />
=== Key Pairs ===<br />
<br />
[[File:Overview_KeyPairs_Circled.png|350px|frameless|border]]<br />
<br />
To get to the Key Pairs page: select the "<tt>Compute</tt>" tab along the top (you should start here at login), then click on "<tt>Key Pairs</tt>" along the top bar as pictured above. If you are logged in already, you can also get to it by this link: [https://redcloud.cac.cornell.edu/dashboard/project/key_pairs/ Key Pairs].<br />
<br />
On the Key Pairs page, you can view the list of available [[OpenStack_Key_Pairs|key pairs]] for your project. From here, you can also [[OpenStack_Key_Pairs#Creating_a_Key_Pair|create]] or [[OpenStack_Key_Pairs#Importing_a_Key_Pair|import]] a key pair. If you do not already have a key pair listed, you can either create one before [[OpenStack#Launch_an_Instance|launching an instance]], or [[OpenStack_Key_Pairs#Selecting_a_Key_Pair_When_Creating_an_Instance|create or upload a key pair]] during instance setup.<br />
<br />
For more information, here is a walk-through on [[OpenStack Key Pairs]].<br />
<br />
=== Security Groups ===<br />
<br />
[[File:Overview_SecurityGroups.png|350px|frameless|border]]<br />
<br />
To get to the Security Groups page: select the "<tt>Network</tt>" drop-down menu along the top, then click on "<tt>Security Groups</tt>" as pictured above. If you are already logged in, you can also get to it by following this link: [https://redcloud.cac.cornell.edu/dashboard/project/security_groups/ Security Groups]<br />
<br />
On the Security Groups page, you can view a list of available [[OpenStack Security Groups|security groups]] for your project, including a default security group. On this page, you can also [[OpenStack_Security_Groups#Creating_a_Security_Group|create]] and delete security groups. It '''is ''not''''' recommended that you use the default security group without [[OpenStack_Security_Groups#Managing_a_Security_Group.27s_Rules|modifying the rules]] to fit your needs. A good security practice is to have one security group per application or one per user. Instances that have no business talking to each other should generally be in separate security groups.<br />
<br />
If you do not already have a security group set up, you will want to [[OpenStack_Security_Groups#Creating_a_Security_Group|create]] one before [[OpenStack#Launch_an_Instance|launching an instance]] because you cannot create one during instance setup. However, you can [[OpenStack_Security_Groups#Assigning_Security_Groups_to_an_Instance|assign a security group]] to an instance later, and even [[OpenStack_Security_Groups#Adding_a_Rule_to_a_Security_Group|add]] or<br />
[[OpenStack_Security_Groups#Managing_a_Security_Group.27s_Rules|modify the rules]] of the security group at any time. <br />
<br />
For more information, here is a walk-through on [[OpenStack Security Groups]].<br />
<br />
== Instances ==<br />
<br />
Each instance is a Virtual Machine (VM) in the cloud. You can select CPU/RAM/disk configurations (called "flavors") for the VM. Note that each vCPU currently equates to one core. The available VM configurations are:<br />
<br />
{| border="1" cellspacing="0" cellpadding="10" align="center" style="text-align:center;"<br />
! Flavor <br />
! vCPUs<br />
! GPUs <br />
! RAM <br />
|-<br />
| c1.m8 || 1 || None || 8 GB<br />
|-<br />
| c2.m16 || 2 || None || 16 GB<br />
|-<br />
| c4.m32 || 4 || None || 32 GB<br />
|-<br />
| c8.m64 || 8 || None || 64 GB<br />
|-<br />
| c14.m112 || 14 || None || 112 GB<br />
|-<br />
| c20.m160 || 20 || None || 160 GB<br />
|-<br />
| c28.m224|| 28 || None || 224 GB<br />
|-<br />
| *''c4.t1.m20'' || 4 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' || 20 GB<br />
|-<br />
| *''c14.g1.m60'' || 14 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' || 60 GB<br />
|-<br />
| colspan="4" style="text-align:left;" | ''* GPU flavors<br />
|}<br />
<br />
When you are first starting an instance, we '''recommend''' that you select the smallest flavor (least number of CPUs) that you think will be able to handle installation and configuration of the software and environment on your instance, and then [[Resizing an Instance|resize the instance]] when you are ready to run. The "c1.m8" flavor will typically be enough, as you will not need much memory or compute power while setting up your software. This way you will save core hours that would otherwise have been spent idle. This method is especially useful when configuring a ''GPU instance'' due to the number of cores. Also note: you can begin with a smaller instance size (or flavor) that does not contain a GPU, and later resize to one that does.<br />
<br />
The root disk size of the instance will default to the size of the [[Images|image]] you select. You have the option to create a [[Volumes|volume]] as the root disk beyond the image size at launch time. Note that we do not oversubscribe physical RAM, CPU cores, or GPUs (hyperthreading is disabled).<br />
<br />
To work with instances, select the "<tt>Instances</tt>" page under the "<tt>Compute</tt>" tab, as pictured below:<br />
<br />
[[File:InstancesMenu.png|350px|frameless|border]]<br />
<br />
=== Launch an Instance ===<br />
<br />
This section is a general walk-through for creating a new instance, which is not specific to an Operating System (OS). For more specific information per OS, see either of these pages:<br />
<br />
:* [[Red Cloud Linux Instances|Linux Instances]]<br />
:* [[Red Cloud Windows Instances|Windows Instances]]<br />
<br />
==== To launch a new instance ====<br />
<br />
# [[#Key_Pairs|Create Key Pair]]<br />
# [[#Security Groups|Create a Security Group]] and be sure that you select the appropriate rule for connecting to your instance (SSH for [[Red Cloud Linux Instances|Linux Instances]] and RDP for [[Red Cloud Windows Instances|Windows Instances]])<br />
# Select "<tt>Launch Instance</tt>" on the top right side of the [https://redcloud.cac.cornell.edu/dashboard/project/instances/ Instances] page [[File:InstancesOptions.png|600px|frameless|border]]<br />
<br />
The full "<tt>Launch Instance</tt>" menu will pop up like this:<br />
<br />
[[File:InstanceLaunchMenuFull.png|700px|frameless|border]]<br />
<br />
:* Tabs that you are required to fill out are marked with a '''*'''<br />
:* It is '''recommended''' that you also select your own Security Group, otherwise the default security group will be selected, which may not be ideal for your work.<br />
:* It is '''necessary''' that you select your own Key Pair, even though this field is not marked required, so that you are able to connect to your instance after creation.<br />
<br />
==== Configuring the Instance ====<br />
<br />
# Under the "<tt>Details</tt>" tab:<br />
#* Enter a name for your instance<br />
#* '''Count''' is the number of identical instances you would like to create (typically 1).<br />
#** Note that if you create multiple instances this way, the names will be identical with a dash and number added at the end.<br />
#** For example, if you set Instance Name to "my_instance" and you set Count to 3, you would start instances named "my_instance-1", "my_instance-2", and "my_instance-3".<br />
# On the "<tt>Source</tt>" tab:<br />
#* You must '''Select Boot Source''', which is described on the page as "the template used to create an instance."<br />
#** It is generally a good idea to start with an [[Images|image]] as the source, unless you want to create an instance from a pre-existing source.<br />
#** For more information on the other options, see [[Images#Creating an Image|Creating an Image]].<br />
#* You can select a specific source under the <tt>Available</tt> list by selecting the up arrow on the right-hand side.<br />
#* Get more details about the specific source by selecting the right-arrow on the left-hand side next to the name.<br />
#* You will have the option to '''Create New Volume''' if you have selected either "<tt>Image</tt>" or "<tt>Instance Snapshot</tt>" as the source (default is "<tt>Yes</tt>"):<br />
#*# '''<tt>Yes</tt>''': If selected, a [[Volumes|volume]] will be created to be the instance's root disk. You will then have the options of extending the size of the volume beyond the image size, and deleting the volume when the instance is deleted.<br />
#*#* '''Volume Size''' is the size of your root [[Volumes|volume]]. The default number will match the size of the [[Images|image]] you select, and can be increased.<br />
#*#* '''Delete Volume on Instance Delete''' determines whether or not the root volume will be deleted automatically when you terminate the instance. The default is "<tt>No</tt>", which prevents your data from being deleted when you delete your instance (perhaps accidentally). However, ''if you do not need this extra level of protection, and you do not intend to re-use the root volume, you could unintentionally incur excess storage usage''. Therefore, it can be a good idea to set this option to "<tt>Yes</tt>" so that the volume is deleted automatically when you terminate your instance. Your alternative is to find and delete the root volume manually, later (it will show up in the list of volumes with a name identical to its arbitrarily assigned ID, unless you give it a different name).<br />
#*#* You can also customize the name of the volume under '''Device Name'''.<br />
#*# '''<tt>No</tt>''': If selected, the instance will boot off a root disk the same size as the image. The root disk will be deleted when the instance is deleted.<br />
# The "<tt>Flavor</tt>" tab is where you select the VM configuration discussed [[#Instances|above]].<br />
#* We '''recommend''' that you select the smallest flavor (least number of CPUs) that you think will be able to handle installation and configuration of the software and environment on your instance, and then [[Resizing an Instance|resize the instance]] when you are ready to run. This way you will save core hours that would otherwise have been spent idle. Also note: you can begin with a smaller instance size (or flavor) that does not contain a GPU, and later resize to one that does.<br />
#* You can select a configuration by selecting the up arrow on the right-hand side.<br />
#* Get more details about the configuration by selecting the right-arrow on the left-hand side next to the name.<br />
#* The "<tt>Total Disk</tt>" will show "0 GB" because the disk size will match the [[Images|image]] you selected on the "<tt>Source</tt>" tab.<br />
# For the "<tt>Networks</tt>" tab, two options are available:<br />
## You can make the instance available on the [[Networks#Public Network|public net]]. This is the simplest and most common selection.<br />
## You can select your own [[Networks#Private Networks|private network]], which has to be set up before you launch an instance. For more information, see the [[Networks]] page.<br />
# On the "<tt>Security Groups</tt>" tab, select the [[#Security Groups|security group]] you already created.<br />
# On the "<tt>Key Pairs</tt>" tab, select the [[#Key Pairs|key pair]] you already created.<br />
<br />
=== Instance States ===<br />
<br />
OpenStack defines several [https://developer.openstack.org/api-guide/compute/server_concepts.html#server-status Server States] through which you can move your instances. You change the state of your instance by making a selection from a drop-down menu under the <tt>Actions</tt> column. Three significant actions to know about are "Resize Instance", "Shelve Instance", and "Unshelve Instance"; these are described below.<br />
<br />
Allowed actions&mdash;i.e., the ones that appear in the drop-down menu&mdash;''depend on the current state of the instance''. For example, the "Resize Instance" action is allowed only for instances that are in the Active state. The figure below shows the possible states in OpenStack and the transitions that are allowed in each case.<br />
<br />
[[File:Openstack-server-states.png|thumb|left|700px|Source: OpenStack[https://docs.openstack.org/nova/latest/reference/vm-states.html]]]<br />
<div style="clear: both"></div><br />
<br />
When your instance has been created, the "<tt>Instances</tt>" tab will list its current state (as well as the state of your other instances) under the "<tt>Status</tt>" column. In the rightmost column called "<tt>Actions</tt>," you will see a drop-down menu for each instance. This menu lists the actions that are allowed for the given instance. Below we describe the typical states and list some of the common actions you will use to change instance state.<br />
<br />
==== Important States ====<br />
<br />
'''''Note: The only state where you are NOT being charged for computational resources is Shelved Offloaded'''''<br />
<br />
:* '''Active''': Instance is active, you can connect to it and are being billed for the computational resources dedicated to it.<br />
:* '''Shelved Offloaded''': The Instance is not resident on the compute host; this means you will not be billed for computational resources while the Instance is in this state (although you will be charged for the storage required to hold it). You can restart the server when you need it again.<br />
:* '''Paused''': In this state, the server state is preserved in RAM, but operations have been stopped and will resume when instructed. You are still being charged for the computational resources dedicated to the Instance.<br />
:* '''Suspended''': Instance state has been stored on disk, including the contents of its RAM. With Red Cloud's configuration, you are still paying for the computational resources you were using.<br />
:* '''Stopped''': This is like powering off a server; in this state, you are still being billed for the computational resources.<br />
:* '''Resized''': At this time, the Instance is being Resized to a different flavor&mdash;that is, a different allocation of vCPUs and RAM&mdash;and is not contactable.<br />
<br />
==== Operations to transition between states ====<br />
<br />
<br />
These options are available, subject to the current state of the Instances, from the dropdown available in the "Actions" column of the Instances page. ''Remember that Shelving is the only operation that will free up the computational resources your Instance has been using so that you stop being charged for them!''<br />
<br />
:* '''Pause Instance:''' Put instance into the Paused state.<br />
:* '''Suspend Instance''': Put instance into the Suspended state.<br />
:* '''Shelve Instance''': This is how you put the Instance aside so that you are no longer charged for computational resources being used; your Instance will still be visible on the Instances page with Status Shelved Offloaded. To get the Instance back up and running, select "Unshelve" from the actions menu.<br />
:* '''Resize Instance''': Allows you to select a new Instance flavor. After you have selected it, Status will be shown as "Confirm or Revert Resize/Migrate" you will have to confirm in the Actions dropdown, after which the Instance will be taken down and then come back up with the new computational resources available to it. On Linux you can check with commands such as <code>lscpu</code> or <code>cat /proc/cpuinfo</code>; on Windows you can, for example, use the "Performance" tab of Task Manager to see the available cores.<br />
:* '''Resume Instance''': Will restart the Instance from states of Paused and Suspended<br />
:* '''Soft Reboot Instance''' or '''Hard Reboot Instance''': Reboot your instance, either through issuing a command to the operation system ("Soft") or as if power-cycling the Instance ("Hard")<br />
:* '''Shut Off Instance''': Like powering off the Instance, an action you will need to confirm in a pop-up window. The Status shown will be Shutoff and you will need to "Restart" to get it back. Additionally, if you are logged into an instance and use an OS command such as <code>sudo poweroff</code> or <code>sudo init</code>, the Instances page will show the same status and you will need to select "Restart" to get the Instance back up and running. In this state, ''you are still being charged for computational resources''.</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=OpenStack_Key_Pairs_Test&diff=3393OpenStack Key Pairs Test2021-01-06T17:53:53Z<p>Pzv2: Minor formatting change</p>
<hr />
<div>The best way to provide secure and easy access to your [[Red Cloud]] [[OpenStack#Instances|instances]] is through the use of key pairs for [https://www.ssh.com/ssh/public-key-authentication SSH authentication]. Key pairs are made up of a private key that only you know, and a public key that is distributed to people and systems with which you would like to have secure communications. Red Cloud allows you to easily generate or upload such key pairs to use with your instances.<br />
<br />
When you [[OpenStack#Launch_an_Instance|create a new instance]], you should specify a key pair to be used for logging in to that instance. '''You can only add a key pair to an instance at the time of its creation''', not afterwards, so it is important not to overlook this step. It is possible to generate a new key pair during the process of creating an instance.<br />
<br />
In [[Red Cloud Linux Instances|Linux instances]], the pair's public key is installed into the root (or ubuntu user) account at the time of its creation, allowing you to login simply by providing the private key. For [[Red Cloud Windows Instances|Windows instances]], you will need to provide the private key to the Red Cloud web interface in order to fetch a valid password for logging in to the instance's administrator account.<br />
<br />
Key pairs are created per user within an account, so other account members will not be able to use the key pairs you create. You will also not be able to use a given key pair in multiple accounts unless you import it into each account.<br />
<br />
__TOC__<br />
<br />
== Creating a Key Pair ==<br />
<br />
Your key pairs can be managed through the Red Cloud web interface by selecting the "Compute" tab [1] and then selecting the "Key Pairs" sub-tab [2]. This will display a list of your current key pairs as well as buttons for creating, importing or deleting key pairs. Begin by clicking "Create Key Pair" [3], which raises a simple wizard dialog.<br />
<br />
[[File:KeyPairList.png|border]]<br />
<br />
Enter a unique and meaningful name for the key pair [1] and then click "Create Keypair" [2]. Note that if the name you entered is invalid, the error message will be displayed in the underlying "Key Pairs" web page. The text for your private key is then displayed in the wizard. It is critical that you copy this text, either by selecting all of the text in the display and using a hot key or context menu item to copy it to the clipboard, or by clicking the "Copy Private Key to Clipboard" button [3]. This will be your only chance to copy the text, so do not forget to do so. When you have copied it, click "Done" [4] to close the wizard.<br />
<br />
[[File:KeyPairWizard.png|border]]<br />
<br />
The newly created key pair will now be shown in the list. It can be deleted using the button on the right of its entry, and clicking on the key pair's name will show more information about it, including its public key.<br />
<br />
You now '''must save the private key that you copied''' to your computer's clipboard into a file having the ".pem" extension. If you save the file with any other extension, you may not get the correct formatting. <br />
<br />
After copying the private key, open any simple text editor, but not a word processing app like Word. On Windows that could be Notepad, on Mac it could be TextEditor, and on Linux that could be any text editor you have installed, like gedit. Next, open a new text file, and paste the private key text into the new file. Make sure to paste all the text you copied from the private key dialogue from Red Cloud. The text you paste should include <code>BEGIN RSA PRIVATE KEY</code> and <code>END RSA PRIVATE KEY</code>, and the accompanying dashes.<br />
<br />
Next, save the file as <code><key name>.pem</code>, where <code><key name></code> is your key name, in an easily accessible directory. Make sure to have only a .pem extension on the saved file, without any extra .txt or such extensions.<br />
<br />
Lastly, if you are on Mac or Linux, make sure to set the file to the appropriate permissions. Open a terminal to access the directory with your saved key file, and enter <code>chmod 600 <key name>.pem</code> to change the permissions.<br />
<br />
The sections below will describe how this file is used to [[#Using_Your_Key_Pair_to_Connect_to_a_Linux_Instance|connect to a Linux instance]] or retrieve the [[#Using_Your_Key_Pair_to_Connect_to_a_Windows_Instance|administrator account's password for a Windows instance]].<br />
<br />
== Importing a Key Pair ==<br />
<br />
If you already have an SSH key pair that you would like to use with Red Cloud, you can import it rather than creating a new one. To do so, click the "Import Key Pair" button [1] on the Key Pairs page. This brings up a dialog for creating a key pair.<br />
<br />
[[File:KeyPairImport.png|border]]<br />
<br />
The Import Key Pair dialog contains some detailed instruction for generating key pairs on your computer. Using either an existing key or one that you generate by following those instructions, enter a unique and meaningful name for the key pair [1] and paste the entire text from its public key into the provided space [2]. This public key text should begin with "ssh-rsa" and end with a name, with a long string of letters and numbers in between. When you have entered those two values, click "Import Key Pair" [3]. They key pair will be imported and will appear in the Key Pairs list.<br />
<br />
[[File:KeyPairImportDialog.png|border]]<br />
<br />
== Selecting a Key Pair When Creating an Instance ==<br />
<br />
During the process of creating an instance you have the opportunity to assign a key pair to the new instances. This happens in the Key Pair tab [1] of the Launch Instance dialog. If you have not previously created or imported a key pair into your project, you can do so here [2]. If you would like to use one of the existing key pairs in the project, click the up arrow button in the list of existing key pairs [3].<br />
<br />
[[File:KeyPairSelection.png|border]]<br />
<br />
== Using Your Key Pair to Connect to a Linux Instance ==<br />
<br />
If you specified a key pair when creating a [[Red Cloud Linux Instances|Linux instance]], the key pair's public key was installed into the initial user account on the instance. When [[Red_Cloud_Linux_Instances#Accessing_Instances|connecting to the instance using the SSH command]], you can pass the corresponding private key to establish a secure connection without need for a password. The following example of the SSH command syntax is for a private key stored in the file "my_key_rsa" and a CentOS system where the initial account is named "centos".<br />
<br />
ssh -i my_key_rsa centos@128.84.8.1<br />
<br />
For more information, see the section on [[Red_Cloud_Linux_Instances#Accessing_Instances|Accessing Instances]] including some [[Red_Cloud_Linux_Instances#Troubleshooting|troubleshooting tips]]. If you would like to connect to a Linux instance using the [[https://www.chiark.greenend.org.uk/~sgtatham/putty/ PuTTY]] application, you will first need to convert your private key from the "pem" format to PuTTY's "ppk" format using the '''puttygen''' tool that is installed with PuTTY.<br />
<br />
== Using Your Key Pair to Connect to a Windows Instance ==<br />
<br />
To log on to a [[Red Cloud Windows Instances|Windows instance]] for the [[Red_Cloud_Windows_Instances#To_Do_On_First_Login|first time]] you will need to use the "admin" account and a password that you can retrieve through the web interface by providing your private key. Under the "Compute" tab and the "Instances" sub-tab, find your Windows instance in the list. With the instance running, open the menu on the right side of its list entry and select the "Retrieve Password" option. This will display a dialog box where you can enter your private key.<br />
<br />
[[File:KeyPairWindows.png|border]]<br />
<br />
The dialog displays the name of the key pair that was assigned when the instance was created, along with the public part of the key pair. You need to provide the private key (in "pem" format) by either choosing a file that contains it [1] or by pasting the text of the private key (including the header and footer) into the space provided [2]. Once the private key is entered, click the "Decrypt Password" button [3]. If the key does not match, an error message will be displayed in the background web page. If the key matches, a password for the "admin" user will be displayed [4]. Copy this password into your computer's clipboard and supply it when logging into your Windows instance using the Remote Desktop application.<br />
<br />
For more information, see the section on [[Red_Cloud_Windows_Instances#Accessing_Instances|Accessing Instances]].</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud_Test&diff=3392Red Cloud Test2021-01-06T17:48:46Z<p>Pzv2: Minor suggestions</p>
<hr />
<div>This wiki provides documentation for [https://{{SERVERNAME}}/redcloud Red Cloud], an on-demand research [https://en.wikipedia.org/wiki/Cloud_computing cloud computing] service maintained and supported by the [https://www.cac.cornell.edu/ CAC]. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on [[OpenStack]].<br />
<br />
Instructions on these pages apply to users who have a [https://www.cac.cornell.edu/services/projects.aspx Red Cloud subscription] they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.<br />
<br />
__TOC__<br />
<br />
== How To Read This Documentation ==<br />
<br />
:* '''Exploratory Account Users (if you received a trial account)'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed below to help you get started managing Red Cloud resources.<br />
:** Pay particular attention to the [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]] section on this page, as your exploratory project ends when you have exhausted your subscription.<br />
:** An important point to remember is that you are ''not'' the PI on your account, so you can ignore any instructions targeting PIs on a project.<br />
:* '''New Users (if you are new to Red Cloud)'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed to help you get started managing Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
:* '''Returning Users (all other users)'''<br />
:** Check out the section dedicated to [[#All Users|all users]].<br />
:** You may also want to look through the [[#Important Pages|Important Pages]] listed to help you manage Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
<br />
'''Note:''' All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.<br />
<br />
=== Important Pages ===<br />
<br />
Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
# '''This page''' - includes information about:<br />
#* The [[#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[#How_to_Access_Red_Cloud_Resources|Accessing resources]]<br />
#* [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# '''[[OpenStack]]''' - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for '''[[Red_Cloud_Linux_Instances|Linux Instances]]''' OR '''[[Red_Cloud_Windows_Instances|Windows Instances]]'''<br />
#* There are special instructions if you intend to use [[MATLAB Parallel Server in Red Cloud]]<br />
#* There is also a [[Linux Tutorial]] for those new to Linux system administration, or if you want a refresher<br />
# If you are planning to use '''GPUs in Red Cloud''':<br />
#* [[GPUs in Red Cloud]]<br />
#* [[Red Cloud GPU Image Usage]]<br />
<!-- #* [[Docker]] - coming soon --><br />
<br />
== New Users ==<br />
<br />
New users would be best served by reading this complete page first, then reading through the pages listed in the [[#Important Pages|Important Pages]] section. New users are also encouraged to explore the [[Getting Started]] page, which includes a lot more general information on using CAC resources beyond Red Cloud.<br />
<br />
'''Note for new Linux users:''' As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system '''must be familiar with Linux system administration'''. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a [[Linux Tutorial]] that should work for RedHat/CentOS and Ubuntu Linux systems. [https://{{SERVERNAME}}/services/ Consulting] is also available to answer general questions about systems administration, or for help on specific software and research problems.<br />
<br />
=== CAC Account First Time Login === <br />
<br />
When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must '''change the automatically generated password immediately''' for security reasons and to access computing resources. Refer to the instructions for [[Getting_Started#Managing_your_password|managing your password]] as needed.<br />
<br />
If you are a PI or a PI's proxy for a new project (If you are not sure what your role is or what you can do, please review [https://www.cac.cornell.edu/Services/projects.aspx this page]), verify that you have added a subscription to your project; see the [https://www.cac.cornell.edu/Services/projects/manage.aspx Manage Projects] page. After waiting up to an hour for account information to propagate, you will then be ready to download the [[OpenStack]] credentials and start managing Red Cloud resources.<br />
<br />
=== How to Create and Manage Red Cloud Resources ===<br />
<br />
Red Cloud is a private research cloud with an '''OpenStack''' backend. Interacting with [[OpenStack]] is how resources can be managed. In this case, resources can refer to [[OpenStack#Instances|instances]] (or [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines]), [[Images|images]], and [[Volumes|volumes]]. <br />
<br />
An instance is a virtual machine (VM), which is the main computational resource. To create an instance, you will need an image. You can use default images to create your instance, or you can create and upload an image to OpenStack and use that to create your instance. A volume is a collection of data that is attached to the instance. You should start by creating an instance. For more information on each resource, click the corresponding links.<br />
<br />
There are two ways to interact with OpenStack:<br />
<br />
:* '''The OpenStack Web Interface (Horizon)'''<br />
:** Go to the [//redcloud.cac.cornell.edu OpenStack Web Interface]<br />
:** For a walk-through, see the [[OpenStack]] page, which includes step-by-step instructions to launch and configure your instance<br />
:* '''The Command-Line Interface (CLI) called the OpenStack CLI'''<br />
:** Linux command-line tools provided by [[OpenStack]]<br />
:** For a walk-through, see the [[OpenStack CLI]] page<br />
:** Also see the [https://docs.openstack.org/python-openstackclient/pike/ official OpenStack CLI documentation]<br />
<br />
'''Note:''' Regardless which method you choose (Web Interface or Command Line Interface), you must first follow the [[#First Time Login | First Time Login]] instructions.<br />
<br />
=== How to Access Instances ===<br />
<br />
Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:<br />
:* [[Red Cloud Linux Instances | Linux Instances]] - especially the [[Red_Cloud_Linux_Instances#Accessing_Instances|accessing instances]] section (also see [[Red_Cloud_Linux_Instances#Troubleshooting|troubleshooting]] if needed)<br />
:* [[Red Cloud Windows Instances | Windows Instances]] - especially the [[Red_Cloud_Windows_Instances#Accessing_Instances|accessing instances]] section<br />
<br />
=== Accounting: Don't Use Up Your Subscription by Accident! ===<br />
<br />
To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to [[OpenStack#Instance_States|control the state]] of system [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines (VMs)], such as start, pause, suspend, shelve, and delete (see [[OpenStack#Instance_States|Instance States]] for a full list). Since starting a VM allocates memory and CPU resources on a physical machine to that VM,''' subscriptions are billed based on the length of time a VM is running, even if it is idle and doing NO work for the user'''. This is fair because your running [[OpenStack#Instances|instance]] will prevent others from using the hardware, even if the hardware is idle.<br />
<br />
Thus, '''the best way to avoid using up your subscription''' needlessly is to make sure you [[OpenStack#Instance_States|'''''shelve''''']] your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]]. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.<br />
<br />
Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various [[OpenStack#Instances|instance sizes]] available. For example, it is usually best to choose a small instance type to do your development work.<br />
<br />
It is worth pointing out that Red Cloud allows the [[Resizing an Instance| instance type]] to be changed if the VM is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we '''recommend''' that you start with the smallest instance size you can to install software and get used to your instance ''before'' [[Resizing an Instance|resizing your instance]] to the full size you would like.<br />
<br />
Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in [[OpenStack]]), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.<br />
<br />
All of the above is true for [[Red Cloud Linux Instances | Linux instances]] and [[Red Cloud Windows Instances | Windows instances]]; note that Cornell users do not need to pay for a [[Red Cloud Windows Instances#Windows_Activation|Windows license]] in Red Cloud.<br />
<br />
We recommend you check your balance frequently using pages provided for [https://{{SERVERNAME}}/services/cu/Memberlimits.aspx Cornell]<br />
or<br />
[https://{{SERVERNAME}}/services/external/Memberlimits.aspx external]<br />
users.<br />
<br />
== All Users ==<br />
<br />
Please refer to the [[OpenStack]] page for more in-depth guidance on how to use Red Cloud, and read either [[Red Cloud Linux Instances | Linux instances]] or [[Red Cloud Windows Instances | Windows instances]] based on what systems will be used. <br />
<br />
The current [https://www.cac.cornell.edu/RedCloud/status/ Red Cloud System Status] can be checked anytime.<br />
<br />
=== Common Tasks ===<br />
<br />
Here are some links to help you with particular aspects of using Red Cloud: <br />
:* [[Linux Tutorial]] - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.<br />
:* Information on choosing [[Instance Types | instance type]] (the size of the virtual machine). [[Resizing volumes]]<!-- this likely needs to be a new page --> is a separate issue, and is somewhat more involved.<br />
:* [//it.cornell.edu/services/ezbackup/ EZ-backup] - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.<br />
:* Data in CAC [[Archival_Storage| Archival Storage]] is intended to be an additional copy of user data; CAC Archival Storage is not backed up or snapshotted.<br />
:* All CAC resources are suitable for unregulated, non-confidential data ([https://it.cornell.edu/security-and-policy/data-types-confidential-regulated-restricted-public reference] for details). <br />
<!-- :* [[GPUs in Red Cloud]] --><br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
== Software on Red Cloud ==<br />
<br />
Generally, new instances launched on Red Cloud will contain basic operating system software and utilities, but will not contain pre-installed scientific applications. It is your responsibility to install any relevant applications either using a built-in package manager or by transferring your application code to the instance (e.g., via scp or sftp). In some cases, however, there are resources available to support running particular applications, as described below:<br />
:* On Linux instances, information on using package managers to install software: ( [[Linux Tutorial#Installing_Software | Using apt on Ubuntu]] ) ([[Linux Tutorial#Installing_Software_2 | Using yum on Centos]] )<br />
:* [[Installing R| Installing R]], a commonly used programming language and statistical analysis environment<br />
:* Running [[MATLAB Parallel Server in Red Cloud | MATLAB Parallel Server in Red Cloud]]<br />
:* Running [[OpenFOAM | OpenFOAM-7 in a Docker container]]<br />
:* Creating and using a [[Red Cloud GPU Image Usage | GPU Instance with pre-installed software]] (CUDA, NVIDIA Driver, Anaconda, Docker, Jupyter, MATLAB, etc.)<br />
<br />
== FAQ ==<br />
<br />
:* [[FAQ#Red_Cloud| Red Cloud FAQ]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud&diff=3180Red Cloud2020-09-08T18:33:00Z<p>Pzv2: Added links to GPU pages</p>
<hr />
<div>This wiki provides documentation for [https://{{SERVERNAME}}/redcloud Red Cloud], an on-demand research [https://en.wikipedia.org/wiki/Cloud_computing cloud computing] service maintained and supported by the [https://www.cac.cornell.edu/ CAC]. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on [[OpenStack]].<br />
<br />
Instructions on these pages apply to users who have a [https://www.cac.cornell.edu/services/projects.aspx Red Cloud subscription] they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.<br />
<br />
__TOC__<br />
<br />
== How To Read This Documentation ==<br />
<br />
:* '''Exploratory Account Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed below to help you get started managing Red Cloud resources.<br />
:** Pay particular attention to the [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]] section on this page, as your exploratory project ends when you have exhausted your subscription.<br />
:** An important point to remember is that you are ''not'' the PI on your account, so you can ignore any instructions targeting PIs on a project.<br />
:* '''New Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed to help you get started managing Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
:* '''Returning Users'''<br />
:** Check out the section dedicated to [[#All Users|all users]].<br />
:** You may also want to look through the [[#Important Pages|Important Pages]] listed to help you manage Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
<br />
'''Note:''' All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.<br />
<br />
=== Important Pages ===<br />
<br />
Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
# '''This page''' - includes information about:<br />
#* The [[#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[#How_to_Access_Red_Cloud_Resources|Accessing resources]]<br />
#* [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# '''[[OpenStack]]''' - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for '''[[Red_Cloud_Linux_Instances|Linux Instances]]''' OR '''[[Red_Cloud_Windows_Instances|Windows Instances]]'''<br />
#* There are special instructions if you intend to use [[MATLAB Parallel Server in Red Cloud]]<br />
#* There is also a [[Linux Tutorial]] for those new to Linux system administration, or if you want a refresher<br />
# If you are planning to use '''GPUs in Red Cloud''':<br />
#* [[GPUs in Red Cloud]]<br />
#* [[Red Cloud GPU Image Usage]]<br />
<!-- #* [[Docker]] - coming soon --><br />
<br />
== New Users ==<br />
<br />
New users would be best served by reading this complete page first, then reading through the pages listed in the [[#Important Pages|Important Pages]] section. New users are also encouraged to explore the [[Getting Started]] page, which includes a lot more general information on using CAC resources beyond Red Cloud.<br />
<br />
'''Note for new Linux users:''' As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system '''must be familiar with Linux system administration'''. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a [[Linux Tutorial]] that should work for RedHat/CentOS and Ubuntu Linux systems. [https://{{SERVERNAME}}/services/ Consulting] is also available to answer general questions about systems administration, or for help on specific software and research problems.<br />
<br />
=== First Time Login === <br />
<br />
When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must '''change the automatically generated password immediately''' for security reasons and to access computing resources. Refer to the instructions for [[Getting_Started#Managing_your_password|managing your password]] as needed.<br />
<br />
If you are a PI or a PI's proxy for a new project, verify that you have added a subscription to your project; see the [https://www.cac.cornell.edu/Services/projects/manage.aspx Manage Projects] page. After waiting up to an hour for account information to propagate, you will then be ready to download the [[OpenStack]] credentials and start managing Red Cloud resources.<br />
<br />
=== How to Create and Manage Red Cloud Resources ===<br />
<br />
Red Cloud is a private research cloud with an '''OpenStack''' backend. Interacting with OpenStack is how resources can be managed. In this case, resources can refer to [[OpenStack#Instances|instances]] (or [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines]), [[Images|images]], and [[Volumes|volumes]]. There are two ways to interact with OpenStack:<br />
<br />
:* '''The OpenStack Web Interface (Horizon)'''<br />
:** Go to the [//redcloud.cac.cornell.edu OpenStack Web Interface]<br />
:** For a walk-through, see the [[OpenStack]] page<br />
:* '''The Command-Line Interface (CLI) called the OpenStack CLI'''<br />
:** Linux command-line tools provided by [[OpenStack]]<br />
:** For a walk-through, see the [[OpenStack CLI]] page<br />
:** Also see the [https://docs.openstack.org/python-openstackclient/pike/ official OpenStack CLI documentation]<br />
<br />
'''Note:''' Regardless which method you choose (Web Interface or Command Line Interface), you must follow the [[#First Time Login | First Time Login]] instructions.<br />
<br />
=== How to Access Instances ===<br />
<br />
Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:<br />
:* [[Red Cloud Linux Instances | Linux Instances]] - especially the [[Red_Cloud_Linux_Instances#Accessing_Instances|accessing instances]] section (also see [[Red_Cloud_Linux_Instances#Troubleshooting|troubleshooting]] if needed)<br />
:* [[Red Cloud Windows Instances | Windows Instances]] - especially the [[Red_Cloud_Windows_Instances#Accessing_Instances|accessing instances]] section<br />
<br />
=== Accounting: Don't Use Up Your Subscription by Accident! ===<br />
<br />
To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to [[OpenStack#Instance_States|control the state]] of system [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines (VMs)], such as start, pause, suspend, shelve, and delete (see [[OpenStack#Instance_States|Instance States]] for a full list). Since starting a VM allocates memory and CPU resources on a physical machine to that VM,''' subscriptions are billed based on the length of time a VM is running, even if it is idle and doing NO work for the user'''. This is fair because your running [[OpenStack#Instances|instance]] will prevent others from using the hardware, even if the hardware is idle.<br />
<br />
Thus, '''the best way to avoid using up your subscription''' needlessly is to make sure you [[OpenStack#Instance_States|'''''shelve''''']] your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]]. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.<br />
<br />
Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various [[OpenStack#Instances|instance sizes]] available. For example, it is usually best to choose a small instance type to do your development work.<br />
<br />
It is worth pointing out that Red Cloud allows the [[Resizing an Instance| instance type]] to be changed if the VM is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we '''recommend''' that you start with the smallest instance size you can to install software and get used to your instance ''before'' [[Resizing an Instance|resizing your instance]] to the full size you would like.<br />
<br />
Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in [[OpenStack]]), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.<br />
<br />
All of the above is true for [[Red Cloud Linux Instances | Linux instances]] and [[Red Cloud Windows Instances | Windows instances]]; note that Cornell users do not need to pay for a [[Red Cloud Windows Instances#Windows_Activation|Windows license]] in Red Cloud.<br />
<br />
We recommend you check your balance frequently using pages provided for [https://{{SERVERNAME}}/services/cu/Memberlimits.aspx Cornell]<br />
or<br />
[https://{{SERVERNAME}}/services/external/Memberlimits.aspx external]<br />
users.<br />
<br />
== All Users ==<br />
<br />
Please refer to the [[OpenStack]] page for more in-depth guidance on how to use Red Cloud, and read either [[Red Cloud Linux Instances | Linux instances]] or [[Red Cloud Windows Instances | Windows instances]] based on what systems will be used. <br />
<br />
The current [https://www.cac.cornell.edu/RedCloud/status/ Red Cloud System Status] can be checked anytime.<br />
<br />
=== Common Tasks ===<br />
<br />
Here are some links to help you with particular aspects of using Red Cloud: <br />
:* [[Linux Tutorial]] - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.<br />
:* Information on choosing [[Instance Types | instance type]] (the size of the virtual machine). [[Resizing volumes]]<!-- this likely needs to be a new page --> is a separate issue, and is somewhat more involved.<br />
:* An example of [[Installing R| Installing R]], a commonly used software package.<br />
:* [//it.cornell.edu/services/ezbackup/ EZ-backup] - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.<br />
:* Data in CAC [[Archival_Storage| Archival Storage]] is intended to be an additional copy of user data; CAC Archival Storage is not backed up or snapshotted.<br />
:* All CAC resources are suitable for unregulated, non-confidential data ([https://it.cornell.edu/security-and-policy/data-types-confidential-regulated-restricted-public reference] for details). <br />
<!-- :* [[GPUs in Red Cloud]] --><br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
== FAQ ==<br />
<br />
:* [[FAQ#Red_Cloud| Red Cloud FAQ]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud&diff=3179Red Cloud2020-09-04T15:12:30Z<p>Pzv2: /* Accounting: Don't Use Up Your Subscription by Accident! */ Reduced proliferation of "virtual machine" by introducing "VM"</p>
<hr />
<div>This wiki provides documentation for [https://{{SERVERNAME}}/redcloud Red Cloud], an on-demand research [https://en.wikipedia.org/wiki/Cloud_computing cloud computing] service maintained and supported by the [https://www.cac.cornell.edu/ CAC]. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on [[OpenStack]].<br />
<br />
Instructions on these pages apply to users who have a [https://www.cac.cornell.edu/services/projects.aspx Red Cloud subscription] they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.<br />
<br />
__TOC__<br />
<br />
== How To Read This Documentation ==<br />
<br />
:* '''Exploratory Account Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed below to help you get started managing Red Cloud resources.<br />
:** Pay particular attention to the [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]] section on this page, as your exploratory project ends when you have exhausted your subscription.<br />
:** An important point to remember is that you are ''not'' the PI on your account, so you can ignore any instructions targeting PIs on a project.<br />
:* '''New Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed to help you get started managing Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
:* '''Returning Users'''<br />
:** Check out the section dedicated to [[#All Users|all users]].<br />
:** You may also want to look through the [[#Important Pages|Important Pages]] listed to help you manage Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
<br />
'''Note:''' All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.<br />
<br />
=== Important Pages ===<br />
<br />
Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
# This page - includes information about:<br />
#* The [[#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[#How_to_Access_Red_Cloud_Resources|Accessing resources]]<br />
#* [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# [[OpenStack]] - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for [[Red_Cloud_Linux_Instances|Linux Instances]] OR [[Red_Cloud_Windows_Instances|Windows Instances]]<br />
#* There are special instructions if you intend to use [[MATLAB Parallel Server in Red Cloud]]<br />
#* There is also a [[Linux Tutorial]] for those new to Linux system administration, or if you want a refresher<br />
<br />
== New Users ==<br />
<br />
New users would be best served by reading this complete page first, then reading through the pages listed in the [[#Important Pages|Important Pages]] section. New users are also encouraged to explore the [[Getting Started]] page, which includes a lot more general information on using CAC resources beyond Red Cloud.<br />
<br />
'''Note for new Linux users:''' As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system '''must be familiar with Linux system administration'''. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a [[Linux Tutorial]] that should work for RedHat/CentOS and Ubuntu Linux systems. [https://{{SERVERNAME}}/services/ Consulting] is also available to answer general questions about systems administration, or for help on specific software and research problems.<br />
<br />
=== First Time Login === <br />
<br />
When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must '''change the automatically generated password immediately''' for security reasons and to access computing resources. Refer to the instructions for [[Getting_Started#Managing_your_password|managing your password]] as needed.<br />
<br />
If you are a PI or a PI's proxy for a new project, verify that you have added a subscription to your project; see the [https://www.cac.cornell.edu/Services/projects/manage.aspx Manage Projects] page. After waiting up to an hour for account information to propagate, you will then be ready to download the [[OpenStack]] credentials and start managing Red Cloud resources.<br />
<br />
=== How to Create and Manage Red Cloud Resources ===<br />
<br />
Red Cloud is a private research cloud with an '''OpenStack''' backend. Interacting with OpenStack is how resources can be managed. In this case, resources can refer to [[OpenStack#Instances|instances]] (or [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines]), [[Images|images]], and [[Volumes|volumes]]. There are two ways to interact with OpenStack:<br />
<br />
:* '''The OpenStack Web Interface (Horizon)'''<br />
:** Go to the [//redcloud.cac.cornell.edu OpenStack Web Interface]<br />
:** For a walk-through, see the [[OpenStack]] page<br />
:* '''The Command-Line Interface (CLI) called the OpenStack CLI'''<br />
:** Linux command-line tools provided by [[OpenStack]]<br />
:** For a walk-through, see the [[OpenStack CLI]] page<br />
:** Also see the [https://docs.openstack.org/python-openstackclient/pike/ official OpenStack CLI documentation]<br />
<br />
'''Note:''' Regardless which method you choose (Web Interface or Command Line Interface), you must follow the [[#First Time Login | First Time Login]] instructions.<br />
<br />
=== How to Access Instances ===<br />
<br />
Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:<br />
:* [[Red Cloud Linux Instances | Linux Instances]] - especially the [[Red_Cloud_Linux_Instances#Accessing_Instances|accessing instances]] section (also see [[Red_Cloud_Linux_Instances#Troubleshooting|troubleshooting]] if needed)<br />
:* [[Red Cloud Windows Instances | Windows Instances]] - especially the [[Red_Cloud_Windows_Instances#Accessing_Instances|accessing instances]] section<br />
<br />
=== Accounting: Don't Use Up Your Subscription by Accident! ===<br />
<br />
To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to [[OpenStack#Instance_States|control the state]] of system [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines (VMs)], such as start, pause, suspend, shelve, and delete (see [[OpenStack#Instance_States|Instance States]] for a full list). Since starting a VM allocates memory and CPU resources on a physical machine to that VM,''' subscriptions are billed based on the length of time a VM is running, even if it is idle and doing NO work for the user'''. This is fair because your running [[OpenStack#Instances|instance]] will prevent others from using the hardware, even if the hardware is idle.<br />
<br />
Thus, '''the best way to avoid using up your subscription''' needlessly is to make sure you [[OpenStack#Instance_States|'''''shelve''''']] your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]]. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.<br />
<br />
Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various [[OpenStack#Instances|instance sizes]] available. For example, it is usually best to choose a small instance type to do your development work.<br />
<br />
It is worth pointing out that Red Cloud allows the [[Resizing an Instance| instance type]] to be changed if the VM is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we '''recommend''' that you start with the smallest instance size you can to install software and get used to your instance ''before'' [[Resizing an Instance|resizing your instance]] to the full size you would like.<br />
<br />
Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in [[OpenStack]]), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.<br />
<br />
All of the above is true for [[Red Cloud Linux Instances | Linux instances]] and [[Red Cloud Windows Instances | Windows instances]]; note that Cornell users do not need to pay for a [[Red Cloud Windows Instances#Windows_Activation|Windows license]] in Red Cloud.<br />
<br />
We recommend you check your balance frequently using pages provided for [https://{{SERVERNAME}}/services/cu/Memberlimits.aspx Cornell]<br />
or<br />
[https://{{SERVERNAME}}/services/external/Memberlimits.aspx external]<br />
users.<br />
<br />
== All Users ==<br />
<br />
Please refer to the [[OpenStack]] page for more in-depth guidance on how to use Red Cloud, and read either [[Red Cloud Linux Instances | Linux instances]] or [[Red Cloud Windows Instances | Windows instances]] based on what systems will be used. <br />
<br />
The current [https://www.cac.cornell.edu/RedCloud/status/ Red Cloud System Status] can be checked anytime.<br />
<br />
=== Common Tasks ===<br />
<br />
Here are some links to help you with particular aspects of using Red Cloud: <br />
:* [[Linux Tutorial]] - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.<br />
:* Information on choosing [[Instance Types | instance type]] (the size of the virtual machine). [[Resizing volumes]]<!-- this likely needs to be a new page --> is a separate issue, and is somewhat more involved.<br />
:* An example of [[Installing R| Installing R]], a commonly used software package.<br />
:* [//it.cornell.edu/services/ezbackup/ EZ-backup] - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.<br />
:* Data in CAC [[Archival_Storage| Archival Storage]] is intended to be an additional copy of user data; CAC Archival Storage is not backed up or snapshotted.<br />
:* All CAC resources are suitable for unregulated, non-confidential data ([https://it.cornell.edu/security-and-policy/data-types-confidential-regulated-restricted-public reference] for details). <br />
<!-- :* [[GPUs in Red Cloud]] --><br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
== FAQ ==<br />
<br />
:* [[FAQ#Red_Cloud| Red Cloud FAQ]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud_GPU_Image_Usage&diff=3172Red Cloud GPU Image Usage2020-09-01T13:24:38Z<p>Pzv2: Moved Introduction to above TOC and removed section heading to follow wiki conventions</p>
<hr />
<div>This wiki provides documentation for creating and using a GPU instance in Red Cloud from a Ubuntu ([https://redcloud.cac.cornell.edu/dashboard/ngdetails/OS::Glance::Image/e096c762-473c-440b-9516-19211c255ad2 gpu-accelerated-ubuntu-2020-08]) or or CentOS-based ([https://redcloud.cac.cornell.edu/dashboard/ngdetails/OS::Glance::Image/516f21bc-07a8-4546-b052-982028a3d04e gpu-accelerated-centos-2020-08]) GPU image. The image features:<br />
<br />
* GPU acceleration via CUDA 10.1. <br />
* Anaconda distribution to facilitate usage of platforms such as Tensorflow, <br />
* Docker-containerized Jupyter Notebook servers, and <br />
* MATLAB R2019a. <br />
<br />
A test application using the Python neural network library Keras, which runs on top of the Tensorflow framework, is provided to check and test GPU utilization.<br />
<br />
The intent is to get users started on using GPU accelerated software with their work quickly with minimal effort.<br />
<br />
__TOC__<br />
<br />
= Create a Server Instance =<br />
<br />
# [https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud#New_Users Get started with with the Openstack web interface]<br />
# [https://www.cac.cornell.edu/wiki/index.php?title=OpenStack#Key_Pairs Create an SSH keypair or upload your public key]<br />
# [https://www.cac.cornell.edu/wiki/index.php?title=OpenStack#Security_Groups Create a custom Security Group if you haven't]<br />
# Launch a new Server Instance<br />
#* In the Openstack web interface Dashboard, navigate to [https://redcloud.cac.cornell.edu/dashboard/project/instances/ Compute > Instances]<br>[[File:InstancesMenu.png|350px|frameless|border]]<br><br />
#* click on “Launch Instance”. <br>[[File:InstancesOptions.png|600px|frameless|border]]<br><br />
#* On the Launch Instance screen, under : <br>[[File:InstanceLaunchMenuFull.png|700px|frameless|border]]<br />
#** Details tab: enter instance name<br />
#** Source tab: <br />
#*** Boot Source = “Image”<br />
#*** Among the list of images, chosose “gpu-accelerated-centos-2020-08” for Centos 7.8 or “gpu-accelerated-ubuntu-2020-08” for Ubuntu 18.04 LTS by clicking on the ⬆️ button.<br />
#*** Volume Size: increase as necessary. For reference, the Anaconda distribution alone is 19GB.<br />
#** Flavor tab: select instance server configurations. Choose between "c4.t1.m20" (Tesla T4 GPU) and "c14.g1.m60" (Tesla V100 GPU for large jobs). If you need multiple GPUs on one instance, please [mailto:help@cac.cornell.edu reach out] for special accommodation.<br />
#** Networks tab: “public” is a good default.<br />
#** Security Groups tab: Click ⬇️ on the “default” group to deselect the default security group. Click ⬆️ to select the custom security group you created above. <br />
#** Key Pair: Select the ssh key pair you had created above.<br />
#** Click the "Launch Instance" button to create your server instance.<br />
#** Note the IP Address of your newly-created Instance.<br />
<br />
= SSH Login =<br />
In a new terminal window, Chrome extension ssh or Putty, SSH into the instance:<br />
* For Ubuntu instance <br />
ssh -i ~/.ssh/id_rsa/id_rsa ubuntu@<IP_Address_from_previous_step><br />
* For CentOS instance: <br />
ssh -i ~/.ssh/id_rsa/id_rsa centos@<IP_Address_from_previous_step><br />
Assumes that your matching private key is saved as <code>~/.ssh/id_rsa</code>. <br />
<br />
== CAC Recommends ==<br />
Monthly security update patching: <br />
* For Ubuntu instance: <br />
sudo apt dist-upgrade<br />
* For CentOS instance: <br />
sudo yum update <br />
<br />
= Installed Software = <br />
== CUDA ==<br />
* Check version with <code>yum info cuda</code> (not applicable for Ubuntu)<br />
* Check CUDA version currently in use with <code>nvcc --version</code><br><br />
(base) ubuntu@shl1-test1:~$ '''nvcc --version'''<br />
nvcc: NVIDIA (R) Cuda compiler driver<br />
Copyright (c) 2005-2019 NVIDIA Corporation<br />
Built on Sun_Jul_28_19:07:16_PDT_2019<br />
Cuda compilation tools, release 10.1, V10.1.243<br />
<br />
== NVIDIA Driver ==<br />
* Check version with dkms status<br />
** Run <code>sudo yum install dkms</code> (CentOS) or <code>sudo apt install dkms</code> if “command not found”<br />
* Check detection of GPU devices by CUDA via NVIDIA’s drivers: <code>nvidia-smi</code><br />
(base) ubuntu@shl1-test1:~$ '''nvidia-smi'''<br />
Sat Aug 29 01:38:55 2020 <br />
+-----------------------------------------------------------------------------+<br />
| NVIDIA-SMI 450.57 Driver Version: 450.57 CUDA Version: 11.0 |<br />
|-------------------------------+----------------------+----------------------+<br />
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br />
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br />
| | | MIG M. |<br />
|===============================+======================+======================|<br />
| 0 Tesla T4 Off | 00000000:00:05.0 Off | 0 |<br />
| N/A 41C P0 27W / 70W | 0MiB / 15109MiB | 6% Default |<br />
| | | N/A |<br />
+-------------------------------+----------------------+----------------------+<br />
<br />
+-----------------------------------------------------------------------------+<br />
| Processes: |<br />
| GPU GI CI PID Type Process name GPU Memory |<br />
| ID ID Usage |<br />
|=============================================================================|<br />
| No running processes found |<br />
+-----------------------------------------------------------------------------+<br />
Note that the CUDA version displayed on top right of this output is not necessarily the CUDA version currently in use.<br />
<br />
== Anaconda ==<br />
* Check overarching Anaconda version with <code>conda list anaconda$</code><br />
* Check “conda” package manager version with <code>conda -V</code><br />
* Check list of packages installed with <code>conda list</code><br />
* If desired, update to latest version with <code>conda update --all</code><br />
* For environment information, such as base paths or Python platform versioning, use <code>conda info</code><br />
* An Anaconda environment should already be activated upon startup, indicated by <code>base</code> preceding each command line prompt.<br />
(base) ubuntu@shl1-test1:~$ <br />
<br />
This “base” environment applies specific settings to your shell for the choice of a Python interpreter and its associated modules and libraries.<br />
Use <code>which python</code> for identifying the environment’s Python location; <code>python -V</code> for Python version.<br />
<br />
== Docker ==<br />
* To start docker using docker:<br />
sudo usermod -aG docker $(whoami)<br />
sudo service docker start<br />
* To view existing images pulled onto this instance, <br />
docker images<br />
* To view existing containers, both running and exited, <br />
docker ps -a<br />
* To enter terminal of a running container, <br />
docker exec -it <Container Name or Container ID> bash<br />
* To restart an exited container (starts from latest first), <br />
docker start `docker ps -q -l`<br />
* To exit container Terminal without exiting or killing the container, press CTRL+p then CTRL+q.<br />
<br />
== MATLAB ==<br />
MATLAB is installed in <code>/usr/local/bin/matlab</code>. See [https://www.mathworks.com/help/parallel-computing/gpu-computing.html GPU Computing in MATLAB] page from MathWorks for more information.<br />
<br />
= Testing with Sample Application with Virtualization Only =<br />
Each of these frameworks can be set up in their own Conda environments. <br />
* Use <code>conda env list</code> to see the list of existing usable environments.<br />
(base) ubuntu@shl1-test1:~$ '''conda env list'''<br />
# conda environments:<br />
#<br />
base * /home/ubuntu/anaconda3<br />
pytorch /home/ubuntu/anaconda3/envs/pytorch<br />
tf_gpu /home/ubuntu/anaconda3/envs/tf_gpu<br />
* Use <code>conda deactivate</code> to exit current environment and return to “base” environment.<br />
<br />
== Keras-GPU on Tensorflow-GPU ==<br />
* Activate the Virtual Environment for Tensorflow<br />
** With the “base” Anaconda environment still activated, <br />
conda create -n tf_gpu tensorflow-gpu<br />
conda activate tf_gpu<br />
* Install necessary packages: <br />
conda install tensorflow-gpu keras-gpu <br />
* Ensure the iPython command-line terminal used is from within the environment, not an external version: <br />
conda install -c anaconda ipython<br />
<code>which ipython</code> should output <code>~/anaconda3/envs/tf_gpu/bin/ipython</code><br />
* Start <code>ipython</code> and paste this sample app at the ipython prompt:<br />
<pre><br />
import tensorflow as tf<br />
from tensorflow import keras<br />
from tensorflow.keras import layers<br />
import numpy as np<br />
tf.config.list_physical_devices('GPU')<br />
</pre><br />
* You should get output like this:<br />
Out[3]: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]<br />
<br />
== PyTorch ==<br />
* Activate the Virtual Environment for PyTorch<br />
** With the “base” Anaconda environment still activated, <br />
conda create -n pytorch<br />
conda activate pytorch<br />
* Install necessary packages: <br />
conda install pytorch torchvision -c pytorch<br />
pip install pycuda<br />
* Ensure the iPython command-line terminal used is from within the environment, not an external version: <br />
conda install -c anaconda ipython<br />
* Start <code>ipython</code> and paste this sample app at the ipython prompt:<br />
<pre>import pycuda <br />
import pycuda.driver as drv <br />
drv.init() <br />
print('CUDA device query (PyCUDA version) \n') <br />
print('Detected {} CUDA Capable device(s) \n'.format(drv.Device.count())) <br />
for i in range(drv.Device.count()): <br />
gpu_device = drv.Device(i) <br />
print('Device {}: {}'.format( i, gpu_device.name() ) ) <br />
compute_capability = float( '%d.%d' % gpu_device.compute_capability() ) <br />
print('\t Compute Capability:{}'.format(compute_capability)) <br />
print('\t Total Memory: {} megabytes'.format(gpu_device.total_memory()//(1024**2 )))<br />
</pre><br />
* You should get the output<br />
<pre><br />
CUDA device query (PyCUDA version) <br />
Detected 1 CUDA Capable device(s) <br />
Device 0: Tesla T4<br />
Compute Capability:7.5<br />
Total Memory: 15109 megabytes<br />
</pre><br />
:or<br />
<pre><br />
CUDA device query (PyCUDA version) <br />
Detected 1 CUDA Capable device(s) <br />
Device 0: Tesla V100-PCIE-16GB<br />
Compute Capability:7.0<br />
Total Memory: 16160 megabytes<br />
</pre><br />
<br />
= Testing with Sample Application: with Virtualization + Docker Containerization =<br />
== Keras GPU on Tensorflow-GPU ==<br />
* Create a new container:<br />
docker run --gpus all -v $(realpath ~/notebooks):/tf/notebooks -p 8000:8000 tensorflow/tensorflow:latest-gpu-jupyter sleep 100000 & <br />
:* The <code>--gpus all</code> tag is for employing devices as detected by nvidia-smi.<br />
:* <code>tensorflow/tensorflow:latest-gpu-jupyter</code> is the specific image used. <br />
:* Read more about this image and other possible tags at [https://hub.docker.com/r/tensorflow/tensorflow/ DockerHub].<br />
:* The container runs a Jupyter Notebook server, exposed on host port 8888, from your own directory (here stated as ~/notebooks).<br />
* Test run the new container, then exit and disconnect:<br />
** Use <code>docker ps</code> to check the <code>Container ID</code> (first column) or <code>Name</code> (last column)<br />
** Run <code>docker exec -it <Container ID or Name> bash</code><br />
** Press CTRL+p and CTRL+q to exit container without killing it.<br />
** Press CTRL+d to end SSH connection to VM.<br />
* Re-connect to the VM via SSH Tunneling, then restart the Docker Tensorflow Container:<br />
ssh -L 8000:localhost:8000 centos@<_IP_> OR ssh -L 8000:localhost:8000 ubuntu@<_IP_><br />
* Then <br />
docker exec -it <Container ID or Name> bash<br />
* Spin up a Jupyter Notebook within this Docker container:<br />
jupyter notebook --ip 0.0.0.0 --port 8000 --allow-root<br />
:Copy the token provided above for use in the next step.<br />
:Note: “No web browser found” error may occur, ignore if the next step is successful. If not, Ctrl+c to stop this current server, and try again.<br />
* Navigate to [http://localhost:8000/ http://localhost:8000/] on a local browser. Enter the token from the previous step in the requested field.<br />
:[[File:JupyterNotebookToken.png|500px|frameless|border]]<br />
* Navigate to notebooks, then create a new Python3 Notebook or use an existing one.<br />
:[[File:SelectNotebook.png|800px|frameless|border]]<br />
* Paste in and run this Keras sample application:<br />
<pre><br />
import tensorflow as tf<br />
from tensorflow import keras<br />
from tensorflow.keras import layers<br />
import numpy as np<br />
tf.config.list_physical_devices('GPU')<br />
</pre><br />
:You should get the following output: <br />
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=GPUs_in_Red_Cloud&diff=3131GPUs in Red Cloud2020-08-24T21:05:21Z<p>Pzv2: /* Launching A GPU Instance */ Formatting</p>
<hr />
<div>''(This page under development)''<br />
<br />
[[Red Cloud]] supports GPU computing and has two different GPU [[OpenStack#Instances|instance]] flavors that allow for GPU computing on virtual machines (VMs). In order to use a GPU in Red Cloud, you will need to select a flavor of machine with a GPU. Currently Red Cloud instances feature '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' and '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' GPU accelerators in these flavors:<br />
<br />
{| border="1" cellspacing="0" cellpadding="10" align="center" style="text-align:center;"<br />
! Flavor <br />
! CPUs<br />
! GPUs <br />
! RAM <br />
|-<br />
| ''c4.t1.m20'' || 4 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' || 20 GB<br />
|-<br />
| ''c14.g1.m60'' || 14 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' || 60 GB<br />
|}<br />
<br />
== Availability ==<br />
As of this writing, there are 20 of the T4s and 4 of the V100s for use in Red Cloud VMs. You can obtain up-to-date GPU usage [https://gpus.redcloud.cac.cornell.edu/usage here] to help you to determine if resources are available to start an instance. If you are new to Red Cloud you should review [[Red_Cloud#How_To_Read_This_Documentation|how to read this documentation]] before launching an instance, especially the section on [[Red_Cloud#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|accounting]]. Because Red Cloud ''does not'' have hyperthreading enabled, GPUs are not oversubscribed. This means when you create a GPU instance with a certain number of GPUs, you are reserving the physical hardware for the duration of the life of your instance unless it is '''''[[OpenStack#Instance_States|shelved]]''''' to free the resources. If the resources are not available when you attempt to start an instance - because someone else has reserved them - then you may receive an error that they cannot be created. Therefore, it is good to check availability before starting an instance, and also shelving instances when not in use.<br />
<br />
== Launching A GPU Instance ==<br />
When '''[[OpenStack#Launch_an_Instance|launching an instance]]''', you can use either the base [[Red_Cloud_Linux_Instances|Linux]] or [[Red_Cloud_Windows_Instances|Windows]] instances and install your own GPU libraries, or select CUDA source images such as (...). Next, select a GPU-enabled flavor and configure the instance as you would any other instance. Once your instance is launched, you will have access to the GPU within the VM and can install software (e.g., pytorch, tensorflow) that will use the GPU.<br />
<br />
For more information on GPU and CUDA computing, see the Cornell Virtual Workshop "'''[https://cvw.cac.cornell.edu/gpu/ Introduction to GPGPU and CUDA Programming: Overview]'''"</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=GPUs_in_Red_Cloud&diff=3128GPUs in Red Cloud2020-08-24T14:51:05Z<p>Pzv2: Added the Availability section</p>
<hr />
<div>''(This page under development)''<br />
<br />
[[Red Cloud]] supports GPU computing and has two different GPU [[OpenStack#Instances|instance]] flavors that allow for GPU computing on virtual machines (VMs). In order to use a GPU in Red Cloud, you will need to select a flavor of machine with a GPU. Currently Red Cloud instances feature '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' and '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' GPU accelerators in these flavors:<br />
<br />
{| border="1" cellspacing="0" cellpadding="10" align="center" style="text-align:center;"<br />
! Flavor <br />
! CPUs<br />
! GPUs <br />
! RAM <br />
|-<br />
| *''c4.t1.m20'' || 4 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' || 20 GB<br />
|-<br />
| *''c14.g1.m60'' || 14 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' || 60 GB<br />
|}<br />
<br />
== Availability ==<br />
As of this writing, there are 20 of the T4s and 4 of the V100s for use in Red Cloud VMs. You can obtain up-to-date GPU usage [https://gpus.redcloud.cac.cornell.edu/usage here] to help you to determine if resources are available to start an instance. If you are new to Red Cloud you should review [[Red_Cloud#How_To_Read_This_Documentation|how to read this documentation]] before launching an instance, especially the section on [[Red_Cloud#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|accounting]]. Because Red Cloud ''does not'' have hyperthreading enabled, GPUs are not oversubscribed. This means when you create a GPU instance with a certain number of GPUs, you are reserving the physical hardware for the duration of the life of your instance unless it is '''''[[OpenStack#Instance_States|shelved]]''''' to free the resources. If the resources are not available when you attempt to start an instance - because someone else has reserved them - then you may receive an error that they cannot be created. Therefore, it is good to check availability before starting an instance, and also shelving instances when not in use.<br />
<br />
== Launching A GPU Instance ==<br />
When '''[[OpenStack#Launch_an_Instance|launching an instance]]''', you can use either the base [[Red_Cloud_Linux_Instances|Linux]] or [[Red_Cloud_Windows_Instances|Windows]] instances and install your own GPU libraries, or select CUDA source images such as (...), select a GPU-enabled flavor, and configure the instance as you would any other instance.<br />
<br />
Once your instance is launched, you will have access to the GPU within the VM and can install software (pytorch, tensorflow) that will use the GPU.<br />
<br />
For more information on GPU and CUDA computing, see the Cornell Virtual Workshop "'''[https://cvw.cac.cornell.edu/gpu/ Introduction to GPGPU and CUDA Programming: Overview]'''"</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=OpenStack&diff=3127OpenStack2020-08-24T14:45:51Z<p>Pzv2: /* Instances */ Added GPUs to the list of things not oversubscribed</p>
<hr />
<div>[https://en.wikipedia.org/wiki/OpenStack OpenStack] is an [https://en.wikipedia.org/wiki/Open-source_model open-source] [https://en.wikipedia.org/wiki/Cloud_computing#Service_models cloud stack] that is currently running on [[Red_Cloud|Red Cloud]]. Also, for more information, see the [https://docs.openstack.org/ Official Documentation for OpenStack].<br />
<br />
This page is intended as a quick walk-through of the most-used features of OpenStack, so it is not comprehensive, but links to a lot of supporting documentation for more thorough explanations and advanced topics.<br />
<br />
__TOC__<br />
<br />
== Using the OpenStack Web Interface (Horizon) ==<br />
<br />
There are two ways to manage [[Red Cloud]] resources:<br />
# [https://redcloud.cac.cornell.edu OpenStack Web Interface]<br />
# [[OpenStack CLI]]<br />
<br />
Most users will use the OpenStack Web Interface (called [https://docs.openstack.org/horizon/latest/ Horizon]). This web-based interface can be used to manage [[#Instances|instances]] and [[Volumes|volumes]]. For [[Red Cloud Linux Instances|Linux Instances]], however, some users may choose to use the OpenStack CLI. This section focuses on the OpenStack Web Interface.<br />
<br />
=== Logging In ===<br />
<br />
Log in to the [https://redcloud.cac.cornell.edu OpenStack Web Interface] to create and manage Red Cloud resources. There are two ways to login: <br />
<br />
[[File:RedCloudCACLogin.png|300px|frameless|border]][[File:White_square.png|100px|frameless]][[File:RedCloudGlobusAuthLogin.png|300px|frameless|border]]<br />
<br />
# [https://www.cac.cornell.edu/services/myacct.aspx CAC Account] - Enter '''cac''' as the "<tt>Domain</tt>" and your [https://www.cac.cornell.edu/services/myacct.aspx CAC username] and password, not your Cornell NetID. If your CAC password has expired, you will need to [https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started#Managing_your_password reset it] before you will be able to login to the OpenStack Web Interface.<br />
# [https://www.globus.org/tags/globus-auth Globus Auth] - Log in through Globus<br />
#* Currently, this feature is '''only available to Aristotle users'''. This feature will be enabled for all users in the future.<br />
#* You must ''link your Cornell account'', or any accounts attached to the projects you are on, in order to have access to them when using Globus Auth.<br />
#* If you can't log in with Globus Auth, it may be that you have not linked your account yet.<br />
<br />
You can use the "<tt>Authenticate using</tt>" drop-down to switch between the two options. Neither option requires you to enter a project ID; you can switch between the projects you are on once logged in.<br />
<br />
=== Overview Page ===<br />
<br />
The Overview page is the first place you will be taken upon logging into Red Cloud.<br />
<br />
:* Provides useful metrics on currently selected project<br />
:* '''Before creating an instance''', you will need to:<br />
:** Select the correct project from the "<tt>Project</tt>" drop-down at the top right of the page (if you are on multiple projects)<br />
:** [[#Key_Pairs|Create a key pair]] - for authentication when you log in the first time<br />
:** [[#Security_Groups|Create a security group]] - defines allowable types of port access for an instance<br />
:** Optional: [[Networks#Private_Networks|Set up a private network]] - if you do not want your instance to be available on the [[Networks#Public Network|public network]]<br />
:* You may also want to:<br />
:** [[Volumes#Create and Attach a Volume|Create and Attach a Volume]] (can also be done when launching an instance)<br />
:** [[Networks#Floating IP Addresses|Associate a Floating IP address]] - a fixed IP address that can be assigned to an instance<br />
<br />
=== Key Pairs ===<br />
<br />
[[File:Overview_KeyPairs_Circled.png|350px|frameless|border]]<br />
<br />
To get to the Key Pairs page: select the "<tt>Compute</tt>" tab along the top (you should start here at login), then click on "<tt>Key Pairs</tt>" along the top bar as pictured above. If you are logged in already, you can also get to it by this link: [https://redcloud.cac.cornell.edu/dashboard/project/key_pairs/ Key Pairs].<br />
<br />
On the Key Pairs page, you can view the list of available [[OpenStack_Key_Pairs|key pairs]] for your project. From here, you can also [[OpenStack_Key_Pairs#Creating_a_Key_Pair|create]] or [[OpenStack_Key_Pairs#Importing_a_Key_Pair|import]] a key pair. If you do not already have a key pair listed, you can either create one before [[OpenStack#Launch_an_Instance|launching an instance]], or [[OpenStack_Key_Pairs#Selecting_a_Key_Pair_When_Creating_an_Instance|create or upload a key pair]] during instance setup.<br />
<br />
For more information, here is a walk-through on [[OpenStack Key Pairs]].<br />
<br />
=== Security Groups ===<br />
<br />
[[File:Overview_SecurityGroups.png|350px|frameless|border]]<br />
<br />
To get to the Security Groups page: select the "<tt>Network</tt>" drop-down menu along the top, then click on "<tt>Security Groups</tt>" as pictured above. If you are already logged in, you can also get to it by following this link: [https://redcloud.cac.cornell.edu/dashboard/project/security_groups/ Security Groups]<br />
<br />
On the Security Groups page, you can view a list of available [[OpenStack Security Groups|security groups]] for your project, including a default security group. On this page, you can also [[OpenStack_Security_Groups#Creating_a_Security_Group|create]] and delete security groups. It '''is ''not''''' recommended that you use the default security group without [[OpenStack_Security_Groups#Managing_a_Security_Group.27s_Rules|modifying the rules]] to fit your needs. A good security practice is to have one security group per application or one per user. Instances that have no business talking to each other should generally be in separate security groups.<br />
<br />
If you do not already have a security group set up, you will want to [[OpenStack_Security_Groups#Creating_a_Security_Group|create]] one before [[OpenStack#Launch_an_Instance|launching an instance]] because you cannot create one during instance setup. However, you can [[OpenStack_Security_Groups#Assigning_Security_Groups_to_an_Instance|assign a security group]] to an instance later, and even [[OpenStack_Security_Groups#Adding_a_Rule_to_a_Security_Group|add]] or<br />
[[OpenStack_Security_Groups#Managing_a_Security_Group.27s_Rules|modify the rules]] of the security group at any time. <br />
<br />
For more information, here is a walk-through on [[OpenStack Security Groups]].<br />
<br />
== Instances ==<br />
<br />
Each instance is a Virtual Machine (VM) in the cloud. You can select CPU/RAM/disk configurations (called "flavors") for the VM. The available VM configurations are:<br />
<br />
{| border="1" cellspacing="0" cellpadding="10" align="center" style="text-align:center;"<br />
! Flavor <br />
! CPUs<br />
! GPUs <br />
! RAM <br />
|-<br />
| c1.m8 || 1 || None || 8 GB<br />
|-<br />
| c2.m16 || 2 || None || 16 GB<br />
|-<br />
| c4.m32 || 4 || None || 32 GB<br />
|-<br />
| c8.m64 || 8 || None || 64 GB<br />
|-<br />
| c14.m112 || 14 || None || 112 GB<br />
|-<br />
| c20.m160 || 20 || None || 160 GB<br />
|-<br />
| c28.m224|| 28 || None || 224 GB<br />
|-<br />
| *''c4.t1.m20'' || 4 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' || 20 GB<br />
|-<br />
| *''c14.g1.m60'' || 14 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' || 60 GB<br />
|-<br />
| colspan="4" style="text-align:left;" | ''* GPU flavors<br />
|}<br />
<br />
When you are first starting an instance, we '''recommend''' that you select the smallest flavor (least number of CPUs) that you think will be able to handle installation and configuration of the software and environment on your instance, and then [[Resizing an Instance|resize the instance]] when you are ready to run. This way you will save core hours that would otherwise have been spent idle. This method is especially useful when configuring a ''GPU instance'' due to the number of cores. Also note: you can begin with a smaller instance size (or flavor) that does not contain a GPU, and later resize to one that does.<br />
<br />
The root disk size of the instance will default to the size of the [[Images|image]] you select. You have the option to create a [[Volumes|volume]] as the root disk beyond the image size at launch time. Note that we do not oversubscribe physical RAM, CPU cores, or GPUs (hyperthreading is disabled).<br />
<br />
To work with instances, select the "<tt>Instances</tt>" page under the "<tt>Compute</tt>" tab, as pictured below:<br />
<br />
[[File:InstancesMenu.png|350px|frameless|border]]<br />
<br />
=== Launch an Instance ===<br />
<br />
This section is a general walk-through for creating a new instance, which is not specific to an Operating System (OS). For more specific information per OS, see either of these pages:<br />
<br />
:* [[Red Cloud Linux Instances|Linux Instances]]<br />
:* [[Red Cloud Windows Instances|Windows Instances]]<br />
<br />
==== To launch a new instance ====<br />
<br />
# [[#Key_Pairs|Create Key Pair]]<br />
# [[#Security Groups|Create a Security Group]] and be sure that you select the appropriate rule for connecting to your instance (SSH for [[Red Cloud Linux Instances|Linux Instances]] and RDP for [[Red Cloud Windows Instances|Windows Instances]])<br />
# Select "<tt>Launch Instance</tt>" on the top right side of the [https://redcloud.cac.cornell.edu/dashboard/project/instances/ Instances] page [[File:InstancesOptions.png|600px|frameless|border]]<br />
<br />
The full "<tt>Launch Instance</tt>" menu will pop up like this:<br />
<br />
[[File:InstanceLaunchMenuFull.png|700px|frameless|border]]<br />
<br />
:* Tabs that you are required to fill out are marked with a '''*'''<br />
:* It is '''recommended''' that you also select your own Security Group, otherwise the default security group will be selected, which may not be ideal for your work.<br />
:* It is '''necessary''' that you select your own Key Pair, even though this field is not marked required, so that you are able to connect to your instance after creation.<br />
<br />
==== Configuring the Instance ====<br />
<br />
# Under the "<tt>Details</tt>" tab:<br />
#* Enter a name for your instance<br />
#* '''Count''' is the number of identical instances you would like to create (typically 1).<br />
#** Note that if you create multiple instances this way, the names will be identical with a dash and number added at the end.<br />
#** For example, if you set Instance Name to "my_instance" and you set Count to 3, you would start instances named "my_instance-1", "my_instance-2", and "my_instance-3".<br />
# On the "<tt>Source</tt>" tab:<br />
#* You must '''Select Boot Source''', which is described on the page as "the template used to create an instance."<br />
#** It is generally a good idea to start with an [[Images|image]] as the source, unless you want to create an instance from a pre-existing source.<br />
#** For more information on the other options, see [[Images#Creating an Image|Creating an Image]].<br />
#* You can select a specific source under the <tt>Available</tt> list by selecting the up arrow on the right-hand side.<br />
#* Get more details about the specific source by selecting the right-arrow on the left-hand side next to the name.<br />
#* You will have the option to '''Create New Volume''' if you have selected either "<tt>Image</tt>" or "<tt>Instance Snapshot</tt>" as the source (default is "<tt>Yes</tt>"):<br />
#*# '''<tt>Yes</tt>''': If selected, a [[Volumes|volume]] will be created to be the instance's root disk. You will then have the options of extending the size of the volume beyond the image size, and deleting the volume when the instance is deleted.<br />
#*#* '''Volume Size''' is the size of your root [[Volumes|volume]]. The default number will match the size of the [[Images|image]] you select, and can be increased.<br />
#*#* '''Delete Volume on Instance Delete''' determines whether or not the root volume will be deleted automatically when you terminate the instance. The default is "<tt>No</tt>", which prevents your data from being deleted when you delete your instance (perhaps accidentally). However, ''if you do not need this extra level of protection, and you do not intend to re-use the root volume, you could unintentionally incur excess storage usage''. Therefore, it can be a good idea to set this option to "<tt>Yes</tt>" so that the volume is deleted automatically when you terminate your instance. Your alternative is to find and delete the root volume manually, later (it will show up in the list of volumes with a name identical to its arbitrarily assigned ID, unless you give it a different name).<br />
#*#* You can also customize the name of the volume under '''Device Name'''.<br />
#*# '''<tt>No</tt>''': If selected, the instance will boot off a root disk the same size as the image. The root disk will be deleted when the instance is deleted.<br />
# The "<tt>Flavor</tt>" tab is where you select the VM configuration discussed [[#Instances|above]].<br />
#* We '''recommend''' that you select the smallest flavor (least number of CPUs) that you think will be able to handle installation and configuration of the software and environment on your instance, and then [[Resizing an Instance|resize the instance]] when you are ready to run. This way you will save core hours that would otherwise have been spent idle. Also note: you can begin with a smaller instance size (or flavor) that does not contain a GPU, and later resize to one that does.<br />
#* You can select a configuration by selecting the up arrow on the right-hand side.<br />
#* Get more details about the configuration by selecting the right-arrow on the left-hand side next to the name.<br />
#* The "<tt>Total Disk</tt>" will show "0 GB" because the disk size will match the [[Images|image]] you selected on the "<tt>Source</tt>" tab.<br />
# For the "<tt>Networks</tt>" tab, two options are available:<br />
## You can make the instance available on the [[Networks#Public Network|public net]]. This is the simplest and most common selection.<br />
## You can select your own [[Networks#Private Networks|private network]], which has to be set up before you launch an instance. For more information, see the [[Networks]] page.<br />
# On the "<tt>Security Groups</tt>" tab, select the [[#Security Groups|security group]] you already created.<br />
# On the "<tt>Key Pairs</tt>" tab, select the [[#Key Pairs|key pair]] you already created.<br />
<br />
=== Instance States ===<br />
<br />
OpenStack defines several [https://developer.openstack.org/api-guide/compute/server_concepts.html#server-status Server States] through which you can move your instances. You change the state of your instance by making a selection from a drop-down menu under the <tt>Actions</tt> column. Three significant actions to know about are "Resize Instance", "Shelve Instance", and "Unshelve Instance"; these are described below.<br />
<br />
Allowed actions&mdash;i.e., the ones that appear in the drop-down menu&mdash;''depend on the current state of the instance''. For example, the "Resize Instance" action is allowed only for instances that are in the Active state. The figure below shows the possible states in OpenStack and the transitions that are allowed in each case.<br />
<br />
[[File:Openstack-server-states.png|thumb|left|700px|Source: OpenStack[https://docs.openstack.org/nova/latest/reference/vm-states.html]]]<br />
<div style="clear: both"></div><br />
<br />
When your instance has been created, the "<tt>Instances</tt>" tab will list its current state (as well as the state of your other instances) under the "<tt>Status</tt>" column. In the rightmost column called "<tt>Actions</tt>," you will see a drop-down menu for each instance. This menu lists the actions that are allowed for the given instance. Below we describe the typical states and list some of the common actions you will use to change instance state.<br />
<br />
==== Important States ====<br />
<br />
'''''Note: The only state where you are NOT being charged for computational resources is Shelved Offloaded'''''<br />
<br />
:* '''Active''': Instance is active, you can connect to it and are being billed for the computational resources dedicated to it.<br />
:* '''Shelved Offloaded''': The Instance is not resident on the compute host; this means you will not be billed for computational resources while the Instance is in this state (although you will be charged for the storage required to hold it). You can restart the server when you need it again.<br />
:* '''Paused''': In this state, the server state is preserved in RAM, but operations have been stopped and will resume when instructed. You are still being charged for the computational resources dedicated to the Instance.<br />
:* '''Suspended''': Instance state has been stored on disk, including the contents of its RAM. With Red Cloud's configuration, you are still paying for the computational resources you were using.<br />
:* '''Stopped''': This is like powering off a server; in this state, you are still being billed for the computational resources.<br />
:* '''Resized''': At this time, the Instance is being Resized to a different flavor&mdash;that is, a different allocation of vCPUs and RAM&mdash;and is not contactable.<br />
<br />
==== Operations to transition between states ====<br />
<br />
<br />
These options are available, subject to the current state of the Instances, from the dropdown available in the "Actions" column of the Instances page. ''Remember that Shelving is the only operation that will free up the computational resources your Instance has been using so that you stop being charged for them!''<br />
<br />
:* '''Pause Instance:''' Put instance into the Paused state.<br />
:* '''Suspend Instance''': Put instance into the Suspended state.<br />
:* '''Shelve Instance''': This is how you put the Instance aside so that you are no longer charged for computational resources being used; your Instance will still be visible on the Instances page with Status Shelved Offloaded. To get the Instance back up and running, select "Unshelve" from the actions menu.<br />
:* '''Resize Instance''': Allows you to select a new Instance flavor. After you have selected it, Status will be shown as "Confirm or Revert Resize/Migrate" you will have to confirm in the Actions dropdown, after which the Instance will be taken down and then come back up with the new computational resources available to it. On Linux you can check with commands such as <code>lscpu</code> or <code>cat /proc/cpuinfo</code>; on Windows you can, for example, use the "Performance" tab of Task Manager to see the available cores.<br />
:* '''Resume Instance''': Will restart the Instance from states of Paused and Suspended<br />
:* '''Soft Reboot Instance''' or '''Hard Reboot Instance''': Reboot your instance, either through issuing a command to the operation system ("Soft") or as if power-cycling the Instance ("Hard")<br />
:* '''Shut Off Instance''': Like powering off the Instance, an action you will need to confirm in a pop-up window. The Status shown will be Shutoff and you will need to "Restart" to get it back. Additionally, if you are logged into an instance and use an OS command such as <code>sudo poweroff</code> or <code>sudo init</code>, the Instances page will show the same status and you will need to select "Restart" to get the Instance back up and running. In this state, ''you are still being charged for computational resources''.</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=GPUs_in_Red_Cloud&diff=3126GPUs in Red Cloud2020-08-24T14:10:33Z<p>Pzv2: Linked several internal pages, and fixed internal link to look internal</p>
<hr />
<div>''(This page under development)''<br />
<br />
[[Red Cloud]] supports GPU computing and has two different GPU [[OpenStack#Instances|instance]] flavors that allow for GPU computing on virtual machines (VMs). In order to use a GPU in Red Cloud, you will need to select a flavor of machine with a GPU. Currently Red Cloud instances feature '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' and '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' GPU accelerators in these flavors:<br />
<br />
{| border="1" cellspacing="0" cellpadding="10" align="center" style="text-align:center;"<br />
! Flavor <br />
! CPUs<br />
! GPUs <br />
! RAM <br />
|-<br />
| *''c4.t1.m20'' || 4 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' || 20 GB<br />
|-<br />
| *''c14.g1.m60'' || 14 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' || 60 GB<br />
|}<br />
<br />
When '''[[OpenStack#Launch_an_Instance|launching an instance]]''', you can use either the base [[Red_Cloud_Linux_Instances|Linux]] or [[Red_Cloud_Windows_Instances|Windows]] instances and install your own GPU libraries, or select CUDA source images such as (...), select a GPU-enabled flavor, and configure the instance as you would any other instance.<br />
<br />
Once your instance is launched, you will have access to the GPU within the VM and can install software (pytorch, tensorflow) that will use the GPU.<br />
<br />
For more information on GPU and CUDA computing, see the Cornell Virtual Workshop "'''[https://cvw.cac.cornell.edu/gpu/ Introduction to GPGPU and CUDA Programming: Overview]'''"</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Mainpage&diff=3064Mainpage2020-05-11T18:05:46Z<p>Pzv2: Updated page name</p>
<hr />
<div><html><br />
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css"><br />
<table role="presentation" style="border:0; margin: 0;" width="100%" cellspacing="10"><br />
<tr> <br />
<td valign="top" id="mainpage_opportunitiescell"><br />
<div class="mainpage_boxtitle">WELCOME TO CAC SUPPORT</div><br />
<br />
<div class="mainpage_boxcontents_small"><br />
<p>This wiki provides Cornell University Center for Advanced Computing <a href="/">(CAC)</a> users with user documentation and other kinds of support information. If you're not a current user and would like to become one, visit <a href="/services/projects.aspx">how to start a project</a>. If you are a PI, visit <a href="/services/projects/manage.aspx">how to manage your current project</a>. Please remember to <a href="/wiki/index.php?title=Acknowledging_CAC">acknowledge CAC support</a> in your publications.</p><br />
</div><br />
</td><br />
</tr><br />
</table><br />
<br />
<br />
<table role="presentation" style="border:0; margin: 0;" width="100%" cellspacing="10"><br />
<tr><br />
<td valign="top" class="mainpage_hubbox"><br />
<br />
<div class="col-sm-4 panel-item"><br />
<a class="panel panel-circle-contrast" href="/wiki/index.php?title=Special:Search"><br />
<div class="panel-icon"><br />
<i class="fa fa-info-circle fa-5x"></i><br />
</div><br />
<div class="panel-body text-center"><br />
<h4 class="panel-title">Search support</h4><br />
<p>Search CAC support site.</p><br />
</div><br />
</a><br />
</div><br />
</td><br />
<td valign="top" class="mainpage_hubbox"><br />
<br />
<div class="col-sm-4 panel-item"><br />
<a class="panel panel-circle-contrast" href="//rt.cac.cornell.edu/index.html"><br />
<div class="panel-icon"><br />
<i class="fa fa-question-circle fa-5x"></i><br />
</div><br />
<div class="panel-body text-center"><br />
<h4 class="panel-title">Contact support</h4><br />
<p>Submit a ticket or call 607-254-8691.</p><br />
</div><br />
</a><br />
</div><br />
</td> <br />
<td valign="top" class="mainpage_hubbox"><br />
<br />
<br />
<div class="col-sm-4 panel-item"><br />
<a class="panel panel-circle-contrast" href="/datafeed/status.aspx"><br />
<div class="panel-icon"><br />
<i class="fa fa-check-circle fa-5x"></i><br />
</div><br />
<div class="panel-body text-center"><br />
<h4 class="panel-title">Check operating status</h4><br />
<p>Plan ahead for CAC infrastructure downtimes.</p><br />
</div><br />
</a><br />
</div><br />
</td> <br />
</tr><br />
</table><br />
<br />
<br />
<table role="presentation" style="border:0; margin: 0;" width="100%" cellspacing="10"><br />
<tr><br />
<br />
<!-- POUR LA PREMIERE COLONNE: USER DOCUMENTATION --><br />
<td valign="top" id="mainpage_opportunitiescell"><br />
<div class="mainpage_boxtitle">USER DOCUMENTATION</div><br />
<div class="mainpage_boxcontents_small"><br />
<ul><br />
<li><a href="/wiki/index.php?title=Getting_Started_on_Private_Clusters">Getting Started on Private Clusters</a><span> - password rules, home directories, and more</span></li><br />
<li><a href="/wiki/index.php?title=Red_Cloud" >Red Cloud</a><span> - on-demand cloud services</span></li><br />
<li><a href="/wiki/index.php?title=Private_Clusters"> Private Clusters</a> <span> - maintained by CAC </span></li><br />
<li><a href="/wiki/index.php?title=Archival_Storage" >Archival Storage</a><span> - how to use and</span><a href="/wiki/index.php?title=Syncing_to_Archival_Storage"> sync directories</a> <span> to CAC Archival Storage</span> </li><br />
<li><a href="/wiki/index.php?title=File_Transfer_using_Globus">File Transfer using Globus</a><span> - high speed file transfers to/from CAC</span></li><br />
<li><a href="/wiki/index.php?title=MATLAB_Parallel_Server_in_Red_Cloud">MATLAB Parallel Server in Red Cloud</a></li><br />
</ul><br />
</div><br />
</td><br />
<br />
<!-- TRAINING & EDUCATION --><br />
<td valign="top" id="mainpage_opportunitiescell"><br />
<div class="mainpage_boxtitle">TRAINING & EDUCATION</div><br />
<div class="mainpage_boxcontents_small"><br />
<ul><br />
<li><a href="/education/Default.aspx"> CAC Education and Outreach</a> <span> - overview </span></li><br />
<li><a href="https://portal.xsede.org/training/course-catalog" >XSEDE Training</a><span> - CAC is training lead (NSF program)</span></li><br />
<li><a href="https://cvw.cac.cornell.edu/topics/">Cornell Virtual Workshop</a><span> - online training </span></li><br />
<li><a href="http://www.math.cornell.edu/~scan/" >SCAN</a><span> - Scientific Computing and Numerics seminar</span></li><br />
<li><a href="http://www.cs.cornell.edu/courses/cs5220/2017fa/">CS 5220</a><span> - Applications of Parallel Computers course</span></li><br />
<li><a href="http://www.cse.cornell.edu/">Program in Computational Science and Engineering</a></li><br />
</ul><br />
</div><br />
</td><br />
<br />
<br />
</tr><br />
</table><br />
<br />
</html></div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started&diff=3063Getting Started2020-05-11T17:13:02Z<p>Pzv2: REDIRECTED this page to Getting Started on Private Clusters page (copy of this page with new title)</p>
<hr />
<div>#REDIRECT [[Getting Started on Private Clusters]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started_on_Private_Clusters&diff=3062Getting Started on Private Clusters2020-05-11T17:03:05Z<p>Pzv2: Created copy of Getting Started page with new title</p>
<hr />
<div>__TOC__<br />
==Managing your password==<br />
CAC has a [[Getting_Started#Rules_for_creating_passwords|password policy]] in effect. The first time that you login to the <tt>cac.cornell.edu</tt> domain, you will be required to change your password. Each password must have at least eight characters and must contain at least three of the following four elements: (1) uppercase letters (2) lowercase letters (3) special characters (4) digits. Your password can be set or changed on any of the CAC login nodes, and the password will be updated on all CAC resources. Passwords expire every six months. Do not share your password. There are more detailed instructions below.<br />
<br />
===Rules for creating passwords===<br />
{{:Rules for Creating Passwords}}<br />
<br />
===Change a password at first login===<br />
{{:Changing password at first login}}<br />
<br />
=== Change password at any time ===<br />
{{:Changing password any time}}<br />
<br />
=== Locked accounts===<br />
<br />
There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:<br />
<br />
:*Mistyping your password several times in a row. <br />
:::<tt>Solution</tt>: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!<br />
:*Trying to login to a Windows login node by using SSH when you have a new or expired password. <br />
:::<tt>Solution</tt>: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.<br />
:*Failing to log off all other sessions connected to login nodes. <br />
:::<tt>Solution</tt>: Log off all remote connections. Disconnecting the sessions is not enough.<br />
:*Failing to disconnect locally mapped drives to the CAC file server before changing your password. <br />
:::<tt>Solution</tt>: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.<br />
<br />
If you can't log on or can't wait you can submit a Password Reset ticket on our [//www.cac.cornell.edu/help issue tracking system]<br />
<br />
==Checking your CAC project ==<br />
Cornell University users can view their account limits at [https://{{SERVERNAME}}/services/cu/memberlimits.aspx CAC Account Limits].<br /><br />
Partner Program members should contact Paul Redfern at [mailto:red@cac.cornell.edu red@cac.cornell.edu] if they need information on their membership limits.<br />
<br />
==Connecting to CAC resources==<br />
<br />
=== Red Cloud ===<br />
'''Please see the [[Red Cloud]] and [[OpenStack]] pages for information on managing, connecting to, and using Red Cloud resources.'''<br />
<br />
The information on the remainder of this page is primarily for users of CAC-maintained [[Private Clusters]] (for particular research groups) and data storage services, especially CAC's online, non-archival data storage.<br />
<br />
However, it is potentially of interest to users of Red Cloud '''''and''''' other CAC services.<br />
<br />
===CAC login nodes===<br />
<br />
There are three types of login nodes:<br />
:* The head nodes for the various Linux-based private clusters<br />
:* Linux login node: <tt>linuxlogin.cac.cornell.edu</tt> <br />
:* Windows login node: <tt>winlogin.cac.cornell.edu</tt><br />
<br />
The general CAC login nodes, '''''linuxlogin''''' and '''''winlogin''''', are mostly intended for researchers who are have procured CAC storage services, apart from Red Cloud and private clusters (see [[Getting Started#Working with CAC file storage|Working with CAC file storage]]). These two login nodes are broadly accessible from the Internet, and they provide a convenient way for researchers to gain access to their files. On the general login nodes, you will find a modest number of software tools installed to aid in working with files (but please do not use these tools for doing production computing there).<br />
<br />
In what follows, we will often use '''''linuxlogin''''' and '''''winlogin''''' as stand-ins for the particular CAC servers that you are trying to access (e.g., in a private cluster or in Red Cloud).<br />
<br />
===Connect to Linux===<br />
{{:Connect to Linux}}<br />
<br />
===Connect to Windows===<br />
{{:Connect to Windows}}<br />
<br />
==Working with CAC file storage==<br />
A dedicated file server named storage03.cac.cornell.edu provides access to much of CAC's file storage, including the home directories for many of the private clusters. To work with your files, you can access this server in a variety of ways from any operating system. The first two of the following methods are covered in detail in this section:<br />
<br />
:# [[Getting Started#Home directory access|Home directory access]] - Mount/map your portion of the storage03 filesystem as network share or network drive. Once the filesystem is mounted, your files on storage03 appear in a folder that you can access just like other folders on your computer. On Linux, use the <tt>mount</tt> command; on Mac, use "Go > Connect to server"; on Windows (including winlogin), enter the UNC address into the address bar of a File Explorer window, or do "Map a network drive".<br />
:# [[Getting Started#File transfer|File transfer]] - Use a file transfer utility like scp or sftp to copy your files to or from storage03. Connect to linuxlogin to do this, because storage03 is not directly accessible. On linuxlogin, your main CAC folder on storage03 is your home folder when connect via ssh, scp, or sftp.<br />
:# - Use Globus to transfer files to or from storage03. The source or destination of the files must also be a Globus endpoint (and note, you can set up any computer to be a personal endpoint). Endpoints at CAC are described on the [[File Transfer using Globus]] page.<br />
<br />
Note: by default, your home directory on linuxlogin and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
===Home directory access===<br />
{{:Home Directory Access}}<br />
<br />
===File transfer===<br />
{{:File transfer}}<br />
<br />
==Linux usage tips==<br />
If you have never used Linux before, we recommend exploring the [[Linux Tutorial]].<br />
<br />
{{:Linux Usage Tips}}<br />
<br />
==More information==<br />
The CAC main website is [https://{{SERVERNAME}}/ here]. There are many useful documents on the Support page at [[ Main Page| CAC documentation]].<br />
<br />
==Acknowledging CAC==<br />
{{:Acknowledging CAC}}<br />
<br />
==FAQ/Troubleshooting==<br />
#[[FAQ#Account| Account FAQ]]<br />
#[[FAQ#Login| Login FAQ]]<br />
# If you have more questions, see [[FAQ| here]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud_Linux_Instances&diff=3039Red Cloud Linux Instances2020-03-11T19:36:23Z<p>Pzv2: Added Instance Maintenance section</p>
<hr />
<div>Linux Instances can be created and maintained on [[Red_Cloud|Red Cloud]] using the [https://redcloud.cac.cornell.edu OpenStack Web Interface]. This documentation assumes a basic understanding of [[OpenStack]], so please review that page as needed. If you are '''new to Linux''', you may want to read the [[Linux Tutorial]] first. It is also a good idea to be familiar with the [[Linux Tutorial]] if you have not previously done '''Linux system administration''', which is an assumed prerequisite to managing Linux Instances. Additionally, you may find the [https://cvw.cac.cornell.edu/Linux/ Introduction to Linux] topic on the [https://cvw.cac.cornell.edu/topics Cornell Virtual Workshop] helpful.<br />
<br />
__TOC__<br />
<br />
== Creating a New Linux Instance ==<br />
<br />
You can boot an instance with most modern Linux distributions. Currently Red Cloud offers pre-made VM images running the following Linux distributions:<br />
<br />
:* CentOS<br />
:* Ubuntu (including [https://wiki.ubuntu.com/LTS LTS])<br />
<br />
=== Steps ===<br />
# Log in to the [https://redcloud.cac.cornell.edu OpenStack Web Interface] (check out [[OpenStack#Logging_In|how to log in]] if you need to)<br />
# If you have not already, [[OpenStack Key Pairs#Creating_a_Key_Pair|create a key pair]]<br />
# If you have not already, [[OpenStack Security Groups#Creating a Security Group|create a security group]]. Note that your security group should include the inbound SSH rule so you can connect to it.<br />
# '''Optional:''' [[Networks#Private Networks|Set up a Private Network]]<br />
# Select <tt>Launch Instance</tt> from the [https://redcloud.cac.cornell.edu/dashboard/project/instances/ Instances] page<br />
# Follow the instructions about [[OpenStack#Launch an Instance|launching a new instance]], and select one of the a CentOS or Ubuntu [[Images|images]] under the <tt>Source</tt> tab<br />
# '''Optional:''' [[Volumes#Create and Attach a Volume|Create and attach a Volume]]<br />
# '''Optional:''' [[Networks#Floating IP Addresses|Associate a Floating IP address]]<br />
<br />
Now that you have created and launched an instance, your next steps will be to [[#Accessing_Instances|connect to it]] and set up a new user account. See the [[Linux_Tutorial#Initial_User_Setup_2|CentOS steps]] or [[Linux_Tutorial#Initial_User_Setup|Ubuntu steps]] for more information on how to set up a new user, update, and install software for each distribution.<br />
<br />
== Accessing Instances ==<br />
<br />
First, establish access to your instance using [[Connect_to_Linux#Using_Secure_Shell|Secure Shell (ssh)]], possibly including [[Connect_to_Linux#Using_X-Windows|X Windows]] for remote graphical display. If you are at all unfamiliar with Linux, we recommend following the [[Linux Tutorial]]. If you would like to have a desktop environment (not available by default for Linux instances), then you may want to follow the steps for [[XFCE Desktop on CentOS]]. Establishing an SSH connection is a prerequisite for creating a Linux desktop environment such as XFCE.<br />
<br />
=== Secure Shell (SSH) ===<br />
<br />
The main requirements for logging in to your instance using secure shell are:<br />
:* The [[OpenStack Security Groups|security group]] for your instance must permit SSH connections (TCP port 22) from your current IP address.<br />
<br />
:* You must use the private key that matches the public key in the [[OpenStack Key Pairs|key pair]] you specified when launching the instance.<br />
<br />
:* You must log in to your instance using the correct initial account name. For CentOS this is <tt>centos</tt> and for Ubuntu it is <tt>ubuntu</tt>.<br />
<br />
To log in through SSH, specify the key pair file (for example <tt>my_key.pem</tt>),<br />
account name and IP address as follows:<br />
<br />
ssh -i my_key.pem <account>@<IP address of your instance><br />
<br />
For more information on how to use SSH, see the [[Connect to Linux]] page.<br />
<br />
Note: Transferring files can also be done over <code>ssh</code>. See the <code>scp</code> and <code>sftp</code> commands, or programs like [//winscp.net/eng/index.php WinSCP] and [//apple.stackexchange.com/questions/25661/whats-a-good-graphical-sftp-utility-for-os-x similar software for Mac OS X].<br />
<br />
=== Troubleshooting ===<br />
<br />
:* Check the instance's console in the [[OpenStack|OpenStack Web Interface]]. Did the instance boot successfully?<br />
:* Do the [[OpenStack Security Groups|security group's]] rules allow incoming ssh connections (TCP port 22) from ''the IP address of your computer''?<br />
:* Is your instance on the [[Networks#Public Network|public network]]? If not, does it have an associated [[Networks#Floating IP Addresses|floating IP address]]?<br />
:* Make sure you are using the correct username:<br />
:** If you are using Ubuntu, did you login as the '''ubuntu''' user instead of root? For more information on that, see the [[Linux_Tutorial#The_.22ubuntu.22_User|Linux Tutorial]].<br />
:** If you are using CentOS, did you login as the '''centos''' user instead of root?<br />
:* Did you [[OpenStack_Key_Pairs#Creating_a_Key_Pair|create a key pair]] and make sure to [[OpenStack_Key_Pairs#Selecting_a_Key_Pair_When_Creating_an_Instance|select it when creating an instance]]?<br />
:* Are you using your key pair in the command? See the [[#Secure_Shell_.28SSH.29|SSH]] instructions above for an example.<br />
:* If you get an unexpected password prompt:<br />
:** Did you use the correct user name? See suggestions above.<br />
:** Make sure your private key matches the public key of the [[OpenStack Key Pairs|key pair]]. <br />
:**# On your computer, run: <code>ssh-keygen -y -f <private key file></code><br />
:**# Does the output match the public key list here? [https://redcloud.cac.cornell.edu/dashboard/project/key_pairs/ https://redcloud.cac.cornell.edu/dashboard/project/key_pairs/]<your keypair name>/<br />
:** Make sure your private key file is saved as a ".pem" extension and that it has the proper permissions: <p><code>chmod 600 <key name>.pem</code></p><br />
<br />
=== XFCE Desktop on CentOS ===<br />
<br />
In case one would prefer a desktop environment over a command-line, this section details the steps to setting up an [https://xfce.org/ XFCE Desktop Environment] on a CentOS 7.4 instance. Other typical Linux desktop environments, such as Gnome, are also available but XFCE is used as an example here. Additionally, setting up a desktop environment may work on Ubuntu instances as well, with some differences. Once the environment is set up, you can run a VNC server and connect to it through an ssh tunnel.<br />
<br />
==== XFCE Setup ====<br />
<br />
# Log in as root via ssh as [[#Secure_Shell_.28ssh.29 | described above]].<br />
# Install needed packages:<br />
#* <code>yum groupinstall xfce</code><br />
#* <code>yum install tigervnc-server</code><br />
# Install some additional software that most users will want. These are only suggestions, and this is not a comprehensive list<br />
#* <code>yum install gedit</code><br />
#* <code>yum install firefox</code><br />
#* [https://www.tecmint.com/install-libreoffice-on-rhel-centos-fedora-debian-ubuntu-linux-mint/ LibreOffice]<br />
<br />
==== VNC User Setup ====<br />
<br />
For each user that will want to use the XFCE Desktop, you will need to set up VNC (Virtual Network Computing) cability. To do this, follow the directions below. Alternatively, there is also an [https://linuxtechlab.com/secure-vnc-server-tls-encryption/ Easy guide to secure VNC server with TLS encryption].<br />
<br />
# Open a shell as that user<br />
# <code>vncpasswd</code><br />
#* Sets the user's VNC password<br />
#* This step is '''not necessary''' for read-only VNC<br />
#* This creates a ~/.vnc folder<br />
# <code>vim ~/.vnc/xstartup</code><br />
#* Paste this text into the file: <br /><tt>#!/bin/bash<br />xrdb $HOME/.Xresources<br />startxfce4 &<br /></tt><br />
# <code>chmod 775 ~/.vnc/xstartup</code><br />
# <code>mkdir ~/bin</code><br />
# <code>vim ~/bin/start_vncserver.sh</code><br />
#* Paste this text into the file: <br /><tt>#!/bin/bash<br />vncserver -geometry 1680x1050<br /></tt><br />
# <code>chmod 775 ~/bin/start_vncserver.sh</code><br />
<br />
==== Using VNC ====<br />
<br />
A brief overview for users is provided here, and for more information please see the [[Getting_Started#Using_VNC|Using VNC section on our Getting Started page]].<br />
<br />
===== Manage the VNC Server =====<br />
<br />
Whenever an instance gets rebooted, you can '''restart''' the VNC server by doing the following<br />
# ssh into the instance<br />
# run <code>~/bin/start_vncserver.sh</code><br />
<br />
You can '''stop''' the NVC server by running <code>vncserver -kill :1</code><br />
<br />
To '''change or reset the VNC password''', you can simply run <code>vncpassword</code><br />
<br />
To '''change the screen resolution''':<br />
:* Permanently: edit the script <code>~/.vnc/xstartup</code><br />
:* For this session only: use Applications->Display<br />
<br />
===== View the Desktop =====<br />
<br />
# Open a local shell and ssh to establish the tunnel (recommended local port 10000):<br />
#* <code>ssh -L 10000:localhost:5901 <remote ip></code><br />
#* '''Note:''' When this shell is closed, the VNC viewer will have to close, too, though VNC is still running.<br />
# In a VNC viewer app, connect to the VNC server <code>localhost:10000</code> (or whichever port you chose above).<br />
#* This should open a window showing the desktop<br />
#* The '''first time''' you do this:<br />
#** For the remote computer, you may have to dismiss a warning dialog<br />
#** You will need to initialize a "panel". Click "Use default config"<br />
<br />
'''Note for Windows users:''' You can also find useful instructions on the [[VNC Tunnel Windows]] page. You may also want to look into [https://fossbytes.com/enable-built-windows-10-openssh-client/ OpenSSH] or [https://www.windowscentral.com/how-install-bash-shell-command-line-windows-10 Bash on Ubuntu on Windows].<br />
<br />
== Instance Maintenance ==<br />
<br />
All self-managed desktops, laptops, servers, and Red Cloud instances, both Windows and Linux, should be updated with Operating System, Adobe Flash, Acrobat, and Reader critical and security updates on a '''''monthly''''' basis. For Linux instances:<br />
# Check for updates<br />
#* Ubuntu: <code>sudo apt update</code><br />
#* CentOS: <code>yum check-update</code><br />
# Install updates<br />
#* Ubuntu: <code>sudo apt upgrade</code><br />
#* CentOS: <code>sudo yum update</code><br />
# Reboot the instance with <code>sudo reboot</code> on both Ubuntu and CentOS<br />
<br />
Before rebooting make sure to save all active work. Rebooting will disconnect you from the instance. Wait a few minutes to allow the instance to restart before reconnecting. When you reconnect, you should verify that the updates were applied by repeating step 1.<br />
<br />
== Initialize and Mount a Volume ==<br />
<br />
The instructions here are for formatting and mounting [[Volumes|attached volumes]], though steps like these can only be performed if you have [[Volumes#Create_and_Attach_a_Volume|allocated and attached the volume]] through OpenStack, which can be done while the instance is running.<br />
<br />
'''Note:''' These instructions assume you are the [[Linux_Tutorial#Definitions|root user]]. If you are not (such as on [[Linux_Tutorial#The_.22ubuntu.22_User|Ubuntu]]), then you may need to prepend <code>sudo</code> where appropriate.<br />
<br />
# Set up file system:<br />
#* <code>mkfs.ext4 /dev/vdb</code><br />
# Make a directory where device will be mounted, for example "<tt>sratch</tt>":<br />
#* <code>mkdir /scratch</code><br />
# Mount the device:<br />
#* <code>mount /dev/vdb /scratch</code><br />
# To have the mount automatically renewed after reboot, add an <code>fstab</code> entry (this is a little dangerous)<br />
#* <code>vim /etc/fstab</code><br />
#* Add a line with tab separations between four fields: disk device, mounted location, "ext4", "defaults":<br />
#** <code>/dev/vdb /scratch ext4 defaults</code></div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=OpenStack&diff=3035OpenStack2020-02-07T21:10:00Z<p>Pzv2: /* Configuring the Instance */ Added a note about selecting a small instance size and resizing later</p>
<hr />
<div>[https://en.wikipedia.org/wiki/OpenStack OpenStack] is an [https://en.wikipedia.org/wiki/Open-source_model open-source] [https://en.wikipedia.org/wiki/Cloud_computing#Service_models cloud stack] that is currently running on [[Red_Cloud|Red Cloud]]. Also, for more information, see the [https://docs.openstack.org/ Official Documentation for OpenStack].<br />
<br />
This page is intended as a quick walk-through of the most-used features of OpenStack, so it is not comprehensive, but links to a lot of supporting documentation for more thorough explanations and advanced topics.<br />
<br />
__TOC__<br />
<br />
== Using the OpenStack Web Interface (Horizon) ==<br />
<br />
There are two ways to manage [[Red Cloud]] resources:<br />
# [https://redcloud.cac.cornell.edu OpenStack Web Interface]<br />
# [[OpenStack CLI]]<br />
<br />
Most users will use the OpenStack Web Interface (called [https://docs.openstack.org/horizon/latest/ Horizon]). This web-based interface can be used to manage [[#Instances|instances]] and [[Volumes|volumes]]. For [[Red Cloud Linux Instances|Linux Instances]], however, some users may choose to use the OpenStack CLI. This section focuses on the OpenStack Web Interface.<br />
<br />
=== Logging In ===<br />
<br />
Log in to the [https://redcloud.cac.cornell.edu OpenStack Web Interface] to create and manage Red Cloud resources. There are two ways to login: <br />
<br />
[[File:RedCloudCACLogin.png|300px|frameless|border]][[File:White_square.png|100px|frameless]][[File:RedCloudGlobusAuthLogin.png|300px|frameless|border]]<br />
<br />
# [https://www.cac.cornell.edu/services/myacct.aspx CAC Account] - Enter '''cac''' as the "<tt>Domain</tt>" and your [https://www.cac.cornell.edu/services/myacct.aspx CAC username] and password, not your Cornell NetID. If your CAC password has expired, you will need to [https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started#Managing_your_password reset it] before you will be able to login to the OpenStack Web Interface.<br />
# [https://www.globus.org/tags/globus-auth Globus Auth] - Log in through Globus<br />
#* Currently, this feature is '''only available to Aristotle users'''. This feature will be enabled for all users in the future.<br />
#* You must ''link your Cornell account'', or any accounts attached to the projects you are on, in order to have access to them when using Globus Auth.<br />
#* If you can't log in with Globus Auth, it may be that you have not linked your account yet.<br />
<br />
You can use the "<tt>Authenticate using</tt>" drop-down to switch between the two options. Neither option requires you to enter a project ID; you can switch between the projects you are on once logged in.<br />
<br />
=== Overview Page ===<br />
<br />
The Overview page is the first place you will be taken upon logging into Red Cloud.<br />
<br />
:* Provides useful metrics on currently selected project<br />
:* '''Before creating an instance''', you will need to:<br />
:** Select the correct project from the "<tt>Project</tt>" drop-down at the top right of the page (if you are on multiple projects)<br />
:** [[#Key_Pairs|Create a key pair]] - for authentication when you log in the first time<br />
:** [[#Security_Groups|Create a security group]] - defines allowable types of port access for an instance<br />
:** Optional: [[Networks#Private_Networks|Set up a private network]] - if you do not want your instance to be available on the [[Networks#Public Network|public network]]<br />
:* You may also want to:<br />
:** [[Volumes#Create and Attach a Volume|Create and Attach a Volume]] (can also be done when launching an instance)<br />
:** [[Networks#Floating IP Addresses|Associate a Floating IP address]] - a fixed IP address that can be assigned to an instance<br />
<br />
=== Key Pairs ===<br />
<br />
[[File:Overview_KeyPairs_Circled.png|350px|frameless|border]]<br />
<br />
To get to the Key Pairs page: select the "<tt>Compute</tt>" tab along the top (you should start here at login), then click on "<tt>Key Pairs</tt>" along the top bar as pictured above. If you are logged in already, you can also get to it by this link: [https://redcloud.cac.cornell.edu/dashboard/project/key_pairs/ Key Pairs].<br />
<br />
On the Key Pairs page, you can view the list of available [[OpenStack_Key_Pairs|key pairs]] for your project. From here, you can also [[OpenStack_Key_Pairs#Creating_a_Key_Pair|create]] or [[OpenStack_Key_Pairs#Importing_a_Key_Pair|import]] a key pair. If you do not already have a key pair listed, you can either create one before [[OpenStack#Launch_an_Instance|launching an instance]], or [[OpenStack_Key_Pairs#Selecting_a_Key_Pair_When_Creating_an_Instance|create or upload a key pair]] during instance setup.<br />
<br />
For more information, here is a walk-through on [[OpenStack Key Pairs]].<br />
<br />
=== Security Groups ===<br />
<br />
[[File:Overview_SecurityGroups.png|350px|frameless|border]]<br />
<br />
To get to the Security Groups page: select the "<tt>Network</tt>" drop-down menu along the top, then click on "<tt>Security Groups</tt>" as pictured above. If you are already logged in, you can also get to it by following this link: [https://redcloud.cac.cornell.edu/dashboard/project/security_groups/ Security Groups]<br />
<br />
On the Security Groups page, you can view a list of available [[OpenStack Security Groups|security groups]] for your project, including a default security group. On this page, you can also [[OpenStack_Security_Groups#Creating_a_Security_Group|create]] and delete security groups. It '''is ''not''''' recommended that you use the default security group without [[OpenStack_Security_Groups#Managing_a_Security_Group.27s_Rules|modifying the rules]] to fit your needs. A good security practice is to have one security group per application or one per user. Instances that have no business talking to each other should generally be in separate security groups.<br />
<br />
If you do not already have a security group set up, you will want to [[OpenStack_Security_Groups#Creating_a_Security_Group|create]] one before [[OpenStack#Launch_an_Instance|launching an instance]] because you cannot create one during instance setup. However, you can [[OpenStack_Security_Groups#Assigning_Security_Groups_to_an_Instance|assign a security group]] to an instance later, and even [[OpenStack_Security_Groups#Adding_a_Rule_to_a_Security_Group|add]] or<br />
[[OpenStack_Security_Groups#Managing_a_Security_Group.27s_Rules|modify the rules]] of the security group at any time. <br />
<br />
For more information, here is a walk-through on [[OpenStack Security Groups]].<br />
<br />
== Instances ==<br />
<br />
Each instance is a Virtual Machine (VM) in the cloud. You can select CPU/RAM/disk configurations (called "flavors") for the VM. The available VM configurations are:<br />
<br />
{| border="1" cellspacing="0" cellpadding="10" align="center" style="text-align:center;"<br />
! Flavor <br />
! CPUs<br />
! GPUs <br />
! RAM <br />
|-<br />
| c1.m8 || 1 || None || 8 GB<br />
|-<br />
| c2.m16 || 2 || None || 16 GB<br />
|-<br />
| c4.m32 || 4 || None || 32 GB<br />
|-<br />
| c8.m64 || 8 || None || 64 GB<br />
|-<br />
| c14.m112 || 14 || None || 112 GB<br />
|-<br />
| c20.m160 || 20 || None || 160 GB<br />
|-<br />
| c28.m240|| 28 || None || 240 GB<br />
|-<br />
| *''c4.t1.m20'' || 4 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' || 20 GB<br />
|-<br />
| *''c14.g1.m60'' || 14 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' || 60 GB<br />
|-<br />
| colspan="4" style="text-align:left;" | ''* GPU flavors<br />
|}<br />
<br />
When you are first starting an instance, we '''recommended''' that you select the smallest flavor (least number of CPUs) that you think will be able to handle installation and configuration of the software and environment on your instance, and then [[Resizing an Instance|resize the instance]] when you are ready to run. This way you will save core hours that would otherwise have been spent idle. This method is especially useful when configuring a ''GPU instance'' due to the number of cores. Also note: you can begin with a smaller instance size (or flavor) that does not contain a GPU, and later resize to one that does.<br />
<br />
The root disk size of the instance will default to the size of the [[Images|image]] you select. You have the option to create a [[Volumes|volume]] as the root disk beyond the image size at launch time. Note that we do not oversubscribe physical RAM or CPU cores (hyperthreading is disabled).<br />
<br />
To work with instances, select the "<tt>Instances</tt>" page under the "<tt>Compute</tt>" tab, as pictured below:<br />
<br />
[[File:InstancesMenu.png|350px|frameless|border]]<br />
<br />
=== Launch an Instance ===<br />
<br />
This section is a general walk-through for creating a new instance, which is not specific to an Operating System (OS). For more specific information per OS, see either of these pages:<br />
<br />
:* [[Red Cloud Linux Instances|Linux Instances]]<br />
:* [[Red Cloud Windows Instances|Windows Instances]]<br />
<br />
==== To launch a new instance ====<br />
<br />
# [[#Key_Pairs|Create Key Pair]]<br />
# [[#Security Groups|Create a Security Group]] and be sure that you select the appropriate rule for connecting to your instance (SSH for [[Red Cloud Linux Instances|Linux Instances]] and RDP for [[Red Cloud Windows Instances|Windows Instances]])<br />
# Select "<tt>Launch Instance</tt>" on the top right side of the [https://redcloud.cac.cornell.edu/dashboard/project/instances/ Instances] page [[File:InstancesOptions.png|600px|frameless|border]]<br />
<br />
The full "<tt>Launch Instance</tt>" menu will pop up like this:<br />
<br />
[[File:InstanceLaunchMenuFull.png|700px|frameless|border]]<br />
<br />
:* Tabs that you are required to fill out are marked with a '''*'''<br />
:* It is '''recommended''' that you also select your own Security Group, otherwise the default security group will be selected, which may not be ideal for your work.<br />
:* It is '''necessary''' that you select your own Key Pair, even though this field is not marked required, so that you are able to connect to your instance after creation.<br />
<br />
==== Configuring the Instance ====<br />
<br />
# Under the "<tt>Details</tt>" tab:<br />
#* Enter a name for your instance<br />
#* '''Count''' is the number of identical instances you would like to create (typically 1).<br />
#** Note that if you create multiple instances this way, the names will be identical with a dash and number added at the end.<br />
#** For example, if you set Instance Name to "my_instance" and you set Count to 3, you would start instances named "my_instance-1", "my_instance-2", and "my_instance-3".<br />
# On the "<tt>Source</tt>" tab:<br />
#* You must '''Select Boot Source''', which is described on the page as "the template used to create an instance."<br />
#** It is generally a good idea to start with an [[Images|image]] as the source, unless you want to create an instance from a pre-existing source.<br />
#** For more information on the other options, see [[Images#Creating an Image|Creating an Image]].<br />
#* You can select a specific source under the <tt>Available</tt> list by selecting the up arrow on the right-hand side.<br />
#* Get more details about the specific source by selecting the right-arrow on the left-hand side next to the name.<br />
#* You will have the option to '''Create New Volume''' if you have selected either "<tt>Image</tt>" or "<tt>Instance Snapshot</tt>" as the source (default is "<tt>Yes</tt>"):<br />
#*# '''<tt>Yes</tt>''': If selected, a [[Volumes|volume]] will be created to be the instance's root disk. You will have the options of extending the size of the volume beyond the image size, and keeping the volume when the instance is deleted (by selecting <tt>No</tt> in the '''Delete Volume on Instance Delete''' field).<br />
#*#* '''Volume Size''' is the size of your root [[Volumes|volume]]. The default number will match the size of the [[Images|image]] you select, and can be increased.<br />
#*#* '''Delete Volume on Instance Delete''' determines whether or not your root volume will be deleted when you terminate the instance. By default, this option is set to "<tt>No</tt>" to prevent accidental deletion of your data. However, ''if you do not intend to re-use the root volume, you could unintentionally incur excess storage usage''. You can either delete the root volume manually later (it will show up in the list of volumes with the ID the same as the name), or select "<tt>Yes</tt>" on this option to automatically delete it when you terminate your instance later.<br />
#*#* You can also customize the name of the volume under '''Device Name'''.<br />
#*# '''<tt>No</tt>''': If selected, the instance will boot off a root disk the same size as the image. The root disk will be deleted when the instance is deleted.<br />
# The "<tt>Flavor</tt>" tab is where you select the VM configuration discussed [[#Instances|above]].<br />
#* We '''recommended''' that you select the smallest flavor (least number of CPUs) that you think will be able to handle installation and configuration of the software and environment on your instance, and then [[Resizing an Instance|resize the instance]] when you are ready to run. This way you will save core hours that would otherwise have been spent idle. Also note: you can begin with a smaller instance size (or flavor) that does not contain a GPU, and later resize to one that does.<br />
#* You can select a configuration by selecting the up arrow on the right-hand side.<br />
#* Get more details about the configuration by selecting the right-arrow on the left-hand side next to the name.<br />
#* The "<tt>Total Disk</tt>" will show "0 GB" because the disk size will match the [[Images|image]] you selected on the "<tt>Source</tt>" tab.<br />
# For the "<tt>Networks</tt>" tab, two options are available:<br />
## You can make the instance available on the [[Networks#Public Network|public net]]. This is the simplest and most common selection.<br />
## You can select your own [[Networks#Private Networks|private network]], which has to be set up before you launch an instance. For more information, see the [[Networks]] page.<br />
# On the "<tt>Security Groups</tt>" tab, select the [[#Security Groups|security group]] you already created.<br />
# On the "<tt>Key Pairs</tt>" tab, select the [[#Key Pairs|key pair]] you already created.<br />
<br />
=== Instance States ===<br />
<br />
OpenStack defines several [https://developer.openstack.org/api-guide/compute/server_concepts.html#server-status Server States] through which you can move your instances. You change the state of your instance by making a selection from a drop-down menu under the <tt>Actions</tt> column. Three significant actions to know about are "Resize Instance", "Shelve Instance", and "Unshelve Instance"; these are described below.<br />
<br />
Allowed actions&mdash;i.e., the ones that appear in the drop-down menu&mdash;''depend on the current state of the instance''. For example, the "Resize Instance" action is allowed only for instances that are in the Active state. The figure below shows the possible states in OpenStack and the transitions that are allowed in each case.<br />
<br />
[[File:Openstack-server-states.png|thumb|left|700px|Source: OpenStack[https://docs.openstack.org/nova/latest/reference/vm-states.html]]]<br />
<div style="clear: both"></div><br />
<br />
When your instance has been created, the "<tt>Instances</tt>" tab will list its current state (as well as the state of your other instances) under the "<tt>Status</tt>" column. In the rightmost column called "<tt>Actions</tt>," you will see a drop-down menu for each instance. This menu lists the actions that are allowed for the given instance. Below we describe the typical states and list some of the common actions you will use to change instance state.<br />
<br />
==== Important States ====<br />
<br />
'''''Note: The only state where you are NOT being charged for computational resources is Shelved Offloaded'''''<br />
<br />
:* '''Active''': Instance is active, you can connect to it and are being billed for the computational resources dedicated to it.<br />
:* '''Shelved Offloaded''': The Instance is not resident on the compute host; this means you will not be billed for computational resources while the Instance is in this state (although you will be charged for the storage required to hold it). You can restart the server when you need it again.<br />
:* '''Paused''': In this state, the server state is preserved in RAM, but operations have been stopped and will resume when instructed. You are still being charged for the computational resources dedicated to the Instance.<br />
:* '''Suspended''': Instance state has been stored on disk, including the contents of its RAM. With Red Cloud's configuration, you are still paying for the computational resources you were using.<br />
:* '''Stopped''': This is like powering off a server; in this state, you are still being billed for the computational resources.<br />
:* '''Resized''': At this time, the Instance is being Resized to a different flavor&mdash;that is, a different allocation of vCPUs and RAM&mdash;and is not contactable.<br />
<br />
==== Operations to transition between states ====<br />
<br />
<br />
These options are available, subject to the current state of the Instances, from the dropdown available in the "Actions" column of the Instances page. ''Remember that Shelving is the only operation that will free up the computational resources your Instance has been using so that you stop being charged for them!''<br />
<br />
:* '''Pause Instance:''' Put instance into the Paused state.<br />
:* '''Suspend Instance''': Put instance into the Suspended state.<br />
:* '''Shelve Instance''': This is how you put the Instance aside so that you are no longer charged for computational resources being used; your Instance will still be visible on the Instances page with Status Shelved Offloaded. To get the Instance back up and running, select "Unshelve" from the actions menu.<br />
:* '''Resize Instance''': Allows you to select a new Instance flavor. After you have selected it, Status will be shown as "Confirm or Revert Resize/Migrate" you will have to confirm in the Actions dropdown, after which the Instance will be taken down and then come back up with the new computational resources available to it. On Linux you can check with commands such as <code>lscpu</code> or <code>cat /proc/cpuinfo</code>; on Windows you can, for example, use the "Performance" tab of Task Manager to see the available cores.<br />
:* '''Resume Instance''': Will restart the Instance from states of Paused and Suspended<br />
:* '''Soft Reboot Instance''' or '''Hard Reboot Instance''': Reboot your instance, either through issuing a command to the operation system ("Soft") or as if power-cycling the Instance ("Hard")<br />
:* '''Shut Off Instance''': Like powering off the Instance, an action you will need to confirm in a pop-up window. The Status shown will be Shutoff and you will need to "Restart" to get it back. Additionally, if you are logged into an instance and use an OS command such as <code>sudo poweroff</code> or <code>sudo init</code>, the Instances page will show the same status and you will need to select "Restart" to get the Instance back up and running. In this state, ''you are still being charged for computational resources''.</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=OpenStack&diff=3034OpenStack2020-02-07T21:07:07Z<p>Pzv2: /* Instances */ Added a note about recommendations to start with a small instance and resize</p>
<hr />
<div>[https://en.wikipedia.org/wiki/OpenStack OpenStack] is an [https://en.wikipedia.org/wiki/Open-source_model open-source] [https://en.wikipedia.org/wiki/Cloud_computing#Service_models cloud stack] that is currently running on [[Red_Cloud|Red Cloud]]. Also, for more information, see the [https://docs.openstack.org/ Official Documentation for OpenStack].<br />
<br />
This page is intended as a quick walk-through of the most-used features of OpenStack, so it is not comprehensive, but links to a lot of supporting documentation for more thorough explanations and advanced topics.<br />
<br />
__TOC__<br />
<br />
== Using the OpenStack Web Interface (Horizon) ==<br />
<br />
There are two ways to manage [[Red Cloud]] resources:<br />
# [https://redcloud.cac.cornell.edu OpenStack Web Interface]<br />
# [[OpenStack CLI]]<br />
<br />
Most users will use the OpenStack Web Interface (called [https://docs.openstack.org/horizon/latest/ Horizon]). This web-based interface can be used to manage [[#Instances|instances]] and [[Volumes|volumes]]. For [[Red Cloud Linux Instances|Linux Instances]], however, some users may choose to use the OpenStack CLI. This section focuses on the OpenStack Web Interface.<br />
<br />
=== Logging In ===<br />
<br />
Log in to the [https://redcloud.cac.cornell.edu OpenStack Web Interface] to create and manage Red Cloud resources. There are two ways to login: <br />
<br />
[[File:RedCloudCACLogin.png|300px|frameless|border]][[File:White_square.png|100px|frameless]][[File:RedCloudGlobusAuthLogin.png|300px|frameless|border]]<br />
<br />
# [https://www.cac.cornell.edu/services/myacct.aspx CAC Account] - Enter '''cac''' as the "<tt>Domain</tt>" and your [https://www.cac.cornell.edu/services/myacct.aspx CAC username] and password, not your Cornell NetID. If your CAC password has expired, you will need to [https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started#Managing_your_password reset it] before you will be able to login to the OpenStack Web Interface.<br />
# [https://www.globus.org/tags/globus-auth Globus Auth] - Log in through Globus<br />
#* Currently, this feature is '''only available to Aristotle users'''. This feature will be enabled for all users in the future.<br />
#* You must ''link your Cornell account'', or any accounts attached to the projects you are on, in order to have access to them when using Globus Auth.<br />
#* If you can't log in with Globus Auth, it may be that you have not linked your account yet.<br />
<br />
You can use the "<tt>Authenticate using</tt>" drop-down to switch between the two options. Neither option requires you to enter a project ID; you can switch between the projects you are on once logged in.<br />
<br />
=== Overview Page ===<br />
<br />
The Overview page is the first place you will be taken upon logging into Red Cloud.<br />
<br />
:* Provides useful metrics on currently selected project<br />
:* '''Before creating an instance''', you will need to:<br />
:** Select the correct project from the "<tt>Project</tt>" drop-down at the top right of the page (if you are on multiple projects)<br />
:** [[#Key_Pairs|Create a key pair]] - for authentication when you log in the first time<br />
:** [[#Security_Groups|Create a security group]] - defines allowable types of port access for an instance<br />
:** Optional: [[Networks#Private_Networks|Set up a private network]] - if you do not want your instance to be available on the [[Networks#Public Network|public network]]<br />
:* You may also want to:<br />
:** [[Volumes#Create and Attach a Volume|Create and Attach a Volume]] (can also be done when launching an instance)<br />
:** [[Networks#Floating IP Addresses|Associate a Floating IP address]] - a fixed IP address that can be assigned to an instance<br />
<br />
=== Key Pairs ===<br />
<br />
[[File:Overview_KeyPairs_Circled.png|350px|frameless|border]]<br />
<br />
To get to the Key Pairs page: select the "<tt>Compute</tt>" tab along the top (you should start here at login), then click on "<tt>Key Pairs</tt>" along the top bar as pictured above. If you are logged in already, you can also get to it by this link: [https://redcloud.cac.cornell.edu/dashboard/project/key_pairs/ Key Pairs].<br />
<br />
On the Key Pairs page, you can view the list of available [[OpenStack_Key_Pairs|key pairs]] for your project. From here, you can also [[OpenStack_Key_Pairs#Creating_a_Key_Pair|create]] or [[OpenStack_Key_Pairs#Importing_a_Key_Pair|import]] a key pair. If you do not already have a key pair listed, you can either create one before [[OpenStack#Launch_an_Instance|launching an instance]], or [[OpenStack_Key_Pairs#Selecting_a_Key_Pair_When_Creating_an_Instance|create or upload a key pair]] during instance setup.<br />
<br />
For more information, here is a walk-through on [[OpenStack Key Pairs]].<br />
<br />
=== Security Groups ===<br />
<br />
[[File:Overview_SecurityGroups.png|350px|frameless|border]]<br />
<br />
To get to the Security Groups page: select the "<tt>Network</tt>" drop-down menu along the top, then click on "<tt>Security Groups</tt>" as pictured above. If you are already logged in, you can also get to it by following this link: [https://redcloud.cac.cornell.edu/dashboard/project/security_groups/ Security Groups]<br />
<br />
On the Security Groups page, you can view a list of available [[OpenStack Security Groups|security groups]] for your project, including a default security group. On this page, you can also [[OpenStack_Security_Groups#Creating_a_Security_Group|create]] and delete security groups. It '''is ''not''''' recommended that you use the default security group without [[OpenStack_Security_Groups#Managing_a_Security_Group.27s_Rules|modifying the rules]] to fit your needs. A good security practice is to have one security group per application or one per user. Instances that have no business talking to each other should generally be in separate security groups.<br />
<br />
If you do not already have a security group set up, you will want to [[OpenStack_Security_Groups#Creating_a_Security_Group|create]] one before [[OpenStack#Launch_an_Instance|launching an instance]] because you cannot create one during instance setup. However, you can [[OpenStack_Security_Groups#Assigning_Security_Groups_to_an_Instance|assign a security group]] to an instance later, and even [[OpenStack_Security_Groups#Adding_a_Rule_to_a_Security_Group|add]] or<br />
[[OpenStack_Security_Groups#Managing_a_Security_Group.27s_Rules|modify the rules]] of the security group at any time. <br />
<br />
For more information, here is a walk-through on [[OpenStack Security Groups]].<br />
<br />
== Instances ==<br />
<br />
Each instance is a Virtual Machine (VM) in the cloud. You can select CPU/RAM/disk configurations (called "flavors") for the VM. The available VM configurations are:<br />
<br />
{| border="1" cellspacing="0" cellpadding="10" align="center" style="text-align:center;"<br />
! Flavor <br />
! CPUs<br />
! GPUs <br />
! RAM <br />
|-<br />
| c1.m8 || 1 || None || 8 GB<br />
|-<br />
| c2.m16 || 2 || None || 16 GB<br />
|-<br />
| c4.m32 || 4 || None || 32 GB<br />
|-<br />
| c8.m64 || 8 || None || 64 GB<br />
|-<br />
| c14.m112 || 14 || None || 112 GB<br />
|-<br />
| c20.m160 || 20 || None || 160 GB<br />
|-<br />
| c28.m240|| 28 || None || 240 GB<br />
|-<br />
| *''c4.t1.m20'' || 4 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-t4/ Nvidia Tesla T4]''' || 20 GB<br />
|-<br />
| *''c14.g1.m60'' || 14 || 1 '''[https://www.nvidia.com/en-us/data-center/tesla-v100/ Nvidia Tesla V100]''' || 60 GB<br />
|-<br />
| colspan="4" style="text-align:left;" | ''* GPU flavors<br />
|}<br />
<br />
When you are first starting an instance, we '''recommended''' that you select the smallest flavor (least number of CPUs) that you think will be able to handle installation and configuration of the software and environment on your instance, and then [[Resizing an Instance|resize the instance]] when you are ready to run. This way you will save core hours that would otherwise have been spent idle. This method is especially useful when configuring a ''GPU instance'' due to the number of cores. Also note: you can begin with a smaller instance size (or flavor) that does not contain a GPU, and later resize to one that does.<br />
<br />
The root disk size of the instance will default to the size of the [[Images|image]] you select. You have the option to create a [[Volumes|volume]] as the root disk beyond the image size at launch time. Note that we do not oversubscribe physical RAM or CPU cores (hyperthreading is disabled).<br />
<br />
To work with instances, select the "<tt>Instances</tt>" page under the "<tt>Compute</tt>" tab, as pictured below:<br />
<br />
[[File:InstancesMenu.png|350px|frameless|border]]<br />
<br />
=== Launch an Instance ===<br />
<br />
This section is a general walk-through for creating a new instance, which is not specific to an Operating System (OS). For more specific information per OS, see either of these pages:<br />
<br />
:* [[Red Cloud Linux Instances|Linux Instances]]<br />
:* [[Red Cloud Windows Instances|Windows Instances]]<br />
<br />
==== To launch a new instance ====<br />
<br />
# [[#Key_Pairs|Create Key Pair]]<br />
# [[#Security Groups|Create a Security Group]] and be sure that you select the appropriate rule for connecting to your instance (SSH for [[Red Cloud Linux Instances|Linux Instances]] and RDP for [[Red Cloud Windows Instances|Windows Instances]])<br />
# Select "<tt>Launch Instance</tt>" on the top right side of the [https://redcloud.cac.cornell.edu/dashboard/project/instances/ Instances] page [[File:InstancesOptions.png|600px|frameless|border]]<br />
<br />
The full "<tt>Launch Instance</tt>" menu will pop up like this:<br />
<br />
[[File:InstanceLaunchMenuFull.png|700px|frameless|border]]<br />
<br />
:* Tabs that you are required to fill out are marked with a '''*'''<br />
:* It is '''recommended''' that you also select your own Security Group, otherwise the default security group will be selected, which may not be ideal for your work.<br />
:* It is '''necessary''' that you select your own Key Pair, even though this field is not marked required, so that you are able to connect to your instance after creation.<br />
<br />
==== Configuring the Instance ====<br />
<br />
# Under the "<tt>Details</tt>" tab:<br />
#* Enter a name for your instance<br />
#* '''Count''' is the number of identical instances you would like to create (typically 1).<br />
#** Note that if you create multiple instances this way, the names will be identical with a dash and number added at the end.<br />
#** For example, if you set Instance Name to "my_instance" and you set Count to 3, you would start instances named "my_instance-1", "my_instance-2", and "my_instance-3".<br />
# On the "<tt>Source</tt>" tab:<br />
#* You must '''Select Boot Source''', which is described on the page as "the template used to create an instance."<br />
#** It is generally a good idea to start with an [[Images|image]] as the source, unless you want to create an instance from a pre-existing source.<br />
#** For more information on the other options, see [[Images#Creating an Image|Creating an Image]].<br />
#* You can select a specific source under the <tt>Available</tt> list by selecting the up arrow on the right-hand side.<br />
#* Get more details about the specific source by selecting the right-arrow on the left-hand side next to the name.<br />
#* You will have the option to '''Create New Volume''' if you have selected either "<tt>Image</tt>" or "<tt>Instance Snapshot</tt>" as the source (default is "<tt>Yes</tt>"):<br />
#*# '''<tt>Yes</tt>''': If selected, a [[Volumes|volume]] will be created to be the instance's root disk. You will have the options of extending the size of the volume beyond the image size, and keeping the volume when the instance is deleted (by selecting <tt>No</tt> in the '''Delete Volume on Instance Delete''' field).<br />
#*#* '''Volume Size''' is the size of your root [[Volumes|volume]]. The default number will match the size of the [[Images|image]] you select, and can be increased.<br />
#*#* '''Delete Volume on Instance Delete''' determines whether or not your root volume will be deleted when you terminate the instance. By default, this option is set to "<tt>No</tt>" to prevent accidental deletion of your data. However, ''if you do not intend to re-use the root volume, you could unintentionally incur excess storage usage''. You can either delete the root volume manually later (it will show up in the list of volumes with the ID the same as the name), or select "<tt>Yes</tt>" on this option to automatically delete it when you terminate your instance later.<br />
#*#* You can also customize the name of the volume under '''Device Name'''.<br />
#*# '''<tt>No</tt>''': If selected, the instance will boot off a root disk the same size as the image. The root disk will be deleted when the instance is deleted.<br />
# The "<tt>Flavor</tt>" tab is where you select the VM configuration discussed [[#Instances|above]].<br />
#* You can select a configuration by selecting the up arrow on the right-hand side.<br />
#* Get more details about the configuration by selecting the right-arrow on the left-hand side next to the name.<br />
#* The "<tt>Total Disk</tt>" will show "0 GB" because the disk size will match the [[Images|image]] you selected on the "<tt>Source</tt>" tab.<br />
# For the "<tt>Networks</tt>" tab, two options are available:<br />
## You can make the instance available on the [[Networks#Public Network|public net]]. This is the simplest and most common selection.<br />
## You can select your own [[Networks#Private Networks|private network]], which has to be set up before you launch an instance. For more information, see the [[Networks]] page.<br />
# On the "<tt>Security Groups</tt>" tab, select the [[#Security Groups|security group]] you already created.<br />
# On the "<tt>Key Pairs</tt>" tab, select the [[#Key Pairs|key pair]] you already created.<br />
<br />
=== Instance States ===<br />
<br />
OpenStack defines several [https://developer.openstack.org/api-guide/compute/server_concepts.html#server-status Server States] through which you can move your instances. You change the state of your instance by making a selection from a drop-down menu under the <tt>Actions</tt> column. Three significant actions to know about are "Resize Instance", "Shelve Instance", and "Unshelve Instance"; these are described below.<br />
<br />
Allowed actions&mdash;i.e., the ones that appear in the drop-down menu&mdash;''depend on the current state of the instance''. For example, the "Resize Instance" action is allowed only for instances that are in the Active state. The figure below shows the possible states in OpenStack and the transitions that are allowed in each case.<br />
<br />
[[File:Openstack-server-states.png|thumb|left|700px|Source: OpenStack[https://docs.openstack.org/nova/latest/reference/vm-states.html]]]<br />
<div style="clear: both"></div><br />
<br />
When your instance has been created, the "<tt>Instances</tt>" tab will list its current state (as well as the state of your other instances) under the "<tt>Status</tt>" column. In the rightmost column called "<tt>Actions</tt>," you will see a drop-down menu for each instance. This menu lists the actions that are allowed for the given instance. Below we describe the typical states and list some of the common actions you will use to change instance state.<br />
<br />
==== Important States ====<br />
<br />
'''''Note: The only state where you are NOT being charged for computational resources is Shelved Offloaded'''''<br />
<br />
:* '''Active''': Instance is active, you can connect to it and are being billed for the computational resources dedicated to it.<br />
:* '''Shelved Offloaded''': The Instance is not resident on the compute host; this means you will not be billed for computational resources while the Instance is in this state (although you will be charged for the storage required to hold it). You can restart the server when you need it again.<br />
:* '''Paused''': In this state, the server state is preserved in RAM, but operations have been stopped and will resume when instructed. You are still being charged for the computational resources dedicated to the Instance.<br />
:* '''Suspended''': Instance state has been stored on disk, including the contents of its RAM. With Red Cloud's configuration, you are still paying for the computational resources you were using.<br />
:* '''Stopped''': This is like powering off a server; in this state, you are still being billed for the computational resources.<br />
:* '''Resized''': At this time, the Instance is being Resized to a different flavor&mdash;that is, a different allocation of vCPUs and RAM&mdash;and is not contactable.<br />
<br />
==== Operations to transition between states ====<br />
<br />
<br />
These options are available, subject to the current state of the Instances, from the dropdown available in the "Actions" column of the Instances page. ''Remember that Shelving is the only operation that will free up the computational resources your Instance has been using so that you stop being charged for them!''<br />
<br />
:* '''Pause Instance:''' Put instance into the Paused state.<br />
:* '''Suspend Instance''': Put instance into the Suspended state.<br />
:* '''Shelve Instance''': This is how you put the Instance aside so that you are no longer charged for computational resources being used; your Instance will still be visible on the Instances page with Status Shelved Offloaded. To get the Instance back up and running, select "Unshelve" from the actions menu.<br />
:* '''Resize Instance''': Allows you to select a new Instance flavor. After you have selected it, Status will be shown as "Confirm or Revert Resize/Migrate" you will have to confirm in the Actions dropdown, after which the Instance will be taken down and then come back up with the new computational resources available to it. On Linux you can check with commands such as <code>lscpu</code> or <code>cat /proc/cpuinfo</code>; on Windows you can, for example, use the "Performance" tab of Task Manager to see the available cores.<br />
:* '''Resume Instance''': Will restart the Instance from states of Paused and Suspended<br />
:* '''Soft Reboot Instance''' or '''Hard Reboot Instance''': Reboot your instance, either through issuing a command to the operation system ("Soft") or as if power-cycling the Instance ("Hard")<br />
:* '''Shut Off Instance''': Like powering off the Instance, an action you will need to confirm in a pop-up window. The Status shown will be Shutoff and you will need to "Restart" to get it back. Additionally, if you are logged into an instance and use an OS command such as <code>sudo poweroff</code> or <code>sudo init</code>, the Instances page will show the same status and you will need to select "Restart" to get the Instance back up and running. In this state, ''you are still being charged for computational resources''.</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Getting_Started&diff=3017Getting Started2019-11-14T17:02:22Z<p>Pzv2: /* Connecting to CAC resources */ Put Red Cloud into a section</p>
<hr />
<div>__TOC__<br />
==Managing your password==<br />
CAC has a [[Getting_Started#Rules_for_creating_passwords|password policy]] in effect. The first time that you login to the <tt>cac.cornell.edu</tt> domain, you will be required to change your password. Each password must have at least eight characters and must contain at least three of the following four elements: (1) uppercase letters (2) lowercase letters (3) special characters (4) digits. Your password can be set or changed on any of the CAC login nodes, and the password will be updated on all CAC resources. Passwords expire every six months. Do not share your password. There are more detailed instructions below.<br />
<br />
===Rules for creating passwords===<br />
{{:Rules for Creating Passwords}}<br />
<br />
===Change a password at first login===<br />
{{:Changing password at first login}}<br />
<br />
=== Change password at any time ===<br />
{{:Changing password any time}}<br />
<br />
=== Locked accounts===<br />
<br />
There have been instances in which user accounts have been locked. Some common causes of locked accounts and the solutions are:<br />
<br />
:*Mistyping your password several times in a row. <br />
:::<tt>Solution</tt>: Wait about a 1/2 hour and then try again. Be sure that your caps lock key is not on!<br />
:*Trying to login to a Windows login node by using SSH when you have a new or expired password. <br />
:::<tt>Solution</tt>: Login to a Windows login node using Remote Desktop Connection or SSH to a linux login node.<br />
:*Failing to log off all other sessions connected to login nodes. <br />
:::<tt>Solution</tt>: Log off all remote connections. Disconnecting the sessions is not enough.<br />
:*Failing to disconnect locally mapped drives to the CAC file server before changing your password. <br />
:::<tt>Solution</tt>: Disconnect all locally mapped drives, wait a 1/2 hour until account is unlocked, and then re-map the drive with the new password.<br />
<br />
If you can't log on or can't wait you can submit a Password Reset ticket on our [//www.cac.cornell.edu/help issue tracking system]<br />
<br />
==Checking your CAC project ==<br />
Cornell University users can view their account limits at [https://{{SERVERNAME}}/services/cu/memberlimits.aspx CAC Account Limits].<br /><br />
Partner Program members should contact Paul Redfern at [mailto:red@cac.cornell.edu red@cac.cornell.edu] if they need information on their membership limits.<br />
<br />
==Connecting to CAC resources==<br />
<br />
=== Red Cloud ===<br />
'''Please see the [[Red Cloud]] and [[OpenStack]] pages for information on managing, connecting to, and using Red Cloud resources.'''<br />
<br />
The information on the remainder of this page is primarily for users of CAC-maintained [[Private Clusters]] (for particular research groups) and data storage services, especially CAC's online, non-archival data storage.<br />
<br />
However, it is potentially of interest to users of Red Cloud '''''and''''' other CAC services.<br />
<br />
===CAC login nodes===<br />
<br />
There are three types of login nodes:<br />
:* The head nodes for the various Linux-based private clusters<br />
:* Linux login node: <tt>linuxlogin.cac.cornell.edu</tt> <br />
:* Windows login node: <tt>winlogin.cac.cornell.edu</tt><br />
<br />
The general CAC login nodes, '''''linuxlogin''''' and '''''winlogin''''', are mostly intended for researchers who are have procured CAC storage services, apart from Red Cloud and private clusters (see [[Getting Started#Working with CAC file storage|Working with CAC file storage]]). These two login nodes are broadly accessible from the Internet, and they provide a convenient way for researchers to gain access to their files. On the general login nodes, you will find a modest number of software tools installed to aid in working with files (but please do not use these tools for doing production computing there).<br />
<br />
In what follows, we will often use '''''linuxlogin''''' and '''''winlogin''''' as stand-ins for the particular CAC servers that you are trying to access (e.g., in a private cluster or in Red Cloud).<br />
<br />
===Connect to Linux===<br />
{{:Connect to Linux}}<br />
<br />
===Connect to Windows===<br />
{{:Connect to Windows}}<br />
<br />
==Working with CAC file storage==<br />
A dedicated file server named storage03.cac.cornell.edu provides access to much of CAC's file storage, including the home directories for many of the private clusters. To work with your files, you can access this server in a variety of ways from any operating system. The first two of the following methods are covered in detail in this section:<br />
<br />
:# [[Getting Started#Home directory access|Home directory access]] - Mount/map your portion of the storage03 filesystem as network share or network drive. Once the filesystem is mounted, your files on storage03 appear in a folder that you can access just like other folders on your computer. On Linux, use the <tt>mount</tt> command; on Mac, use "Go > Connect to server"; on Windows (including winlogin), enter the UNC address into the address bar of a File Explorer window, or do "Map a network drive".<br />
:# [[Getting Started#File transfer|File transfer]] - Use a file transfer utility like scp or sftp to copy your files to or from storage03. Connect to linuxlogin to do this, because storage03 is not directly accessible. On linuxlogin, your main CAC folder on storage03 is your home folder when connect via ssh, scp, or sftp.<br />
:# - Use Globus to transfer files to or from storage03. The source or destination of the files must also be a Globus endpoint (and note, you can set up any computer to be a personal endpoint). Endpoints at CAC are described on the [[File Transfer using Globus]] page.<br />
<br />
Note: by default, your home directory on linuxlogin and its contents will be readable and executable by all other users of CAC systems. If this is not what you want, you can change the permissions of the home directory and its files and subdirectories via the standard Linux or Windows mechanisms. However, be aware that this may lead to conflicts for cross-platform applications, as Windows and Linux permissions are not 100% compatible.<br />
<br />
===Home directory access===<br />
{{:Home Directory Access}}<br />
<br />
===File transfer===<br />
{{:File transfer}}<br />
<br />
==Linux usage tips==<br />
If you have never used Linux before, we recommend exploring the [[Linux Tutorial]].<br />
<br />
{{:Linux Usage Tips}}<br />
<br />
==More information==<br />
The CAC main website is [https://{{SERVERNAME}}/ here]. There are many useful documents on the Support page at [[ Main Page| CAC documentation]].<br />
<br />
==Acknowledging CAC==<br />
{{:Acknowledging CAC}}<br />
<br />
==FAQ/Troubleshooting==<br />
#[[FAQ#Account| Account FAQ]]<br />
#[[FAQ#Login| Login FAQ]]<br />
# If you have more questions, see [[FAQ| here]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=FAQ&diff=3016FAQ2019-11-14T16:54:05Z<p>Pzv2: Update Red Cloud links for updated sections</p>
<hr />
<div>=Account=<br />
<br />
====How can I obtain a CAC account?====<br />
See [https://{{SERVERNAME}}/services/projects.aspx How to Start a Project].<br />
<br />
====How can I determine the number of hours I have left before I reach my project limit?====<br />
Check links from the [https://{{SERVERNAME}}/services/projects.aspx CAC Projects page].<br />
<br />
====My account is locked.====<br />
If it was locked after repeated password failures, it should automatically unlock after 30 minutes. Otherwise: {{ContactCAC}}<br />
<br />
====I forgot my password, or have problems with a new password, or need a password reset.====<br />
{{ContactCAC}}<br />
<br />
====Are my login id and password the same for all machines?====<br />
Yes. For an ssh connection give your login id at the prompt. With a Windows GUI, specify the username as CTC_ITH\<login_id> or <login_id>@tc.cornell.edu.<br />
<br />
====When I try to use a Remote Desktop client to connect to winlogin, it tells me that my username/password are incorrect.====<br />
Make sure that you are logging using the CTC_ITH domain. If you just put your username in the "username" box, it will try to log you into winlogin as a local user, which won't work. Put CTC_ITH\<username> in the "username" box.<br />
<br />
<br />
=Files=<br />
<br />
====How can I copy files to my desktop from H:?====<br />
Use SSH client to sftp files. See [[File_Transfer_To_Clusters]].<br />
<br />
====Can't use scp to transfer files to the CAC.====<br />
Use sftp.<br />
<br />
====Problems using WinSCP.====<br />
Use sftp. <br />
<br />
====Needed to share a file with a colleague outside the university. This is typically available on to CAC personnel.====<br />
Showed how to use outgoing ftp folder and sent detailed instructions by email.<br />
<br />
====Can't access files.====<br />
System problem. Send email to consult@tc.cornell.edu.<br />
<br />
====Can see files in explorer, but sees files only in home directory with dir at command prompt.====<br />
User had navigated Start | Run, then typed the command command. Needs to use the command cmd.<br />
<br />
====How Do I Transfer Files To and From CAC Machines?====<br />
# '''Use a program to send them''' - [[SecureShell]]<br />
#* Faster over slower connections.<br />
#* Less hassle.<br />
# '''Make your CAC home directory look like a local drive''' - [[FileAccess]]<br />
#* Works fine on campus.<br />
#* Convenient for editing.<br />
<br />
If you have any questions, please [mailto:help@cac.cornell.edu?subject=CAC Web site contact Send email] or call 607.254.8686.<br />
<br />
====Why use a temporary directory====<br />
'''''It is faster to perform local file I/O and copy complete data files to/from $HOME at the beginning and the end of the job, rather than perform I/O over the network ($HOME is network mounted on the compute nodes).''''' <br />
'''<br />
<br />
* Torque creates a uniquely named directory (/tmp/$PBS_JOBID) when a job starts and stores the path of this directory in the $TMPDIR environment variable. This directory is cleaned up when the job exits.<br />
** To use this feature, reference $TMPDIR<br />
<br />
* You may create directories for file read/writes outside your /tmp/$PBS_JOBID in /tmp. You do risk leaving any data there; it may be deleted at any time we see /tmp getting full.<br />
<br />
= Red Cloud =<br />
<br />
== Getting Started ==<br />
<br />
[[Red_Cloud#New_Users|New to Red Cloud]]? The best way to get started is to read the documentation and try things out. Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
=== Suggested Reading ===<br />
<br />
# [[Red Cloud]] - includes information about:<br />
#* The [[Red_Cloud#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[Red_Cloud#How_to_Create_and_Manage_Red_Cloud_Resources|Create/Manage resources]]<br />
#* [[Red_Cloud#How_to_Access_Instances|Accessing instances]]<br />
#* [[Red_Cloud#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# [[OpenStack]] - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for [[Red_Cloud_Linux_Instances|Linux Instances]] OR [[Red_Cloud_Windows_Instances|Windows Instances]]<br />
<br />
=== Other Useful References ===<br />
<br />
:* [[Linux Tutorial]]<br />
:* [[Resizing an Instance|Resizing your instance]]<br />
:* [[OpenStack Key Pairs| Key Pairs]]<br />
:* [[OpenStack Security Groups| Security Groups]]<br />
:* [[Volumes]]<br />
:* [[Images]]<br />
:* [[Networks]]<br />
:* [[OpenStack CLI]]<br />
<br />
== Connecting to Instances ==<br />
<br />
First, ensure that the [[OpenStack#Instances|instance]] has finished being created by checking in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]] or the [[OpenStack CLI]]. Next, refer to the available documentation for [[Red_Cloud_Linux_Instances#Accessing_Instances|accessing Linux instances]] or [[Red_Cloud_Windows_Instances#Accessing_Instances|accessing Windows instances]]. If you are having trouble connecting to your instance, please review this documentation first to ensure you're following the correct steps. If you have created a [[Red_Cloud_Linux_Instances|Linux instance]] and are having trouble connecting via <code>ssh</code>, try the [[Red_Cloud_Linux_Instances#Troubleshooting|troubleshooting steps]]. If you are still having trouble, {{ContactCAC}}.<br />
<br />
=Linux Batch=<br />
==Scheduler Frequently Asked Questions==<br />
{{ContactCAC}}<br />
====Why are you using Maui and Torque now?====<br />
We have switched to using a nationally recognized resource manager and scheduler in order to make the usage of our systems align more closely with the national community. This also allows us to leverage the considerable capabilities of the Maui software to ensure optimal and flexible use of our systems.<br />
====When's my job going to run?====<br />
If you have already submitted your job and you'd like to know that, use the '''showstart''' command to find estimated start times. If you are trying to decide where to run your job so that it runs the soonest, you'll want to examine the '''showbf''' command. This allows you to search for when a job with particular resource requirements will run.<br />
====Why is my job stuck in the queue?====<br />
Sometimes your job doesn't run, even though it looks like it should. Maybe there are few jobs running in the cluster, and your job still won't run.<br />
# Find your jobids with "showq -u username"<br />
# Use "checkjob -v jobid" to examine one of the jobs. [[Examining Checkjob -v]] discusses how to read this output.<br />
Jobs in the "Batch Hold" state initiate emails to the system administrators. For other problems, contact CAC help.<br />
====Why is my job deferred?====<br />
There can be several reasons for a job to defer. Sometimes when the Maui scheduler's queue is full, two jobs attempt to start on a node at the same time, and one will switch to being deferred. On this occasion, if you type "checkjob -v <jobid>", you will see, at the bottom, the message:<br />
Message[0] job rejected by RM 'scheduler' - job started on hostlist<br />
compute-3-40.v4linux,compute-3-37.v4linux,compute-3-35.v4linux,compute-3-34.v4linux<br />
at time 13:11:22_07/20, job reported idle at time 13:11:53_07/20 (see RM logs for details)<br />
In this case, the only way to make this job run is to notify help at CAC.<br />
====What are the queues/affiliations?====<br />
Affiliations was the term used by the vsched scheduler to indicate the name of the queue that jobs were submitted to. Most schedulers use the term queue (The scheduler also uses the term "class" to represent the same entity), so you can substitute the word you prefer. V4 queues are listed on the [[v4 Linux Cluster]] page.<br />
<br />
====When I try to run mpdboot I get an error regarding bad python version====<br />
This type of message goes on to say, "You can't run mpdboot on ['compute-3-44.v4linux'] version of python must be >= 2.4, current..." Mpdboot uses python and ssh to start MPI daemons on all nodes of your job. It begins by using ssh to ask what version of python is running on each node.<br />
<br />
Usually, this error means that ssh is having a problem establishing communication for the<br />
mpds. First, make sure you added "-r ssh" to your mpdboot line. If that<br />
looks OK, then try to rename (mv) the .ssh directory in your home directory<br />
to something like .ssh_bak. Log out, and log back in. A new .ssh<br />
directory should be recreated for you automatically (you can verify with<br />
"ls -la") which should have valid keys in it.<br />
<br />
You may also get this error if you are using a version of Python which does not work with mpdboot. In general, mpdboot needs python 2.3 or newer, but it gets very picky about versions newer than 2.4, as well. If you are trying to run Python 2.5 or 2.6 from your own directory, sometimes mpdboot will find only older versions when it does ssh to the other nodes in your job (because a non-interactive ssh can have a different path). One way to ensure mpdboot runs properly in this case is to ensure it uses the system copy of python. In bash, you can set the path for a command before you invoke it, here so that the system Python is used.<br />
PATH=/usr/bin:/bin:/opt/intel/impi/3.1/bin64/ mpdboot ...<br />
<br />
====What variables does PBS define in the job script?====<br />
Some of the variables are listed in [http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/commands/qsub.php qsub documentation] but a good way to see the working environment is to submit a batch job which just does "env>variables.txt" and look for the ones starting in "PBS_".<br />
<br />
====No Job Control Warning for CSH and TCSH====<br />
The output file from the script starts with the error:<br />
Warning: no access to tty (Bad file descriptor).<br />
Thus no job control in this shell.<br />
This warning means that the <tt>fg</tt>, <tt>bg</tt>, and ampersand will not work in your script files. If your default user shell is csh or tcsh, the job will try to execute your script using csh or tcsh, and you'll get this warning. Bash doesn't have this problem.<br />
<br />
You can force your script to start with the Bash shell using a PBS directive:<br />
#PBS -S /bin/sh<br />
When Torque starts your job, it will now use Bash, but it won't actually call your .bashrc. If you have any startup files to modify the path or set other variables, you can add to the start of your script, after the PBS directives:<br />
source ~/.bashrc<br />
<br />
Another nice way to ensure your favorite variables are defined is to submit the script with the -V option:<br />
nsub -V batch.sh<br />
This option copied whatever environment variables you have defined on the command line to the script when it runs. In short, if you could run something interactively, it should run when the scheduler executes the job.<br />
<br />
====Mpiexec Won't Accept -ppn Argument====<br />
The default MPI, Intel MPI, requires that you put the -ppn argument before the -np argument.<br />
The nodes have at least three versions of mpiexec installed. The default is Intel MPI under /opt/intel. If you modify your shell's path, in .bashrc or .cshrc, to put /usr/local/bin before the default path, then you may be getting the [http://www.osc.edu/~pw/mpiexec/ OSC mpiexec]. This version does not depend on mpdboot. It talks directly with Torque to start jobs. A drawback is that the OSC mpiexec, on our system, cannot start more than one job per node. That's why it's not the default one to use.<br />
<br />
====I cannot find my output file====<br />
If you do not specify an output file when submitting a batch script, then it will automatically produce a file with a name like 110432.scheduler.v4linux.OU in the directory which was the working directory when you submitted your job. If you specify an output file with a command like "#PBS -o out.txt", then that file will be in your $HOME directory. This behavior has changed in recent versions of the scheduler.<br />
<br />
<br />
{{Template:ContactCAC}}<br />
<br />
=Microsoft Visual Studio=<br />
====Has CAC installed Visual Studio and the Intel compilers on winlogin?====<br />
No, not at the present time. This section of the FAQ pertains to Red Cloud users who have installed this software.<br />
<br />
====Where is nmake?====<br />
C:\Program Files\Microsoft Visual Studio\VC98\bin\nmake. Call setup_visualc.bat <br />
<br />
====How can you find the cl compiler?====<br />
Call setup_visualc.bat<br />
<br />
====Can't find uuid.lib.====<br />
It's in C:\Program Files\Microsoft SDK\lib.<br />
<br />
====LINK fatal error LNK1201: error writing to program database H:\users\...\some.pdb; check for insufficient disk space, invalid path, or insufficient privilege.====<br />
Suspicion is that there is an older version of the file some.pdb. Delete that file and rebuild.<br />
<br />
====How do I use Intel Fortran at the command line?====<br />
First, call setup_intelf32.bat. The compilation command is ifort.<br />
<br />
====Fortran program gives an access violation. What to do? forrtl: severe (157): Program Exception - access violation====<br />
Segmentation fault. Look for a place where you are writing past the end of an array. <br />
<br />
====Fortran program gives stack overflow. What to do? forrtl: severe (170): Program Exception - stack overflow====<br />
Increase the space available on the stack with the flag /F, where is the size of the stack in bytes. The default is 1000000. Try /F10000000. Increase as necessary.<br />
<br />
====What is the command line syntax to compile a Fortran code with OpenMP?====<br />
See the info provided by "ifort -h". There are 4 options beginning with /Qopenmp.<br />
<br />
====Fortran program gives convergence errors when compiled with with /O1, /O2, /O3.====<br />
Add /Op flag to enable better floating point precision.<br />
<br />
====For a Fortran code, how do I set up debugging, either for the Release version in VS or at a command prompt?====<br />
Let's say you would like to debug an optimized Intel Fortran code, created either as a Release version in Visual Studio (VS) or at a command prompt with /O2. A Debug version in VS sets the correct debugging flags, but disables optimization. Add the command-line flags /Zi /debug:full /traceback to the Release version. Specify the linker option /pdbfile:filename.pdb to create the program database file. This file and the executable must be copied into the same directory when you run the program.<br />
<br />
==== Can the Intel C compiler handle makefile dependencies without having to use cygwin's makedepend?====<br />
Yes. You can use the /QMM compiler option, which is OFF by default.<br />
* /QM - Generates makefile dependency lines for each source file, based on the #include lines found in the source file.<br />
* /QMD - Preprocess and compile. Generate output file (.d extension) containing dependency information.<br />
* /QMF file - Generate makefile dependency information in file. Must specify /QM or /QMM.<br />
* /QMG - Similar to /QM, but treats missing header files as generated files.<br />
* /QMM - Similar to /QM, but does not include system header files.<br />
* /QMMD - Similar to /QMD, but does not include system header files.</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud&diff=3015Red Cloud2019-11-14T16:48:33Z<p>Pzv2: /* How to Access Instances */</p>
<hr />
<div>This wiki provides documentation for [https://{{SERVERNAME}}/redcloud Red Cloud], an on-demand research [https://en.wikipedia.org/wiki/Cloud_computing cloud computing] service maintained and supported by the [https://www.cac.cornell.edu/ CAC]. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on [[OpenStack]].<br />
<br />
Instructions on these pages apply to users who have a [https://www.cac.cornell.edu/services/projects.aspx Red Cloud subscription] they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.<br />
<br />
__TOC__<br />
<br />
== How To Read This Documentation ==<br />
<br />
:* '''Exploratory Account Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed below to help you get started managing Red Cloud resources.<br />
:** Pay particular attention to the [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]] section on this page, as your exploratory project ends when you have exhausted your subscription.<br />
:** An important point to remember is that you are ''not'' the PI on your account, so you can ignore any instructions targeting PIs on a project.<br />
:* '''New Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed to help you get started managing Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
:* '''Returning Users'''<br />
:** Check out the section dedicated to [[#All Users|all users]].<br />
:** You may also want to look through the [[#Important Pages|Important Pages]] listed to help you manage Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
<br />
'''Note:''' All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.<br />
<br />
=== Important Pages ===<br />
<br />
Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
# This page - includes information about:<br />
#* The [[#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[#How_to_Access_Red_Cloud_Resources|Accessing resources]]<br />
#* [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# [[OpenStack]] - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for [[Red_Cloud_Linux_Instances|Linux Instances]] OR [[Red_Cloud_Windows_Instances|Windows Instances]]<br />
#* There are special instructions if you intend to use [[MATLAB Parallel Server in Red Cloud]]<br />
#* There is also a [[Linux Tutorial]] for those new to Linux system administration, or if you want a refresher<br />
<br />
== New Users ==<br />
<br />
New users would be best served by reading this complete page first, then reading through the pages listed in the [[#Important Pages|Important Pages]] section. New users are also encouraged to explore the [[Getting Started]] page, which includes a lot more general information on using CAC resources beyond Red Cloud.<br />
<br />
'''Note for new Linux users:''' As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system '''must be familiar with Linux system administration'''. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a [[Linux Tutorial]] that should work for RedHat/CentOS and Ubuntu Linux systems. [https://{{SERVERNAME}}/services/ Consulting] is also available to answer general questions about systems administration, or for help on specific software and research problems.<br />
<br />
=== First Time Login === <br />
<br />
When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must '''change the automatically generated password immediately''' for security reasons and to access computing resources. Refer to the instructions for [[Getting_Started#Managing_your_password|managing your password]] as needed.<br />
<br />
If you are a PI or a PI's proxy for a new project, verify that you have added a subscription to your project; see the [https://www.cac.cornell.edu/Services/projects/manage.aspx Manage Projects] page. After waiting up to an hour for account information to propagate, you will then be ready to download the [[OpenStack]] credentials and start managing Red Cloud resources.<br />
<br />
=== How to Create and Manage Red Cloud Resources ===<br />
<br />
Red Cloud is a private research cloud with an '''OpenStack''' backend. Interacting with OpenStack is how resources can be managed. In this case, resources can refer to [[OpenStack#Instances|instances]] (or [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines]), [[Images|images]], and [[Volumes|volumes]]. There are two ways to interact with OpenStack:<br />
<br />
:* '''The OpenStack Web Interface (Horizon)'''<br />
:** Go to the [//redcloud.cac.cornell.edu OpenStack Web Interface]<br />
:** For a walk-through, see the [[OpenStack]] page<br />
:* '''The Command-Line Interface (CLI) called the OpenStack CLI'''<br />
:** Linux command-line tools provided by [[OpenStack]]<br />
:** For a walk-through, see the [[OpenStack CLI]] page<br />
:** Also see the [https://docs.openstack.org/python-openstackclient/pike/ official OpenStack CLI documentation]<br />
<br />
'''Note:''' Regardless which method you choose (Web Interface or Command Line Interface), you must follow the [[#First Time Login | First Time Login]] instructions.<br />
<br />
=== How to Access Instances ===<br />
<br />
Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:<br />
:* [[Red Cloud Linux Instances | Linux Instances]] - especially the [[Red_Cloud_Linux_Instances#Accessing_Instances|accessing instances]] section (also see [[Red_Cloud_Linux_Instances#Troubleshooting|troubleshooting]] if needed)<br />
:* [[Red Cloud Windows Instances | Windows Instances]] - especially the [[Red_Cloud_Windows_Instances#Accessing_Instances|accessing instances]] section<br />
<br />
=== Accounting: Don't Use Up Your Subscription by Accident! ===<br />
<br />
To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to [[OpenStack#Instance_States|control the state]] of system [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines], such as start, pause, suspend, shelve, and delete (see [[OpenStack#Instance_States|Instance States]] for a full list). Since starting a virtual machine allocates memory and CPU resources on a physical machine to that virtual machine,''' subscriptions are billed based on the length of time a virtual machine is running, even if it is idle and doing NO work for the user'''. This is fair because your running [[OpenStack#Instances|instance]] will prevent others from using the hardware, even if the hardware is idle.<br />
<br />
Thus, '''the best way to avoid using up your subscription''' needlessly is to make sure you [[OpenStack#Instance_States|'''''shelve''''']] your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]]. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.<br />
<br />
Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various [[OpenStack#Instances|instance sizes]] available. For example, it is usually best to choose a small instance type to do your development work.<br />
<br />
It is worth pointing out that Red Cloud allows the [[Resizing an Instance| instance type]] to be changed if the virtual machine is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we '''recommend''' that you start with the smallest instance size you can to install software and get used to your instance ''before'' [[Resizing an Instance|resizing your instance]] to the full size you would like.<br />
<br />
Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in [[OpenStack]]), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.<br />
<br />
All of the above is true for [[Red Cloud Linux Instances | Linux instances]] and [[Red Cloud Windows Instances | Windows instances]]; note that Cornell users do not need to pay for a [[Red Cloud Windows Instances#Windows_Activation|Windows license]] in Red Cloud.<br />
<br />
We recommend you check your balance frequently using pages provided for [https://{{SERVERNAME}}/services/cu/Memberlimits.aspx Cornell]<br />
or<br />
[https://{{SERVERNAME}}/services/external/Memberlimits.aspx external]<br />
users.<br />
<br />
== All Users ==<br />
<br />
Please refer to the [[OpenStack]] page for more in-depth guidance on how to use Red Cloud, and read either [[Red Cloud Linux Instances | Linux instances]] or [[Red Cloud Windows Instances | Windows instances]] based on what systems will be used. <br />
<br />
The current [https://www.cac.cornell.edu/RedCloud/status/ Red Cloud System Status] can be checked anytime.<br />
<br />
=== Common Tasks ===<br />
<br />
Here are some links to help you with particular aspects of using Red Cloud: <br />
:* [[Linux Tutorial]] - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.<br />
:* Information on choosing [[Instance Types | instance type]] (the size of the virtual machine). [[Resizing volumes]]<!-- this likely needs to be a new page --> is a separate issue, and is somewhat more involved.<br />
:* An example of [[Installing R| Installing R]], a commonly used software package.<br />
:* [//it.cornell.edu/services/ezbackup/ EZ-backup] - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.<br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
== FAQ ==<br />
<br />
:* [[FAQ#Red_Cloud| Red Cloud FAQ]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud&diff=3014Red Cloud2019-11-14T16:47:26Z<p>Pzv2: /* New Users */ Slight restructure for clarification: Changed "Access" section to "Create and Manage" because it was inaccurate, and created separate</p>
<hr />
<div>This wiki provides documentation for [https://{{SERVERNAME}}/redcloud Red Cloud], an on-demand research [https://en.wikipedia.org/wiki/Cloud_computing cloud computing] service maintained and supported by the [https://www.cac.cornell.edu/ CAC]. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on [[OpenStack]].<br />
<br />
Instructions on these pages apply to users who have a [https://www.cac.cornell.edu/services/projects.aspx Red Cloud subscription] they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.<br />
<br />
__TOC__<br />
<br />
== How To Read This Documentation ==<br />
<br />
:* '''Exploratory Account Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed below to help you get started managing Red Cloud resources.<br />
:** Pay particular attention to the [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]] section on this page, as your exploratory project ends when you have exhausted your subscription.<br />
:** An important point to remember is that you are ''not'' the PI on your account, so you can ignore any instructions targeting PIs on a project.<br />
:* '''New Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed to help you get started managing Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
:* '''Returning Users'''<br />
:** Check out the section dedicated to [[#All Users|all users]].<br />
:** You may also want to look through the [[#Important Pages|Important Pages]] listed to help you manage Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
<br />
'''Note:''' All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.<br />
<br />
=== Important Pages ===<br />
<br />
Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
# This page - includes information about:<br />
#* The [[#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[#How_to_Access_Red_Cloud_Resources|Accessing resources]]<br />
#* [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# [[OpenStack]] - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for [[Red_Cloud_Linux_Instances|Linux Instances]] OR [[Red_Cloud_Windows_Instances|Windows Instances]]<br />
#* There are special instructions if you intend to use [[MATLAB Parallel Server in Red Cloud]]<br />
#* There is also a [[Linux Tutorial]] for those new to Linux system administration, or if you want a refresher<br />
<br />
== New Users ==<br />
<br />
New users would be best served by reading this complete page first, then reading through the pages listed in the [[#Important Pages|Important Pages]] section. New users are also encouraged to explore the [[Getting Started]] page, which includes a lot more general information on using CAC resources beyond Red Cloud.<br />
<br />
'''Note for new Linux users:''' As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system '''must be familiar with Linux system administration'''. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a [[Linux Tutorial]] that should work for RedHat/CentOS and Ubuntu Linux systems. [https://{{SERVERNAME}}/services/ Consulting] is also available to answer general questions about systems administration, or for help on specific software and research problems.<br />
<br />
=== First Time Login === <br />
<br />
When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must '''change the automatically generated password immediately''' for security reasons and to access computing resources. Refer to the instructions for [[Getting_Started#Managing_your_password|managing your password]] as needed.<br />
<br />
If you are a PI or a PI's proxy for a new project, verify that you have added a subscription to your project; see the [https://www.cac.cornell.edu/Services/projects/manage.aspx Manage Projects] page. After waiting up to an hour for account information to propagate, you will then be ready to download the [[OpenStack]] credentials and start managing Red Cloud resources.<br />
<br />
=== How to Create and Manage Red Cloud Resources ===<br />
<br />
Red Cloud is a private research cloud with an '''OpenStack''' backend. Interacting with OpenStack is how resources can be managed. In this case, resources can refer to [[OpenStack#Instances|instances]] (or [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines]), [[Images|images]], and [[Volumes|volumes]]. There are two ways to interact with OpenStack:<br />
<br />
:* '''The OpenStack Web Interface (Horizon)'''<br />
:** Go to the [//redcloud.cac.cornell.edu OpenStack Web Interface]<br />
:** For a walk-through, see the [[OpenStack]] page<br />
:* '''The Command-Line Interface (CLI) called the OpenStack CLI'''<br />
:** Linux command-line tools provided by [[OpenStack]]<br />
:** For a walk-through, see the [[OpenStack CLI]] page<br />
:** Also see the [https://docs.openstack.org/python-openstackclient/pike/ official OpenStack CLI documentation]<br />
<br />
'''Note:''' Regardless which method you choose (Web Interface or Command Line Interface), you must follow the [[#First Time Login | First Time Login]] instructions.<br />
<br />
=== How to Access Instances ===<br />
<br />
Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:<br />
:* [[Red Cloud Linux Instances | Linux Instances]] - especially the [[Red_Cloud_Linux_Instances#Accessing_Instances|accessing instances]] section (also see [[Red_Cloud_Linux_Instances#Troubleshooting|troubleshooting]] if needed)<br />
:* [[Red Cloud Windows Instances | Windows Instances]] - especially the [[Red_Cloud_Windows_Instances#Accessing_Instances]] section<br />
<br />
=== Accounting: Don't Use Up Your Subscription by Accident! ===<br />
<br />
To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to [[OpenStack#Instance_States|control the state]] of system [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines], such as start, pause, suspend, shelve, and delete (see [[OpenStack#Instance_States|Instance States]] for a full list). Since starting a virtual machine allocates memory and CPU resources on a physical machine to that virtual machine,''' subscriptions are billed based on the length of time a virtual machine is running, even if it is idle and doing NO work for the user'''. This is fair because your running [[OpenStack#Instances|instance]] will prevent others from using the hardware, even if the hardware is idle.<br />
<br />
Thus, '''the best way to avoid using up your subscription''' needlessly is to make sure you [[OpenStack#Instance_States|'''''shelve''''']] your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]]. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.<br />
<br />
Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various [[OpenStack#Instances|instance sizes]] available. For example, it is usually best to choose a small instance type to do your development work.<br />
<br />
It is worth pointing out that Red Cloud allows the [[Resizing an Instance| instance type]] to be changed if the virtual machine is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we '''recommend''' that you start with the smallest instance size you can to install software and get used to your instance ''before'' [[Resizing an Instance|resizing your instance]] to the full size you would like.<br />
<br />
Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in [[OpenStack]]), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.<br />
<br />
All of the above is true for [[Red Cloud Linux Instances | Linux instances]] and [[Red Cloud Windows Instances | Windows instances]]; note that Cornell users do not need to pay for a [[Red Cloud Windows Instances#Windows_Activation|Windows license]] in Red Cloud.<br />
<br />
We recommend you check your balance frequently using pages provided for [https://{{SERVERNAME}}/services/cu/Memberlimits.aspx Cornell]<br />
or<br />
[https://{{SERVERNAME}}/services/external/Memberlimits.aspx external]<br />
users.<br />
<br />
== All Users ==<br />
<br />
Please refer to the [[OpenStack]] page for more in-depth guidance on how to use Red Cloud, and read either [[Red Cloud Linux Instances | Linux instances]] or [[Red Cloud Windows Instances | Windows instances]] based on what systems will be used. <br />
<br />
The current [https://www.cac.cornell.edu/RedCloud/status/ Red Cloud System Status] can be checked anytime.<br />
<br />
=== Common Tasks ===<br />
<br />
Here are some links to help you with particular aspects of using Red Cloud: <br />
:* [[Linux Tutorial]] - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.<br />
:* Information on choosing [[Instance Types | instance type]] (the size of the virtual machine). [[Resizing volumes]]<!-- this likely needs to be a new page --> is a separate issue, and is somewhat more involved.<br />
:* An example of [[Installing R| Installing R]], a commonly used software package.<br />
:* [//it.cornell.edu/services/ezbackup/ EZ-backup] - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.<br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
== FAQ ==<br />
<br />
:* [[FAQ#Red_Cloud| Red Cloud FAQ]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud&diff=3013Red Cloud2019-11-14T00:03:10Z<p>Pzv2: /* Important Pages */ Added Linux Tutorial</p>
<hr />
<div>This wiki provides documentation for [https://{{SERVERNAME}}/redcloud Red Cloud], an on-demand research [https://en.wikipedia.org/wiki/Cloud_computing cloud computing] service maintained and supported by the [https://www.cac.cornell.edu/ CAC]. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on [[OpenStack]].<br />
<br />
Instructions on these pages apply to users who have a [https://www.cac.cornell.edu/services/projects.aspx Red Cloud subscription] they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.<br />
<br />
__TOC__<br />
<br />
== How To Read This Documentation ==<br />
<br />
:* '''Exploratory Account Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed below to help you get started managing Red Cloud resources.<br />
:** Pay particular attention to the [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]] section on this page, as your exploratory project ends when you have exhausted your subscription.<br />
:** An important point to remember is that you are ''not'' the PI on your account, so you can ignore any instructions targeting PIs on a project.<br />
:* '''New Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed to help you get started managing Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
:* '''Returning Users'''<br />
:** Check out the section dedicated to [[#All Users|all users]].<br />
:** You may also want to look through the [[#Important Pages|Important Pages]] listed to help you manage Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
<br />
'''Note:''' All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.<br />
<br />
=== Important Pages ===<br />
<br />
Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
# This page - includes information about:<br />
#* The [[#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[#How_to_Access_Red_Cloud_Resources|Accessing resources]]<br />
#* [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# [[OpenStack]] - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for [[Red_Cloud_Linux_Instances|Linux Instances]] OR [[Red_Cloud_Windows_Instances|Windows Instances]]<br />
#* There are special instructions if you intend to use [[MATLAB Parallel Server in Red Cloud]]<br />
#* There is also a [[Linux Tutorial]] for those new to Linux system administration, or if you want a refresher<br />
<br />
== New Users ==<br />
<br />
New users would be best served by reading this complete page first, then reading through the pages listed in the [[#Important Pages|Important Pages]] section. New users are also encouraged to explore the [[Getting Started]] page, which includes a lot more general information on using CAC resources beyond Red Cloud.<br />
<br />
'''Note for new Linux users:''' As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system '''must be familiar with Linux system administration'''. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a [[Linux Tutorial]] that should work for RedHat/CentOS and Ubuntu Linux systems. [https://{{SERVERNAME}}/services/ Consulting] is also available to answer general questions about systems administration, or for help on specific software and research problems.<br />
<br />
=== First Time Login === <br />
<br />
When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must '''change the automatically generated password immediately''' for security reasons and to access computing resources. Refer to the instructions for [[Getting_Started#Managing_your_password|managing your password]] as needed.<br />
<br />
If you are a PI or a PI's proxy for a new project, verify that you have added a subscription to your project; see the [https://www.cac.cornell.edu/Services/projects/manage.aspx Manage Projects] page. After waiting up to an hour for account information to propagate, you will then be ready to download the [[OpenStack]] credentials and start managing Red Cloud resources.<br />
<br />
=== How to Access Red Cloud Resources ===<br />
<br />
Red Cloud is a private research cloud with an '''OpenStack''' backend. Interacting with OpenStack is how resources can be managed. In this case, resources can refer to [[OpenStack#Instances|instances]] (or [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines]), [[Images|images]], and [[Volumes|volumes]]. There are two ways to interact with OpenStack:<br />
<br />
:* '''The OpenStack Web Interface (Horizon)'''<br />
:** Go to the [//redcloud.cac.cornell.edu OpenStack Web Interface]<br />
:** For a walk-through, see the [[OpenStack]] page<br />
:* '''The Command-Line Interface (CLI) called the OpenStack CLI'''<br />
:** Linux command-line tools provided by [[OpenStack]]<br />
:** For a walk-through, see the [[OpenStack CLI]] page<br />
:** Also see the [https://docs.openstack.org/python-openstackclient/pike/ official OpenStack CLI documentation]<br />
<br />
'''Note:''' Regardless which method you choose (Web Interface or Command Line Interface), you must follow the [[#First Time Login | First Time Login]] instructions.<br />
<br />
Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:<br />
:* [[Red Cloud Linux Instances | Linux Instances]]<br />
:* [[Red Cloud Windows Instances | Windows Instances]]<br />
<br />
=== Accounting: Don't Use Up Your Subscription by Accident! ===<br />
<br />
To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to [[OpenStack#Instance_States|control the state]] of system [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines], such as start, pause, suspend, shelve, and delete (see [[OpenStack#Instance_States|Instance States]] for a full list). Since starting a virtual machine allocates memory and CPU resources on a physical machine to that virtual machine,''' subscriptions are billed based on the length of time a virtual machine is running, even if it is idle and doing NO work for the user'''. This is fair because your running [[OpenStack#Instances|instance]] will prevent others from using the hardware, even if the hardware is idle.<br />
<br />
Thus, '''the best way to avoid using up your subscription''' needlessly is to make sure you [[OpenStack#Instance_States|'''''shelve''''']] your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]]. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.<br />
<br />
Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various [[OpenStack#Instances|instance sizes]] available. For example, it is usually best to choose a small instance type to do your development work.<br />
<br />
It is worth pointing out that Red Cloud allows the [[Resizing an Instance| instance type]] to be changed if the virtual machine is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we '''recommend''' that you start with the smallest instance size you can to install software and get used to your instance ''before'' [[Resizing an Instance|resizing your instance]] to the full size you would like.<br />
<br />
Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in [[OpenStack]]), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.<br />
<br />
All of the above is true for [[Red Cloud Linux Instances | Linux instances]] and [[Red Cloud Windows Instances | Windows instances]]; note that Cornell users do not need to pay for a [[Red Cloud Windows Instances#Windows_Activation|Windows license]] in Red Cloud.<br />
<br />
We recommend you check your balance frequently using pages provided for [https://{{SERVERNAME}}/services/cu/Memberlimits.aspx Cornell]<br />
or<br />
[https://{{SERVERNAME}}/services/external/Memberlimits.aspx external]<br />
users.<br />
<br />
== All Users ==<br />
<br />
Please refer to the [[OpenStack]] page for more in-depth guidance on how to use Red Cloud, and read either [[Red Cloud Linux Instances | Linux instances]] or [[Red Cloud Windows Instances | Windows instances]] based on what systems will be used. <br />
<br />
The current [https://www.cac.cornell.edu/RedCloud/status/ Red Cloud System Status] can be checked anytime.<br />
<br />
=== Common Tasks ===<br />
<br />
Here are some links to help you with particular aspects of using Red Cloud: <br />
:* [[Linux Tutorial]] - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.<br />
:* Information on choosing [[Instance Types | instance type]] (the size of the virtual machine). [[Resizing volumes]]<!-- this likely needs to be a new page --> is a separate issue, and is somewhat more involved.<br />
:* An example of [[Installing R| Installing R]], a commonly used software package.<br />
:* [//it.cornell.edu/services/ezbackup/ EZ-backup] - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.<br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
== FAQ ==<br />
<br />
:* [[FAQ#Red_Cloud| Red Cloud FAQ]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Red_Cloud&diff=3012Red Cloud2019-11-13T23:46:42Z<p>Pzv2: /* Accounting: Don't Use Up Your Subscription by Accident! */ updating links to renamed page</p>
<hr />
<div>This wiki provides documentation for [https://{{SERVERNAME}}/redcloud Red Cloud], an on-demand research [https://en.wikipedia.org/wiki/Cloud_computing cloud computing] service maintained and supported by the [https://www.cac.cornell.edu/ CAC]. At present, Red Cloud is an Infrastructure as a Service (IaaS) based on [[OpenStack]].<br />
<br />
Instructions on these pages apply to users who have a [https://www.cac.cornell.edu/services/projects.aspx Red Cloud subscription] they are managing, though some instructions may also apply to users of subscriptions managed by someone else. Individuals who manage a Red Cloud subscription can create, administer, and delete virtual servers and storage in Red Cloud.<br />
<br />
__TOC__<br />
<br />
== How To Read This Documentation ==<br />
<br />
:* '''Exploratory Account Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed below to help you get started managing Red Cloud resources.<br />
:** Pay particular attention to the [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]] section on this page, as your exploratory project ends when you have exhausted your subscription.<br />
:** An important point to remember is that you are ''not'' the PI on your account, so you can ignore any instructions targeting PIs on a project.<br />
:* '''New Users'''<br />
:** Read through all the sections on this page targeting New Users.<br />
:** Look through the [[#Important Pages|Important Pages]] listed to help you get started managing Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
:* '''Returning Users'''<br />
:** Check out the section dedicated to [[#All Users|all users]].<br />
:** You may also want to look through the [[#Important Pages|Important Pages]] listed to help you manage Red Cloud resources.<br />
:** If you are ''not'' the PI on your account, you can ignore any instructions targeting PIs.<br />
<br />
'''Note:''' All links on our wiki are colored red and underlined. Each of the external links will have dotted underlining and an icon next to them that looks like an arrow pointing out of a box, whereas internal links have a solid underline and do not have any icon. This can help you navigate by knowing that the external links are not part of our documentation or "how to" instructions.<br />
<br />
=== Important Pages ===<br />
<br />
Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
# This page - includes information about:<br />
#* The [[#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[#How_to_Access_Red_Cloud_Resources|Accessing resources]]<br />
#* [[#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# [[OpenStack]] - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for [[Red_Cloud_Linux_Instances|Linux Instances]] OR [[Red_Cloud_Windows_Instances|Windows Instances]]<br />
#* There are special instructions if you intend to use [[MATLAB Parallel Server in Red Cloud]]<br />
<br />
== New Users ==<br />
<br />
New users would be best served by reading this complete page first, then reading through the pages listed in the [[#Important Pages|Important Pages]] section. New users are also encouraged to explore the [[Getting Started]] page, which includes a lot more general information on using CAC resources beyond Red Cloud.<br />
<br />
'''Note for new Linux users:''' As the root user, you will have complete control over access to the system, such as setting up users and their permissions, defining the firewall, and more. This means that the primary user of a Linux system '''must be familiar with Linux system administration'''. Aside from the basics of using the command line, this includes familiarity with: creating and modifying users, installing software, configuring software for remote logins, and managing/transferring data. For users that want to use Red Cloud, but do not have much system administration experience, we've written a [[Linux Tutorial]] that should work for RedHat/CentOS and Ubuntu Linux systems. [https://{{SERVERNAME}}/services/ Consulting] is also available to answer general questions about systems administration, or for help on specific software and research problems.<br />
<br />
=== First Time Login === <br />
<br />
When you are added to a CAC project, you will receive an e-mail confirming your Red Cloud access. You must '''change the automatically generated password immediately''' for security reasons and to access computing resources. Refer to the instructions for [[Getting_Started#Managing_your_password|managing your password]] as needed.<br />
<br />
If you are a PI or a PI's proxy for a new project, verify that you have added a subscription to your project; see the [https://www.cac.cornell.edu/Services/projects/manage.aspx Manage Projects] page. After waiting up to an hour for account information to propagate, you will then be ready to download the [[OpenStack]] credentials and start managing Red Cloud resources.<br />
<br />
=== How to Access Red Cloud Resources ===<br />
<br />
Red Cloud is a private research cloud with an '''OpenStack''' backend. Interacting with OpenStack is how resources can be managed. In this case, resources can refer to [[OpenStack#Instances|instances]] (or [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines]), [[Images|images]], and [[Volumes|volumes]]. There are two ways to interact with OpenStack:<br />
<br />
:* '''The OpenStack Web Interface (Horizon)'''<br />
:** Go to the [//redcloud.cac.cornell.edu OpenStack Web Interface]<br />
:** For a walk-through, see the [[OpenStack]] page<br />
:* '''The Command-Line Interface (CLI) called the OpenStack CLI'''<br />
:** Linux command-line tools provided by [[OpenStack]]<br />
:** For a walk-through, see the [[OpenStack CLI]] page<br />
:** Also see the [https://docs.openstack.org/python-openstackclient/pike/ official OpenStack CLI documentation]<br />
<br />
'''Note:''' Regardless which method you choose (Web Interface or Command Line Interface), you must follow the [[#First Time Login | First Time Login]] instructions.<br />
<br />
Depending on which operating system you are planning on running on your instances, you should also refer to one of the following pages:<br />
:* [[Red Cloud Linux Instances | Linux Instances]]<br />
:* [[Red Cloud Windows Instances | Windows Instances]]<br />
<br />
=== Accounting: Don't Use Up Your Subscription by Accident! ===<br />
<br />
To understand how billing works, it is necessary to understand a bit about how Red Cloud operates. Red Cloud enables the user to [[OpenStack#Instance_States|control the state]] of system [//en.wikipedia.org/wiki/Virtual_machine#Definitions virtual machines], such as start, pause, suspend, shelve, and delete (see [[OpenStack#Instance_States|Instance States]] for a full list). Since starting a virtual machine allocates memory and CPU resources on a physical machine to that virtual machine,''' subscriptions are billed based on the length of time a virtual machine is running, even if it is idle and doing NO work for the user'''. This is fair because your running [[OpenStack#Instances|instance]] will prevent others from using the hardware, even if the hardware is idle.<br />
<br />
Thus, '''the best way to avoid using up your subscription''' needlessly is to make sure you [[OpenStack#Instance_States|'''''shelve''''']] your Red Cloud instance any time you are not using it. It is very simple to do this via the menu in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]]. You can always start the instance again later, and the disk contents will be unchanged. It is just like shutting down your laptop.<br />
<br />
Whenever you have one or more instances that are up and running, the amount that is deducted from your Red Cloud subscription is: the length of time that your instances are running, multiplied by the number of cores that you are occupying with those instances. This implies that you should also take advantage of the various [[OpenStack#Instances|instance sizes]] available. For example, it is usually best to choose a small instance type to do your development work.<br />
<br />
It is worth pointing out that Red Cloud allows the [[Resizing an Instance| instance type]] to be changed if the virtual machine is stopped (i.e. shut down). This allows you to "scale up" an instance at any time by stopping it, choosing a larger size for it, and starting it back up. You can shrink an instance in the same way. If you intend to use a large instance, we '''recommend''' that you start with the smallest instance size you can to install software and get used to your instance ''before'' [[Resizing an Instance|resizing your instance]] to the full size you would like.<br />
<br />
Here are a couple of motivating examples for you. Let's say you have an exploratory account, with just 165 core hours to start. If you leave a 1-core node running around the clock, you will use up the entire account in a little less than a week. Similarly, let's say you are on a CAC project with a Red Cloud subscription (8,585 core hours). If you start up an instance with 4 cores (sometimes called CPUs in [[OpenStack]]), and you leave the instance running for a week, or 168 hours, you will use up (168 hours)*(4 cores) or 672 core hours, or 8% of the subscription.<br />
<br />
All of the above is true for [[Red Cloud Linux Instances | Linux instances]] and [[Red Cloud Windows Instances | Windows instances]]; note that Cornell users do not need to pay for a [[Red Cloud Windows Instances#Windows_Activation|Windows license]] in Red Cloud.<br />
<br />
We recommend you check your balance frequently using pages provided for [https://{{SERVERNAME}}/services/cu/Memberlimits.aspx Cornell]<br />
or<br />
[https://{{SERVERNAME}}/services/external/Memberlimits.aspx external]<br />
users.<br />
<br />
== All Users ==<br />
<br />
Please refer to the [[OpenStack]] page for more in-depth guidance on how to use Red Cloud, and read either [[Red Cloud Linux Instances | Linux instances]] or [[Red Cloud Windows Instances | Windows instances]] based on what systems will be used. <br />
<br />
The current [https://www.cac.cornell.edu/RedCloud/status/ Red Cloud System Status] can be checked anytime.<br />
<br />
=== Common Tasks ===<br />
<br />
Here are some links to help you with particular aspects of using Red Cloud: <br />
:* [[Linux Tutorial]] - This may help you get up and running with some basic systems administration tasks. It is not intended to be comprehensive.<br />
:* Information on choosing [[Instance Types | instance type]] (the size of the virtual machine). [[Resizing volumes]]<!-- this likely needs to be a new page --> is a separate issue, and is somewhat more involved.<br />
:* An example of [[Installing R| Installing R]], a commonly used software package.<br />
:* [//it.cornell.edu/services/ezbackup/ EZ-backup] - a CIT solution for backups. Data stored on Red Cloud is not backed up by default; users are responsible for their own backups.<br />
<br />
===Acknowledging CAC===<br />
{{:Acknowledging CAC}}<br />
<br />
== FAQ ==<br />
<br />
:* [[FAQ#Red_Cloud| Red Cloud FAQ]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=FAQ&diff=3011FAQ2019-11-13T23:45:20Z<p>Pzv2: /* Other Useful References */ Update link to match new page name</p>
<hr />
<div>=Account=<br />
<br />
====How can I obtain a CAC account?====<br />
See [https://{{SERVERNAME}}/services/projects.aspx How to Start a Project].<br />
<br />
====How can I determine the number of hours I have left before I reach my project limit?====<br />
Check links from the [https://{{SERVERNAME}}/services/projects.aspx CAC Projects page].<br />
<br />
====My account is locked.====<br />
If it was locked after repeated password failures, it should automatically unlock after 30 minutes. Otherwise: {{ContactCAC}}<br />
<br />
====I forgot my password, or have problems with a new password, or need a password reset.====<br />
{{ContactCAC}}<br />
<br />
====Are my login id and password the same for all machines?====<br />
Yes. For an ssh connection give your login id at the prompt. With a Windows GUI, specify the username as CTC_ITH\<login_id> or <login_id>@tc.cornell.edu.<br />
<br />
====When I try to use a Remote Desktop client to connect to winlogin, it tells me that my username/password are incorrect.====<br />
Make sure that you are logging using the CTC_ITH domain. If you just put your username in the "username" box, it will try to log you into winlogin as a local user, which won't work. Put CTC_ITH\<username> in the "username" box.<br />
<br />
<br />
=Files=<br />
<br />
====How can I copy files to my desktop from H:?====<br />
Use SSH client to sftp files. See [[File_Transfer_To_Clusters]].<br />
<br />
====Can't use scp to transfer files to the CAC.====<br />
Use sftp.<br />
<br />
====Problems using WinSCP.====<br />
Use sftp. <br />
<br />
====Needed to share a file with a colleague outside the university. This is typically available on to CAC personnel.====<br />
Showed how to use outgoing ftp folder and sent detailed instructions by email.<br />
<br />
====Can't access files.====<br />
System problem. Send email to consult@tc.cornell.edu.<br />
<br />
====Can see files in explorer, but sees files only in home directory with dir at command prompt.====<br />
User had navigated Start | Run, then typed the command command. Needs to use the command cmd.<br />
<br />
====How Do I Transfer Files To and From CAC Machines?====<br />
# '''Use a program to send them''' - [[SecureShell]]<br />
#* Faster over slower connections.<br />
#* Less hassle.<br />
# '''Make your CAC home directory look like a local drive''' - [[FileAccess]]<br />
#* Works fine on campus.<br />
#* Convenient for editing.<br />
<br />
If you have any questions, please [mailto:help@cac.cornell.edu?subject=CAC Web site contact Send email] or call 607.254.8686.<br />
<br />
====Why use a temporary directory====<br />
'''''It is faster to perform local file I/O and copy complete data files to/from $HOME at the beginning and the end of the job, rather than perform I/O over the network ($HOME is network mounted on the compute nodes).''''' <br />
'''<br />
<br />
* Torque creates a uniquely named directory (/tmp/$PBS_JOBID) when a job starts and stores the path of this directory in the $TMPDIR environment variable. This directory is cleaned up when the job exits.<br />
** To use this feature, reference $TMPDIR<br />
<br />
* You may create directories for file read/writes outside your /tmp/$PBS_JOBID in /tmp. You do risk leaving any data there; it may be deleted at any time we see /tmp getting full.<br />
<br />
= Red Cloud =<br />
<br />
== Getting Started ==<br />
<br />
[[Red_Cloud#New_Users|New to Red Cloud]]? The best way to get started is to read the documentation and try things out. Here is a suggested list of pages to look over to help with getting started managing resources.<br />
<br />
=== Suggested Reading ===<br />
<br />
# [[Red Cloud]] - includes information about:<br />
#* The [[Red_Cloud#First_Time_Login|first time you login]] to your [https://www.cac.cornell.edu/services/myacct.aspx CAC Account]<br />
#* [[Red_Cloud#How_to_Access_Red_Cloud_Resources|Accessing resources]]<br />
#* [[Red_Cloud#Accounting:_Don.27t_Use_Up_Your_Subscription_by_Accident.21|Accounting]]<br />
# [[OpenStack]] - a '''highly recommended''' quick-start page including instructions for:<br />
#* The [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|Web interface]]<br />
#* [[OpenStack#Instances|Managing instances]] including:<br />
#** [[OpenStack#Launching an Instance|launching a new instance]]<br />
#** [[OpenStack#Instance_States|changing instance state]]<br />
# Either instructions for [[Red_Cloud_Linux_Instances|Linux Instances]] OR [[Red_Cloud_Windows_Instances|Windows Instances]]<br />
<br />
=== Other Useful References ===<br />
<br />
:* [[Linux Tutorial]]<br />
:* [[Resizing an Instance|Resizing your instance]]<br />
:* [[OpenStack Key Pairs| Key Pairs]]<br />
:* [[OpenStack Security Groups| Security Groups]]<br />
:* [[Volumes]]<br />
:* [[Images]]<br />
:* [[Networks]]<br />
:* [[OpenStack CLI]]<br />
<br />
== Connecting to Instances ==<br />
<br />
First, ensure that the [[OpenStack#Instances|instance]] has finished being created by checking in the [[OpenStack#Using_the_OpenStack_Web_Interface_.28Horizon.29|OpenStack Web Interface]] or the [[OpenStack CLI]]. Next, refer to the available documentation for [[Red_Cloud_Linux_Instances#Accessing_Instances|accessing Linux instances]] or [[Red_Cloud_Windows_Instances#Accessing_Instances|accessing Windows instances]]. If you are having trouble connecting to your instance, please review this documentation first to ensure you're following the correct steps. If you have created a [[Red_Cloud_Linux_Instances|Linux instance]] and are having trouble connecting via <code>ssh</code>, try the [[Red_Cloud_Linux_Instances#Troubleshooting|troubleshooting steps]]. If you are still having trouble, {{ContactCAC}}.<br />
<br />
=Linux Batch=<br />
==Scheduler Frequently Asked Questions==<br />
{{ContactCAC}}<br />
====Why are you using Maui and Torque now?====<br />
We have switched to using a nationally recognized resource manager and scheduler in order to make the usage of our systems align more closely with the national community. This also allows us to leverage the considerable capabilities of the Maui software to ensure optimal and flexible use of our systems.<br />
====When's my job going to run?====<br />
If you have already submitted your job and you'd like to know that, use the '''showstart''' command to find estimated start times. If you are trying to decide where to run your job so that it runs the soonest, you'll want to examine the '''showbf''' command. This allows you to search for when a job with particular resource requirements will run.<br />
====Why is my job stuck in the queue?====<br />
Sometimes your job doesn't run, even though it looks like it should. Maybe there are few jobs running in the cluster, and your job still won't run.<br />
# Find your jobids with "showq -u username"<br />
# Use "checkjob -v jobid" to examine one of the jobs. [[Examining Checkjob -v]] discusses how to read this output.<br />
Jobs in the "Batch Hold" state initiate emails to the system administrators. For other problems, contact CAC help.<br />
====Why is my job deferred?====<br />
There can be several reasons for a job to defer. Sometimes when the Maui scheduler's queue is full, two jobs attempt to start on a node at the same time, and one will switch to being deferred. On this occasion, if you type "checkjob -v <jobid>", you will see, at the bottom, the message:<br />
Message[0] job rejected by RM 'scheduler' - job started on hostlist<br />
compute-3-40.v4linux,compute-3-37.v4linux,compute-3-35.v4linux,compute-3-34.v4linux<br />
at time 13:11:22_07/20, job reported idle at time 13:11:53_07/20 (see RM logs for details)<br />
In this case, the only way to make this job run is to notify help at CAC.<br />
====What are the queues/affiliations?====<br />
Affiliations was the term used by the vsched scheduler to indicate the name of the queue that jobs were submitted to. Most schedulers use the term queue (The scheduler also uses the term "class" to represent the same entity), so you can substitute the word you prefer. V4 queues are listed on the [[v4 Linux Cluster]] page.<br />
<br />
====When I try to run mpdboot I get an error regarding bad python version====<br />
This type of message goes on to say, "You can't run mpdboot on ['compute-3-44.v4linux'] version of python must be >= 2.4, current..." Mpdboot uses python and ssh to start MPI daemons on all nodes of your job. It begins by using ssh to ask what version of python is running on each node.<br />
<br />
Usually, this error means that ssh is having a problem establishing communication for the<br />
mpds. First, make sure you added "-r ssh" to your mpdboot line. If that<br />
looks OK, then try to rename (mv) the .ssh directory in your home directory<br />
to something like .ssh_bak. Log out, and log back in. A new .ssh<br />
directory should be recreated for you automatically (you can verify with<br />
"ls -la") which should have valid keys in it.<br />
<br />
You may also get this error if you are using a version of Python which does not work with mpdboot. In general, mpdboot needs python 2.3 or newer, but it gets very picky about versions newer than 2.4, as well. If you are trying to run Python 2.5 or 2.6 from your own directory, sometimes mpdboot will find only older versions when it does ssh to the other nodes in your job (because a non-interactive ssh can have a different path). One way to ensure mpdboot runs properly in this case is to ensure it uses the system copy of python. In bash, you can set the path for a command before you invoke it, here so that the system Python is used.<br />
PATH=/usr/bin:/bin:/opt/intel/impi/3.1/bin64/ mpdboot ...<br />
<br />
====What variables does PBS define in the job script?====<br />
Some of the variables are listed in [http://www.adaptivecomputing.com/resources/docs/torque/2-5-9/commands/qsub.php qsub documentation] but a good way to see the working environment is to submit a batch job which just does "env>variables.txt" and look for the ones starting in "PBS_".<br />
<br />
====No Job Control Warning for CSH and TCSH====<br />
The output file from the script starts with the error:<br />
Warning: no access to tty (Bad file descriptor).<br />
Thus no job control in this shell.<br />
This warning means that the <tt>fg</tt>, <tt>bg</tt>, and ampersand will not work in your script files. If your default user shell is csh or tcsh, the job will try to execute your script using csh or tcsh, and you'll get this warning. Bash doesn't have this problem.<br />
<br />
You can force your script to start with the Bash shell using a PBS directive:<br />
#PBS -S /bin/sh<br />
When Torque starts your job, it will now use Bash, but it won't actually call your .bashrc. If you have any startup files to modify the path or set other variables, you can add to the start of your script, after the PBS directives:<br />
source ~/.bashrc<br />
<br />
Another nice way to ensure your favorite variables are defined is to submit the script with the -V option:<br />
nsub -V batch.sh<br />
This option copied whatever environment variables you have defined on the command line to the script when it runs. In short, if you could run something interactively, it should run when the scheduler executes the job.<br />
<br />
====Mpiexec Won't Accept -ppn Argument====<br />
The default MPI, Intel MPI, requires that you put the -ppn argument before the -np argument.<br />
The nodes have at least three versions of mpiexec installed. The default is Intel MPI under /opt/intel. If you modify your shell's path, in .bashrc or .cshrc, to put /usr/local/bin before the default path, then you may be getting the [http://www.osc.edu/~pw/mpiexec/ OSC mpiexec]. This version does not depend on mpdboot. It talks directly with Torque to start jobs. A drawback is that the OSC mpiexec, on our system, cannot start more than one job per node. That's why it's not the default one to use.<br />
<br />
====I cannot find my output file====<br />
If you do not specify an output file when submitting a batch script, then it will automatically produce a file with a name like 110432.scheduler.v4linux.OU in the directory which was the working directory when you submitted your job. If you specify an output file with a command like "#PBS -o out.txt", then that file will be in your $HOME directory. This behavior has changed in recent versions of the scheduler.<br />
<br />
<br />
{{Template:ContactCAC}}<br />
<br />
=Microsoft Visual Studio=<br />
====Has CAC installed Visual Studio and the Intel compilers on winlogin?====<br />
No, not at the present time. This section of the FAQ pertains to Red Cloud users who have installed this software.<br />
<br />
====Where is nmake?====<br />
C:\Program Files\Microsoft Visual Studio\VC98\bin\nmake. Call setup_visualc.bat <br />
<br />
====How can you find the cl compiler?====<br />
Call setup_visualc.bat<br />
<br />
====Can't find uuid.lib.====<br />
It's in C:\Program Files\Microsoft SDK\lib.<br />
<br />
====LINK fatal error LNK1201: error writing to program database H:\users\...\some.pdb; check for insufficient disk space, invalid path, or insufficient privilege.====<br />
Suspicion is that there is an older version of the file some.pdb. Delete that file and rebuild.<br />
<br />
====How do I use Intel Fortran at the command line?====<br />
First, call setup_intelf32.bat. The compilation command is ifort.<br />
<br />
====Fortran program gives an access violation. What to do? forrtl: severe (157): Program Exception - access violation====<br />
Segmentation fault. Look for a place where you are writing past the end of an array. <br />
<br />
====Fortran program gives stack overflow. What to do? forrtl: severe (170): Program Exception - stack overflow====<br />
Increase the space available on the stack with the flag /F, where is the size of the stack in bytes. The default is 1000000. Try /F10000000. Increase as necessary.<br />
<br />
====What is the command line syntax to compile a Fortran code with OpenMP?====<br />
See the info provided by "ifort -h". There are 4 options beginning with /Qopenmp.<br />
<br />
====Fortran program gives convergence errors when compiled with with /O1, /O2, /O3.====<br />
Add /Op flag to enable better floating point precision.<br />
<br />
====For a Fortran code, how do I set up debugging, either for the Release version in VS or at a command prompt?====<br />
Let's say you would like to debug an optimized Intel Fortran code, created either as a Release version in Visual Studio (VS) or at a command prompt with /O2. A Debug version in VS sets the correct debugging flags, but disables optimization. Add the command-line flags /Zi /debug:full /traceback to the Release version. Specify the linker option /pdbfile:filename.pdb to create the program database file. This file and the executable must be copied into the same directory when you run the program.<br />
<br />
==== Can the Intel C compiler handle makefile dependencies without having to use cygwin's makedepend?====<br />
Yes. You can use the /QMM compiler option, which is OFF by default.<br />
* /QM - Generates makefile dependency lines for each source file, based on the #include lines found in the source file.<br />
* /QMD - Preprocess and compile. Generate output file (.d extension) containing dependency information.<br />
* /QMF file - Generate makefile dependency information in file. Must specify /QM or /QMM.<br />
* /QMG - Similar to /QM, but treats missing header files as generated files.<br />
* /QMM - Similar to /QM, but does not include system header files.<br />
* /QMMD - Similar to /QMD, but does not include system header files.</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Instance_Types&diff=3010Instance Types2019-11-13T23:43:58Z<p>Pzv2: Pzv2 moved page Instance Types to Resizing an Instance: More descriptive of page contents</p>
<hr />
<div>#REDIRECT [[Resizing an Instance]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Resizing_an_Instance&diff=3009Resizing an Instance2019-11-13T23:43:57Z<p>Pzv2: Pzv2 moved page Instance Types to Resizing an Instance: More descriptive of page contents</p>
<hr />
<div>A variety of [[OpenStack#Instances|instance sizes]] are available on [[Red Cloud]]. The instance size (or type) defines how much memory (RAM) is available, the amount of per-instance storage (typically available through /dev/vdb in [[Red Cloud Linux Instances | Linux instances]]), and the number of CPU cores available.<br />
<br />
Resizing an instance allows you to dynamically control your work process. During periods of heavy development, you may only want a small instance type to develop on, but during periods of heavy computational activity, a large instance (or multiple large instances) may be desirable.<br />
<br />
The options can be found by clicking on the Resize Instance option in the menu for the instance on the right side of the instances listing page:<br />
<br />
[[File:White_square.png|100px|frameless]][[File:Resize Instance Menu.png|150px|frameless|border]]<br />
<br />
This option is only available for instances whose [[OpenStack#Instance States|state]] is either active or stopped. The dialog for resizing the instance type is shown below:<br />
<br />
[[File:Resize Instance Dialog.png|500px|frameless|border]]<br />
<br />
Simply select the new flavor you would like and then select "Resize".</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=MATLAB_Parallel_Server_in_Red_Cloud&diff=3008MATLAB Parallel Server in Red Cloud2019-11-04T18:41:11Z<p>Pzv2: Link Eucalyptus -> OpenStack</p>
<hr />
<div>== What You Need to Know About This Service ==<br />
'''MATLAB Parallel Server''' (formerly termed MATLAB Distributed Computing Server, or MDCS, in releases prior to R2019a) must be used in conjunction with the '''Parallel Computing Toolbox (PCT)''' in your MATLAB client. Accordingly, it is necessary to own and be familiar with PCT. Gaining PCT knowledge is to your advantage, though, because it is the best way for MATLAB to make effective use of multiple CPUs on any system--even the multiple cores in your laptop. One starting point for learning PCT is [[Tutorial:_Using_MATLAB_PCT_and_Parallel_Server_in_Red_Cloud | CAC's tutorial]]. Extending PCT's basic concepts to Red Cloud should be natural and easy. Furthermore, if you are a member of the Cornell community, you probably already have PCT, as it is included in Cornell's normal site license. <br />
<br />
[[Red_Cloud | Red Cloud]] offers the following advantages for your PCT computations.<br />
*Up to 64 parallel workers are available in total.<br />
*Licenses for the workers are included in your subscription.<br />
*Workers have exclusive access to their allocated cores and memory.<br />
*Data are readily transferred through the campus network at no extra cost.<br />
<br />
==== Assumptions ====<br />
#You are a member of an academic community with access to an academic MATLAB '''R2017a''' or '''R2019a''' client.<br />
#Your group has [https://www.cac.cornell.edu/services/projects.aspx started a CAC project] giving you access to [[Red_Cloud | Red Cloud]], and you are familiar with its [[OpenStack]] console.<br />
#The MATLAB '''R2017a''' or '''R2019a''' client, including the [http://www.mathworks.com/products/parallel-computing/ Parallel Computing Toolbox] (PCT), is installed on your local workstation.<br />
#[[Red_Cloud#First_Time_Login | First time Red Cloud login]] has been completed.<br />
#[[OpenStack#Key_Pairs | Create a Red Cloud key pair]] has been completed.<br />
<br />
== Create a Security Group ==<br />
In the [https://redcloud.cac.cornell.edu Red Hat OpenStack Platform console], you will need to create a Security Group for your MATLAB instances. Its purpose is to open up certain TCP ports so your client has proper access to your MATLAB Parallel Server(s). <br />
<br />
Choose "Networks > Security Groups", then "Create Security Group". Then add the Rules below, following the instructions on the [[OpenStack Security Groups]] page. Where a port range is required, be sure to use the "Open Port" drop-down menu to change "Port" to "Port Range".<br />
<br />
{| border="1" cellspacing="0" cellpadding="5" align="center" style="text-align:left;"<br />
! Rule<br />
! Port Range <br />
! CIDR<br />
|-<br />
| Custom TCP Rule || 27350 - 28000 || <your client IP address>/32<br />
|-<br />
| SSH (''for terminal access, file transfers'') || N/A (''will be port 22'') || <your client IP address>/32<br />
|}<br />
<br />
You must click "Add" at the bottom of the pop-up after making each new entry. You must also click "Create Security Group" at the end, after adding all rules.<br />
<br />
As indicated in the table, the most secure CIDR of all is <your client's exact IP address>/32. But your laptop's address (e.g.) can be tricky to determine if you are on the campus Wi-Fi, or if you are connecting via VPN from off campus. Therefore, CAC recommends allowing connections from a broader set of addresses representing the 3 main types of internal networks at Cornell. All of this is explained on the [[OpenStack Security Groups]] page. You will need to repeat the steps for each CIDR (i.e., set of allowed source addresses) that you want to include.<br />
<br />
If you wish, you can keep adding rules to the security group until all possible Cornell IP addresses are covered. Here is the full list of CIDR entries, starting with the 3 main ones:<br />
* 10.0.0.0/8, <-- this one is sufficient for VPN and on-campus Wi-Fi<br />
* 128.84.0.0/16,<br />
* 128.253.0.0/16,<br />
* 132.236.0.0/16,<br />
* 192.35.82.0/24,<br />
* 192.122.235.0/24, and<br />
* 192.122.236.0/24<br />
<br />
If all of the above are included, then access is permitted from anywhere on Cornell network ([https://it.cornell.edu/dns/ip-addresses-and-subnets-cornell#section-6 reference 1], [https://it.cornell.edu/dns/what-10-space-and-what-does-it-do#section-1 reference 2]). However, you should be aware that in this case, any Cornell user who has knowledge of the IP address(es) of your Red Cloud instance(s) and the MJS name (see below) will be able to submit MATLAB PCT jobs to your instance(s).<br />
<br />
Note, these same port ranges have to be open on the client side, too. For example, if you have Windows Firewall enabled, you will need to set up special rules that allow inbound TCP and UDP connections to MATLAB through/from any port (a typical client-side firewall will not have outbound restrictions). You may need to consult MathWorks documentation to see what to do in your particular case.<br />
<br />
== Start the MATLAB Cluster ==<br />
Currently only one-node clusters are supported. However, a single node can support many workers, up to the total number of cores that you assigned to your instance. (Multi-node clusters are also possible, and they could be supported, if a demand for them arises.)<br />
<br />
In the [https://redcloud.cac.cornell.edu Red Hat OpenStack Platform console for Red Cloud]:<br />
<br />
# From the Compute > Instances tab, click on the "Launch Instance" button.<br />
# Give your instance an easy-to-remember name; click "Next".<br />
# For the ''Source'', select the image matching your Matlab client version:<br />
#* Scroll to matlab-201xx where xx matches your client and click its up-arrow, OR<br />
#* Type matlab-201xx in the filter field to find the right image and click the up-arrow.<br />
#* Increase "Volume Size (GB)" if you think you will be uploading large data files to the instance.<br />
#* Say "Yes" to "Delete Volume on Instance Delete" if you don't want a long-term backup of any customizations or data.<br />
#* When you're done making your selections, click "Next".<br />
# For the ''Flavor'', select the desired instance type. '''One MATLAB worker will be started per CPU in the instance.''' Click "Next".<br />
# In ''Networks'', select the "public" network by clicking its up-arrow, and click "Next".<br />
# No ''Network Ports'' need to be added, so click "Next".<br />
# In ''Security Groups'', find the MATLAB security group that you created. Click its up-arrow, then "Next".<br />
# ''Key Pair'' is the only other setting that is recommended.<br />
#* Assigning a key pair lets you access your instance via ssh, which can be handy for troubleshooting.<br />
#* It also lets you move files back and forth to your instance using sftp or scp.<br />
# Click on "Launch Instance".<br />
#* After the instance is running, the MATLAB cluster should be reachable at the public IP address of the instance within a minute or two.<br />
<br />
Note: after you finish using your MATLAB cluster, remember to Shelve or Delete the instance to stop charges against your Red Cloud subscription.<br />
<br />
== Connect to Your MATLAB Cluster ==<br />
<br />
Perform the following steps in your local MATLAB client:<br />
<br />
* Open Parallel > Manage Cluster Profiles.<br />
* Choose Add > Custom > MATLAB Job Scheduler (MJS).<br />
* In the warning dialog that comes up, click "OK".<br />
* In the lower right corner of the scheduler, click "Edit".<br />
* Enter values for at least the top three values in Properties:<br />
*# Description: '''Red Cloud''' or another name of your choosing.<br />
*# Host: '''128.84.8.XXX''' (where XXX matches the public address of your Red Cloud instance)<br />
*# Name: '''EC2_job_manager'''<br />
* ''Optional:'' if you want each worker to have more than the standard allotment of memory or disk per core, scroll down and set NumWorkersRange to have a maximum value which is less than the number of cores in your cluster. (In that case, you may also choose to set NumThreads > 1.)<br />
* Click "Done". Click "Rename" in the toolbar to give a new scheduler a better name. This name will appear in your MATLAB client.<br />
* Click "Validate" in the toolbar to ensure the scheduler is configured properly. As each stage completes, a green circle with a check mark in it should be displayed.<br />
<br />
==== Possible validation issues ====<br />
<br />
Validation may fail for a number of reasons. Here is a short list of things to try if it does:<br />
<br />
# If the first validation stage fails, it is most likely because nothing in your list of Security Groups is allowing access from your client's IP address.<br />
#* Log into the [https://redcloud.cac.cornell.edu console] and go to Network > Security Groups.<br />
#* Choose "Manage Rules" for the Security Group you created, then add one or more of the above rules.<br />
#* Wait a minute until the new rules take effect in your running instance (you may also want to restart your MATLAB client).<br />
#* Run the validation test again to ensure your MATLAB cluster passes all the stages.<br />
# If the client is able to connect to the cluster, but the second stage of validation fails, check the results ("Show Details").<br />
#* If you see an error message saying, "This MATLAB Job Scheduler does not support running jobs for MATLAB release...", this just means that the workers are not yet ready.<br />
#* Wait a few more minutes and re-try validation.<br />
# If you still cannot pass validation, and error messages such as "Could not contact an MJS lookup service on host..." persist, it means your network connection is being blocked.<br />
#* Double-check your Security Group and firewall settings as described above.<br />
#* Then contact your departmental IT support, as there may be port blocking in effect on departmental routers. (Campus Wi-Fi connections should be sufficiently open.)<br />
<br />
== Test Your MATLAB Cluster ==<br />
<br />
Finally, you can run this quick "Hello, world" test from the command line in your client.<br />
In the first line, supply the name of your scheduler. If you did not rename the scheduler when you created it, its name appears in the Cluster Profiles Manager dialog.<br />
<br />
pool = parpool('Red Cloud')<br />
spmd; fprintf('Hello from lab %d of %d', labindex, numlabs); end<br />
delete(pool)<br />
<br />
The number of replies should equal the number of workers ("labs"), which by default is equal to the number of cores in your instance. Note that the labs are numbered starting from 1, not 0.<br />
<br />
== Upload Large Files to Your MATLAB Cluster ==<br />
<br />
MATLAB PCT provides built-in mechanisms for uploading data files so they can be accessed by your MATLAB Parallel Server's workers. The primary ones are the AttachedFiles keyword in functions like parpool() and createJob(), and the addAttachedFiles() function for an existing parallel pool. Unfortunately, these mechanisms are not suitable for large files, because they generate a separate copy of the file for each worker. This is inefficient and unnecessary in Red Cloud, where in most cases, all the workers share a file system on the same instance. Here we present two alternatives that should help you to make data files available to your MATLAB Parallel Server's workers.<br />
<br />
Prerequisites:<br />
* You must have [[OpenStack#Key_Pairs | created a Red Cloud key pair]] before starting your instance, and you must have specified this key pair when the instance was launched.<br />
* You should also be familiar with how public key authentication works in Linux.<br />
* Finally, in order to connect to the instance using ssh, sftp, or scp, the Red Cloud security group should include a rule to allow incoming connections to port 22 from the address of the computer that is trying to connect.<br />
<br />
==== Alternative 1: Upload to /tmp on your instance ====<br />
<br />
This method is probably the simpler of the two. Any files you upload will persist on your instance until you terminate it. The only tricky part is knowing how to authenticate with the key pair when you connect to your instance with a file transfer client. It is straightforward to do this type of authentication from the command line in Linux or MacOS, if you use either sftp or scp:<br />
<br />
sftp -i ~/.ssh/myCACid-key.pem root@128.84.8.NNN<br />
sftp> put file.txt /tmp<br />
<br />
scp -i ~/.ssh/myCACid-key.pem file.txt root@128.84.8.NNN:/tmp<br />
<br />
The above examples assume you have stored the the key pair (or at least the private-key portion of it) in your local .ssh folder in Linux or MacOS. If sftp or scp does not accept the -i option in your OS, you can try using ssh-agent and ssh-add to make the private key available to these commands.<br />
<br />
In Windows, the PuTTY client comes with a psftp client that you might want to try. (First you'll have to use the PuTTYgen application to import your .pem file and save it as a .ppk private key.) In the Windows cmd environment, the syntax for psftp would look something like this:<br />
<br />
C:\Users\srl6>"C:\Program Files (x86)\PuTTY\psftp.exe" -i C:\Users\srl6\SSHkeys\srl6-key.ppk root@128.84.8.NNN<br />
<br />
For exceptionally large files, you can make use of your instance's ephemeral storage, which is located at /dev/vdb. You will need to format it and create a mount point for it. The volume persists only as long as the instance is running, but it is large (100 GB minimum) and fast (local RAID 5).<br />
<br />
==== Alternative 2: Upload to your CAC home folder ====<br />
<br />
Your Red Cloud subscription comes with 50GB of storage, part of which can be used to store data files in your home folder at CAC. On Cornell networks, your home folder is available as a network share located at //linuxlogin.cac.cornell.edu/myCACid, where myCACid is your CAC username. (More storage can be added to your subscription if desired.) To upload files to your home folder, use your favorite file transfer client such as WinSCP, or a command-line utility such as sftp or scp. Point your file transfer client or utility to the above address, making sure to provide your CAC username and password.<br />
<br />
But this CAC home folder is not automatically available to your Red Cloud instances. The preferred way to make it accessible is to mount the network share using Samba/CIFS. First log in to your instance as root, which you do with your private key:<br />
<br />
ssh -i ~/.ssh/myCACid-key.pem root@128.84.8.NNN<br />
<br />
The above example again assumes you have stored the the key pair in your .ssh folder in Linux or MacOS. In Windows, you may wish to use PuTTY as the ssh client (in which case you will have to generate a .ppk file from the .pem file using PuTTYgen.) After you are logged in, issue the following commands:<br />
<br />
yum install cifs-utils<br />
mkdir /home/<myCACid><br />
mount -t cifs //storage03.cac.cornell.edu/<myCACid> /home/<myCACid> -o user=<myCACid>,domain=CTC_ITH,vers=2.1<br />
<supply your CAC password when prompted><br />
<br />
At this point all files in your home folder should be available to all MATLAB workers, via a path starting with /home/myCACid/.<br />
<br />
If you stop this Red Cloud instance and start it back up, the mount command will have to be executed anew. To make the Samba mount automatic during restarts, add an appropriate entry to /etc/fstab in the instance.<br />
<br />
==== Fast examples of file I/O ====<br />
<br />
''Example 1.'' Let's say you have copied a file, file.txt, to /tmp on your instance by using scp as described in Alternative 1 above. Let's also suppose this file contains 3 lines (or any arbitrary number) with 1 integer per line. If you'd like to have all your MATLAB workers read this file into vector b and print b to the MATLAB console, you can do the following:<br />
<br />
spmd; fid=fopen('/tmp/file.txt'); b=fscanf(fid,'%d'); disp(b); end<br />
<br />
Vector b is now available in the workspace of all the workers, where it can be used for further parallel computations. Note: from your MATLAB client, you can also use spmd in combination with system(), pwd, etc., in order to explore the environment of your MATLAB workers in Red Cloud. (Or you can just use ssh to take a look around.)<br />
<br />
''Example 2.'' Now let's say file.txt is located in your CAC home folder, which you have mounted on your instance as shown in Alternative 2 above. If the following MATLAB function is assigned to a task in a parallel job, then a parallel worker in Red Cloud will read file.txt and return its contents:<br />
<br />
function b = echofile()<br />
fid=fopen('/home/<myCACid>/file.txt');<br />
b=fscanf(fid,'%d');<br />
end<br />
<br />
Assuming the above function is saved into a local file named echofile.m, you can enter following commands in your MATLAB client to run echofile() on your cluster in Red Cloud, then fetch and display the contents of file.txt:<br />
<br />
clust = parcluster()<br />
job = createJob(clust)<br />
task = createTask(job,@echofile,1,{})<br />
submit(job)<br />
wait(job)<br />
bvals = fetchOutputs(job)<br />
bvals{1}<br />
<br />
Again, the echofile() function is just one of many ways that you can imagine interacting with the files in your CAC home folder, using either MATLAB built-in commands or shell commands invoked through system().</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Linux_Tutorial&diff=3007Linux Tutorial2019-11-04T18:37:23Z<p>Pzv2: </p>
<hr />
<div>This tutorial is intended as a basic introduction to [https://en.wikipedia.org/wiki/Linux Linux] for users of CAC's Linux resources, especially those who are managing [[Red_Cloud|Red Cloud]] services and are creating or using a [[Red Cloud Linux Instances|Linux Instance]]. There are two [https://en.wikipedia.org/wiki/Linux_distribution Linux distributions] (AKA distros) available for [[Images|images]] on Red Cloud: [https://en.wikipedia.org/wiki/Ubuntu_(operating_system) Ubuntu] and [https://en.wikipedia.org/wiki/CentOS CentOS]. In this tutorial, you will learn how to add a user, install software using the distribution's [https://en.wikipedia.org/wiki/Package_manager package manager], and enable remote password logins, along with a number of related tips. The tutorial begins with [[Linux_Tutorial#Basic_Useful_Commands|common commands]] between both distributions, followed by sections for the specific commands you are likely to need on [[Linux_Tutorial#Ubuntu|Ubuntu]] and [[Linux_Tutorial#CentOS|CentOS]]. While many commands are identical across Linux systems, there are some areas that will differ from distribution to distribution, such as package management, service control, and (to a lesser extent) user management.<br />
<br />
If you want more help with Linux, you may find the [https://cvw.cac.cornell.edu/Linux/ Introduction to Linux] topic on the [https://cvw.cac.cornell.edu/topics Cornell Virtual Workshop] useful. For a really thorough treatment, including guidance on how to write scripts in bash, you can check out "Learning the Bash Shell" from the [https://www.library.cornell.edu Cornell Library] (available as an e-book).<br />
<br />
__TOC__<br />
<br />
== Definitions ==<br />
<br />
This section contains some basic working definitions to help you through this tutorial if you have never used Linux before. This list and the definitions in it should not be considered authoritative.<br />
<br />
'''directory''' - folder<br />
<br />
'''path''' - the sequence of directories leading to a particular subdirectory or file<br />
<br />
'''shell''' (A.K.A. '''console''' or '''terminal''') - a text-only user interface for interacting with an operating system's programs and services. This is where commands are entered.<br />
<br />
'''command''' - a task for the computer to execute that is entered via the shell<br />
<br />
'''environment''' - the set of all variables defined in the current shell. The special environment variable <tt>PATH</tt> shows the sequence of paths that will be searched to find the commands that you enter.<br />
<br />
'''package''' - an archive of software and metadata that can be downloaded, installed, and removed via a package manager<br />
<br />
'''root''' - the system administrative account with all the highest privileges, also known as the superuser. By default, most Linux distros have a single root account when installed, and no user accounts.<br />
<br />
'''sudo''' - a program that allows a user to run commands with the privileges of another user, most often the root or superuser account. This is typically used by typing sudo before a command.<br />
<br />
'''root directory''' - the top-level directory of the system, denoted <code>/</code> (forward slash). It is the start of most paths. This is not the same as the root user.<br />
<br />
'''home directory''' - the top-level directory of the user, denoted <code>~</code> (tilde). It is the start of most paths where the user will store work.<br />
<br />
== Basic Useful Commands ==<br />
<br />
This list is not comprehensive, but rather a starting point.<br />
<br />
; <code>pwd</code> : print working directory - specifically, print the full path to the current working directory<br />
; <code>ls</code> : list directory contents<br />
; <code>cd</code> : change directory <p>Example: <code>cd ~</code> will take you to your home directory</p><br />
; <code>mkdir <name></code> : make a directory with the specified name<br />
; <code>man <command></code> : display a manual pages for the specified command<br />
; <code>which <command></code> : show the full path to the given command, as found from the paths in the PATH environment variable<br />
; <code>history</code> : display a list of commands that have been executed via the terminal<br />
; <code>cat <file></code> : output (concatenate) the contents of a file to the terminal, with many other options available (check out <code>man cat</code> for more info)<br />
; <code>less <file></code> : output the contents of a file one screenful at a time, with page-advance (spacebar) and search (/) functions. The <code>more</code> command works similarly.<br />
; <code>grep <pattern> <file></code> : print lines from the file that match the specified pattern. To search multiple files, use the <code>*</code> wildcard. To search a directory tree, use the <code>-r</code> option. The <code>-x</code> option prints lines that DON'T match the pattern.<br />
; <code><command> | less</code><br />
; <code><command> | grep <pattern></code> : join commands with a <code>|</code> in order to "pipe" the output from the first command into the second, e.g., into <code>less</code> to paginate it, or into <code>grep</code> to search it. <p>Example: <code>history | grep mkdir</code> would search the history output for each time the <code>mkdir</code> command was executed, thus determining all the directories you had created.</p><br />
; <code>export VAR=value</code> : set an environment variable (VAR in this example) to have a certain value<br />
; <code>ssh</code> : [[Connect_to_Linux#Using_Secure_Shell|Secure Shell (ssh)]] is great for getting a quick command-line interface where you can enter bash shell commands. It also allows you to do X11 forwarding, which enables you to interact graphically with your instance using [[Connect_to_Linux#Using_X-Windows|X Windows]]. If you have not already, it would also be good to familiarize yourself with how to [[Connect_to_Linux|connect to Linux machines remotely]].<br />
<br />
=== Text Editors === <br />
<br />
Since the default interaction with a [[Red Cloud Linux Instances|Linux Instance]] is through a terminal, it may be useful to familiarize yourself with at least one text editor that can be used in the terminal. Here are a few, with links to get more information about them, but there are more.<br />
<br />
; <code>vim</code> : [https://en.wikipedia.org/wiki/Vim_(text_editor) Vim] is often already installed with many Linux distros, and is therefore useful to learn. There are many online tutorials, but you can also simply type <code>vimtutor</code> in the terminal to learn how to use vim.<br />
; <code>emacs</code> : [https://en.wikipedia.org/wiki/Emacs Emacs] is a family of text editors including the very popular [https://en.wikipedia.org/wiki/GNU_Emacs GNU Emacs]. If you want to use it, it may be helpful to take a [https://www.gnu.org/software/emacs/tour/ guided tour] or to consult the [https://www.gnu.org/software/emacs/manual/html_node/emacs/index.html manual].<br />
; <code>nano</code> : [https://en.wikipedia.org/wiki/GNU_nano GNU nano] is a simpler text editor than something like vim because it doesn't have modes, you simply type when it opens. If you'd like more information, consult the [https://www.nano-editor.org/docs.php documentation].<br />
<br />
== Ubuntu ==<br />
<br />
This section has specific instructions for Ubuntu images on how to create your [[#Initial User Setup| first user with sudo privileges]], [[#Additional Users|create additional users]], and [[#Installing Software|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup|Initial User Setup]] steps. If you do want to create [[#Additional Users|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used Ubuntu before, please read this whole section.<br />
<br />
=== The "ubuntu" User ===<br />
<br />
Since the Ubuntu distribution of Linux [https://help.ubuntu.com/community/RootSudo locks the root account] by default, you cannot use that account to ssh when you first setup a new image. Instead, there is a default account with the username <code>ubuntu</code>, with a blank password, that has sudo privileges. Unless you are the sole user of your machine, it is still recommended that you create a new [https://help.ubuntu.com/lts/serverguide/user-management.html user account], for which the steps are detailed below.<br />
<br />
=== Initial User Setup ===<br />
<br />
These steps create a new sudo user, and must all be '''completed in order''':<br />
# <code>ssh -i <keyname>.pem ubuntu@<ip of instance></code><br />
#* Connects to the instance via ssh as the [[Linux_Tutorial#The_.22ubuntu.22_user|ubuntu account]].<br />
# <code>sudo adduser <username></code><br />
#* You will be prompted to enter & verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one. You will also be prompted for some information (i.e. name, phone number, etc.) which is optional. If you do not wish to add this information, simply hit "enter".<br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members.<br />
#* This adds a new user with the name <username>.<br />
# <code>sudo adduser <username> sudo</code><br />
#* This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>sudo mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh.<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>sudo cp ~/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>sudo chown -hR <username>:<username> /home/<username></code><br />
#* Changes the ownership of the user's home directory and subdirectories (including .ssh) to the user.<br />
# <code>sudo chmod 700 -R ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the folder and all files contained within.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* Verify the line that says <code>PasswordAuthentication</code> has a <code>no</code> next to it (this should be the default).<br />
#* '''Change this to <code>yes</code> only if''' you intend to have multiple users and wish to allow them to connect via ssh with a password, without requiring a [[OpenStack#Key Pairs| key pair]].<br />
#* You could also use your preferred text editor<br />
# <code>sudo systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to connect via ssh.<br />
# <code>sudo apt update</code> and <code>sudo apt upgrade</code><br />
#* This makes sure the system is up-to-date.<br />
#* You can now begin [[Linux_Tutorial#Installing_Software|Installing Software]].<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo chown -hR <username> /home/<username></code> <br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo adduser <username> sudo</code><br />
# <code>sudo chown -hR <username> /home/<username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code> <p>'''Or''' you will be prompted for a password when you initially create the user with <code>sudo adduser <username></code></p><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to edit the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and change the line that says <code>PasswordAuthentication no</code> to <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for Ubuntu is called [https://en.wikipedia.org/wiki/APT_(Debian) apt] (also see the Ubuntu docs on [https://help.ubuntu.com/lts/serverguide/apt.html apt] and [https://help.ubuntu.com/lts/serverguide/aptitude.html aptitude]). Here are some basic commands worth making sure you understand (again, <code>man apt</code> will help here):<br />
<br />
:* <code>sudo apt update</code><br />
:* <code>sudo apt upgrade</code><br />
:* <code>apt search <package></code><br />
:* <code>sudo apt install <package></code><br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance. <br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
To find available packages (from currently installed repositories), the following command may be used: <code>apt search <package></code>. For instance, here are the first 6 results for <code>apt search python</code>:<br />
<br />
p bpython - fancy interface to the Python interpreter<br />
p bpython-gtk - fancy interface to the Python interpreter<br />
p bpython-urwid - fancy interface to the Python interpreter<br />
p bpython3 - fancy interface to the Python3 interpreter<br />
p cairo-dock-plug-ins-dbus-interf - Python interface to interact with Cairo-Do<br />
p cantor-backend-python - Python backend for Cantor<br />
<br />
Note that the ‘p’ in the first column means that no trace of package exists on the system (run <code>man apt</code> for more details).<br />
<br />
== CentOS ==<br />
<br />
This section has specific instructions for CentOS images on how to create your [[#Initial User Setup_2| first user with sudo privileges]], [[#Additional Users_2|create additional users]], and [[#Installing Software_2|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup_2|Initial User Setup]] steps. If you do want to create [[#Additional Users_2|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used CentOS before, please read this whole section.<br />
<br />
=== Initial User Setup ===<br />
Once you have started a [[Red Cloud Linux Instances|Linux Instance]], you will want to connect as the [http://en.wikipedia.org/wiki/Superuser#Unix_and_Unix-like root user] and set up a new user account that you can use for your day-to-day work. The way to do this depends on the CentOS base image: on some you can log in directly as ‘root’, but on others you first have to log in as user ‘centos’, then switch to ‘root’ as shown below. In the latter case you could choose to make ‘centos’ the account you use each time you want to connect, but it is recommended that you follow the steps below so you can pick a separate username of your own choosing. <br />
<br />
# <code>ssh -i <keyname>.pem centos@<ip of instance></code><br />
#* Connects to the instance via ssh as the ‘centos’ account (note, by default in Red Cloud, direct login by root is disabled )<br />
# <code>sudo su -</code><br />
#* Subsequent steps are easier if you become root, so you don't have to type ‘sudo’ ahead of each command.<br />
# <code>adduser <username></code><br />
#* Adds a new user with the name <username><br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members<br />
#* Multiple users may be added at the instance owner’s discretion (see below). <br />
# <code>passwd <username></code><br />
#* This will prompt you to set and verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one.<br />
#* '''Note''': if you do not run this command, a password will not be set for the user!<br />
# <code>usermod -aG wheel <username></code><br />
#*This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>cp ~centos/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>chown -hR <username>:<username> /home/<username>/.ssh</code><br />
#* Changes the ownership of the user's .ssh directory and all the files and subdirectories in it to the user.<br />
# <code>chmod 700 ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the .ssh folder to the recommended level.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* '''Skip this step if''' you never intend for users to connect via ssh with a password and instead want them to use a [[OpenStack#Key Pairs| key pair]].<br />
#* Uncomment the line that says <code>PasswordAuthentication yes</code>.<br />
#* Comment out the line that says <code>PasswordAuthentication no</code>.<br />
#* Note: you may need to install vim first by running <code>yum install vim</code>. See [[#Installing_Software_2|Installing Software]] for more information.<br />
#* You could also use your preferred text editor<br />
# <code>systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to ssh<br />
# <code>sudo yum update</code><br />
#* This makes sure the system is up-to-date<br />
#* You may notice that certain packages do not get updated; don't be alarmed, as this is expected<br />
#* It's a good idea to restart the instance after the update completes<br />
#* You can now begin [[#Installing_Software_2|Installing Software]]<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
# <code>usermod -aG wheel <username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to check the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and ensure the line that says <code>PasswordAuthentication</code> says <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for CentOS is called [https://en.wikipedia.org/wiki/Yum_(software) yum]. Here are some basic commands worth making sure you understand (again, <code>man yum</code> will help here):<br />
<br />
:* <code>yum check-update</code><br />
:* <code>sudo yum update</code><br />
:* <code>yum search <package></code><br />
:* <code>sudo yum install <package></code><br />
<br />
You may notice that certain packages do not get updated with <code>sudo yum update</code>; don't be alarmed, as this is expected. It's a good idea to restart the instance after the update completes.<br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance.<br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
=== SSH Security ===<br />
<br />
Once you have set up a user with sudo privileges and ensured that you can indeed login and perform sudo commands successfully (it would be good to test this to be sure), you may want to secure the root login by disabling it.<br />
<br />
'''Disable root login:'''<br />
This must be done while logged in either as root or your user with sudo privileges.<br />
<br />
# <code>vim /etc/ssh/sshd_config</code><br />
# Change the the line <code>PermitRootLogin yes</code> to <code>PermitRootLogin no</code><br />
# '''Note''': if this line is commented out (with a <code>#</code> character in the front), you will need to uncomment it.<br />
# <code>systemctl restart sshd</code><br />
<br />
When you exit, you should verify that you cannot login as root, but that you can still login as your user.<br />
<br />
For more information on SSH Security, see the [https://wiki.centos.org/HowTos/Network/SecuringSSH CentOS guide on Securing OpenSSH].</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Linux_Tutorial&diff=3006Linux Tutorial2019-11-04T18:35:38Z<p>Pzv2: /* Initial User Setup */ Fixed typo in links</p>
<hr />
<div>This tutorial is intended as a basic introduction to [https://en.wikipedia.org/wiki/Linux Linux] for users of CAC's Linux resources, especially those who are managing [[Red_Cloud|Red Cloud]] services and are creating or using a [[Red Cloud Linux Instances|Linux Instance]]. There are two [https://en.wikipedia.org/wiki/Linux_distribution Linux distributions] (AKA distros) available for [[Images|images]] on Red Cloud: [https://en.wikipedia.org/wiki/Ubuntu_(operating_system) Ubuntu] and [https://en.wikipedia.org/wiki/CentOS CentOS]. In this tutorial, you will learn how to add a user, install software using the distribution's [https://en.wikipedia.org/wiki/Package_manager package manager], and enable remote password logins, along with a number of related tips. The tutorial begins with [[Linux_Tutorial#Basic_Useful_Commands|common commands]] between both distributions, followed by sections for the specific commands you are likely to need on [[Linux_Tutorial#Ubuntu|Ubuntu]] and [[Linux_Tutorial#CentOS|CentOS]]. While many commands are identical across Linux systems, there are some areas that will differ from distribution to distribution, such as package management, service control, and (to a lesser extent) user management.<br />
<br />
If you want more help with Linux, you may find the [https://cvw.cac.cornell.edu/Linux/ Introduction to Linux] topic on the [https://cvw.cac.cornell.edu/topics Cornell Virtual Workshop] useful. For a really thorough treatment, including guidance on how to write scripts in bash, you can check out "Learning the Bash Shell" from the [https://www.library.cornell.edu Cornell Library] (available as an e-book).<br />
<br />
__TOC__<br />
<br />
== Definitions ==<br />
<br />
This section contains some basic working definitions to help you through this tutorial if you have never used Linux before. This list and the definitions in it should not be considered authoritative.<br />
<br />
'''directory''' - folder<br />
<br />
'''path''' - the sequence of directories leading to a particular subdirectory or file<br />
<br />
'''shell''' (A.K.A. '''console''' or '''terminal''') - a text-only user interface for interacting with an operating system's programs and services. This is where commands are entered.<br />
<br />
'''command''' - a task for the computer to execute that is entered via the shell<br />
<br />
'''environment''' - the set of all variables defined in the current shell. The special environment variable <tt>PATH</tt> shows the sequence of paths that will be searched to find the commands that you enter.<br />
<br />
'''package''' - an archive of software and metadata that can be downloaded, installed, and removed via a package manager<br />
<br />
'''root''' - the system administrative account with all the highest privileges, also known as the superuser. By default, most Linux distros have a single root account when installed, and no user accounts.<br />
<br />
'''sudo''' - a program that allows a user to run commands with the privileges of another user, most often the root or superuser account. This is typically used by typing sudo before a command.<br />
<br />
'''root directory''' - the top-level directory of the system, denoted <code>/</code> (forward slash). It is the start of most paths. This is not the same as the root user.<br />
<br />
'''home directory''' - the top-level directory of the user, denoted <code>~</code> (tilde). It is the start of most paths where the user will store work.<br />
<br />
== Basic Useful Commands ==<br />
<br />
This list is not comprehensive, but rather a starting point.<br />
<br />
; <code>pwd</code> : print working directory - specifically, print the full path to the current working directory<br />
; <code>ls</code> : list directory contents<br />
; <code>cd</code> : change directory <p>Example: <code>cd ~</code> will take you to your home directory</p><br />
; <code>mkdir <name></code> : make a directory with the specified name<br />
; <code>man <command></code> : display a manual pages for the specified command<br />
; <code>which <command></code> : show the full path to the given command, as found from the paths in the PATH environment variable<br />
; <code>history</code> : display a list of commands that have been executed via the terminal<br />
; <code>cat <file></code> : output (concatenate) the contents of a file to the terminal, with many other options available (check out <code>man cat</code> for more info)<br />
; <code>less <file></code> : output the contents of a file one screenful at a time, with page-advance (spacebar) and search (/) functions. The <code>more</code> command works similarly.<br />
; <code>grep <pattern> <file></code> : print lines from the file that match the specified pattern. To search multiple files, use the <code>*</code> wildcard. To search a directory tree, use the <code>-r</code> option. The <code>-x</code> option prints lines that DON'T match the pattern.<br />
; <code><command> | less</code><br />
; <code><command> | grep <pattern></code> : join commands with a <code>|</code> in order to "pipe" the output from the first command into the second, e.g., into <code>less</code> to paginate it, or into <code>grep</code> to search it. <p>Example: <code>history | grep mkdir</code> would search the history output for each time the <code>mkdir</code> command was executed, thus determining all the directories you had created.</p><br />
; <code>export VAR=value</code> : set an environment variable (VAR in this example) to have a certain value<br />
; <code>ssh</code> : [[Connect_to_Linux#Using_Secure_Shell|Secure Shell (ssh)]] is great for getting a quick command-line interface where you can enter bash shell commands. It also allows you to do X11 forwarding, which enables you to interact graphically with your instance using [[Connect_to_Linux#Using_X-Windows|X Windows]]. If you have not already, it would also be good to familiarize yourself with how to [[Connect_to_Linux|connect to Linux machines remotely]].<br />
<br />
=== Text Editors === <br />
<br />
Since the default interaction with a [[Red Cloud Linux Instances|Linux Instance]] is through a terminal, it may be useful to familiarize yourself with at least one text editor that can be used in the terminal. Here are a few, with links to get more information about them, but there are more.<br />
<br />
; <code>vim</code> : [https://en.wikipedia.org/wiki/Vim_(text_editor) Vim] is often already installed with many Linux distros, and is therefore useful to learn. There are many online tutorials, but you can also simply type <code>vimtutor</code> in the terminal to learn how to use vim.<br />
; <code>emacs</code> : [https://en.wikipedia.org/wiki/Emacs Emacs] is a family of text editors including the very popular [https://en.wikipedia.org/wiki/GNU_Emacs GNU Emacs]. If you want to use it, it may be helpful to take a [https://www.gnu.org/software/emacs/tour/ guided tour] or to consult the [https://www.gnu.org/software/emacs/manual/html_node/emacs/index.html manual].<br />
; <code>nano</code> : [https://en.wikipedia.org/wiki/GNU_nano GNU nano] is a simpler text editor than something like vim because it doesn't have modes, you simply type when it opens. If you'd like more information, consult the [https://www.nano-editor.org/docs.php documentation].<br />
<br />
== Ubuntu ==<br />
<br />
This section has specific instructions for Ubuntu images on how to create your [[#Initial User Setup| first user with sudo privileges]], [[#Additional Users|create additional users]], and [[#Installing Software|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup|Initial User Setup]] steps. If you do want to create [[#Additional Users|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used Ubuntu before, please read this whole section.<br />
<br />
=== The "ubuntu" User ===<br />
<br />
Since the Ubuntu distribution of Linux [https://help.ubuntu.com/community/RootSudo locks the root account] by default, you cannot use that account to ssh when you first setup a new image. Instead, there is a default account with the username <code>ubuntu</code>, with a blank password, that has sudo privileges. Unless you are the sole user of your machine, it is still recommended that you create a new [https://help.ubuntu.com/lts/serverguide/user-management.html user account], for which the steps are detailed below.<br />
<br />
=== Initial User Setup ===<br />
<br />
These steps create a new sudo user, and must all be '''completed in order''':<br />
# <code>ssh -i <keyname>.pem ubuntu@<ip of instance></code><br />
#* Connects to the instance via ssh as the [[Linux_Tutorial#The_.22ubuntu.22_user|ubuntu account]].<br />
# <code>sudo adduser <username></code><br />
#* You will be prompted to enter & verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one. You will also be prompted for some information (i.e. name, phone number, etc.) which is optional. If you do not wish to add this information, simply hit "enter".<br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members.<br />
#* This adds a new user with the name <username>.<br />
# <code>sudo adduser <username> sudo</code><br />
#* This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>sudo mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh.<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>sudo cp ~/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>sudo chown -hR <username>:<username> /home/<username></code><br />
#* Changes the ownership of the user's home directory and subdirectories (including .ssh) to the user.<br />
# <code>sudo chmod 700 -R ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the folder and all files contained within.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* Verify the line that says <code>PasswordAuthentication</code> has a <code>no</code> next to it (this should be the default).<br />
#* '''Change this to <code>yes</code> only if''' you intend to have multiple users and wish to allow them to connect via ssh with a password, without requiring a [[OpenStack#Key Pairs| key pair]].<br />
#* You could also use your preferred text editor<br />
# <code>sudo systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to connect via ssh.<br />
# <code>sudo apt update</code> and <code>sudo apt upgrade</code><br />
#* This makes sure the system is up-to-date.<br />
#* You can now begin [[Linux_Tutorial#Installing_Software|Installing Software]].<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo chown -hR <username> /home/<username></code> <br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo adduser <username> sudo</code><br />
# <code>sudo chown -hR <username> /home/<username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code> <p>'''Or''' you will be prompted for a password when you initially create the user with <code>sudo adduser <username></code></p><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to edit the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and change the line that says <code>PasswordAuthentication no</code> to <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for Ubuntu is called [https://en.wikipedia.org/wiki/APT_(Debian) apt] (also see the Ubuntu docs on [https://help.ubuntu.com/lts/serverguide/apt.html apt] and [https://help.ubuntu.com/lts/serverguide/aptitude.html aptitude]). Here are some basic commands worth making sure you understand (again, <code>man apt</code> will help here):<br />
<br />
:* <code>sudo apt update</code><br />
:* <code>sudo apt upgrade</code><br />
:* <code>apt search <package></code><br />
:* <code>sudo apt install <package></code><br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance. <br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
To find available packages (from currently installed repositories), the following command may be used: <code>apt search <package></code>. For instance, here are the first 6 results for <code>apt search python</code>:<br />
<br />
p bpython - fancy interface to the Python interpreter<br />
p bpython-gtk - fancy interface to the Python interpreter<br />
p bpython-urwid - fancy interface to the Python interpreter<br />
p bpython3 - fancy interface to the Python3 interpreter<br />
p cairo-dock-plug-ins-dbus-interf - Python interface to interact with Cairo-Do<br />
p cantor-backend-python - Python backend for Cantor<br />
<br />
Note that the ‘p’ in the first column means that no trace of package exists on the system (run <code>man apt</code> for more details).<br />
<br />
== CentOS ==<br />
<br />
This section has specific instructions for CentOS images on how to create your [[#Initial User Setup_2| first user with sudo privileges]], [[#Additional Users_2|create additional users]], and [[#Installing Software_2|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup_2|Initial User Setup]] steps. If you do want to create [[#Additional Users_2|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used CentOS before, please read this whole section.<br />
<br />
=== Initial User Setup ===<br />
Once you have started a [[Red Cloud Linux_Instances|Linux Instance]], you will want to connect as the [http://en.wikipedia.org/wiki/Superuser#Unix_and_Unix-like root user] and set up a new user account that you can use for your day-to-day work. The way to do this depends on the CentOS base image: on some you can log in directly as ‘root’, but on others you first have to log in as user ‘centos’, then switch to ‘root’ as shown below. In the latter case you could choose to make ‘centos’ the account you use each time you want to connect, but it is recommended that you follow the steps below so you can pick a separate username of your own choosing. <br />
<br />
# <code>ssh -i <keyname>.pem centos@<ip of instance></code><br />
#* Connects to the instance via ssh as the ‘centos’ account (note, by default in Red Cloud, direct login by root is disabled )<br />
# <code>sudo su -</code><br />
#* Subsequent steps are easier if you become root, so you don't have to type ‘sudo’ ahead of each command.<br />
# <code>adduser <username></code><br />
#* Adds a new user with the name <username><br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members<br />
#* Multiple users may be added at the instance owner’s discretion (see below). <br />
# <code>passwd <username></code><br />
#* This will prompt you to set and verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one.<br />
#* '''Note''': if you do not run this command, a password will not be set for the user!<br />
# <code>usermod -aG wheel <username></code><br />
#*This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>cp ~centos/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>chown -hR <username>:<username> /home/<username>/.ssh</code><br />
#* Changes the ownership of the user's .ssh directory and all the files and subdirectories in it to the user.<br />
# <code>chmod 700 ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the .ssh folder to the recommended level.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* '''Skip this step if''' you never intend for users to connect via ssh with a password and instead want them to use a [[OpenStack#Key Pairs| key pair]].<br />
#* Uncomment the line that says <code>PasswordAuthentication yes</code>.<br />
#* Comment out the line that says <code>PasswordAuthentication no</code>.<br />
#* Note: you may need to install vim first by running <code>yum install vim</code>. See [[#Installing_Software_2|Installing Software]] for more information.<br />
#* You could also use your preferred text editor<br />
# <code>systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to ssh<br />
# <code>sudo yum update</code><br />
#* This makes sure the system is up-to-date<br />
#* You may notice that certain packages do not get updated; don't be alarmed, as this is expected<br />
#* It's a good idea to restart the instance after the update completes<br />
#* You can now begin [[#Installing_Software_2|Installing Software]]<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
# <code>usermod -aG wheel <username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to check the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and ensure the line that says <code>PasswordAuthentication</code> says <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for CentOS is called [https://en.wikipedia.org/wiki/Yum_(software) yum]. Here are some basic commands worth making sure you understand (again, <code>man yum</code> will help here):<br />
<br />
:* <code>yum check-update</code><br />
:* <code>sudo yum update</code><br />
:* <code>yum search <package></code><br />
:* <code>sudo yum install <package></code><br />
<br />
You may notice that certain packages do not get updated with <code>sudo yum update</code>; don't be alarmed, as this is expected. It's a good idea to restart the instance after the update completes.<br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance.<br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
=== SSH Security ===<br />
<br />
Once you have set up a user with sudo privileges and ensured that you can indeed login and perform sudo commands successfully (it would be good to test this to be sure), you may want to secure the root login by disabling it.<br />
<br />
'''Disable root login:'''<br />
This must be done while logged in either as root or your user with sudo privileges.<br />
<br />
# <code>vim /etc/ssh/sshd_config</code><br />
# Change the the line <code>PermitRootLogin yes</code> to <code>PermitRootLogin no</code><br />
# '''Note''': if this line is commented out (with a <code>#</code> character in the front), you will need to uncomment it.<br />
# <code>systemctl restart sshd</code><br />
<br />
When you exit, you should verify that you cannot login as root, but that you can still login as your user.<br />
<br />
For more information on SSH Security, see the [https://wiki.centos.org/HowTos/Network/SecuringSSH CentOS guide on Securing OpenSSH].</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Linux_Tutorial&diff=3005Linux Tutorial2019-11-04T18:20:25Z<p>Pzv2: Fixed old links</p>
<hr />
<div>This tutorial is intended as a basic introduction to [https://en.wikipedia.org/wiki/Linux Linux] for users of CAC's Linux resources, especially those who are managing [[Red_Cloud|Red Cloud]] services and are creating or using a [[Red Cloud Linux Instances|Linux Instance]]. There are two [https://en.wikipedia.org/wiki/Linux_distribution Linux distributions] (AKA distros) available for [[Images|images]] on Red Cloud: [https://en.wikipedia.org/wiki/Ubuntu_(operating_system) Ubuntu] and [https://en.wikipedia.org/wiki/CentOS CentOS]. In this tutorial, you will learn how to add a user, install software using the distribution's [https://en.wikipedia.org/wiki/Package_manager package manager], and enable remote password logins, along with a number of related tips. The tutorial begins with [[Linux_Tutorial#Basic_Useful_Commands|common commands]] between both distributions, followed by sections for the specific commands you are likely to need on [[Linux_Tutorial#Ubuntu|Ubuntu]] and [[Linux_Tutorial#CentOS|CentOS]]. While many commands are identical across Linux systems, there are some areas that will differ from distribution to distribution, such as package management, service control, and (to a lesser extent) user management.<br />
<br />
If you want more help with Linux, you may find the [https://cvw.cac.cornell.edu/Linux/ Introduction to Linux] topic on the [https://cvw.cac.cornell.edu/topics Cornell Virtual Workshop] useful. For a really thorough treatment, including guidance on how to write scripts in bash, you can check out "Learning the Bash Shell" from the [https://www.library.cornell.edu Cornell Library] (available as an e-book).<br />
<br />
__TOC__<br />
<br />
== Definitions ==<br />
<br />
This section contains some basic working definitions to help you through this tutorial if you have never used Linux before. This list and the definitions in it should not be considered authoritative.<br />
<br />
'''directory''' - folder<br />
<br />
'''path''' - the sequence of directories leading to a particular subdirectory or file<br />
<br />
'''shell''' (A.K.A. '''console''' or '''terminal''') - a text-only user interface for interacting with an operating system's programs and services. This is where commands are entered.<br />
<br />
'''command''' - a task for the computer to execute that is entered via the shell<br />
<br />
'''environment''' - the set of all variables defined in the current shell. The special environment variable <tt>PATH</tt> shows the sequence of paths that will be searched to find the commands that you enter.<br />
<br />
'''package''' - an archive of software and metadata that can be downloaded, installed, and removed via a package manager<br />
<br />
'''root''' - the system administrative account with all the highest privileges, also known as the superuser. By default, most Linux distros have a single root account when installed, and no user accounts.<br />
<br />
'''sudo''' - a program that allows a user to run commands with the privileges of another user, most often the root or superuser account. This is typically used by typing sudo before a command.<br />
<br />
'''root directory''' - the top-level directory of the system, denoted <code>/</code> (forward slash). It is the start of most paths. This is not the same as the root user.<br />
<br />
'''home directory''' - the top-level directory of the user, denoted <code>~</code> (tilde). It is the start of most paths where the user will store work.<br />
<br />
== Basic Useful Commands ==<br />
<br />
This list is not comprehensive, but rather a starting point.<br />
<br />
; <code>pwd</code> : print working directory - specifically, print the full path to the current working directory<br />
; <code>ls</code> : list directory contents<br />
; <code>cd</code> : change directory <p>Example: <code>cd ~</code> will take you to your home directory</p><br />
; <code>mkdir <name></code> : make a directory with the specified name<br />
; <code>man <command></code> : display a manual pages for the specified command<br />
; <code>which <command></code> : show the full path to the given command, as found from the paths in the PATH environment variable<br />
; <code>history</code> : display a list of commands that have been executed via the terminal<br />
; <code>cat <file></code> : output (concatenate) the contents of a file to the terminal, with many other options available (check out <code>man cat</code> for more info)<br />
; <code>less <file></code> : output the contents of a file one screenful at a time, with page-advance (spacebar) and search (/) functions. The <code>more</code> command works similarly.<br />
; <code>grep <pattern> <file></code> : print lines from the file that match the specified pattern. To search multiple files, use the <code>*</code> wildcard. To search a directory tree, use the <code>-r</code> option. The <code>-x</code> option prints lines that DON'T match the pattern.<br />
; <code><command> | less</code><br />
; <code><command> | grep <pattern></code> : join commands with a <code>|</code> in order to "pipe" the output from the first command into the second, e.g., into <code>less</code> to paginate it, or into <code>grep</code> to search it. <p>Example: <code>history | grep mkdir</code> would search the history output for each time the <code>mkdir</code> command was executed, thus determining all the directories you had created.</p><br />
; <code>export VAR=value</code> : set an environment variable (VAR in this example) to have a certain value<br />
; <code>ssh</code> : [[Connect_to_Linux#Using_Secure_Shell|Secure Shell (ssh)]] is great for getting a quick command-line interface where you can enter bash shell commands. It also allows you to do X11 forwarding, which enables you to interact graphically with your instance using [[Connect_to_Linux#Using_X-Windows|X Windows]]. If you have not already, it would also be good to familiarize yourself with how to [[Connect_to_Linux|connect to Linux machines remotely]].<br />
<br />
=== Text Editors === <br />
<br />
Since the default interaction with a [[Red Cloud Linux Instances|Linux Instance]] is through a terminal, it may be useful to familiarize yourself with at least one text editor that can be used in the terminal. Here are a few, with links to get more information about them, but there are more.<br />
<br />
; <code>vim</code> : [https://en.wikipedia.org/wiki/Vim_(text_editor) Vim] is often already installed with many Linux distros, and is therefore useful to learn. There are many online tutorials, but you can also simply type <code>vimtutor</code> in the terminal to learn how to use vim.<br />
; <code>emacs</code> : [https://en.wikipedia.org/wiki/Emacs Emacs] is a family of text editors including the very popular [https://en.wikipedia.org/wiki/GNU_Emacs GNU Emacs]. If you want to use it, it may be helpful to take a [https://www.gnu.org/software/emacs/tour/ guided tour] or to consult the [https://www.gnu.org/software/emacs/manual/html_node/emacs/index.html manual].<br />
; <code>nano</code> : [https://en.wikipedia.org/wiki/GNU_nano GNU nano] is a simpler text editor than something like vim because it doesn't have modes, you simply type when it opens. If you'd like more information, consult the [https://www.nano-editor.org/docs.php documentation].<br />
<br />
== Ubuntu ==<br />
<br />
This section has specific instructions for Ubuntu images on how to create your [[#Initial User Setup| first user with sudo privileges]], [[#Additional Users|create additional users]], and [[#Installing Software|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup|Initial User Setup]] steps. If you do want to create [[#Additional Users|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used Ubuntu before, please read this whole section.<br />
<br />
=== The "ubuntu" User ===<br />
<br />
Since the Ubuntu distribution of Linux [https://help.ubuntu.com/community/RootSudo locks the root account] by default, you cannot use that account to ssh when you first setup a new image. Instead, there is a default account with the username <code>ubuntu</code>, with a blank password, that has sudo privileges. Unless you are the sole user of your machine, it is still recommended that you create a new [https://help.ubuntu.com/lts/serverguide/user-management.html user account], for which the steps are detailed below.<br />
<br />
=== Initial User Setup ===<br />
<br />
These steps create a new sudo user, and must all be '''completed in order''':<br />
# <code>ssh -i <keyname>.pem ubuntu@<ip of instance></code><br />
#* Connects to the instance via ssh as the [[Linux_Tutorial#The_.22ubuntu.22_user|ubuntu account]].<br />
# <code>sudo adduser <username></code><br />
#* You will be prompted to enter & verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one. You will also be prompted for some information (i.e. name, phone number, etc.) which is optional. If you do not wish to add this information, simply hit "enter".<br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members.<br />
#* This adds a new user with the name <username>.<br />
# <code>sudo adduser <username> sudo</code><br />
#* This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>sudo mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh.<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>sudo cp ~/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>sudo chown -hR <username>:<username> /home/<username></code><br />
#* Changes the ownership of the user's home directory and subdirectories (including .ssh) to the user.<br />
# <code>sudo chmod 700 -R ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the folder and all files contained within.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* Verify the line that says <code>PasswordAuthentication</code> has a <code>no</code> next to it (this should be the default).<br />
#* '''Change this to <code>yes</code> only if''' you intend to have multiple users and wish to allow them to connect via ssh with a password, without requiring a [[OpenStack#Key Pairs| key pair]].<br />
#* You could also use your preferred text editor<br />
# <code>sudo systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to connect via ssh.<br />
# <code>sudo apt update</code> and <code>sudo apt upgrade</code><br />
#* This makes sure the system is up-to-date.<br />
#* You can now begin [[Linux_Tutorial#Installing_Software|Installing Software]].<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo chown -hR <username> /home/<username></code> <br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>sudo adduser <username></code> <p>'''Note:''' you will be prompted to create a password. See the [[#Passwords|Passwords]] section for recommendations.</p><br />
# <code>sudo adduser <username> sudo</code><br />
# <code>sudo chown -hR <username> /home/<username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code> <p>'''Or''' you will be prompted for a password when you initially create the user with <code>sudo adduser <username></code></p><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to edit the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and change the line that says <code>PasswordAuthentication no</code> to <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for Ubuntu is called [https://en.wikipedia.org/wiki/APT_(Debian) apt] (also see the Ubuntu docs on [https://help.ubuntu.com/lts/serverguide/apt.html apt] and [https://help.ubuntu.com/lts/serverguide/aptitude.html aptitude]). Here are some basic commands worth making sure you understand (again, <code>man apt</code> will help here):<br />
<br />
:* <code>sudo apt update</code><br />
:* <code>sudo apt upgrade</code><br />
:* <code>apt search <package></code><br />
:* <code>sudo apt install <package></code><br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance. <br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
To find available packages (from currently installed repositories), the following command may be used: <code>apt search <package></code>. For instance, here are the first 6 results for <code>apt search python</code>:<br />
<br />
p bpython - fancy interface to the Python interpreter<br />
p bpython-gtk - fancy interface to the Python interpreter<br />
p bpython-urwid - fancy interface to the Python interpreter<br />
p bpython3 - fancy interface to the Python3 interpreter<br />
p cairo-dock-plug-ins-dbus-interf - Python interface to interact with Cairo-Do<br />
p cantor-backend-python - Python backend for Cantor<br />
<br />
Note that the ‘p’ in the first column means that no trace of package exists on the system (run <code>man apt</code> for more details).<br />
<br />
== CentOS ==<br />
<br />
This section has specific instructions for CentOS images on how to create your [[#Initial User Setup_2| first user with sudo privileges]], [[#Additional Users_2|create additional users]], and [[#Installing Software_2|install software]]. If you are the sole user of your instance, you only need to follow the [[#Initial User Setup_2|Initial User Setup]] steps. If you do want to create [[#Additional Users_2|Additional Users]], be sure to consider what privilege level you would prefer your users to have, and whether you wish to require a key pair for authentication (should be unique per user). If you have not used CentOS before, please read this whole section.<br />
<br />
=== Initial User Setup ===<br />
Once you have started a [[Red Cloud Linux_Instances|Linux Instance]], you will want to connect as the [http://en.wikipedia.org/wiki/Superuser#Unix_and_Unix-like root user] and set up a new user account that you can use for your day-to-day work. The way to do this depends on the CentOS base image: on some you can log in directly as ‘root’, but on others you first have to log in as user ‘centos’, then switch to ‘root’ as shown below. In the latter case you could choose to make ‘centos’ the account you use each time you want to connect, but it is recommended that you follow the steps below so you can pick a separate username of your own choosing. <br />
<br />
# <code>ssh -i <keyname>.pem centos@<ip of instance></code><br />
#* Connects to the instance via ssh as the ‘centos’ account (note, by default in Red Cloud, direct login by root is disabled )<br />
# <code>sudo su -</code><br />
#* Subsequent steps are easier if you become root, so you don't have to type ‘sudo’ ahead of each command.<br />
# <code>adduser <username></code><br />
#* Adds a new user with the name <username><br />
#* Note that <username> could be e.g. ‘bob’, it doesn’t need to be (and really should not be) a Cornell netid, since you can optionally configure your instances to allow use of netid and netid passwords for project members<br />
#* Multiple users may be added at the instance owner’s discretion (see below). <br />
# <code>passwd <username></code><br />
#* This will prompt you to set and verify a password for the user. You should review the [[Rules for Creating Passwords]] before setting one.<br />
#* '''Note''': if you do not run this command, a password will not be set for the user!<br />
# <code>usermod -aG wheel <username></code><br />
#*This will add <username> to the sudo group, which will enable <username> to easily install software and perform other administrative tasks. This has the advantage of making it more difficult to accidentally do something unfortunate to the system.<br />
# <code>mkdir ~<username>/.ssh</code><br />
#* Creates a directory for the user to hold the public encryption key used in ssh<br />
#* '''Note''': The <code>.ssh</code> folder is hidden to the <code>ls</code> command by default because of the "." at the beginning. You can see all folders by sending the <code>ls -a</code> command.<br />
# <code>cp ~centos/.ssh/authorized_keys ~<username>/.ssh</code><br />
#* This copies the public key to the correct place for the user to be able to ssh.<br />
#* '''Only do this if''' you intend to ssh from the same computer with the private key every time you access the instance. This is recommended, since it is more secure.<br />
# <code>chown -hR <username>:<username> /home/<username>/.ssh</code><br />
#* Changes the ownership of the user's .ssh directory and all the files and subdirectories in it to the user.<br />
# <code>chmod 700 ~<username>/.ssh/</code><br />
#* Changes the [https://en.wikipedia.org/wiki/Chmod access permissions] on the .ssh folder to the recommended level.<br />
# <code>vim /etc/ssh/sshd_config</code><br />
#* '''Skip this step if''' you never intend for users to connect via ssh with a password and instead want them to use a [[OpenStack#Key Pairs| key pair]].<br />
#* Uncomment the line that says <code>PasswordAuthentication yes</code>.<br />
#* Comment out the line that says <code>PasswordAuthentication no</code>.<br />
#* Note: you may need to install vim first by running <code>yum install vim</code>. See [[l#Installing_Software_2|Installing Software]] for more information.<br />
#* You could also use your preferred text editor<br />
# <code>systemctl restart sshd</code><br />
# <code>exit</code><br />
# <code>ssh -i <keyname>.pem <username>@<ip></code><br />
#* At this point your user should be set up to ssh<br />
# <code>sudo yum update</code><br />
#* This makes sure the system is up-to-date<br />
#* You may notice that certain packages do not get updated; don't be alarmed, as this is expected<br />
#* It's a good idea to restart the instance after the update completes<br />
#* You can now begin [[l#Installing_Software_2|Installing Software]]<br />
<br />
=== Additional Users ===<br />
<br />
A normal user account on a Ubuntu system does not have sudo privileges, so they cannot install software or perform administrative tasks. These steps create a '''new user without sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
<br />
<br />
It is also possible to create new user accounts with sudo privileges, which enables them to easily install software and perform other administrative tasks without needing a root (or the ubuntu) login. These steps create a '''new user with sudo privileges''':<br />
# <code>adduser <username></code><br />
# <code>passwd <username></code> '''This is optional''' - See the [[#Passwords_2|Passwords]] section below for more information.<br />
# <code>usermod -aG wheel <username></code><br />
<br />
<br />
It is recommended, for security purposes, to require all users to authenticate with a unique SSH key pair when connecting to Red Cloud instances. However, it can be simpler to allow users to authenticate with a password, particularly for users who are inexperienced with SSH. Below we describe how to set up both types of authentication. Choose what's right for you and your users.<br />
<br />
==== Passwords ====<br />
<br />
It's best to assign a different temporary password to each user. When notifying users of their new passwords, remember to ask them to log in and change their passwords right away with <code>passwd</code> (just the plain command, with no arguments).<br />
<br />
# <code>sudo passwd <username></code><br />
# Assign a temporary password like <tt>ch@ngeM3!</tt><br />
# Retype the temporary password when prompted<br />
<br />
Be sure to notify your users of the [[Rules for Creating Passwords]] before they set one.<br />
<br />
==== SSH Key Pairs ====<br />
<br />
Do the following steps to create an SSH key pair for each user. If you have set up passwords and [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], then users can also perform these steps themselves.<br />
<br />
# <code>sudo su <username></code> <p>Omit this step if you are already logged in as the user</p><br />
# <code>mkdir ~/.ssh</code><br />
# <code>ssh-keygen -t rsa</code> <p>Respond to all prompts by hitting enter</p><br />
# <code>cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys</code><br />
# <code>chmod 700 -R ~/.ssh</code><br />
# <code>exit</code> <p>Omit this step if you are already logged in as the user</p><br />
<br />
The resulting keys created can be used to login to the instance without a password. This private key should be stored where it can be found by the user's local SSH client. If [[#Enabling SSH Logins With Passwords_2|logins with passwords are enabled]], each user can retrieve the file <code>~/.ssh/id_rsa</code> using <code>scp</code> or <code>sftp</code>. Otherwise, the trick will be to distribute the private keys to each user in some other secure manner. '''They should not just be attached to emails!'''<br />
<br />
==== Enabling SSH Logins With Passwords ====<br />
<br />
If you are intending to allow users to connect to the instance with only a password, then you will have to check the <code>/etc/ssh/sshd_config</code> file with your preferred text editor, and ensure the line that says <code>PasswordAuthentication</code> says <code>yes</code>. If this is set to <code>no</code>, password authentication will fail for all users, even with the correct password.<br />
<br />
'''Note:''' It is a good idea to check that you can connect to the new user's account via ssh, even if the account is not for you, to ensure it was set up correctly.<br />
<br />
=== Installing Software ===<br />
<br />
The '''package manager''' for CentOS is called [https://en.wikipedia.org/wiki/Yum_(software) yum]. Here are some basic commands worth making sure you understand (again, <code>man yum</code> will help here):<br />
<br />
:* <code>yum check-update</code><br />
:* <code>sudo yum update</code><br />
:* <code>yum search <package></code><br />
:* <code>sudo yum install <package></code><br />
<br />
You may notice that certain packages do not get updated with <code>sudo yum update</code>; don't be alarmed, as this is expected. It's a good idea to restart the instance after the update completes.<br />
<br />
<br />
It is '''recommended''' that you:<br />
:* Ensure your system is up-to-date after beginning an instance.<br />
:* Check for and install updates monthly.<br />
:* Install a screen-saving program like [http://en.wikipedia.org/wiki/Tmux tmux] or [https://www.nixtutor.com/linux/introduction-to-gnu-screen/ screen] ([http://hyperpolyglot.org/multiplexers comparison]), which is often useful in case your connection is dropped (either intentionally or unintentionally) or if you want to have multiple terminals available without needing to login each time. Either one has advantages over trying to manipulate processes that are [http://www.basicallytech.com/blog/archive/70/Shell-stuff-job-control-and-screen/ running in the background] - though this too is good to understand and know how to do!<br />
<br />
=== SSH Security ===<br />
<br />
Once you have set up a user with sudo privileges and ensured that you can indeed login and perform sudo commands successfully (it would be good to test this to be sure), you may want to secure the root login by disabling it.<br />
<br />
'''Disable root login:'''<br />
This must be done while logged in either as root or your user with sudo privileges.<br />
<br />
# <code>vim /etc/ssh/sshd_config</code><br />
# Change the the line <code>PermitRootLogin yes</code> to <code>PermitRootLogin no</code><br />
# '''Note''': if this line is commented out (with a <code>#</code> character in the front), you will need to uncomment it.<br />
# <code>systemctl restart sshd</code><br />
<br />
When you exit, you should verify that you cannot login as root, but that you can still login as your user.<br />
<br />
For more information on SSH Security, see the [https://wiki.centos.org/HowTos/Network/SecuringSSH CentOS guide on Securing OpenSSH].</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Docker&diff=2982Docker2019-10-07T18:55:04Z<p>Pzv2: </p>
<hr />
<div>'''This page is under construction'''<br />
<br />
Intro<br />
<br />
Only available in [[Red Cloud]], not available on CAC Private Clusters.<br />
<br />
__TOC__<br />
<br />
<br />
== Getting Started ==<br />
<br />
=== Installation ===<br />
<br />
<br />
<br />
=== Using A Docker Image ===<br />
<br />
<br />
=== Useful Commands ===<br />
<br />
<br />
<br />
== Creating a Dockerfile ==</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2981Singularity2019-10-07T18:54:06Z<p>Pzv2: </p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
[https://en.wikipedia.org/wiki/Singularity_(software) Singularity] is a [https://en.wikipedia.org/wiki/Containerization containerization] technology, similar to [[Docker]], [https://singularity.lbl.gov/about originally developed] at Lawrence Berkeley National Lab with the requirements of scientific computing and high performance computing (HPC) in mind, especially [https://sylabs.io/guides/3.4/user-guide/security.html security]. A company called [https://sylabs.io/ Sylabs] has since been started and continues to maintain and update Singularity (see the [https://github.com/sylabs/singularity GitHub Repository]). In addition to this page, you may want to explore the [https://sylabs.io/guides/3.4/user-guide/ official documentation].<br />
<br />
'''Please note:''' examples and information in this documentation are based on Singularity version 3.4.0.<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Recipes ===<br />
<br />
=== Images ===<br />
<br />
<br />
[https://github.com/sylabs/singularity GitHub Repository]<br />
[https://singularity-hub.org/ SingularityHub]<br />
<br />
<br />
== Using Singularity ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===<br />
<br />
<br />
=== Creating Images ===<br />
<br />
<br />
== Docker vs. Singularity ==<br />
<br />
[https://sylabs.io/guides/3.4/user-guide/singularity_and_docker.html#best-practices Best Practices]<br />
[https://sylabs.io/guides/3.4/user-guide/singularity_and_docker.html#singularity-definition-file-vs-dockerfile File Differences]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2980Singularity2019-10-07T18:52:27Z<p>Pzv2: </p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
[https://en.wikipedia.org/wiki/Singularity_(software) Singularity] is a [https://en.wikipedia.org/wiki/Containerization containerization] technology, similar to [[Docker]], [https://singularity.lbl.gov/about originally developed] at Lawrence Berkeley National Lab with the requirements of scientific computing and high performance computing (HPC) in mind, especially [https://sylabs.io/guides/3.4/user-guide/security.html security]. A company called [https://sylabs.io/ Sylabs] has since been started and continues to maintain and update Singularity (see the [https://github.com/sylabs/singularity GitHub Repository]). In addition to this page, you may want to explore the [https://sylabs.io/guides/3.4/user-guide/ official documentation].<br />
<br />
'''Please note:''' this documentation uses examples tested with version 3.4.0<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Recipes ===<br />
<br />
=== Images ===<br />
<br />
<br />
[https://github.com/sylabs/singularity GitHub Repository]<br />
[https://singularity-hub.org/ SingularityHub]<br />
<br />
<br />
== Using Singularity ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===<br />
<br />
<br />
=== Creating Images ===<br />
<br />
<br />
== Docker vs. Singularity ==<br />
<br />
[https://sylabs.io/guides/3.4/user-guide/singularity_and_docker.html#best-practices Best Practices]<br />
[https://sylabs.io/guides/3.4/user-guide/singularity_and_docker.html#singularity-definition-file-vs-dockerfile File Differences]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2979Singularity2019-10-07T18:51:37Z<p>Pzv2: </p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
[https://en.wikipedia.org/wiki/Singularity_(software) Singularity] is a [https://en.wikipedia.org/wiki/Containerization containerization] technology, similar to [[Docker]], [https://singularity.lbl.gov/about originally developed] at Lawrence Berkeley National Lab with the requirements of scientific computing and high performance computing (HPC) in mind, especially [https://sylabs.io/guides/3.3/user-guide/security.html security]. A company called [https://sylabs.io/ Sylabs] has since been started and continues to maintain and update Singularity (see the [https://github.com/sylabs/singularity GitHub Repository]). In addition to this page, you may want to explore the [https://sylabs.io/guides/3.4/user-guide/ official documentation].<br />
<br />
'''Please note:''' this documentation uses examples tested with version 3.4.0<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Recipes ===<br />
<br />
=== Images ===<br />
<br />
<br />
[https://github.com/sylabs/singularity GitHub Repository]<br />
[https://singularity-hub.org/ SingularityHub]<br />
<br />
<br />
== Using Singularity ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===<br />
<br />
<br />
=== Creating Images ===<br />
<br />
<br />
== Docker vs. Singularity ==<br />
<br />
[https://sylabs.io/guides/3.4/user-guide/singularity_and_docker.html#best-practices Best Practices]<br />
[https://sylabs.io/guides/3.4/user-guide/singularity_and_docker.html#singularity-definition-file-vs-dockerfile File Differences]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2978Singularity2019-10-07T18:50:43Z<p>Pzv2: Added Docker vs. Singularity section</p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
[https://en.wikipedia.org/wiki/Singularity_(software) Singularity] is a [https://en.wikipedia.org/wiki/Containerization containerization] technology, similar to [[Docker]], [https://singularity.lbl.gov/about originally developed] at Lawrence Berkeley National Lab with the requirements of scientific computing and high performance computing (HPC) in mind, especially [https://sylabs.io/guides/3.3/user-guide/security.html security]. A company called [https://sylabs.io/ Sylabs] has since been started and continues to maintain and update Singularity (see the [https://github.com/sylabs/singularity GitHub Repository]). In addition to this page, you may want to explore the [https://sylabs.io/guides/3.3/user-guide/ official documentation].<br />
<br />
'''Please note:''' this documentation uses examples tested with version 3.4.0<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Recipes ===<br />
<br />
=== Images ===<br />
<br />
<br />
[https://github.com/sylabs/singularity GitHub Repository]<br />
[https://singularity-hub.org/ SingularityHub]<br />
<br />
<br />
== Using Singularity ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===<br />
<br />
<br />
=== Creating Images ===<br />
<br />
<br />
== Docker vs. Singularity ==<br />
<br />
[https://sylabs.io/guides/3.4/user-guide/singularity_and_docker.html#best-practices Best Practices]<br />
[https://sylabs.io/guides/3.4/user-guide/singularity_and_docker.html#singularity-definition-file-vs-dockerfile File Differences]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2977Singularity2019-10-07T18:47:51Z<p>Pzv2: </p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
[https://en.wikipedia.org/wiki/Singularity_(software) Singularity] is a [https://en.wikipedia.org/wiki/Containerization containerization] technology, similar to [[Docker]], [https://singularity.lbl.gov/about originally developed] at Lawrence Berkeley National Lab with the requirements of scientific computing and high performance computing (HPC) in mind, especially [https://sylabs.io/guides/3.3/user-guide/security.html security]. A company called [https://sylabs.io/ Sylabs] has since been started and continues to maintain and update Singularity (see the [https://github.com/sylabs/singularity GitHub Repository]). In addition to this page, you may want to explore the [https://sylabs.io/guides/3.3/user-guide/ official documentation].<br />
<br />
'''Please note:''' this documentation uses examples tested with version 3.4.0<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Recipes ===<br />
<br />
=== Images ===<br />
<br />
<br />
[https://github.com/sylabs/singularity GitHub Repository]<br />
[https://singularity-hub.org/ SingularityHub]<br />
<br />
<br />
== Using Singularity ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===<br />
<br />
<br />
== Creating Images ===</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2943Singularity2019-09-13T18:36:36Z<p>Pzv2: Version note</p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
[https://en.wikipedia.org/wiki/Singularity_(software) Singularity] is a [https://en.wikipedia.org/wiki/Containerization containerization] technology, similar to [[Docker]], [https://singularity.lbl.gov/about originally developed] at Lawrence Berkeley National Lab with the requirements of scientific computing and high performance computing (HPC) in mind, especially [https://sylabs.io/guides/3.3/user-guide/security.html security]. A company called [https://sylabs.io/ Sylabs] has since been started and continues to maintain and update Singularity (see the [https://github.com/sylabs/singularity GitHub Repository]). In addition to this page, you may want to explore the [https://sylabs.io/guides/3.3/user-guide/ official documentation].<br />
<br />
'''Please note:''' this documentation uses examples tested with version 3.3.0<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Recipes ===<br />
<br />
=== Images ===<br />
<br />
<br />
[https://github.com/sylabs/singularity GitHub Repository]<br />
[https://singularity-hub.org/ SingularityHub]<br />
<br />
<br />
== Using Singularity ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===<br />
<br />
<br />
== Creating Images ===</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Private_Clusters&diff=2914Private Clusters2019-09-05T14:51:17Z<p>Pzv2: /* General Documentation */ Removed outdated link</p>
<hr />
<div>===Restricted Use - Privately owned computer resources (in alphabetical order)===<br />
<br />
<br />
* [[AIDA Cluster]] <br />
* [[ASTRA Cluster]]<br />
* [[ATLAS2 Cluster]] <br />
* [[CAPECRYSTAL Cluster]]<br />
* [[CLAL cluster]]<br />
* [[ECCO Cluster]]<br />
* [[Gu Lab]]<br />
* [[HD Human Neuroscience Institute (HD-HNI) Computing]]<br />
* [[MARVIN Cluster]]<br />
* [[OH10 Cluster]]<br />
* [[pool Cluster]]<br />
* [[TARDIS Cluster]]<br />
* [[THECUBE Cluster]] - OpenHPC with Slurm<br />
* [[VEGA Cluster]]<br />
* [[WALLER Cluster]]<br />
* [[ICSE Cluster Transition]]<br />
<br />
===General Documentation===<br />
*[[Rules for Creating Passwords]]<br />
*[[Linux Tutorial]]<br />
*[[Connect to Linux]]<br />
*[[Linux Tips and Tricks]]<br />
*[[FAQ|Troubleshooting]]<br />
*[[Slurm]]<br />
*[[Modules (Lmod)]]</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2913Singularity2019-09-04T22:04:11Z<p>Pzv2: Added intro</p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
[https://en.wikipedia.org/wiki/Singularity_(software) Singularity] is a [https://en.wikipedia.org/wiki/Containerization containerization] technology, similar to [[Docker]], [https://singularity.lbl.gov/about originally developed] at Lawrence Berkeley National Lab with the requirements of scientific computing and high performance computing (HPC) in mind, especially [https://sylabs.io/guides/3.3/user-guide/security.html security]. A company called [https://sylabs.io/ Sylabs] has since been started and continues to maintain and update Singularity (see the [https://github.com/sylabs/singularity GitHub Repository]). In addition to this page, you may want to explore the [https://sylabs.io/guides/3.3/user-guide/ official documentation].<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Recipes ===<br />
<br />
=== Images ===<br />
<br />
<br />
[https://github.com/sylabs/singularity GitHub Repository]<br />
[https://singularity-hub.org/ SingularityHub]<br />
<br />
<br />
== Using Singularity ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===<br />
<br />
<br />
== Creating Images ===</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2912Singularity2019-09-04T21:48:41Z<p>Pzv2: </p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
Intro<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
=== Recipes ===<br />
<br />
=== Images ===<br />
<br />
<br />
[https://github.com/sylabs/singularity GitHub Repository]<br />
[https://singularity-hub.org/ SingularityHub]<br />
<br />
<br />
== Using Singularity ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===<br />
<br />
<br />
== Creating Images ===</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Docker&diff=2911Docker2019-09-04T21:48:33Z<p>Pzv2: Started page</p>
<hr />
<div>'''This page is under construction'''<br />
<br />
Intro<br />
<br />
Only available in [Red Cloud], not available <br />
<br />
__TOC__<br />
<br />
<br />
== Getting Started ==<br />
<br />
=== Installation ===<br />
<br />
<br />
<br />
=== Using A Docker Image ===<br />
<br />
<br />
=== Useful Commands ===<br />
<br />
<br />
<br />
== Creating a Dockerfile ==</div>Pzv2https://www.cac.cornell.edu/wiki/index.php?title=Singularity&diff=2910Singularity2019-09-04T21:23:35Z<p>Pzv2: Started page</p>
<hr />
<div>'''This page is currently under construction'''<br />
<br />
Intro<br />
<br />
__TOC__<br />
<br />
== Getting Started ==<br />
<br />
<br />
<br />
== Using Singularity Images ==<br />
<br />
=== On CAC Private Clusters ===<br />
<br />
<br />
<br />
=== On Red Cloud ===</div>Pzv2