scuff-em in the cloud: Running scuff-em calculations on Amazon AWS Nodes

Although many nontrivial scuff-em calculations---including all of the tutorial examples in the scuff-em documentation---can be run quickly (few minutes or less) on a typical laptop or desktop workstation, eventually you may want to run some larger-scale calculations. For example,

  • you may want to crank up the resolution of the surface meshes in a scattering problem to get high numerical precision, or

  • you may want to [run large-scale parameter sweeps][ParameterSweep] involving thousands of scuff-em calculations at various values of geometrical or other parameters, or

  • you may be computing quantities that require both [frequency and Brillouin-zone integration][HeatTransferProblem], thus involving thousands of scuff-em calculations at various frequencies and Bloch vectors.

A very convenient way to run large-scale calculations without building your own supercomputer is to use an on-demand cloud-computing resource offering pay-as-you-go access to remote compute nodes. One example of such a resource is Amazon Web Services (AWS), where you can buy a few hour's worth of access to a quite powerful server at inexpensive prices. For example, as discussed below, the cost of a machine with 16 Haswell cores and 30 GB memory can be as low as $0.10/hour (10 cents per hour). (This is using AWS spot instances, for which current pricing information is available here.)

Launch an AWS instance with the Simpetus Platform

The very first time you ever do this you will need to spend about 10 minutes going through a totally straightforward procedure to set up an AWS account and "launch an EC2 instance running the Simpetus Electromagnetic Simulation Platform." I put this in quotes for the benefit of cloud-computing neophytes (like me!) to communicate that you don't have to understand all the jargon here, or to follow precisely what is happening---all you need to know is that, at the end of this process, a server will have been magically created somewhere in the cloud for you (and only for you, with no other users) with pre-installed versions of scuff-em and other modeling tools like meep and mpb. You will have an IP address that you can use to log in to this server, upload files to this server, run calculations on this server, and download the results. Once you're finished, you will "terminate the EC2 instance", whereupon, poof! The server instantly disintegrates back into the primordial soup of the cloud, taking with it all files you uploaded or that were produced by your calculations. You will be charged for only the time elapsed between launch and termination.

The entire procedure for launching your AWS instance is carefully explained in this series of follow-along tutorial walkthrough videos on the Simpetus youtube channel. If you are already familiar with things like SSH keypairs you can probably start with this video, which will get you up and running with an EC2 instance.

In what follows I will refer to the EC2 instance created in this way as your "cloud server."

Upload and run initialization script on cloud server

Once your cloud server is running, the next step is to upload and run the following bash script:

This script fetches the latest version of scuff-em from the scuff-em GitHub page and builds it on the EC2 instance with appropriate compilation options. (As a bonus, the script also fetches and installs the awesome openblas package for automatically architecture-optimized linear algebra, significantly accelerating large-scale scuff-em calculations.)

Assuming you are logging in to your EC2 instance from a terminal on your Linux or Mac laptop, the process of uploading and running the script looks like this.

1. Copy script from your local machine to the cloud server

 mylaptop% scp -i keypair.pem InitializeSCUFFonAWS.sh ubuntu@123.45.67.89:

Here 123.45.67.89 is the IP address of the server you launched, and keypair.pem is the SSH key-pair file you will have created in the launching process. Note the colon (:) at the far right end of the command line!

Of course, you could also use a GUI-based file-transfer tool like winscp to upload the script from Windows without using a terminal.

2. Login to cloud server

mylaptop % ssh -i keypair.pem ubuntu@123.45.67.89

Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-91-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Wed Oct 12 01:30:34 UTC 2016

...
...
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@ip-172-31-0-191:~$

3. Run initialization script on cloud server

ubuntu@ip-172-31-0-191:~$ . InitializeSCUFFOnAWS.sh

*
* Building OpenBLAS...
*
...
...
...
make[1]: Leaving directory `/home/ubuntu/scuff-em'

*
* SUCCESS building SCUFF-EM 
*
26 ip-172-31-0-191 /home/ubuntu/scuff-em <>

This script takes about 10 minutes to run (on a node with 16 CPU cores). Once it is finished, the scuff-em installation will be present starting at your root filesystem, i.e.

  • the binary executable programs will be in ${HOME}/bin

  • the examples will be in ${HOME}/share/scuff-em,

etc.

Also, the script adds ${HOME}/bin to your path, so that you can run scuff-em commands from any directory without needing to specify the ${HOME}/bin prefix.

If you would like to run any of the scuff-em examples mentioned in this tutorial, you will want to execute the following additional steps:

4. Download (to your laptop or desktop) the following archive file containing all meshes, geometry files, and other data needed to run the tutorial examples:

5. From your laptop, upload this file to the cloud server: 

 mylaptop% scp -i keypair.pem SCUFFTutorial.tar.bz2 .sh ubuntu@123.45.67.89:

6. Unpack this file on the cloud server: 

290 ip-172-31-3-53 /tmp <> bunzip2 -c SCUFFTutorial.tar.bz2 | tar xvf -
SCUFFTutorial/
SCUFFTutorial/SpiralArrays/
SCUFFTutorial/SpiralArrays/ScreenTransformations
SCUFFTutorial/SpiralArrays/MakeSpiralArray.jl
...
SCUFFTutorial/PolarizationSensitiveAntenna/geoFiles/MakeMeshes
SCUFFTutorial/PolarizationSensitiveAntenna/geoFiles/Triangle.geo

Now the various subdirectories referenced by the discussions in this tutorial will be present on your cloud server in the SCUFFTutorial folder.

Special instructions for BU symposium participants

For today's symposium I've prepared several AWS machines on which the latest scuff-em/buff-em/ versions and the input files for today's tutorials are pre-installed. Here are instructions for how to access these machines:

  1. Install gmsh on your laptop.

    • Windows/Mac users: Download installer from gmsh website.

    • Ubuntu users: 

 % apt-get install gmsh

  1. Windows users: Install SSH/SCP clients like PuTTY and WinSCP. You will also need the PuTTY key generator, PuTTYgen.

  2. Download private key file for BU symposium AWS machines:

4a. Windows users: Use PuTTYgen to convert seoul-keypair.pem to PuTTY format. If you have trouble, here is some guidance: Connecting to your Linux Instance from Windows using PuTTY.

4b. Linux/Mac users: Set correct access permissions for the private key file:

 % chmod 400 seoul-keypair.pem
  1. Log on the cloud server using PuTTY (Windows users) or using ssh (Linux/Mac users):
mylaptop % ssh -Y -i seoul-keypair.pem ubuntu@XX.XX.XX.XX

288 ip-172-31-3-53 /home/ubuntu

Here XX.XX.XX.XX is the IP address you were assigned for today's tutorial.