You will learn how to transfer data to and from Kupe.
Transferring small amounts of data
For small amounts of data, SSH based utilities such as SecureCopy (
scp) can be used.
scp should be readily available on a terminal on Linux and MacOS, or via WinSCP and MobaXterm on Windows. Note that two-factor authentication will be required for file transfer sessions.
The lander node filesystem is distinct from the Kupe filesystem, and so copying a file from your desktop to Kupe is by default a two-step process. If you are using an OpenSSH based ssh, which will be the case for most MacOSX and Linux users, then this inconvenience can be avoided by setting up an SSH config file such as:
Host kupe User your_username Hostname login.kupe.niwa.co.nz ProxyCommand ssh -W %h:%p lander.nesi.org.nz
With that file present at
~/.ssh/config, the command
ssh kupe should take you there directly, jumping across the lander node on the way, and
scp will do the same for your file transfers.
There is also the problem of having to repeatedly provide both authentication factors every time you use
scp. The following method allows passwordless
scp while you have an
ssh session open, but it does bring some complications and so is not recommended unless you often use
scp. First create the directory
~/.ssh/sockets on your machine, and then set up your
ControlPath ~/.ssh/sockets/%r@%h:%p Host kupe User your_username Hostname login.kupe.niwa.co.nz ProxyCommand ssh -W %h:%p lander.nesi.org.nz ControlMaster auto ControlPersist 1
For full details on this see https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing
Transferring large amounts of data (500GB or more)
Instructions on how to use Globus Data Transfer Node (DTN)
Globus provides a fast file transfer service that is suitable for large data volumes. Globus requires two endpoints, on at your institution, and on the HPC. Data transfer sessions can be set up and monitored on the Globus webpage. Globus also provides APIs:
There is a Software Development Kit (SDK) in Python to work with the API.
Please note: Currently DTN is not yet connected to Kupe. If you have access to Fitzroy and wish to use Globus to transfer files, you can use SCP (via Fitzroy) to copy them to/from Kupe.