NeSI Filesystem

Overview

Teaching: 15 min
Exercises: 5 min

Questions

• Where is the best place to store my data?

• How do I recover deleted files?

• How do I find out how much disk space I have?

Objectives

• Learn about the NeSI filesystems, and when to use each one.

The NeSI filesystem looks something like this:

The directories that are relevant to us are.

Managing your data and storage (backups and quotas)

NeSI performs backups of the /home and /nesi/project (persistent) filesystems. However, backups are only captured once per day. So, if you edit or change code or data and then immediately delete it, it likely cannot be recovered. Note, as the name suggests, NeSI does not backup the /nesi/nobackup filesystem.

Protecting critical data from corruption or deletion is primarily your responsibility. Ensure you have a data management plan and stick to the plan to reduce the chance of data loss. Also important is managing your storage quota. To check your quotas, use the nn_storage_quota command, eg

$ nn_storage_quota
Quota Location                    Available         Used      Use%     State       Inodes        IUsed     IUse%    IState
home_johndoe                            20G       14.51G    72.57%        OK      1000000       112084    11.21%        OK
project_nesi99991                      100G         101G    101.00%       LOCKED  100000           194     0.19%        OK
nobackup_nesi99991                      10T            0     0.00%        OK      1000000           14     0.00%        OK

As well as disk space, ‘inodes’ are also tracked, this is the number of files.

Notice that the project space for this user is over quota and has been locked, meaning no more data can be added. When your space is locked you will need to move or remove data. Also note that none of the nobackup space is being used. Likely data from project can be moved to nobackup. nn_storage_quota uses cached data, and so will no immediately show changes to storage use.

For more details on our persistent and nobackup storage systems, including data retention and the nobackup autodelete schedule, please see our Filesystem and Quota documentation.

Working Directory

We will be working from the directory introhpc2410.

[yourUsername@mahuika ~]$ cd /nesi/nobackup/nesi99991/introhpc2410

Creating directories

As previously mentioned, it is general useful to organise your work in a hierarchical file structure to make managing and finding files easier. It is also is especially important when working within a shared directory with colleagues, such as a project, to minimise the chance of accidentally affecting your colleagues work. So for this workshop you will each make a directory using the mkdir command within the workshops directory for you to personally work from.

[yourUsername@mahuika ~]$ mkdir <username>

You should then be able to see your new directory is there using ls.

[yourUsername@mahuika ~]$ ls /nesi/nobackup/nesi99991/introhpc2410

 sum_matrix.r   usr123  usr345

Create a text file

Now let’s create a file. To do this we will use a text editor called Nano to create a file called draft.txt:

We will want to do this from inside the directory we just created.

[yourUsername@mahuika ~]$ cd <username>
[yourUsername@mahuika ~]$ nano draft.txt

Which Editor?

When we say, ‘nano is a text editor’ we really do mean ‘text’: it can only work with plain character data, not tables, images, or any other human-friendly media. We use it in examples because it is one of the least complex text editors. However, because of this trait, it may not be powerful enough or flexible enough for the work you need to do after this workshop. On Unix systems (such as Linux and macOS), many programmers use Emacs or Vim (both of which require more time to learn), or a graphical editor such as Gedit. On Windows, you may wish to use Notepad++. Windows also has a built-in editor called notepad that can be run from the command line in the same way as nano for the purposes of this lesson.

No matter what editor you use, you will need to know where it searches for and saves files. If you start it from the shell, it will (probably) use your current working directory as its default location. If you use your computer’s start menu, it may want to save files in your desktop or documents directory instead. You can change this by navigating to another directory the first time you ‘Save As…’

Let’s type in a few lines of text. Once we’re happy with our text, we can press Ctrl+O (press the Ctrl or Control key and, while holding it down, press the O key) to write our data to disk (we’ll be asked what file we want to save this to: press Return to accept the suggested default of draft.txt).

Once our file is saved, we can use Ctrl+X to quit the editor and return to the shell.

Control, Ctrl, or ^ Key

The Control key is also called the ‘Ctrl’ key. There are various ways in which using the Control key may be described. For example, you may see an instruction to press the Control key and, while holding it down, press the X key, described as any of:

• Control-X
• Control+X
• Ctrl-X
• Ctrl+X
• ^X
• C-x

In nano, along the bottom of the screen you’ll see ^G Get Help ^O WriteOut. This means that you can use Control-G to get help and Control-O to save your file.

nano doesn’t leave any output on the screen after it exits, but ls now shows that we have created a file called draft.txt:

[yourUsername@mahuika ~]$ ls

draft.txt

Copying files and directories

In a future lesson, we will be running the R script /nesi/nobackup/nesi99991/introhpc2410/sum_matrix.r, but as we can’t all work on the same file at once you will need to take your own copy. This can be done with the copy command cp, at least two arguments are needed the file (or directory) you want to copy, and the directory (or file) where you want the copy to be created. We will be copying the file into the directory we made previously, as this should be your current directory the second argument can be a simple ..

[yourUsername@mahuika ~]$ cp /nesi/nobackup/nesi99991/introhpc2410/sum_matrix.r  .

We can check that it did the right thing using ls

[yourUsername@mahuika ~]$ ls

draft.txt   sum_matrix.r 

Key Points