Building Container Images

Overview

Teaching: 10 min
Exercises: 25 min

Questions

• How do I create my own Apptainer images?

Objectives

• Understand the different Apptainer container file formats.

• Understand how to build and share your own Apptainer containers.

So far you’ve been able to work with Apptainer from your own user account as a non-privileged user.

This part of the Apptainer material requires that you use Apptainer in an environment where you have administrative (root) access.

There are a couple of different ways to work around this restriction.

We’ll focus on the last option in this part of the course - Simulate root access using the --fakeroot feature.

Building container via Slurm

The new Mahuika Extension nodes can be used to build Apptainer containers using the fakeroot feature. This functionality is only available on these nodes at the moment due to their operating system version.

Since the Mahuika Extension nodes are not directly accessable, you will have to create a slurm script. Create a new file called build.sh and enter the following.

#!/bin/bash -e
#SBATCH --job-name=apptainer_build
#SBATCH --partition=milan
#SBATCH --time=0-00:15:00
#SBATCH --mem=4GB
#SBATCH --cpus-per-task=2

module purge
module load Apptainer

apptainer build --fakeroot my_container.sif my_container.def

Submit your new script with, The module purge command will remove unnecessary modules we may have loaded that would interfere with our build.

[username@mahuika01]$ sbatch build.sh

Submitted batch job 33031078

You can check the status of your job using sacct

[username@mahuika01]$ sacct -X

JobID           JobName          Alloc     Elapsed     TotalCPU  ReqMem   MaxRSS State      
--------------- ---------------- ----- ----------- ------------ ------- -------- ---------- 
33031074        spawner-jupyter+     2    00:27:29     00:00:00      4G          RUNNING    
33031277        apptainer_build      2    00:00:06     00:00:00      4G          RUNNING    

Note, the first job shown there is your Jupyter session.

Once the job is finished you should see the built container file my_container.sif.

[username@mahuika01]$ ls

apptainer_cache  apptainer_tmp  build.sh  lolcow_latest.sif  my_container.def  my_container.sif  python-3.9.6.sif slurm-33031491.out ubuntu_latest.sif

Note the slurm output slurm-33031491.out, if you don’t have my_container.sif you will want to check here first.

We can test our new container by running.

module load Apptainer
./my_container.sif

Hello World! Hello from our custom Apptainer image!

We can also inspect our new container, confirm everything looks as it should.

[username@mahuika01]$ apptainer inspect my_container.sif

org.label-schema.build-arch: amd64
org.label-schema.build-date: Friday_17_February_2023_11:52:9_NZDT
org.label-schema.schema-version: 1.0
org.label-schema.usage.apptainer.version: 1.1.5-dirty
org.label-schema.usage.singularity.deffile.bootstrap: docker
org.label-schema.usage.singularity.deffile.from: ubuntu:20.04
org.opencontainers.image.ref.name: ubuntu
org.opencontainers.image.version: 20.04

[username@mahuika01]$ apptainer inspect ir my_container.sif

{: .language-bash}

#!/bin/sh

python3 -c 'print("Hello World! Hello from our custom Apptaine image!")' ```

Known limitations

This method, ( i.e using fakeroot), is known to not work for all types of Apptainer/Singularity containers.

If your container uses RPM to install packages, i.e. is based on CentOS or Rocky Linux, you need to disable the APPTAINER_TMPDIR environment variable (use unset APPTAINER_TMPDIR) and request more memory for your Slurm job. Otherwise, RPM will crash due to an incompatibility with the nobackup filesystem.

Remote build

Apptainer remote builder

Currently there are no freely available remote builders for Apptainer.

Apptainer offers the option to run build remotely, using a Remote Builder we will be using the default provided by Sylabs; You will need a Sylabs account and a token to use this feature.

[username@mahuika01]$ apptainer remote login

Generate an API Key at https://cloud.sylabs.io/auth/tokens, and paste here:
API Key:

Now paste the token you had copied to the clipboard end press Enter:

INFO:    API Key Verified!

With this set up, you may use apptainer build -r to start the remote build. Once finished, the image will be downloaded so that it’s ready to use:

[username@mahuika01]$ apptainer build -r lolcow_remote.sif lolcow.def

INFO:    Remote "default" added.
INFO:    Authenticating with remote: default
INFO:    API Key Verified!
INFO:    Remote "default" now in use.
INFO:    Starting build...
[..]
INFO:    Running post scriptlet
[..]
INFO:    Adding help info
INFO:    Adding labels
INFO:    Adding environment to container
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: /tmp/image-699539270
WARNING: Skipping container verifying
 67.07 MiB / 67.07 MiB  100.00% 14.18 MiB/s 4s

At the time of writing, when using the Remote Builder you won’t be able to use the %files header in the def file, to copy host files into the image.

You are now ready to push your image to the Cloud Library, e.g. via apptainer push:

[username@mahuika01]$ apptainer push -U lolcow.sif library://<YOUR-SYLABS-USERNAME>/default/lolcow:30oct19

WARNING: Skipping container verifying
 67.08 MiB / 67.08 MiB [==================================================================================================================================] 100.00% 6.37 MiB/s 10s

Note the use of the flag -U to allow pushing unsigned containers (see further down).
Also note once again the format for the registry: //:.

Finally, you (or other peers) are now able to pull your image from the Cloud Library:

[username@mahuika01]$ apptainer pull -U library://<YOUR-SYLABS-USERNAME>/default/lolcow:30oct19

INFO:    Downloading library image
 67.07 MiB / 67.07 MiB [===================================================================================================================================] 100.00% 8.10 MiB/s 8s
WARNING: Skipping container verification
INFO:    Download complete: lolcow_30oct19.sif

Key Points

• Apptainer definition files are used to define the build process and configuration for an image.

• Apptainer’s Docker container provides a way to build images on a platform where Apptainer is not installed but Docker is available.

• Existing images from remote registries such as Docker Hub and Singularity Hub can be used as a base for creating new Apptainer images.