splitbrain.org

electronic brain surgery since 2001

Off-Site Backup with Borgmatic

As the saying goes “RAID is not a backup”. So I needed a backup solution for my new NAS. Some of my files are already synced with my Google drive1) but I needed proper backup for the rest2).

For my personal server setup, I had decided on Borg and borgmatic. So that's what I did for my NAS as well. Here's how.

Hetzner Storage Box

The first question is where to store the backup data. I picked Hetzner's storage box which provides good value for a low price and has all the sync options you'd want.

To use it for borg backups, you need two things:

  1. a directory where the borg repo will reside
  2. a SSH key that can connect password-less

My borg backups are to be stored in /home/borg/<hostname>. So I created a directory for my NAS:

ssh -p 23 uXXXXXX@uXXXXXX.your-storagebox.de 'mkdir -p /home/borg/nas'

I will run borgmatic via docker in /data/docker/borgmatic. The ssh key will be mounted as a volume. I decided to create a dedicated key just for the backup:

mkdir -p /data/docker/borgmatic/conf/ssh
ssh-keygen -t ed25519 -C 'borgmatic@nas' -f /data/docker/borgmatic/conf/ssh/id_ed25519

To add it to the authorized_keys entry in the storage box, we can use ssh-copy-id with the -s option to use the SFTP protocol:

ssh-copy-id -i /data/docker/borgmatic/conf/ssh/id_ed25519 -p 23 -s uXXXXXX@uXXXXXX.your-storagebox.de

Borgmatic Docker Compose Setup

I am using the borgmatic-collective Docker image with Docker Compose. The compose.yaml is relatively straight forward. Besides a few mounts for borg to store its caches and such, you need to mount the ssh key directory and a directory where the configuration will reside. And of course the container needs access to all data it should back up.

Simply mount the directories to back up into the /mnt/source/ dir in the container.

borgmatic/compose.yaml
services:
  borgmatic:
    image: ghcr.io/borgmatic-collective/borgmatic:2
    container_name: borgmatic
    restart: unless-stopped
    volumes:
      # mount folders to backup into /mnt/source
      - /data/docker:/mnt/source/docker:ro
      - /data/photo:/mnt/source/photo:ro
      - /home/andi:/mnt/source/andi:ro
      # config      
      - ./conf/borgmatic:/etc/borgmatic.d/
      - ./conf/ssh:/root/.ssh
      # state
      - ./volumes/borg:/root/.config/borg
      - ./volumes/cache:/root/.cache/borg
    environment:
      TZ: Europe/Berlin
      BORG_PASSPHRASE: ${BORG_PASSPHRASE}
      HEALTHCHECKS_PINGURL: ${HEALTHCHECKS_PINGURL}

In conf/borgmatic/ are two files: a crontab.txt to define when the backup should be run and a borgmatic configuration file.

borgmatic/conf/borgmatic/config.yaml
source_directories:
    - /mnt/source/
repositories:
    - path: ssh://u408911@u408911.your-storagebox.de:23/home/borg/nas/repo
      label: hetzner
 
# we disable this to not have to specify all the mounts individually:
one_file_system: false
 
#   Passphase is set in varibable $BORG_PASSPHRASE
#   encryption_passphrase: "DoNotForgetToChangeYourPassphrase"
compression: zlib
archive_name_format: 'backup-{now}'

exclude_patterns:
  - '*/log/*'
  - '*/logs/*'
  - '*.log'
  - '*/cache/*'
  - '*.cache'
 
# Exclude directories that contain a CACHEDIR.TAG file. See
# http://www.brynosaurus.com/cachedir/spec.html for details. Defaults
# to false.
exclude_caches: true
 
# Exclude directories that contain a file with the given filenames.
# Defaults to not set.
exclude_if_present:
  - .nobackup
 
# what to keep
keep_daily: 1
keep_weekly: 1
keep_monthly: 1

checks:
  - name: repository
    frequency: 2 weeks
  - name: archives
    frequency: always
  - name: extract
    frequency: 2 weeks
  - name: data
    frequency: 1 month

apprise:
    states:
        - start
        - finish
        - fail

    services:
        - url: mailto://192.168.1.33?to=splitbrain@gmail.com&from=borgmatic@69b.place
          label: mail

    start:
        title: "⚙️ Backup nas: started"
        body: Starting backup process.

    finish:
        title: "✅ Backup nas: SUCCESS"
        body: Backups successfully made.

    fail:
        title: "❌ Backup nas: FAILED"
        body: Your backups have failed.

healthchecks:
    ping_url: ${HEALTHCHECKS_PINGURL}

Most of the stuff is self explanatory, but you can always reference the manual.

But let me point out a couple of choices I made:

I set one_file_system: false. Because we mount different directories into /mnt/source, each of them is technically a different file system. You could specify each one in the source_directories array, but I prefer to only specify what I want to backup once – in the compose.yaml.

I exclude a couple of typical cache and log file directories and also make use of the .nobackup marker. The latter is useful to exclude a few things in my home directory.

Finally I use two different notification mechanisms to know when the backups fail.

The apprise config sends mails. Currently I let it send mails for all occasions, just to keep an eye on the backup. Once I am sure it works as I want it, I will remove the start and finish states.

The healthchecks entry makes use of the healthchecks.io Service. The nice thing about that service is that it will also send you an alert when no data is received for a while. This will let you know when your cron job is not running for any reason (eg. your container was accidentally stopped). The URL itself is defined as an environment variable.

The very last bit of configuration you need, is an .env containing your backup passphrase. Something like this:

borgmatic/.env
BORG_PASSPHRASE='My secrete passphrase!!1!1'
HEALTHCHECKS_PINGURL='https://hc-ping.com/xxx-xxx-xxx-xxx'

Finally start the container:

cd /data/docker/borgmatic
docker compose up -d

Initialize the Repo

Before the first backup, you need to initialize the borg repo. This is simple.

You need to open a shell within the container and run the init command:

docker compose exec borgmatic /bin/bash
borgmatic init --encryption repokey
exit

The above command will initialize the configured repository using the repokey method. This means the actual encryption key is stored in the repository (at Hetzner). So all you need to decrypt the data is your passphrase. Borg supports a different mode where the encryption key is only stored on the local machine, but then you need to make sure you do backup the key separately. repokey seems simpler to me.

Finally you may want to execute your first backup manually, to check that everything works. Again, do it from a shell within the container:

docker compose up -d
borgmatic --stats -v 1 --files
exit

Verify the Backup

A backup is only useful if you know how to access and restore data from it. Of course you can use the borgmatic container or the borg command line tool to access the data. However to me it was easiest to check if I could access the backup via the Vorta GUI client on my local machine.

yay -S vorta

In Vorta, click the + button in the repository tab, select “existing repo” and add the repo you configured in your borgmatic config. Eg. ssh://uXXXXXX@uXXXXXX.your-storagebox.de:23/home/borg/nas/repo. Give your passphrase in the “password” field.

You should now see your backups in the “Archives” tab. Select one and use the “Mount…” button to mount it to a local directory. You now can browse that directory and copy files from it. That is verification enough for me that the backup is working.

Tags:
backup, borg, borgmatic, linux, howto
Similar posts:
1)
Let me know if you're interested in that setup
2)
actually not all of it, some data is not backed up again because it is either replaceable, like music or movies, or is already a backup of data located on a different machine

Comments