Backup

Workshop

In this workshop, we will create a bash script to run as a cron job for our system backups. We have installed and run restic for one-off backups and we would like to schedule them on a daily basis. The backup takes place between anarchaserver and a remote server, which is the backup repository. But it can be also applied between our PC and a raspberry-pi or any home based file server.

Meeting Sunday 04 April 2021 - 3PM CET Time here: https://bbb.futuretic.fr/b/spi-utv-eo9-lqe

En este taller, vamos a crear un script bash para ejecutar como un trabajo cron para nuestras copias de seguridad del sistema. Hemos instalado y ejecutado restic para copias de seguridad puntuales y nos gustaría programarlas diariamente. La copia de seguridad tiene lugar entre anarchaserver y un servidor remoto, que es el repositorio de la copia de seguridad. Pero también se puede aplicar entre nuestro PC y una raspberry-pi o un servidor de archivos casero.

Cita el Domingo 04 de Abril - 3PM CET time aqui: https://bbb.futuretic.fr/b/spi-utv-eo9-lqe

Current setup

Now the backup of Binti is done with a script that relies on Restic tool

We used to do backups with duplicity but we opted for restic as it is easier to install than duplicity. However duplicity has more options on how to fine tune backups.

Simplest tool for backups is rsync (https://rsync.samba.org/) which copies files, which can be archived or not, and checks if a file has been modified, or is removed or created and updates the backup target repo accordingly. But it doesn't create snapshots and thus we cannot access the state of a system's files of a specific date in the past. Dedicated backup tools create snapshots and retain a history of the changes, so we can choose what version to restore. However we need to be aware of the backup's size storage and remove older snapshots.

Steps we follow to backup AS LXC containers

Note: all system paths and parameters in this HOWTO are fictional.

Process

• stop the containers (because AS containers were created and managed by root user, they need root privlege to stop them. We solved this by allowing the specific lxc stop/start commands for the backup user via the sudoers.d configuration.)
• backup the containers
• restart the containers

System setup

• a backup 'user'
• ssh keys to access remote backup repo
• password for restic backup command
• bash script with the necessary commands to run the process above
• cron job added via the backup user to run the bash script on regular intervals

Configuration of the system setup

1. How to access the remote backup repo?

• create new ssh keys for ssh access to the remote backup repo, and give directory flag for saving the keys under the desired directory.
• scp the new pub key in the remote backup repo, under the remote user's home .ssh/authorized_keys. Remote user is the one we use to ssh to the remote machine.

2. How to run the backup command without root privelege

when some of the filesystem to be backed-up is accessed only by root, aka execute a binary meant for root without being root? The idea is to execute the restic binary from backup user's home or from /usr/bin.

creates the new user and a group with the same name

useradd backupuser

makes user root and group backupuser owners of the restic binary

chown root:backupuser /usr/bin/restic

user root has now read, write, execute permissions, and users in backupuser group can execute and read the restic binary

chmod 750 /usr/bin/restic

assigns capabilities to backup the whole system

setcap cap_dac_read_search=+ep ~backupuser/bin/restic 

Ref: https://restic.readthedocs.io/en/stable/080_examples.html#backing-up-your-system-without-running-restic-as-root

3. How to run specific root commands by a non root user?

Note: we needed that for stopping, checking status and starting of the lxc containers We should run our LXC containers rootless. This requires to change configuration of containers, lxc config files (to be checked/implemented in the future, tasks to do for AS, to be added in tasks in gitlab; Mika can you create a git issue for this too?)

Ref: https://www.cyberciti.biz/faq/how-to-create-unprivileged-linux-containers-on-ubuntu-linux/

But for now we will give the backupuser restricted root privelege for the specific lxc commands we need to run in the backup script.

usermod -aG sudo backupuser

Give the backup user the acces for specific commands to be executed as 'root'. Add these commands in a new file under /etc/sudoers.d/

vi /etc/sudoers.d/00-backupuser 
bintibackup ALL=(ALL) NOPASSWD: /usr/bin/lxc-stop, /usr/bin/lxc-start, /usr/bin/lxc-info, /usr/bin/lxc-ls

Ref: https://www.cyberciti.biz/faq/linux-unix-running-sudo-command-without-a-password/

++++

chown root:bintibackup /usr/bin/lxc-stop 

> this is for changing the property to usr/bin/lxc-stop to root group and bintibackup

first, we are trying to make this happen with a bintibackup user with no root privilege - if it does no work, will switch to possibility to give the bintibackup user root provileges, but this will imply to have a vert strict security management policy for that userthis above scratched steps didn't work! ++++

One line command to run backups from terminal

Needs the following parameters:

path to password file:

 PASSWORD="~backupuser/pass"

absolute path to the remote backup repo:

 remote-repo ="/var/backups"

abosulte path to the directory we want to backup:

 local-repo="/var/foo"

remote host:

 $host="backups.org"
 $user="backuphost"

So the command becomes:

restic -p ~/backup/pass -r sftp:backuphost@backups.org:/var/backups --verbose backup /var/foo

OR:

restic -p $PASSWORD -r sftp:$user@$host:$remote-path --verbose backup

Final bash script (the exact paths haave been changed to fictional ones)

#!/bin/bash
# a backup to our backup server of the LXC containers

set -e
PASSWORD=/root/mypassword_enter
# Destination
DEST="sftp:binti-backups:/server/backups"
STATIC_OPTIONS="--verbose backup"
ROOT="/var/lib/lxc"
# list containers
CONTAINERS="$(sudo lxc-ls)"
declare -a STOPPED_CONTAINERS

# Check if containers are running and then stop the containers
echo $containers
for container in ${CONTAINERS[@]}; do
    echo $container
    status="$(sudo lxc-info $container | grep 'State' | xargs | cut -d' ' -f2)"
    echo $status
    if [[ $status == "STOPPED" ]]; then
        STOPPED_CONTAINERS+=("$container")
    elif [[ $status == "RUNNING" ]]; then
        sudo lxc-stop $container
        # Update the list of stopped containers
        STOPPED_CONTAINERS+=("$container")
    fi
done
set +e
restic -p $PASSWORD -r  $DEST $STATIC_OPTIONS $ROOT

BACKUP_SUCCESS=$?
set -e

if [[ $BACKUP_SUCCESS -eq 0 ]]; then
    echo "BACKUP succeeded"
else
    # Send email to admins
    echo "BACKUP failed"
    sendemail admin@backups.org < ./mail.txt
fi

for container in ${STOPPED_CONTAINERS[@]}; do
        echo $container
        sudo lxc-start $container
done

Resources for backup scripts and helpful tips

restic for backups

• https://restic.readthedocs.io/en/stable/040_backup.html

duplicity

• https://blog.xmatthias.com/duplicity_backup_script/

What to backup

• https://www.debian.org/doc/manuals/debian-reference/ch10.en.html#_backup_and_recovery

Read password from a file:

• https://www.foxinfotech.in/2019/03/reading-a-password-from-a-file-in-linux.html

Old setup

We currently make a nightly backup of all containers from our virtual machine (called anoia) to the virtual machine of calafou.org (called ebro).

We use the backup tool called duplicity through a package from the Debian repository.

We wrote a script that performs that backup, which is located at /usr/local/bin/backup3.sh

The backup script runs every night around 4am, because we run it via cron (configured with sudo crontab -e command).

In the cron job we redirect the script output to /var/log/backup.log

The script stops ALL the containers (to have enough RAM for running duplicity and to make sure that the databases and other volatile files inside the containers are not changing during the backup), backs up the containers, then starts the containers again. Obviously, while the containers are stopped and the backup is running, the anarchaserver services are not available. We have many ideas about how to do a backup without service interruption, but we have not implemented them yet.

There are many possible improvements to this system:

• Change remote: switch from calafou virtual machine to S14
• Optimise the script: rewrite the script to be a more beautiful program
• Test backup recovery: try to restore the backup to make sure it is possible
• Change the backup tool: use borgbackup instead of duplicity
• Use a backup manager such as backupninja
• Etc.

We also have a S14 for setting up the tunnel for backup and openvpn, info about back-tunnel here.

Restore a backup

duplicity scp://anoia@ebro.tachanka.org//home/anoia/backup/repository /var/lib/lxc --ssh-options="-oIdentityFile=/var/backups/.ssh/id_binti" --verbosity 6

Useful links

• How to back up a Debian system using backupninja? http://xmodulo.com/backup-debian-system-backupninja.html
• Using Duplicity to backup LXC containers https://www.savjee.be/2017/05/Using-Duplicity-to-Backup-LXC-Containers/
• Implement a backup system of livingdata and repository containers (gitlab issue on systerserver) https://git.systerserver.net/collective-anarchaserver/organizing/issues/1
• Roadmap https://git.systerserver.net/collective-anarchaserver/organizing/issues/1

Older notes

STEP 1 check data usage df -h

STEP 2 check access to another virtual machine

STEP 3 start screen session in root

STEP 4 copy data to virtual machine

rsync -ravz --progress --exclude "/proc" --exclude "/sys" --exclude "/dev" -e "ssh -p 8022" / root@188.210.92.35:/var/backups/vm/anarcha

r (recursive) a (archive - for special files) v (verbose) z (compressed)

STEP 5 diagnostics

watch watch -n 180 "du -sh /var/backups/vm/anarcha"

proc/kcore is a special directory which is created each time the computer boots so we have to exclude it from the rsync command

bwm-ng we want to know whether the data is arriving and at which speed, data amount http://linux.die.net/man/1/bwm-ng

apt-get install bwm-ng