Anarchaserver - User contributions [en]

Backup

2019-12-16T20:56:55Z

Maxigas: /* Current setup */ Make the working less ambiguous

= Current setup =

We currently make a nightly backup of all containers from our virtual machine (called <code>anoia</code>) to the virtual machine of calafou.org (called <code>ebro</code>).

We use the backup tool called <code>duplicity</code> through a package from the Debian repository.

We wrote a script that performs that backup, which is located at <code>/usr/local/bin/backup.sh</code>

The backup script runs every night around 4am, because we run it via <code>cron</code> (configured with <code>sudo crontab -e</code> command).

In the cron job we redirect the script output to <code>/var/log/backup.log</code>

The script stops ALL the containers (to have enough RAM for running <code>duplicity</code> and to make sure that the databases and other volatile files inside the containers are not changing during the backup), backs up the containers, then starts the containers again. Obviously, while the containers are stopped and the backup is running, the anarchaserver services are not available. We have many ideas about how to do a backup without service interruption, but we have not implemented them yet.

There are many possible improvements to this system:

* Change remote: switch from calafou virtual machine to S14
* Optimise the script: rewrite the script to be a more beautiful program
* Test backup recovery: try to restore the backup to make sure it is possible
* Change the backup tool: use <code>borgbackup</code> instead of <code>duplicity</code>
* Use a backup manager such as <code>backupninja</code>
* Etc.

= Useful links =

* How to back up a Debian system using backupninja? http://xmodulo.com/backup-debian-system-backupninja.html
* Using Duplicity to backup LXC containers https://www.savjee.be/2017/05/Using-Duplicity-to-Backup-LXC-Containers/
* Implement a backup system of livingdata and repository containers (gitlab issue on systerserver) https://git.systerserver.net/collective-anarchaserver/organizing/issues/1
* Roadmap https://git.systerserver.net/collective-anarchaserver/organizing/issues/1

= Older notes =

STEP 1
check data usage
df -h

STEP 2
check access to another virtual machine

STEP 3
start screen session in root

STEP 4
copy data to virtual machine

rsync -ravz --progress --exclude "/proc" --exclude "/sys" --exclude "/dev" -e "ssh -p 8022" / root@188.210.92.35:/var/backups/vm/anarcha

r (recursive)
a (archive - for special files)
v (verbose)
z (compressed)

STEP 5
diagnostics

watch
watch -n 180 "du -sh /var/backups/vm/anarcha"

proc/kcore is a special directory which is created each time the computer boots
so we have to exclude it from the rsync command

bwm-ng
we want to know whether the data is arriving and at which speed, data amount
http://linux.die.net/man/1/bwm-ng

apt-get install bwm-ng

Backup

2019-08-13T23:12:45Z

Maxigas: Add useful links

= Current setup =

We currently make a nightly backup of all containers from our virtual machine (called <code>ebro</code>) to the virtual machine of calafou.org (called <code>anoia</code>).

We use the backup tool called <code>duplicity</code> through a package from the Debian repository.

We wrote a script that performs that backup, which is located at <code>/usr/local/bin/backup.sh</code>

The backup script runs every night around 4am, because we have put it in <code>cron</code> (using the <code>sudo crontab -e</code> command).

In the cron job we redirect the script output to <code>/var/log/backup.log</code>

The script stops ALL the containers (to have enough RAM for running <code>duplicity</code> and to make sure that the databases and other volatile files inside the containers are not changing during the backup), backs up the containers, then starts the containers again. Obviously, while the containers are stopped and the backup is running, the anarchaserver services are not available. We have many ideas about how to do a backup without service interruption, but we have not implemented them yet.

There are many possible improvements to this system:

* Change remote: switch from calafou virtual machine to S14
* Optimise the script: rewrite the script to be a more beautiful program
* Test backup recovery: try to restore the backup to make sure it is possible
* Change the backup tool: use <code>borgbackup</code> instead of <code>duplicity</code>
* Use a backup manager such as <code>backupninja</code>
* Etc.

= Useful links =

* How to back up a Debian system using backupninja? http://xmodulo.com/backup-debian-system-backupninja.html
* Using Duplicity to backup LXC containers https://www.savjee.be/2017/05/Using-Duplicity-to-Backup-LXC-Containers/
* Implement a backup system of livingdata and repository containers (gitlab issue on systerserver) https://git.systerserver.net/collective-anarchaserver/organizing/issues/1
* Roadmap https://git.systerserver.net/collective-anarchaserver/organizing/issues/1

= Older notes =

STEP 1
check data usage
df -h

STEP 2
check access to another virtual machine

STEP 3
start screen session in root

STEP 4
copy data to virtual machine

rsync -ravz --progress --exclude "/proc" --exclude "/sys" --exclude "/dev" -e "ssh -p 8022" / root@188.210.92.35:/var/backups/vm/anarcha

r (recursive)
a (archive - for special files)
v (verbose)
z (compressed)

STEP 5
diagnostics

watch
watch -n 180 "du -sh /var/backups/vm/anarcha"

proc/kcore is a special directory which is created each time the computer boots
so we have to exclude it from the rsync command

bwm-ng
we want to know whether the data is arriving and at which speed, data amount
http://linux.die.net/man/1/bwm-ng

apt-get install bwm-ng

Backup

2019-08-13T15:35:15Z

Maxigas: Describe current backup situation

= Current setup =

We currently make a nightly backup of all containers from our virtual machine (called <code>ebro</code>) to the virtual machine of calafou.org (called <code>anoia</code>).

We use the backup tool called <code>duplicity</code> through a package from the Debian repository.

We wrote a script that performs that backup, which is located at <code>/usr/local/bin/backup.sh</code>

The backup script runs every night around 4am, because we have put it in <code>cron</code> (using the <code>sudo crontab -e</code> command).

In the cron job we redirect the script output to <code>/var/log/backup.log</code>

The script stops ALL the containers (to have enough RAM for running <code>duplicity</code> and to make sure that the databases and other volatile files inside the containers are not changing during the backup), backs up the containers, then starts the containers again. Obviously, while the containers are stopped and the backup is running, the anarchaserver services are not available. We have many ideas about how to do a backup without service interruption, but we have not implemented them yet.

There are many possible improvements to this system:

* Change remote: switch from calafou virtual machine to S14
* Optimise the script: rewrite the script to be a more beautiful program
* Test backup recovery: try to restore the backup to make sure it is possible
* Change the backup tool: use <code>borgbackup</code> instead of <code>duplicity</code>
* Use a backup manager such as <code>backupninja</code>
* Etc.

= Older notes =

STEP 1
check data usage
df -h

STEP 2
check access to another virtual machine

STEP 3
start screen session in root

STEP 4
copy data to virtual machine

rsync -ravz --progress --exclude "/proc" --exclude "/sys" --exclude "/dev" -e "ssh -p 8022" / root@188.210.92.35:/var/backups/vm/anarcha

r (recursive)
a (archive - for special files)
v (verbose)
z (compressed)

STEP 5
diagnostics

watch
watch -n 180 "du -sh /var/backups/vm/anarcha"

proc/kcore is a special directory which is created each time the computer boots
so we have to exclude it from the rsync command

bwm-ng
we want to know whether the data is arriving and at which speed, data amount
http://linux.die.net/man/1/bwm-ng

apt-get install bwm-ng