Moving to new machine binti
Installation of the new machine "binti" in new server
Anoia > Anna (US) Moving to Binti > Ursula (SE)
Binti as the main character of a science fiction novella written by Nnedi Okorafor (2015). https://en.wikipedia.org/wiki/Binti_(novel)
We have setup a virtual server (KVM) on our host server for you:
Host server: ursula.tachanka.org Virtual server: binti.tachanka.org Storage: 120G Memory: 4096M CPU: 2
Network configuration for your virtual server: Public IP: 198.167.222.149/32 (we did not put 32 cause was not accepted, only 198.167.222.149) Gateway: 198.167.222.1 Host name: binti DNS Server: 9.9.9.10 Domain name: anarchaserver.org
ssh binti@ursula.tachanka.org
Steps to do:
Install Debian Stable
Set up full disk encryption in your virtual machine during the installation, and keep encrypted backups of the passphrase as well as backups of any important data, as we do not keep backups of your data by default.
Add ssh keys inside binti
Next steps:
Install and configure some security things : ssh server iptable + ufw fail2ban things like chkrootkit rkhunter etckeeper ? configure an everyday mail report sent to sysadmins ? Install and configure some softwares on the host : apache2 LXC
Reinstall AS by testing current back up system > shutting donw anoia and passing it over to tachanka
Documentation:
Add new tech documentation Reframe current wiki page: https://alexandria.anarchaserver.org/index.php/Machine
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
b01 : yes, I could add the keys in/au /home/binti/.sshthorized_keys I can connect to the server with ssh binti@ursula.tachanka.org
Once connected with ssh binti@ursula.tachanka.org it is possible to reach the installation screen with :
screen -x
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Installation of debian stable 10 buster
Network configuration for your virtual server: IP: 198.167.222.149/32 (we did not put 32 cause was not accepted, only 198.167.222.149) Gateway: 198.167.222.1 Host name: binti DNS Server: 9.9.9.10 Domain name: anarchaserver.org
B01 creates a new password for root in binti > 40 characters and will share with sysadmin over GPG
A user Anarcha is created for non administrative tasks inside the server > b01 creates a new password > 40 characters and will share with sysadmin over GPG
Setting up full disk encryption in Binti during the installation - create a swap of 8 GB ? Pass phrase is created for encryption > b01 creates a new password > 40 characters and will share with sysadmin over GPG
We answer yes to be part of the debian package survey
Grub created and located in /dev/vda/
We could decrypt and log in binti with passwords created
Put the LUKS passphrase when the VM boot
Connect to the virtual screen session to see the boot process (an admin must have put your public key in the rights place)
ssh binti@ursula.tachanka.org
You should see a prompt that ask for the LUKS passphrase, put it and then to detach from the screen session :
"C-\ d" to detach
Improve the security
ssh
basic sshd_config same as previous :
PasswordAuthentication no X11Forwarding no PermitRootLogin no
Cammbiamos puerto al 2212, ahora nos logueamos con
ssh -p 2212 user@binti.tachanka.org
Folder ssh keys of AS sysadmin added inside binti Process: Created users for all syadmin, add each at sudo group and added related SSH key (the one used for anoia) so you can access the server, from there you can decide to keep old key or to generate a new one for binti as described here https://alexandria.anarchaserver.org/index.php/Access_server
unattended upgrades
Activar unattended upgrades > se hace con el script que actualiza todas los containers mejor, hay que configurarlo, no?
sudo apt install unattended-upgrades sudo apt install apt-listchanges Unattended-Upgrade::Mail "root"; Unattended-Upgrade::MailOnlyOnError "false";
Si queremos que nos llege un mail cada dia con las actualizaciones que se hicieron hay que configurar un servicio de correo (no lo hicimos)
En /etc/apt/apt.conf.d/20auto-upgrades:
APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended-Upgrade "1";
fail2ban
Activar fail2ban:
apt-get install fail2ban cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
borramos las jails que no usamos
Activamos las jails: sshd , apache y apache badbots (dejamos para investigar como protejer los containers)
Firewall
ufw ufw default deny incoming ufw default allow outgoing ufw allow 80/tcp ufw allow 443/tcp ufw allow 2212/tcp -> ufw allow 8000/tcp
Security -References
Resources on security and devops/sysadmin:
- https://dev.to/shosta/security-headers-to-use-on-your-webserver-3id5
- https://content-security-policy.com/
- http://opsreportcard.com/section/11
- https://the-sysadmin-book.com/
About unattended upgrades
- https://haydenjames.io/how-to-enable-unattended-upgrades-on-ubuntu-debian/
- https://wiki.debian.org/UnattendedUpgrades
Fail2ban
- https://www.a2hosting.com/kb/security/hardening-a-server-with-fail2ban
- https://labekka.red/servidoras-feministas/2019/10/09/fanzine-parte-4.html#os
UFW
Apache2 web server
Installation of apache2 server
apt install apache2
Specific configuration : add some modules
a2enmod ssl a2enmod rewrite a2enmod proxy_http a2ensite > to activate the imported virtual hosts
Some changes has been made to increase the MaxWorkers numbers, see this article
Find which Multi-Processing Module (MPM) is currently in use:
apache2ctl -V | grep MPM Server MPM: event
Edit the appropriated file :
nano /etc/apache2/mods-enabled/mpm_event.conf
<IfModule mpm_event_module>
StartServers 3
MinSpareThreads 5
MaxSpareThreads 10
ServerLimit 250
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 1250
MaxConnectionsPerChild 0
</IfModule>
Check the numbers of connexions (to check if the server is not under DDoS attack) :
netstat -an | egrep ':80|:443' | grep ESTABLISHED | awk '{print $5}' | grep -o -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -n | uniq -c | sort -nr
Install LXC containers
Installing the base of LXC following :
Restoration of the container from the previous machine with duplicity
Config of the containers with this type of config (at the end, the most basic one), there has been changes of the config file with debian 10 buster :
nano /var/lib/lxc/livingdata/config lxc.net.0.type = veth lxc.net.0.hwaddr = 00:16:3e:ff:1f:df lxc.net.0.link = virbr0 lxc.net.0.flags = up lxc.apparmor.profile = generated lxc.apparmor.allow_nesting = 1 lxc.rootfs.path = /var/lib/lxc/livingdata/rootfs # Common configuration lxc.include = /usr/share/lxc/config/debian.common.conf # Container specific configuration lxc.tty.max = 4 lxc.uts.name = livingdata lxc.arch = amd64 lxc.pty.max = 1024 lxc.start.auto = 1
is working and giving adresse 192.168.122.*
vhost apache proxy to the containers
Restoration of the apache2 vhost to proxy the traffic to the containers, exemple of transition subdomain / container :
nano /etc/apache2/sites-enabled/transitional.conf
<VirtualHost *:80>
ServerAdmin webmaster@localhost
ServerName transitional.anarchaserver.org
ErrorLog ${APACHE_LOG_DIR}/transitional-error.log
CustomLog ${APACHE_LOG_DIR}/transitional-access.log combined
ProxyPreserveHost On
ProxyRequests Off
ProxyPass / http://192.168.x.x/
ProxyPassReverse http://192.168.x.x/ /
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
RewriteEngine on
RewriteCond %{SERVER_NAME} =transitional.anarchaserver.org
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>nano /etc/apache2/sites-enabled/transitional-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerAdmin webmaster@localhost
ServerName transitional.anarchaserver.org
ErrorLog ${APACHE_LOG_DIR}/transitional-error.log
CustomLog ${APACHE_LOG_DIR}/transitional-access.log combined
ProxyPreserveHost On
ProxyRequests Off
ProxyPass / https://transitional.anarchaserver.org/
ProxyPassReverse / https://transitional.anarchaserver.org/
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
SSLEngine on
SSLProxyEngine On
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
RewriteEngine on
SSLCertificateFile /etc/letsencrypt/live/anarchaserver.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/anarchaserver.org/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
# vim: syntax=apache ts=4 sw=4 sts=4 sr noet
</IfModule>Add the ip of the container in /etc/hosts if the ssl proxy is a dns :
nano /etc/hosts 192.168.x.x transitional.anarchaserver.org
SSL certificates
Utilisation of certbot with gandi plug-in (our registrar) to create and update the certificates for the main domain and all subdomains (wildcard) : One certificate for all the subdomains
- Certbot wildcard tab : https://certbot.eff.org/lets-encrypt/debianbuster-apache
- Plug in Gandi for automatic renew https://github.com/obynio/certbot-plugin-gandi
- How to get the Gandi API key https://doc.livedns.gandi.net/
Install Gandi plug-in on Debian 10 :
apt remove python-cryptography pip install cryptography pip install certbot-plugin-gandi
Activate mod ssl for Apache :
a2enmod ssl systemctl restart apache2
Create and Install the certificates :
certbot certonly -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini -d anarchaserver.org -d "*.anarchaserver.org" --server https://acme-v02.api.letsencrypt.org/directory
To renew them automatically, and restart apache2 add in crontab as root :
crontab -e 0 0 * * 0 /usr/local/bin/certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory --post-hook "systemctl restart apache2"
Don't forget to restart apache
systemctl restart apache2
Also remove the Timer driven by systemd (installed by default with certbot threw the package mamager) to let the cron job do the renew process :
- List Existing Timers
systemctl list-timers systemctl list-timers --all
- remove the certbot.timer and disable certbot.service
rm /etc/systemd/system/timers.target.wants/certbot.timer systemctl disable certbot.service
- Check if the crontab works
crontab -e cat /var/mail/root (a mail is sent each time the cronjob is done tail -f /var/log/letsencrypt/letsencrypt.log
If certbot: error: unrecognized arguments: --certbot-plugin-gandi:dns-credentials so the crontab doesn't use the good certbot version, to check :
# certbot --version -bash: /usr/local/bin/certbot: No such file or directory <- Former location of certbot 0.35 installed via pip # which certbot /usr/bin/certbot <- Location of certbot 0.28 installed with apt # whereis certbot certbot: /usr/bin/certbot /usr/share/man/man1/certbot.1.gz # echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
In anarchaserver case, the cron job was using the bad version of certbot (/usr/bin/certbot was throwing the error), let's remove them :
apt remove certbot python3-certbot
Indicate full path for the good certbotin the cronjob
*/5 * * * * /usr/local/bin/certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /root/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory
Maybe we should add a hook to restart apache ?
Netdata monitoring and alert system
Installation of Netdata to monitor the server activity and to send alert once something is not working, now available at https://netdata.anarchaserver.org
bash <(curl -Ss https://my-netdata.io/kickstart.sh) --stable-channel --disable-telemetry
- Proxy behind existing apache server https://learn.netdata.cloud/docs/agent/running-behind-apache
- Edit config https://learn.netdata.cloud/docs/agent/configuration-guide
/etc/netdata/edit-config health_alarm_notify.conf
- Configuration of the alerts https://learn.netdata.cloud/docs/agent/health/notifications/ at the moment configured to send alert to b01 by mail
To restric access to netdata with an authetification Utilisation of auth Digest with apache :
a2enmod auth_digest
Create a user
htdigest -c /etc/apache2/.htpasswd netdata netdata
Add this conf to the vhost netdata-le-ssl.conf
<Proxy *>
AuthType Digest
AuthName "netdata"
AuthDigestProvider file
AuthUserFile /etc/apache2/.htpasswd
Require valid-user
</Proxy>force redirection to https in the other vhost
Serveur Sonification
It uses promiscuous~ from Pure Data / ext13 and create a stream > serveur icecast 2
apt-get install puredata pd-pdogg pd-ext13 icecast2
Pour v2, ajouter
apt install pd-ggee pd-zexy pd-iemlib
The PD patch : anarchasonification.pd and to check the audio device :
puredata -nogui -listdev
To launch the patch :
puredata -nogui anarchasonification.pd
The result can be listened here : http://anarchaserver.org:8000/anarchasonification.ogg.m3u
To launch the patch at the server start-up with systemctl
nano /lib/systemd/system/anarchasonification.service
[Unit] Description=Server Sonification net activity [Service] Type=simple ExecStart=/usr/bin/pd -nogui -noaudio /data/pnode/algo/pnodelocalenetsonification.pd StandardOutput=null Restart=on-failure # Configures the time to wait before service is stopped forcefully. TimeoutStopSec=30 [Install] WantedBy=multi-user.target Alias=serversonification.service
take into account the modification int the service
sudo systemctl daemon-reload
launch it
sudo systemctl start omnioussonification.service
activate it at start-up
sudo systemctl enable omnioussonification.service
