# -*- mode: org-mode; truncate-lines: nil -*- #+TITLE: Systems documentation #+AUTHOR: Fripost -- the Free E-mail Association #+DESCRIPTION: Systems documentation for Fripost, the Free E-mail Association #+KEYWORDS: #+LANGUAGE: en #+OPTIONS: H:3 num:t toc:t \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t #+OPTIONS: TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc #+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js #+EXPORT_SELECT_TAGS: export #+EXPORT_EXCLUDE_TAGS: noexport #+LINK_UP: #+LINK_HOME: #+XSLT: #+DRAWERS: HIDDEN STATE PROPERTIES CONTENT Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts and no Back-Cover Texts. A copy of the license is included in a separate file called "COPYING". This is the documentation of the server configuration used by the free e-mail association, given here to provide a transparent system. Debian GNU/Linux lenny is the target system. The complete documentation is the actual configuration files on the servers. This document intends to give a general idea of the setup and be of help if we need to recreate a crashed server. Also, if an administrator goes AWOL, it should be easy to pick up where he left of. The steps taken here will not necessarily give a perfect replica of our systems. We are constantly (yes, constantly) working on improving the security and reliability of our systems. We do not think of security as a shoot and forget sort of thing but instead as an ongoing effort. Thus, while we strive to document all configuration that we consider stable enough, the documentation may sometimes lag behind. We welcome all criticism, suggestions for improvements, additions etc. Please send them to skangas@skangas.se. * BASIC SETUP -- Checklist after having installed a new Debian GNU/Linux-server - Do not install any "tasks" during installation (web server etc.). - If using expert install, you might want to choose to install "Base system". - Make sure to answer "yes" to shadow passwords and MD5. - Disable root account. ** Install etckeeper Used to keep track of /etc. Install ASAP after install! - /etc/etckeeper/etckeeper.conf AVOID_COMMIT_BEFORE_INSTALL=1 - cd /etc && sudo etckeeper init && sudo etckeeper commit "first commit" ** Uninstall a bunch of unnecessary packages sudo aptitude remove --purge debian-faq dictionaries-common doc-debian \ doc-linux-text iamerican ibritish ispell laptop-detect nfs-common \ openbsd-inetd portmap tasksel tasksel-data w3m ** Packages to install *** Administrative - sudo aptitude install openssh-server molly-guard ntp ntpdate screen If the system is on a dynamic IP (e.g. using DHCP): - sudo aptitude install resolvconf *** Security - sudo aptitude install logcheck syslog-summary harden-servers NB: harden-clients conflicts with telnet, which as we know is very handy during configuration. Therefore, optionally: - sudo aptitude install harden-clients ** Configure sshd First, make sure you have put your private key in ~/.ssh/authorized_keys2 - /etc/ssh/sshd_config :HIDDEN: # Add relevant users here AllowUsers xx yy zz # Change these settings PermitRootLogin no PasswordAuthentication no X11Forwarding no :END: - /etc/init.d/ssh restart Without closing the current connection, try to connect to the server, verifying that you can still connect. ** Configure sudo If you disabled root account during installation, the default account is already in the sudo group. Otherwise, follow these steps: - Add relevant users to the sudo group - EDITOR="emacs" sudo visudo %sudo ALL= (ALL) ALL ** Configure logcheck - sudo aptitude install logcheck syslog-summary - /etc/logcheck/logcheck.conf INTRO=0 SENDMAILTO="skangas@skangas.se" - /etc/logcheck/ignore.d.server/ntp :HIDDEN: - ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync (disabled|enabled) [0-9]+$ + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync (disabled|enabled|status( change)?) [0-9]+$ :END: - /etc/logcheck/ignore.d.server/ssh [until logcheck 1.3.7 hits stable] :HIDDEN: + ^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ sshd\[[[:digit:]]+\]: Received disconnect from [:[:xdigit:].]+: [[:digit:]]+: disconnected by user$ :END: - /etc/logcheck/ignore.d.server/rsyslog [until rsyslog 4.2.0-2 hits stable] :HIDDEN: ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel:( \[[[:digit:]]+\.[[:digit:]]+\])? imklog [0-9.]+, log source = /proc/kmsg started.$ ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ rsyslogd: \[origin software="rsyslogd" swVersion="[0-9.]+" x-pid="[0-9]+" x-info="http://www.rsyslog.com"\] restart$ ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: Kernel logging (proc) stopped.$ :END: - /etc/logcheck/ignore.d.server/ddclient :HIDDEN: + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ddclient\[[0-9]+\]: WARNING: file /var/cache/ddclient/ddclient.cache, line [0-9]+: Invalid Value for keyword 'ip' = ''$ + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ddclient\[[0-9]+\]: WARNING: updating [._[:alnum:]-]+: nochg: No update required; unnecessary attempts to change to the current address are considered abusive$ :END: ** Configuring aptitude and friends We're going for a setup where we install many security updates automatically using the package "unattended-upgrades". Automated upgrades are in general not a very good idea, but "unattended-upgrades" takes steps to mitigate the problems with this kind of setup. Given the Debian security teams track record in recent years we believe the positives outweigh the negatives. For the situations when unattended-upgrades fails (e.g. when there are configuration changes), we should e-mail the administrator. We will be using apticron to do this until the version of unattended-upgrades in stable supports mailing when an upgrade fails (the one in unstable does). - sudo aptitude install apticron unattended-upgrades - /etc/apt/apt.conf :CONTENT: // Limit download speed //Acquire::http::Dl-Limit "70"; /* Unsupported in the version of unattended-upgrades that is in stable, * but will later send an e-mail when an upgrade fails. * Until this works in stable, we will use apticron. */ //Unattended-Upgrade::Mail "skangas@skangas.se"; APT { // Increase cache size to some arbitrary size. // Remove this line completely once we have apt v0.7.26 in stable. (it defaults to no limit) Cache-Limit "33554432"; // Configuration for /etc/cron.daily/apt Periodic { // Do "apt-get update" automatically every n-days (0=disable) Update-Package-Lists "1"; // Do "apt-get autoclean" every n-days (0=disable) AutocleanInterval "1"; // Do "apt-get upgrade --download-only" every n-days (0=disable) Download-Upgradeable-Packages "1"; // Run the "unattended-upgrade" security upgrade script every n days Unattended-Upgrade "1"; } }; Aptitude { UI { Autoclean-After-Update: true; Auto-Fix-Broken: false; Keep-Recommends: true; Recommends-Important: true; Description-Visible-By-Default: false; HelpBar false; Menubar-Autohide true; Purge-Unused: true; Prompt-On-Exit false; } } :END: - /etc/apticron/apticron.conf EMAIL="skangas@skangas.se" * NEXT STEPS ** Configuring the backup solution General idea [[http://wikis.sun.com/display/BigAdmin/Using+rdist+rsync+with+sudo+for+remote+updating][from here]]. This is just a basic setup for now, will need to be changed to rsnapshot or perhaps something even more sophisticated like bacula. 1. Install rsync - sudo aptitude install rsync 2. Create a key on the backup computer - ssh-keygen -N "" -b 4096 -f ~/.ssh/backup_key - cat .ssh/backup_key.pub 3. Create a user on the computer that will be backed up - sudo adduser remupd - sudo passwd -d remupd - add the public key from above to ~remupd/.ssh/authorized_keys2 prefix with: no-X11-forwarding,no-agent-forwarding,no-port-forwarding - test the key: ssh -i ~/.ssh/backup_key -l remupd example.com - add remupd to sudo: Cmnd_Alias RSYNCDIST=/usr/bin/rsync remupd ALL=NOPASSWD:RSYNCDIST 3. Create a script on the backup computer to automatically backup 4. Add script to crontab ** Configuring the e-mail servers *** Introduction **** Overview We will be using one main mail storage server, accessible by users via IMAP. This server should be referred to as the main `IMAP server'. We will have two or more mail gateways that will relay e-mail to the main server over secure connections. These are called `smarthosts'. The main server will also be responsible for keeping all users in an MySQL database that will be replicated using MySQL. **** Definitions IMAP server = the main storage server smarthost = the server receiving email from the internet (configured as MX) *** Configuring an SSH tunnel between two hosts Definitions: originating host = the host that will be connecting destination host = the host that runs some service Begin by setting a few environment variables: TUNNEL_KEY="my_tunnel_key" TUNNEL_USER="tunneluser" TUNNEL_HOME="/home/$TUNNEL_USER" DEST_PORT="25" ORIGIN_PORT="1917" **** Prepare origin 1. Create a key on the originating host: sudo ssh-keygen -N "" -b 4096 -f /root/.ssh/$TUNNEL_KEY sudo cat /root/.ssh/$TUNNEL_KEY.pub **** Prepare destination 2a. Install necessary software on the destination host: sudo aptitude install netcat-openbsd 2b. Create a new user on the destination host: sudo adduser --home=$TUNNEL_HOME --shell=`type rbash|cut -d' ' -f3` \ --disabled-password $TUNNEL_USER echo "exit" | sudo -u $TUNNEL_USER tee $TUNNEL_HOME/.bash_profile # Also, make sure to add this user to AllowUsers in /etc/ssh/sshd_config. # Note: We need bash, so we can not change the shell to something else. 2c. Add the public key from above to this user: THE_PUBLIC_KEY="ssh-rsa xxxxxxxxxxx" sudo -u $TUNNEL_USER mkdir $TUNNEL_HOME/.ssh echo "command=\"nc localhost $DEST_PORT\",no-X11-forwarding,no-agent-forwarding,\ no-port-forwarding $THE_PUBLIC_KEY" | sudo -u $TUNNEL_USER tee $TUNNEL_HOME/.ssh/authorized_keys2 **** Set up the tunnel 4. Test the key on the originating host: sudo ssh -v -l $TUNNEL_USER -i /root/.ssh/$TUNNEL_KEY destination.example.com 5. Configure openbsd-inetd on the originating host: # Comment: We use inetd instead of ssh -L because, among other things, ssh # -L tends to hang. sudo aptitude install openbsd-inetd - /etc/inetd.conf :HIDDEN: 127.0.0.1:$ORIGIN_PORT stream tcp nowait root /usr/bin/ssh -q -T -i /root/.ssh/tunnel_key smtptunnel@example.com :END: sudo /etc/init.d/openbsd-inetd restart You should now be able to connect through the tunnel from the originating host using something like: telnet localhost $ORIGIN_PORT *** Basic configuration of MySQL **** Installing MySQL - sudo apt-get install mysql-server - generate a long (25 characters) password for the mysql root user - /etc/mysql/my.cnf :HIDDEN: skip-innodb :END: **** MySQL on the main IMAP server - create database mail; We will use four tables `alias', `domain', `log' and `mailbox'. // FIXME; add description of tables :HIDDEN: mysql> show tables; mysql> describe alias; mysql> describe domain; mysql> describe log; mysql> describe mailbox; :END: - sudo mysql -u root -p --database=mail :HIDDEN: DROP TABLE IF EXISTS `alias`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `alias` ( `address` varchar(255) NOT NULL default '', `goto` text NOT NULL, `domain` varchar(255) NOT NULL default '', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `change_date` datetime NOT NULL default '0000-00-00 00:00:00', `active` tinyint(4) NOT NULL default '1', PRIMARY KEY (`address`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Aliases - mysql_virtual_\nalias_maps'; SET character_set_client = @saved_cs_client; DROP TABLE IF EXISTS `domain`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `domain` ( `domain` varchar(255) NOT NULL default '', `description` varchar(255) NOT NULL default '', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `change_date` datetime NOT NULL default '0000-00-00 00:00:00', `active` tinyint(4) NOT NULL default '1', PRIMARY KEY (`domain`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Domains - mysql_virtual_\ndomains_maps'; SET character_set_client = @saved_cs_client; DROP TABLE IF EXISTS `log`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `log` ( `id` int(11) NOT NULL auto_increment, `user` varchar(20) NOT NULL default '', `event` text NOT NULL, `date` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, PRIMARY KEY (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=106 DEFAULT CHARSET=utf8 COMMENT='log table'; SET character_set_client = @saved_cs_client; DROP TABLE IF EXISTS `mailbox`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `mailbox` ( `username` varchar(255) NOT NULL default '', `password` varchar(255) NOT NULL default '', `name` varchar(255) NOT NULL default '', `maildir` varchar(255) NOT NULL default '', `domain` varchar(255) NOT NULL default '', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `change_date` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `active` tinyint(4) NOT NULL default '1', PRIMARY KEY (`username`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Mailboxes - mysql_virtua\nl_mailbox_maps'; SET character_set_client = @saved_cs_client; :END: **** Configuring the MySQL replication [[http://dev.mysql.com/doc/refman/5.0/en/replication.html][MySQL 5.0 Reference Manual :: 16 Replication]] We will use MySQL replication to keep the MySQL user data on the smarthosts in sync with the data held on the main IMAP server. We begin by setting up an SSH tunnel. This process is described above. The rest is fairly straight-forward (instructions below adapted from [[http://www.howtoforge.com/mysql_database_replication][here]]). - Set up the SSH tunnel. ***** Configure the master - Add this to my.cnf: :HIDDEN: server-id = 1 log_bin = /var/log/mysql/mysql-bin.log expire_logs_days = 10 max_binlog_size = 100M binlog_do_db = mail :END: - /etc/init.d/mysql restart - Enter MySQL shell and create user with replication privileges: mysql -u root -p When in shell, do the following (replace with something better): create database mail; GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY ''; FLUSH PRIVILEGES; # Is this only needed when using "load data from master"? grant reload, super, replication client on *.* to 'slave_user'; USE mail; FLUSH TABLES WITH READ LOCK; SHOW MASTER STATUS; unlock tables; quit; ***** Configure the slave - Enter the MySQL shell and create the database: mysql -u root -p Enter password: CREATE DATABASE mail; quit; - /etc/mysql/my.cnf :HIDDEN: tmpdir = /var/lib/mysql/tmp # Note that the server-id must be different on all hosts server-id = 2 master-host = 127.0.0.1 master-port = 1949 master-user = slave_user master-password = master-connect-retry = 60 replicate-do-db = mail :END: - create the temporary directory: mkdir /var/lib/mysql/tmp chown mysql:mysql !$ chmod 0750 !$ - /etc/init.d/mysql restart - Enter the MySQL shell and make the replication: mysql -u root -p Enter password: LOAD DATA FROM MASTER; quit; A strange bug bit me at this point, notes: start slave; stop slave; show slave status\G [[http://dev.mysql.com/doc/refman/5.0/en/change-master-to.html][12.5.2.1. CHANGE MASTER TO Syntax]] CHANGE MASTER TO MASTER_PORT=1949, MASTER_CONNECT_RETRY=60; *** Configuring the main IMAP server to receive e-mail to Maildir First setup the tables like above. - sudo apt-get install maildrop - /etc/postfix/main.cf :HIDDEN: # Not really needed until we switch to using Courier maildrop maildrop_destination_recipient_limit = 1 virtual_mailbox_base = /home/mail/virtual :END: - sudo mkdir -p /home/mail/virtual/fripost.org/example/ - sudo maildirmake /home/mail/virtual/fripost.org/example/Maildir - mysql -u root -p INSERT INTO mailbox (username,password,name,maildir,domain) VALUES ('exempel@fripost.org','test666','Exempelanvändare','fripost.org/exempel/Maildir/','fripost.org'); - /etc/init.d/postfix restart Now it should work to send an e-mail to exempel@fripost.org *** Configuring a new smarthost to relay e-mail to the main IMAP server Definitions: IMAP server = the main storage server smarthost = the receiving server (configured as MX) First setup an SSH tunnel between the hosts according to instructions given above in this document. Next, you need to configure postfix on the smarthost to relay emails through the tunnel: One quick-and-dirty example to try it out is: - /etc/postfix/main.cf relay_domains = fripost.org transport_maps = hash:/etc/postfix/transport - /etc/postfix/transport fripost.org smtp:localhost:1917 - sudo postmap hash:/etc/postfix/transport *** Setting up dovecot - sudo apt-get install dovecot - /etc/dovecot/dovecot.conf Note: These settings are already in the file but commented out or set to other values. :HIDDEN: protocols = imaps protocol imap { ssl_listen = *:993 } disable_plaintext_auth = yes mail_location = maildir:/home/mail/virtual/%d/%u/Maildir # Set this to something that works for the Maildirs first_valid_uid = XXX first_valid_gid = XXX # Allow clients to be fancy if they want to mechanisms = plain cram-md5 #passdb pam <--- comment this stuff out # uncomment this stuff passdb sql { args = /etc/dovecot/dovecot-sql.conf } #userdb passwd <--- comment this stuff out # uncomment this stuff userdb sql { args = /etc/dovecot/dovecot-sql.conf } # Do not needlessly run as root user = nobody :END: - /etc/dovecot/dovecot-sql.conf :HIDDEN: driver = mysql connect = host=127.0.0.1 port=3306 user=XXX password=XXX dbname=mail # Salted MD5 default_pass_scheme = SMD5 password_query = SELECT username AS user, password FROM mailbox WHERE username = '%u' AND domain = '%d' # replace XXX with relevant numbers for the system user_query = SELECT concat('/home/mail/virtual/',maildir) AS mail, XXX AS uid, XXX AS gid FROM mailbox WHERE username = '%u' AND domain = '%d' :END: - sudo /etc/init.d/dovecot restart After adding a user, you should be able to login using any IMAP client. ** Configuring the webserver - sudo apt-get install apache2 ** Necessary stuff to fix for security *** Firewall rules TODO: Add nice rules. ** Ideas for improved security *** Increased rate of backups when the IMAP server goes down *** Bacula for backups Also has tripwire-like capabilities. *** Some kind of IDS *** Monitoring * NEED TO KNOW FOR SERVER ADMINS ** Use etckeeper We keep /etc in a git repository using the tool etckeeper. This means that every time you make changes to any files in /etc, you are expected to commit them using a descriptive commit message. Please add a signature (initials or your username) since all commits will be made as root. $ etckeeper commit "postfix: enable to relay messages to remote hosts via smtp /skangas" If you do not commit your changes, the next system upgrade will fail and whoever makes the upgrade will have to commit your changes for you. They may have to guess as to why you made your changes. Please do not put your co-administrators in this uncomfortable position. It is also possible to use simple git commands in /etc, e.g. `git log'. `etckeeper' has the benefit of keeping track of file permissions, which git by itself will not. *** Warn when /etc has uncommitted changes on logout echo "sudo etckeeper unclean && echo \"WARNING: You have uncommitted changes in /etc\" && sudo git diff" >> ~/.bash_logout