# -*- mode: org-mode; truncate-lines: nil -*- #+TITLE: Systems documentation #+AUTHOR: Fripost -- the Free E-mail Association #+DESCRIPTION: Systems documentation for Fripost, the Free E-mail Association #+KEYWORDS: #+LANGUAGE: en #+OPTIONS: H:3 num:t toc:t \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t #+OPTIONS: TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc #+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js #+EXPORT_SELECT_TAGS: export #+EXPORT_EXCLUDE_TAGS: noexport #+LINK_UP: #+LINK_HOME: #+XSLT: #+DRAWERS: HIDDEN STATE PROPERTIES CONTENT #+STARTUP: indent Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts and no Back-Cover Texts. A copy of the license is included in a separate file called "COPYING". This is the documentation of the server configuration used by the free e-mail association, given here to provide a transparent system. Debian GNU/Linux lenny is the current target system. The complete documentation is the actual configuration files on the servers. This document intends to give a general idea of the setup and be of help if we need to recreate a crashed server. Also, if an administrator goes AWOL, it should be easy to pick up where he left of. The steps taken here will not necessarily give a perfect replica of our systems. We are constantly (yes, constantly) working on improving the security and reliability of our systems. We do not think of security as a shoot and forget sort of thing but instead as an ongoing effort. Thus, while we strive to document all configuration that we consider stable enough, the documentation may sometimes lag behind. We welcome all criticism, suggestions for improvements, additions etc. Please send them to skangas@skangas.se. * BASIC SETUP -- Checklist after having installed a new Debian GNU/Linux-server ** Basic installation instructions - Use expert install to maximize fun. - Preferably, only install the "Standard system utilities" and "SSH Server" tasks. - Make sure to answer "yes" to shadow passwords and MD5. - Do disable the root account. ** Install etckeeper Used to keep track of /etc. Install ASAP after install! :: /etc/etckeeper/etckeeper.conf AVOID_COMMIT_BEFORE_INSTALL=1 # not needed on squeeze: cd /etc && sudo etckeeper init && sudo etckeeper commit "first commit" ** Uninstall a bunch of unnecessary packages sudo aptitude remove --purge debian-faq dictionaries-common doc-debian \ doc-linux-text iamerican ibritish ispell laptop-detect nfs-common \ openbsd-inetd portmap tasksel tasksel-data w3m ** Packages to install *** Administrative sudo aptitude install openssh-server molly-guard ntp ntpdate screen # If the system is on a dynamic IP (e.g. using DHCP): sudo aptitude install resolvconf *** Security sudo aptitude install logcheck syslog-summary harden-servers # NB: harden-clients conflicts with telnet, which as we know is very handy # during configuration. Therefore, only optionally: sudo aptitude install harden-clients ** Configure sshd Make sure your private key is in ~/.ssh/authorized_keys2 :: /etc/ssh/sshd_config # Add relevant users here AllowUsers xx yy zz # Change these settings PermitRootLogin no PasswordAuthentication no X11Forwarding no sudo /etc/init.d/ssh restart # Without closing the current connection, try to connect to the server, # verifying that you can still connect. ** Configure sudo # If you disabled root account during installation, the default account is # already in the sudo group. Otherwise, follow these steps: sudo adduser myuser sudo sudo EDITOR="emacs" visudo %sudo ALL= (ALL) ALL ** Configure logcheck sudo aptitude install logcheck syslog-summary :: /etc/logcheck/logcheck.conf INTRO=0 SENDMAILTO="skangas@skangas.se" :: /etc/logcheck/ignore.d.server/ntp - ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync (disabled|enabled) [0-9]+$ + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync (disabled|enabled|status( change)?) [0-9]+$ :: /etc/logcheck/ignore.d.server/ssh [until logcheck 1.3.7 hits stable] + ^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ sshd\[[[:digit:]]+\]: Received disconnect from [:[:xdigit:].]+: [[:digit:]]+: disconnected by user$ :: /etc/logcheck/ignore.d.server/rsyslog [until rsyslog 4.2.0-2 hits stable] ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel:( \[[[:digit:]]+\.[[:digit:]]+\])? imklog [0-9.]+, log source = /proc/kmsg started.$ ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ rsyslogd: \[origin software="rsyslogd" swVersion="[0-9.]+" x-pid="[0-9]+" x-info="http://www.rsyslog.com"\] restart$ ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: Kernel logging \(proc\) stopped.$ /etc/logcheck/ignore.d.server/ddclient :HIDDEN: + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ddclient\[[0-9]+\]: WARNING: file /var/cache/ddclient/ddclient.cache, line [0-9]+: Invalid Value for keyword 'ip' = ''$ + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ddclient\[[0-9]+\]: WARNING: updating [._[:alnum:]-]+: nochg: No update required; unnecessary attempts to change to the current address are considered abusive$ :END: ** Configuring aptitude and friends # We are going to automatically install many security updates using the package # "unattended-upgrades". Automated upgrades are in general not a very good # idea, but "unattended-upgrades" takes steps to mitigate the problems with this # approach. Given the Debian security teams track record in recent years we # believe the positives outweigh the negatives. # # For the situations when unattended-upgrades fails (e.g. when there are # configuration changes), there is an e-mail sent to the administrator. # sudo aptitude install unattended-upgrades :: /etc/apt/apt.conf :CONTENT: APT { // Increase cache size to some arbitrary size. // Remove this line completely once we have apt v0.7.26 in stable. (it defaults to no limit) Cache-Limit "33554432"; // Configuration for /etc/cron.daily/apt Periodic { // Do "apt-get update" automatically every n-days (0=disable) Update-Package-Lists "1"; // Do "apt-get autoclean" every n-days (0=disable) AutocleanInterval "1"; // Do "apt-get upgrade --download-only" every n-days (0=disable) Download-Upgradeable-Packages "1"; // Run the "unattended-upgrade" security upgrade script every n days Unattended-Upgrade "1"; } }; Aptitude { UI { Autoclean-After-Update: true; Auto-Fix-Broken: false; Keep-Recommends: true; Recommends-Important: true; Description-Visible-By-Default: false; HelpBar false; Menubar-Autohide true; Purge-Unused: true; Prompt-On-Exit false; } } :END: # Using Debian squeeze: :: /etc/apt/apt.conf.d/50unattended-upgrades Unattended-Upgrade::Mail "admin@fripost.org"; Acquire::http::Dl-Limit "70"; # Using Debian lenny: sudo aptitude install apticron :: /etc/apticron/apticron.conf EMAIL="admin@fripost.org" ** Reconfigure exim # FIXME: fix for squeeze sudo dpkg-reconfigure exim4-config # - select "mail sent by smarthost; no local mail" # - hostname: # host.example.com # - listen on: # 127.0.0.1 # - other destinations: # [empty] # - visible domain name: # host.example.com # - address of outgoing smarthost # smtp.bredband.net [or whatever the ISP uses] # - number of DNS queries minimal? # no # - split configuration? # no * NEXT STEPS ** Configuring the backup solution *** Bacula configuration *** Simple rsync solution General idea [[http://wikis.sun.com/display/BigAdmin/Using+rdist+rsync+with+sudo+for+remote+updating][from here]]. This is just a basic setup for now, will need to be changed to rsnapshot or perhaps something even more sophisticated like bacula. 1. Install rsync - sudo aptitude install rsync 2. Create a key on the backup computer - ssh-keygen -N "" -b 4096 -f ~/.ssh/backup_key - cat .ssh/backup_key.pub 3. Create a user on the computer that will be backed up - sudo adduser remupd - sudo passwd -d remupd - add the public key from above to ~remupd/.ssh/authorized_keys2 prefix with: no-X11-forwarding,no-agent-forwarding,no-port-forwarding - test the key: ssh -i ~/.ssh/backup_key -l remupd example.com - add remupd to sudo: Cmnd_Alias RSYNCDIST=/usr/bin/rsync remupd ALL=NOPASSWD:RSYNCDIST 3. Create a script on the backup computer to automatically backup 4. Add script to crontab ** Configuring the e-mail servers *** Introduction **** Overview We will be using one main mail storage server, accessible by users via IMAP. This server should be referred to as the main `IMAP server'. We will have two or more mail gateways that will relay e-mail to the main server over secure connections. These are called `smarthosts'. The main server will also be responsible for keeping all users in an MySQL database that will be replicated using MySQL. **** Definitions IMAP server = the main storage server smarthost = the server receiving email from the internet (configured as MX) *** Configuring an SSH tunnel between two hosts Definitions: originating host = the host that will be connecting destination host = the host that runs some service Begin by setting a few environment variables: TUNNEL_KEY="my_tunnel_key" TUNNEL_USER="tunneluser" TUNNEL_HOME="/home/$TUNNEL_USER" DEST_PORT="25" ORIGIN_PORT="1917" **** Prepare origin 1. Create a key on the originating host: sudo ssh-keygen -N "" -b 4096 -f /root/.ssh/$TUNNEL_KEY sudo cat /root/.ssh/$TUNNEL_KEY.pub **** Prepare destination 2a. Install necessary software on the destination host: sudo aptitude install netcat-openbsd 2b. Create a new user on the destination host: sudo adduser --home=$TUNNEL_HOME --shell=`type rbash|cut -d' ' -f3` \ --disabled-password $TUNNEL_USER echo "exit" | sudo -u $TUNNEL_USER tee -a $TUNNEL_HOME/.bash_profile # Also, make sure to add this user to AllowUsers in /etc/ssh/sshd_config. # Note: We need bash, so we can not change the shell to something else. 2c. Add the public key from above to this user: THE_PUBLIC_KEY="ssh-rsa xxxxxxxxxxx" sudo -u $TUNNEL_USER mkdir -p $TUNNEL_HOME/.ssh echo "command=\"nc localhost $DEST_PORT\",no-X11-forwarding,no-agent-forwarding,\ no-port-forwarding $THE_PUBLIC_KEY" | sudo -u $TUNNEL_USER tee -a $TUNNEL_HOME/.ssh/authorized_keys2 **** Set up the tunnel 4. Test the key on the originating host: sudo ssh -v -l $TUNNEL_USER -i /root/.ssh/$TUNNEL_KEY destination.example.com 5. Configure openbsd-inetd on the originating host: # Comment: We use inetd instead of ssh -L because, among other things, ssh # -L tends to hang. sudo aptitude install openbsd-inetd - /etc/inetd.conf :HIDDEN: 127.0.0.1:$ORIGIN_PORT stream tcp nowait root /usr/bin/ssh -q -T -i /root/.ssh/tunnel_key smtptunnel@example.com :END: sudo /etc/init.d/openbsd-inetd restart You should now be able to connect through the tunnel from the originating host using something like: telnet localhost $ORIGIN_PORT *** Installing MySQL - sudo apt-get install mysql-server - generate a long (25 characters) password for the mysql root user - /etc/mysql/my.cnf: skip-innodb *** MySQL on the main IMAP server **** Overview We will use four tables `alias', `domain', `log' and `mailbox'. ***** mysql> show tables; +----------------+ | Tables_in_mail | +----------------+ | alias | | domain | | log | | mailbox | +----------------+ 4 rows in set (0.00 sec) ***** mysql> describe alias; +-------------+--------------+------+-----+---------------------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------------------+-------+ | address | varchar(255) | NO | PRI | | | | goto | text | NO | | NULL | | | domain | varchar(255) | NO | | | | | create_date | datetime | NO | | 0000-00-00 00:00:00 | | | change_date | timestamp | NO | | CURRENT_TIMESTAMP | | | active | tinyint(4) | NO | | 1 | | +-------------+--------------+------+-----+---------------------+-------+ 6 rows in set (0.00 sec) ***** mysql> describe domain; +-------------+--------------+------+-----+---------------------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------------------+-------+ | domain | varchar(255) | NO | PRI | | | | description | varchar(255) | NO | | | | | create_date | datetime | NO | | 0000-00-00 00:00:00 | | | change_date | timestamp | NO | | CURRENT_TIMESTAMP | | | active | tinyint(4) | NO | | 1 | | +-------------+--------------+------+-----+---------------------+-------+ 5 rows in set (0.00 sec) ***** mysql> describe log; +-------+-------------+------+-----+-------------------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------+-------------+------+-----+-------------------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | user | varchar(20) | NO | | | | | event | text | NO | | NULL | | | date | timestamp | NO | | CURRENT_TIMESTAMP | | +-------+-------------+------+-----+-------------------+----------------+ 4 rows in set (0.00 sec) ***** mysql> describe mailbox; +-------------+--------------+------+-----+---------------------+-------+ | Field | Type | Null | Key | Default | Extra | +-------------+--------------+------+-----+---------------------+-------+ | username | varchar(255) | NO | PRI | | | | password | varchar(255) | NO | | | | | name | varchar(255) | NO | | | | | maildir | varchar(255) | NO | | | | | domain | varchar(255) | NO | | | | | create_date | datetime | NO | | 0000-00-00 00:00:00 | | | change_date | timestamp | NO | | CURRENT_TIMESTAMP | | | active | tinyint(4) | NO | | 1 | | +-------------+--------------+------+-----+---------------------+-------+ 8 rows in set (0.00 sec) **** Steps to produce it mysql -u root -p create database mail; sudo mysql -u root -p --database=mail FIXME: Not 100 % up to date :HIDDEN: DROP TABLE IF EXISTS `alias`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `alias` ( `address` varchar(255) NOT NULL default '', `goto` text NOT NULL, `domain` varchar(255) NOT NULL default '', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `change_date` datetime NOT NULL default '0000-00-00 00:00:00', `active` tinyint(4) NOT NULL default '1', PRIMARY KEY (`address`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Aliases - mysql_virtual_\nalias_maps'; SET character_set_client = @saved_cs_client; DROP TABLE IF EXISTS `domain`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `domain` ( `domain` varchar(255) NOT NULL default '', `description` varchar(255) NOT NULL default '', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `change_date` datetime NOT NULL default '0000-00-00 00:00:00', `active` tinyint(4) NOT NULL default '1', PRIMARY KEY (`domain`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Domains - mysql_virtual_\ndomains_maps'; SET character_set_client = @saved_cs_client; DROP TABLE IF EXISTS `log`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `log` ( `id` int(11) NOT NULL auto_increment, `user` varchar(20) NOT NULL default '', `event` text NOT NULL, `date` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, PRIMARY KEY (`id`) ) ENGINE=MyISAM AUTO_INCREMENT=106 DEFAULT CHARSET=utf8 COMMENT='log table'; SET character_set_client = @saved_cs_client; DROP TABLE IF EXISTS `mailbox`; SET @saved_cs_client = @@character_set_client; SET character_set_client = utf8; CREATE TABLE `mailbox` ( `username` varchar(255) NOT NULL default '', `password` varchar(255) NOT NULL default '', `name` varchar(255) NOT NULL default '', `maildir` varchar(255) NOT NULL default '', `domain` varchar(255) NOT NULL default '', `create_date` datetime NOT NULL default '0000-00-00 00:00:00', `change_date` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP, `active` tinyint(4) NOT NULL default '1', PRIMARY KEY (`username`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Mailboxes - mysql_virtua\nl_mailbox_maps'; SET character_set_client = @saved_cs_client; :END: mysql -u root -p # Create triggers use mail; DELIMITER $$ CREATE TRIGGER alias_set_created_on_insert before insert on alias for each row begin set new.create_date = current_timestamp; end$$ CREATE TRIGGER domain_set_created_on_insert before insert on domain for each row begin set new.create_date = current_timestamp; end$$ CREATE TRIGGER mailbox_set_created_on_insert before insert on mailbox for each row begin set new.create_date = current_timestamp; end$$ DELIMITER ; # Create mail user CREATE USER 'mail'@'localhost' IDENTIFIED BY 'mijhl9hniiMu5WxvvtdgsacxZ'; GRANT SELECT ON mail.alias TO 'mail'@'localhost'; GRANT SELECT ON mail.domain TO 'mail'@'localhost'; GRANT SELECT ON mail.mailbox TO 'mail'@'localhost'; *** Configuring the MySQL replication ***** Overview [[http://dev.mysql.com/doc/refman/5.0/en/replication.html][MySQL 5.0 Reference Manual :: 16 Replication]] We will use MySQL replication to keep the MySQL user data on the smarthosts in sync with the data held on the main IMAP server. These instructions are mainly adapted from the MySQL manual. ***** Configure the master :: /etc/mysql/my.cnf: server-id = 1 log_bin = /var/log/mysql/mysql-bin.log expire_logs_days = 10 max_binlog_size = 100M binlog_do_db = mail /etc/init.d/mysql restart ***** Configure the slave ****** Set up an SSH tunnel We begin by setting up an SSH tunnel from the slave to the master, as described [[Configuring an SSH tunnel between two hosts][above]]. ****** Preparing steps to take on master # Enter MySQL shell and create a user with replication privileges. # NB: Use only ASCII for the mysql -u root -p GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'localhost' IDENTIFIED BY ''; FLUSH PRIVILEGES; USE mail; FLUSH TABLES WITH READ LOCK; quit; # Make a database dump. mysqldump -u root -p --opt mail > mydump.sql # Now, copy this file to the slave. # Save the output of the SHOW MASTER STATUS COMMAND. mysql -u root -p SHOW MASTER STATUS; unlock tables; quit; ****** Slave configuration # Create a new temporary directory. # NOTE: It has to be outside of /tmp so the replication is not screwed up on e.g. power outage. TMP_DIR=/var/lib/mysql/tmp sudo mkdir $TMP_DIR sudo chown mysql:mysql $TMP_DIR sudo chmod 0750 $TMP_DIR :: /etc/mysql/my.cnf tmpdir = /var/lib/mysql/tmp # Note that the server-id must be different on all hosts server-id = 2 /etc/init.d/mysql restart # Enter the MySQL shell and create the database: mysql -u root -p CREATE DATABASE mail; quit; mysql -u root -p --database=mail < mydump.sql # [[http://dev.mysql.com/doc/refman/5.0/en/change-master-to.html][12.5.2.1. CHANGE MASTER TO Syntax]] # NOTE: fill in these values using output from SHOW MASTER STATUS; above # NOTE: filling this in my.cnf is deprecated mysql -u root -p SLAVE STOP; CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_PORT=1949, MASTER_USER='slave_user', MASTER_PASSWORD='', MASTER_LOG_FILE='mysql-bin.000013', MASTER_LOG_POS=98; START SLAVE; show slave status\G # If it seems OK, just: quit; *** Configuring the main IMAP server **** /etc/postfix/main.cf TODO: add file contents **** Test delivery sudo mkdir -p /home/mail/virtual/fripost.org/ mysql -u root -p INSERT INTO mailbox (username,password,name,maildir,domain) VALUES ('exempel@fripost.org','test666','Exempelanvändare','fripost.org/exempel/Maildir/','fripost.org'); sudo /etc/init.d/postfix restart echo "test at `date`"|mail -s "test" exempel@fripostorg **** Configuring dovecot sudo aptitude install dovecot-imapd :: /etc/dovecot/dovecot.conf # Note: These settings are already in the file but commented out or set to other # values. :HIDDEN: protocols = imaps protocol imap { ssl_listen = *:993 } disable_plaintext_auth = yes mail_location = maildir:/home/mail/virtual/%d/%u/Maildir # Set this to something that works for the Maildirs first_valid_uid = XXX first_valid_gid = XXX # Allow clients to be fancy if they want to mechanisms = plain cram-md5 #passdb pam <--- comment this stuff out # uncomment this stuff passdb sql { args = /etc/dovecot/dovecot-sql.conf } #userdb passwd <--- comment this stuff out # uncomment this stuff userdb sql { args = /etc/dovecot/dovecot-sql.conf } # Do not needlessly run as root user = nobody :END: :: /etc/dovecot/dovecot-sql.conf :HIDDEN: driver = mysql connect = host=127.0.0.1 port=3306 user=XXX password=XXX dbname=mail # Salted MD5 default_pass_scheme = SMD5 password_query = SELECT username AS user, password FROM mailbox WHERE username = '%u' AND domain = '%d' # replace XXX with relevant numbers for the system user_query = SELECT concat('/home/mail/virtual/',maildir) AS mail, XXX AS uid, XXX AS gid FROM mailbox WHERE username = '%u' AND domain = '%d' :END: sudo /etc/init.d/dovecot restart # Provided there is a user, you should now be able to login using any IMAP # client. *** Configuring a new smarthost to relay e-mail to the main IMAP server **** Overview We relay mail from our smarthosts to the main IMAP server. This is to avoid having a single poin of failure and to separate concerns. The IMAP server then only needs to deal with authenticated clients and the smarthosts. **** Prerequisites Before this can work we must make sure that: - the MySQL replication is working - there is an SSH tunnel for the smtp If they are both setup, we can configure postfix on the smarthost to relay emails through the tunnel. **** Configuration files TODO: add the necessary configuration files ** Configuring the webserver - sudo apt-get install apache2 ** Logging *** Overview We want to limit how much we log for privacy reasons. At the same time we want to be able to debug technical problems and detect intrusions. For the webmail, we only log messages of priority warn or higher. *** Configuration :: /etc/rsyslog.conf *.*;auth,authpriv.none;mail.err -/var/log/syslog # NOTE: /var/log/mail.{err,warn} can be kept at the default # values since they do not contain any sensitive information. :: /etc/logrotate.d/rsyslog /var/log/mail.log /var/log/mail.info { rotate 3 daily missingok ifempty compress delaycompress sharedscripts postrotate invoke-rc.d rsyslog reload > /dev/null endscript } ** Necessary stuff to fix for security *** Bacula for backups Also has tripwire-like capabilities. *** OSSEC *** Firewall rules TODO: Add nice rules. ** Ideas for improved security *** Monitoring * NEED TO KNOW FOR SERVER ADMINS ** Use etckeeper We keep /etc in a git repository using the tool etckeeper. This means that every time you make changes to any files in /etc, you are expected to commit them using a descriptive commit message. Please add a signature (initials or your username) since all commits will be made as root. $ etckeeper commit "postfix: enable to relay messages to remote hosts via smtp /skangas" If you do not commit your changes, the next system upgrade will fail and whoever makes the upgrade will have to commit your changes for you. They may have to guess as to why you made your changes. Please do not put your co-administrators in this uncomfortable position. It is also possible to use simple git commands in /etc, e.g. `git log'. `etckeeper' has the benefit of keeping track of file permissions, which git by itself will not.