# -*- mode: org-mode; truncate-lines: nil -*-
#+TITLE: Systems documentation
#+AUTHOR: Fripost -- the Free E-mail Association
#+DESCRIPTION: Systems documentation for Fripost, the Free E-mail Association
#+KEYWORDS: 
#+LANGUAGE:  en
#+OPTIONS:   H:3 num:t toc:t \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t
#+OPTIONS:   TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc
#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
#+EXPORT_SELECT_TAGS: export
#+EXPORT_EXCLUDE_TAGS: noexport
#+LINK_UP:   
#+LINK_HOME: 
#+XSLT: 
#+DRAWERS: HIDDEN STATE PROPERTIES CONTENT
#+STARTUP: indent

Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License,
Version 1.3 or any later version published by the Free Software
Foundation; with no Invariant Sections, no Front-Cover Texts and
no Back-Cover Texts.  A copy of the license is included in a
separate file called "COPYING".

This is the documentation of the server configuration used by the free e-mail
association, given here to provide a transparent system.

Debian GNU/Linux lenny is the current target system.

The complete documentation is the actual configuration files on the servers.
This document intends to give a general idea of the setup and be of help if we
need to recreate a crashed server.  Also, if an administrator goes AWOL, it
should be easy to pick up where he left of.

The steps taken here will not necessarily give a perfect replica of our systems.
We are constantly (yes, constantly) working on improving the security and
reliability of our systems.  We do not think of security as a shoot and forget
sort of thing but instead as an ongoing effort.  Thus, while we strive to
document all configuration that we consider stable enough, the documentation may
sometimes lag behind.

We welcome all criticism, suggestions for improvements, additions etc.  Please
send them to skangas@skangas.se.

* BASIC SETUP -- Checklist after having installed a new Debian GNU/Linux-server

** Basic installation instructions

- Use expert install to maximize fun.
- Preferably, only install the "Standard system utilities" and "SSH Server" tasks.
- Make sure to answer "yes" to shadow passwords and MD5.
- Do disable the root account.

** Install etckeeper

Used to keep track of /etc.  Install ASAP after install!

:: /etc/etckeeper/etckeeper.conf

     AVOID_COMMIT_BEFORE_INSTALL=1

# not needed on squeeze:
cd /etc && sudo etckeeper init && sudo etckeeper commit "first commit"

** Uninstall a bunch of unnecessary packages

   sudo aptitude remove --purge debian-faq dictionaries-common doc-debian \
   doc-linux-text iamerican ibritish ispell laptop-detect nfs-common \
   openbsd-inetd portmap tasksel tasksel-data w3m

** Packages to install
*** Administrative

sudo aptitude install openssh-server molly-guard ntp ntpdate screen

# If the system is on a dynamic IP (e.g. using DHCP):
sudo aptitude install resolvconf

*** Security

sudo aptitude install logcheck syslog-summary harden-servers

# NB: harden-clients conflicts with telnet, which as we know is very handy
# during configuration.  Therefore, only optionally:
sudo aptitude install harden-clients

** Configure sshd

Make sure your private key is in ~/.ssh/authorized_keys2

:: /etc/ssh/sshd_config

    # Add relevant users here
    AllowUsers xx yy zz
    
    # Change these settings
    PermitRootLogin no
    PasswordAuthentication no
    X11Forwarding no
    
sudo /etc/init.d/ssh restart
   
# Without closing the current connection, try to connect to the server,
# verifying that you can still connect.
 
** Configure sudo

# If you disabled root account during installation, the default account is
# already in the sudo group.  Otherwise, follow these steps:

sudo adduser myuser sudo

sudo EDITOR="emacs" visudo

     %sudo ALL= (ALL) ALL

** Configure logcheck

sudo aptitude install logcheck syslog-summary

:: /etc/logcheck/logcheck.conf

     INTRO=0
     SENDMAILTO="skangas@skangas.se"

:: /etc/logcheck/ignore.d.server/ntp

    - ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync (disabled|enabled) [0-9]+$
    + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync (disabled|enabled|status( change)?) [0-9]+$
    
:: /etc/logcheck/ignore.d.server/ssh [until logcheck 1.3.7 hits stable]

    + ^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ sshd\[[[:digit:]]+\]: Received disconnect from [:[:xdigit:].]+: [[:digit:]]+: disconnected by user$

:: /etc/logcheck/ignore.d.server/rsyslog [until rsyslog 4.2.0-2 hits stable]

    ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel:( \[[[:digit:]]+\.[[:digit:]]+\])? imklog [0-9.]+, log source = /proc/kmsg started.$
    ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ rsyslogd: \[origin software="rsyslogd" swVersion="[0-9.]+" x-pid="[0-9]+" x-info="http://www.rsyslog.com"\] restart$
    ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: Kernel logging (proc) stopped.$
    
/etc/logcheck/ignore.d.server/ddclient
:HIDDEN:

    + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ddclient\[[0-9]+\]: WARNING:  file /var/cache/ddclient/ddclient.cache, line [0-9]+: Invalid Value for keyword 'ip' = ''$
    + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ddclient\[[0-9]+\]: WARNING:  updating [._[:alnum:]-]+: nochg: No update required; unnecessary attempts to change to the current address are considered abusive$
    :END:

** Configuring aptitude and friends

# We are going to automatically install many security updates using the package
# "unattended-upgrades".  Automated upgrades are in general not a very good
# idea, but "unattended-upgrades" takes steps to mitigate the problems with this
# approach.  Given the Debian security teams track record in recent years we
# believe the positives outweigh the negatives.
#
# For the situations when unattended-upgrades fails (e.g. when there are
# configuration changes), there is an e-mail sent to the administrator.
#
sudo aptitude install unattended-upgrades

:: /etc/apt/apt.conf

     :CONTENT:
APT
{
  // Increase cache size to some arbitrary size.
  // Remove this line completely once we have apt v0.7.26 in stable. (it defaults to no limit)
  Cache-Limit "33554432";

  // Configuration for /etc/cron.daily/apt
  Periodic
  {
     // Do "apt-get update" automatically every n-days (0=disable)
     Update-Package-Lists "1";
     // Do "apt-get autoclean" every n-days (0=disable)
     AutocleanInterval "1";
     // Do "apt-get upgrade --download-only" every n-days (0=disable)
     Download-Upgradeable-Packages "1";
     // Run the "unattended-upgrade" security upgrade script every n days
     Unattended-Upgrade "1";
  }
};

Aptitude
{
  UI
  {
     Autoclean-After-Update:         true;
     Auto-Fix-Broken:                false;
     Keep-Recommends:                true;
     Recommends-Important:           true;
     Description-Visible-By-Default: false;
     HelpBar                         false;
     Menubar-Autohide                true;
     Purge-Unused:                   true;
     Prompt-On-Exit                  false;
  }
}
     :END:

# Using Debian squeeze:
:: /etc/apt/apt.conf.d/50unattended-upgrades

     Unattended-Upgrade::Mail "admin@fripost.org";
     Acquire::http::Dl-Limit "70";

# Using Debian lenny:
sudo aptitude install apticron
:: /etc/apticron/apticron.conf

     EMAIL="admin@fripost.org"

** Reconfigure exim

sudo dpkg-reconfigure exim4-config

# - select "mail sent by smarthost; no local mail"
# - hostname:
#   host.example.com
# - listen on:
#   127.0.0.1
# - other destinations:
#   [empty]
# - visible domain name:
#   host.example.com
# - address of outgoing smarthost
#   smtp.bredband.net [or whatever the ISP uses]
# - number of DNS queries minimal?
#   no
# - split configuration?
#   no

* NEXT STEPS
** Configuring the backup solution

   General idea [[http://wikis.sun.com/display/BigAdmin/Using+rdist+rsync+with+sudo+for+remote+updating][from here]].  This is just a basic setup for now, will need to be
   changed to rsnapshot or perhaps something even more sophisticated like
   bacula.

   1. Install rsync
      - sudo aptitude install rsync
   2. Create a key on the backup computer
      - ssh-keygen -N "" -b 4096 -f ~/.ssh/backup_key
      - cat .ssh/backup_key.pub
   3. Create a user on the computer that will be backed up
      - sudo adduser remupd
      - sudo passwd -d remupd
      - add the public key from above to ~remupd/.ssh/authorized_keys2
        prefix with: no-X11-forwarding,no-agent-forwarding,no-port-forwarding
      - test the key:
        ssh -i ~/.ssh/backup_key -l remupd example.com
      - add remupd to sudo:
        Cmnd_Alias      RSYNCDIST=/usr/bin/rsync
        remupd	ALL=NOPASSWD:RSYNCDIST
   3. Create a script on the backup computer to automatically backup
   4. Add script to crontab

** Configuring the e-mail servers

*** Introduction
**** Overview

    We will be using one main mail storage server, accessible by users via IMAP.
    This server should be referred to as the main `IMAP server'. We will have two
    or more mail gateways that will relay e-mail to the main server over secure
    connections.  These are called `smarthosts'.

    The main server will also be responsible for keeping all users in an MySQL
    database that will be replicated using MySQL.

**** Definitions

IMAP server = the main storage server

smarthost = the server receiving email from the internet (configured as MX)

*** Configuring an SSH tunnel between two hosts

    Definitions:
    originating host = the host that will be connecting
    destination host = the host that runs some service

    Begin by setting a few environment variables:

    TUNNEL_KEY="my_tunnel_key"
    TUNNEL_USER="tunneluser"
    TUNNEL_HOME="/home/$TUNNEL_USER"
    DEST_PORT="25"
    ORIGIN_PORT="1917"

**** Prepare origin

   1. Create a key on the originating host:

      sudo ssh-keygen -N "" -b 4096 -f /root/.ssh/$TUNNEL_KEY
      sudo cat /root/.ssh/$TUNNEL_KEY.pub

**** Prepare destination

   2a. Install necessary software on the destination host:

      sudo aptitude install netcat-openbsd

   2b. Create a new user on the destination host:

      sudo adduser --home=$TUNNEL_HOME --shell=`type rbash|cut -d' ' -f3` \
                   --disabled-password $TUNNEL_USER
      echo "exit" | sudo -u $TUNNEL_USER tee $TUNNEL_HOME/.bash_profile

      # Also, make sure to add this user to AllowUsers in /etc/ssh/sshd_config.

      # Note: We need bash, so we can not change the shell to something else.

   2c. Add the public key from above to this user:

      THE_PUBLIC_KEY="ssh-rsa xxxxxxxxxxx"

      sudo -u $TUNNEL_USER mkdir $TUNNEL_HOME/.ssh
      echo "command=\"nc localhost $DEST_PORT\",no-X11-forwarding,no-agent-forwarding,\
no-port-forwarding $THE_PUBLIC_KEY" | sudo -u $TUNNEL_USER tee $TUNNEL_HOME/.ssh/authorized_keys2

**** Set up the tunnel

   4. Test the key on the originating host:

      sudo ssh -v -l $TUNNEL_USER -i /root/.ssh/$TUNNEL_KEY destination.example.com

   5. Configure openbsd-inetd on the originating host:

      # Comment: We use inetd instead of ssh -L because, among other things, ssh
      #          -L tends to hang.

      sudo aptitude install openbsd-inetd

      - /etc/inetd.conf
:HIDDEN:
127.0.0.1:$ORIGIN_PORT  stream  tcp     nowait  root    /usr/bin/ssh    -q -T -i /root/.ssh/tunnel_key smtptunnel@example.com
:END:
      sudo /etc/init.d/openbsd-inetd restart

   You should now be able to connect through the tunnel from the originating
   host using something like:

   telnet localhost $ORIGIN_PORT

*** Installing MySQL
     - sudo apt-get install mysql-server
     - generate a long (25 characters) password for the mysql root user
     - /etc/mysql/my.cnf
:HIDDEN:
skip-innodb
:END:
*** Configuring the MySQL replication
***** Overview
     [[http://dev.mysql.com/doc/refman/5.0/en/replication.html][MySQL 5.0 Reference Manual :: 16 Replication]]

     We will use MySQL replication to keep the MySQL user data on the smarthosts
     in sync with the data held on the main IMAP server.

     We begin by setting up an SSH tunnel, as described above.  The rest is
     fairly straight-forward.  Here are instructions adapted from the MySQL
     manual.

     - Set up the SSH tunnel.

***** Configure the master

     - Add this to my.cnf:
:HIDDEN:
server-id		= 1
log_bin			= /var/log/mysql/mysql-bin.log
expire_logs_days	= 10
max_binlog_size         = 100M
binlog_do_db		= mail
:END:

     - /etc/init.d/mysql restart

     - Enter MySQL shell and create user with replication privileges:
       mysql -u root -p

       # use only ASCII for <password>

       GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'localhost' IDENTIFIED BY '<password>';
       FLUSH PRIVILEGES;

       USE mail;
       FLUSH TABLES WITH READ LOCK;

       # Save the output of this command:
       SHOW MASTER STATUS;

       unlock tables;
       quit;

       # Copy this file to the slave:
       mysqldump -u root -p --opt mail > mydump.sql
       
***** Configure the slave

      - Enter the MySQL shell and create the database:

        mysql -u root -p
        CREATE DATABASE mail;
        quit;
  
        mysql -u root -p --database=mail < mydump.sql
  
      - create a new temporary directory:
        
        sudo mkdir /var/lib/mysql/tmp
        sudo chown mysql:mysql !$
        sudo chmod 0750 !$

      - /etc/mysql/my.cnf
:HIDDEN:
tmpdir		= /var/lib/mysql/tmp
# Note that the server-id must be different on all hosts
server-id		= 2
:END:
      - /etc/init.d/mysql restart

      SLAVE STOP;
      # [[http://dev.mysql.com/doc/refman/5.0/en/change-master-to.html][12.5.2.1. CHANGE MASTER TO Syntax]]
      # NOTE: fill in these values using output from SHOW MASTER STATUS; above

      CHANGE MASTER TO
      MASTER_HOST='127.0.0.1',
      MASTER_PORT=1949,
      MASTER_USER='slave_user',
      MASTER_PASSWORD='<password>', MASTER_LOG_FILE='mysql-bin.000013', MASTER_LOG_POS=98;

      START SLAVE;
      quit;

***** Useful commands while debugging
       start slave; stop slave;
       show slave status\G

*** Configuring the main IMAP server
**** /etc/postfix/main.cf

**** MySQL on the main IMAP server

     - create database mail;

     We will use four tables `alias', `domain', `log' and `mailbox'.
     
     // FIXME; add description of tables
     :HIDDEN:
mysql> show tables;

mysql> describe alias;

mysql> describe domain;

mysql> describe log;

mysql> describe mailbox;
     :END:

     - sudo mysql -u root -p --database=mail
       :HIDDEN:
DROP TABLE IF EXISTS `alias`;
SET @saved_cs_client     = @@character_set_client;
SET character_set_client = utf8;
CREATE TABLE `alias` (
  `address` varchar(255) NOT NULL default '',
  `goto` text NOT NULL,
  `domain` varchar(255) NOT NULL default '',
  `create_date` datetime NOT NULL default '0000-00-00 00:00:00',
  `change_date` datetime NOT NULL default '0000-00-00 00:00:00',
  `active` tinyint(4) NOT NULL default '1',
  PRIMARY KEY  (`address`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Aliases - mysql_virtual_\nalias_maps';
SET character_set_client = @saved_cs_client;

DROP TABLE IF EXISTS `domain`;
SET @saved_cs_client     = @@character_set_client;
SET character_set_client = utf8;
CREATE TABLE `domain` (
  `domain` varchar(255) NOT NULL default '',
  `description` varchar(255) NOT NULL default '',
  `create_date` datetime NOT NULL default '0000-00-00 00:00:00',
  `change_date` datetime NOT NULL default '0000-00-00 00:00:00',
  `active` tinyint(4) NOT NULL default '1',
  PRIMARY KEY  (`domain`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Domains - mysql_virtual_\ndomains_maps';
SET character_set_client = @saved_cs_client;

DROP TABLE IF EXISTS `log`;
SET @saved_cs_client     = @@character_set_client;
SET character_set_client = utf8;
CREATE TABLE `log` (
  `id` int(11) NOT NULL auto_increment,
  `user` varchar(20) NOT NULL default '',
  `event` text NOT NULL,
  `date` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=106 DEFAULT CHARSET=utf8 COMMENT='log table';
SET character_set_client = @saved_cs_client;

DROP TABLE IF EXISTS `mailbox`;
SET @saved_cs_client     = @@character_set_client;
SET character_set_client = utf8;
CREATE TABLE `mailbox` (
  `username` varchar(255) NOT NULL default '',
  `password` varchar(255) NOT NULL default '',
  `name` varchar(255) NOT NULL default '',
  `maildir` varchar(255) NOT NULL default '',
  `domain` varchar(255) NOT NULL default '',
  `create_date` datetime NOT NULL default '0000-00-00 00:00:00',
  `change_date` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
  `active` tinyint(4) NOT NULL default '1',
  PRIMARY KEY  (`username`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COMMENT='Virtual Mailboxes - mysql_virtua\nl_mailbox_maps';
SET character_set_client = @saved_cs_client;
        :END:
     - mysql -u root -p
       CREATE USER 'mail'@'localhost' IDENTIFIED BY 'secret';
       GRANT SELECT ON mail.alias   TO 'mail'@'localhost';
       GRANT SELECT ON mail.domain  TO 'mail'@'localhost';
       GRANT SELECT ON mail.mailbox TO 'mail'@'localhost';

**** Test delivery

    - /etc/postfix/main.cf
:HIDDEN:
# Not really needed until we switch to using Courier maildrop
maildrop_destination_recipient_limit = 1

virtual_mailbox_base = /home/mail/virtual
:END:
    - sudo mkdir -p /home/mail/virtual/fripost.org/example/
    - mysql -u root -p
      INSERT INTO mailbox (username,password,name,maildir,domain)
      VALUES ('exempel@fripost.org','test666','Exempelanvändare','fripost.org/exempel/Maildir/','fripost.org');
    - /etc/init.d/postfix restart

    Now it should work to send an e-mail to exempel@fripost.org

**** Setting up dovecot

    - sudo aptitude install dovecot-imapd
    - /etc/dovecot/dovecot.conf

Note: These settings are already in the file but commented out or set to other
      values.
:HIDDEN:
protocols = imaps
protocol imap {
	ssl_listen = *:993
}
disable_plaintext_auth = yes
mail_location = maildir:/home/mail/virtual/%d/%u/Maildir

# Set this to something that works for the Maildirs
first_valid_uid = XXX
first_valid_gid = XXX

# Allow clients to be fancy if they want to
mechanisms = plain cram-md5

#passdb pam <--- comment this stuff out

# uncomment this stuff
passdb sql {
  args = /etc/dovecot/dovecot-sql.conf
}

#userdb passwd  <--- comment this stuff out

# uncomment this stuff
userdb sql {
   args = /etc/dovecot/dovecot-sql.conf
}

# Do not needlessly run as root
user = nobody
:END:
    - /etc/dovecot/dovecot-sql.conf
:HIDDEN:
driver = mysql
connect = host=127.0.0.1 port=3306 user=XXX password=XXX dbname=mail

# Salted MD5
default_pass_scheme = SMD5

password_query = SELECT username AS user, password FROM mailbox WHERE username = '%u' AND domain = '%d'

# replace XXX with relevant numbers for the system
user_query = SELECT concat('/home/mail/virtual/',maildir) AS mail, XXX AS uid, XXX AS gid FROM mailbox WHERE username = '%u' AND domain = '%d'
:END:
    - sudo /etc/init.d/dovecot restart

    Provided there is a user, you should now be able to login using any IMAP
    client.

*** Configuring a new smarthost to relay e-mail to the main IMAP server

    First setup an SSH tunnel between the hosts according to instructions given
    above in this document.

    Next, you need to configure postfix on the smarthost to relay emails through
    the tunnel:
       
       One quick-and-dirty example to try it out is:
       - /etc/postfix/main.cf
         relay_domains  = fripost.org
         transport_maps = hash:/etc/postfix/transport
       - /etc/postfix/transport
         fripost.org smtp:localhost:1917
       - sudo postmap hash:/etc/postfix/transport

** Configuring the webserver

   - sudo apt-get install apache2

** Necessary stuff to fix for security

*** Firewall rules
TODO: Add nice rules.

** Ideas for improved security
*** Increased rate of backups when the IMAP server goes down 
*** Bacula for backups
    Also has tripwire-like capabilities.
*** Some kind of IDS
*** Monitoring



* NEED TO KNOW FOR SERVER ADMINS

** Use etckeeper

   We keep /etc in a git repository using the tool etckeeper.

   This means that every time you make changes to any files in /etc, you are
   expected to commit them using a descriptive commit message.  Please add a
   signature (initials or your username) since all commits will be made as root.

   $ etckeeper commit "postfix: enable to relay messages to remote hosts via smtp /skangas"

   If you do not commit your changes, the next system upgrade will fail and
   whoever makes the upgrade will have to commit your changes for you.  They may
   have to guess as to why you made your changes.  Please do not put your
   co-administrators in this uncomfortable position.

   It is also possible to use simple git commands in /etc, e.g. `git log'.
   `etckeeper' has the benefit of keeping track of file permissions, which git
   by itself will not.