path: root/fripost-docs.org

# -*- mode: org-mode; truncate-lines: nil -*-
#+TITLE: Systems documentation
#+AUTHOR: The Free E-mail Association
#+DESCRIPTION: Systems documentation for The Free E-mail Association
#+KEYWORDS: 
#+LANGUAGE:  en
#+OPTIONS:   H:3 num:t toc:t \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t
#+OPTIONS:   TeX:t LaTeX:nil skip:nil d:nil todo:t pri:nil tags:not-in-toc
#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
#+EXPORT_SELECT_TAGS: export
#+EXPORT_EXCLUDE_TAGS: noexport
#+LINK_UP:   
#+LINK_HOME: 
#+XSLT: 
#+DRAWERS: HIDDEN STATE PROPERTIES CONTENT

Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU Free Documentation License,
Version 1.3 or any later version published by the Free Software
Foundation; with no Invariant Sections, no Front-Cover Texts and
no Back-Cover Texts.  A copy of the license is included in a
separate file called "COPYING".

This is documentation of the server configuration used by the free e-mail association, given here in general interest of transparency.

The complete documentation is the actual configuration files on the servers.  This document intends to give a general idea of the setup and be of help if we need to recreate a crashed server.  Also, if an administrator goes AWOL, it should be easy to pick up where he left of.

We welcome all critisism, suggestions for improvements, additions etc.  Please send them to skangas@skangas.se.

* BASIC SETUP -- Checklist after having installed a new Debian GNU/Linux-server

  NB! Do not install any "tasks" during installation (web server etc.).
  Make sure to answer "yes" to shadow passwords and MD5.

** Uninstall a bunch of unecessary packages, among them:

   sudo aptitude remove --purge openbsd-inetd portmap 

** Install etckeeper
   Used to keep track of /etc.  Install ASAP after install!
   - /etc/etckeeper/etckeeper.conf
     AVOID_COMMIT_BEFORE_INSTALL=1

** Packages to install
*** Administrative

    sudo aptitude install openssh-server ntp ntpdate screen

*** Security

    sudo aptitude install logcheck harden-clients harden-servers

    NB harden-clients conflicts with telnet, which as we know is very handy during configuration

** Configure sshd
   - /etc/ssh/sshd_config
     PermitRootLogin no
     PasswordAuthentication no
     X11Forwarding no

** Configure sudo
   - Add relevant users to the sudo group
   - sudo visudo
     %sudo ALL= (ALL) ALL

** Configure logcheck

   - sudo aptitude install logcheck syslog-summary

   - /etc/logcheck/logcheck.conf

     INTRO=0
     SENDMAILTO="skangas@skangas.se"

   - /etc/logcheck/ignore.d.server/ntpd

     - ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync (disabled|enabled) [0-9]+$
     + ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ ntpd\[[0-9]+\]: kernel time sync (disabled|enabled|status( change)?) [0-9]+$

   - /etc/logcheck/ignore.d.server/ssh [until logcheck 1.3.7 hits stable]

     + ^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ sshd\[[[:digit:]]+\]: Received disconnect from [:[:xdigit:].]+: [[:digit:]]+: disconnected by user$

   - /etc/logcheck/ignore.d.server/rsyslog [until rsyslog 4.2.0-2 hits stable]

   ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel:( \[[[:digit:]]+\.[[:digit:]]+\])? imklog [0-9.]+, log source = /proc/kmsg started.$
   ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ rsyslogd: \[origin software="rsyslogd" swVersion="[0-9.]+" x-pid="[0-9]+" x-info="http://www.rsyslog.com"\] restart$
   ^\w{3} [ :0-9]{11} [._[:alnum:]-]+ kernel: Kernel logging (proc) stopped.$


** Configuring aptitude and friends
   We're going for a setup where we install many security updates automatically using the package "unattended-upgrades".  Automated upgrades are in general not a very good idea, but "unattended-upgrades" takes steps to minimize the issues with this kind of setup.  Given the Debian security teams track record we believe the positives outweigh the negatives.

   For the situations when unattended-upgrades fails (e.g. when there are configuration changes), we should e-mail the administrator.  We will be using apticron to do this until the version of unattended-upgrades in stable supports mailing when an upgrade fails (the one in unstable does).

   - sudo aptitude install apticron unattended-upgrades
   - /etc/apt/apt.conf
     :CONTENT:
// Limit download speed
//Acquire::http::Dl-Limit "70";

/* Unsupported in the version of unattended-upgrades that is in stable,
 * but will later send an e-mail when an upgrade fails.
 * Until this works in stable, we will use apticron. */
//Unattended-Upgrade::Mail "skangas@skangas.se";

APT
{
  // Increase cache size to some arbitrary size.
  // Remove this line completely once we have apt v0.7.26 in stable. (it defaults to no limit)
  Cache-Limit "33554432";

  // Configuration for /etc/cron.daily/apt
  Periodic
  {
     // Do "apt-get update" automatically every n-days (0=disable)
     Update-Package-Lists "1";
     // Do "apt-get autoclean" every n-days (0=disable)
     AutocleanInterval "1";
     // Do "apt-get upgrade --download-only" every n-days (0=disable)
     Download-Upgradeable-Packages "1";
     // Run the "unattended-upgrade" security upgrade script every n days
     Unattended-Upgrade "1"
  }
};

Aptitude
{
  UI
  {
     Autoclean-After-Update:         true;
     Auto-Fix-Broken:                false;
     Keep-Recommends:                true;
     Recommends-Important:           true;
     Description-Visible-By-Default: false;
     HelpBar                         false;
     Menubar-Autohide                true;
     Purge-Unused:                   true;
     Prompt-On-Exit                  false;
  }
}
     :END:
   - /etc/apticron/apticron.conf
     EMAIL="skangas@skangas.se"


* NEXT STEPS

** Configuring the backup solution
   General idea [[http://wikis.sun.com/display/BigAdmin/Using+rdist+rsync+with+sudo+for+remote+updating][from here]].  This is just a basic setup for now, will need to be changed to rsnapshot or perhaps something even more sophisticated like bacula.

   1. Install rsync
      - sudo aptitude install rsync
   2. Create a key on the backup computer
      - ssh-keygen -N "" -b 4096 -f ~/.ssh/backup_key
      - cat .ssh/backup_key.pub
   3. Create a user on the computer that will be backed up
      - sudo adduser remupd
      - turn of password using sudo vipw -s
      - add the key to ~remupd/.ssh/authorized_keys2
        prefix with: no-X11-forwarding,no-agent-forwarding,no-port-forwarding
      - test the key:
        ssh -i ~/.ssh/backup_key -l remupd host
      - add remupd to sudo:
        Cmnd_Alias      RSYNCDIST=/usr/bin/rsync
        remupd	ALL=NOPASSWD:RSYNCDIST
   3. Create a script on the backup computer to automatically backup
   4. Add script to crontab


** Configuring the e-mail servers

   We will be using one main mail storage server, accessible by users via IMAP.
   This server should be referred to as the main `IMAP server'. We will have two
   or more mail gateways that will relay e-mail to the main server over secure
   connections.  These are called `smarthosts'.

   The main server will also be responsible for keeping all users in an MySQL
   database that will be replicated using MySQL.

*** Configuring an SSH tunnel between two hosts

    Definitons:
    originating host = the host that will be connecting
    destination host = the host that runs some service

**** Preparing steps on the destination

   1a. Install necessary software on the destination host:

      - sudo aptitude install openbsd-netcat

   1b. Create a new user on the destination host:

      - sudo adduser smtptunnel
      - echo "exit" | sudo -u smtptunnel tee ~smtptunnel/.bash_profile
      - disable the password: sudo vipw -s

        Note: We need bash, so we can not change the shell to something else.

**** Preparing steps on the originating server

   1c. Install necessary software on the originating server:

      - sudo aptitude install openbsd-inetd

        Comment: We use inetd instead of ssh -L because, among other things, ssh
        -L tends to hang.

**** Steps to set up a new tunnel 

   2. Create a key on the originating server:
      - sudo su
      - ssh-keygen -N "" -b 4096 -f ~/.ssh/tunnel_key
      - cat .ssh/tunnel_key.pub

   3. Add this key to the user `smtptunnel' on the IMAP server
      - echo "<thekey>" | sudo tee .ssh/authorized_keys2
      - Add this before "ssh-rsa" in authorized_keys2:
        command="nc localhost 25",no-X11-forwarding,no-agent-forwarding,no-port-forwarding

   4. Test the key on the smarthost:
      - sudo ssh -l smtptunnel -i /root/.ssh/tunnel_key example.com

   5. Configure openbsd-inetd on the smarthost:

      - /etc/inetd.conf
:HIDDEN:
127.0.0.1:1917  stream  tcp     nowait  root    /usr/bin/ssh    -q -T -i /root/.ssh/tunnel_key smtptunnel@example.com
:END:
      - sudo /etc/init.d/openbsd-inetd restart

   You should now be able to connect through the tunnel using something like:

   telnet localhost 1917

*** Basic configuration of MySQL
**** Installing MySQL
     - sudo apt-get-install mysql-server
     - generate a long (25 characters) password for the mysql root user

**** Configuring the main IMAP server

     - create database mail;

     We will use four tables `alias', `domain', `log' and `mailbox'.
     
     // FIXME; add description of tables
     :HIDDEN:
mysql> show tables;

mysql> describe alias;

mysql> describe domain;

mysql> describe log;

mysql> describe mailbox;
     :END:

      - to set up the tables, use the following schema // FIXME: Add schema
        :HIDDEN:

        :END:

**** Configuring the MySQL replication
     
     [[http://dev.mysql.com/doc/refman/5.0/en/replication.html][MySQL 5.0 Reference Manual :: 16 Replication]]

     We will use MySQL replication to keep the MySQL user data on the smarthosts
     in sync with the data held on the main IMAP server.

     We begin by setting up an SSH tunnel.  This process is described above.
     The rest is fairly straight-forward (instructions below adapted from [[http://www.howtoforge.com/mysql_database_replication][here]]).

     - Set up the SSH tunnel.

***** Configure the master

     - Add this to my.cnf:
:HIDDEN:
server-id		= 1
log_bin			= /var/log/mysql/mysql-bin.log
expire_logs_days	= 10
max_binlog_size         = 100M
binlog_do_db		= mail
:END:

     - /etc/init.d/mysql restart

     - Enter MySQL shell and create user with replication privileges:
       mysql -u root -p

       When in shell, do the following (replace <password> with something better):

       create database mail;

       GRANT REPLICATION SLAVE ON *.* TO 'slave_user'@'%' IDENTIFIED BY '<password>';
       FLUSH PRIVILEGES;

       # Is this only needed when using "load data from master"?
       grant reload, super, replication client on *.* to 'slave_user';

       USE mail;
       FLUSH TABLES WITH READ LOCK;
       SHOW MASTER STATUS;

       unlock tables;
       quit;
       
***** Configure the slave

      - Enter the MySQL shell and create the database:

      mysql -u root -p
      Enter password:
      CREATE DATABASE mail;
      quit;

      - /etc/mysql/my.cnf
:HIDDEN:
tmpdir		= /var/lib/mysql/tmp
# Note that the server-id must be different on all hosts
server-id		= 2
master-host		= 127.0.0.1
master-port		= 1949
master-user		= slave_user
master-password		= <password>
master-connect-retry	= 60
replicate-do-db		= mail
:END:
      - create the temporary directory:
        
        mkdir /var/lib/mysql/tmp
        chown mysql:mysql !$
        chmod 0750 !$

      - /etc/init.d/mysql restart
      - Enter the MySQL shell and make the replication:

      mysql -u root -p
      Enter password:
      LOAD DATA FROM MASTER;
      quit;

      A strange bug bit me at this point, notes:

      start slave; stop slave;
      show slave status\G

      [[http://dev.mysql.com/doc/refman/5.0/en/change-master-to.html][12.5.2.1. CHANGE MASTER TO Syntax]]

      CHANGE MASTER TO MASTER_PORT=1949, MASTER_CONNECT_RETRY=60;


*** Configuring a new smarthost to relay e-mail to the main IMAP server

    Definitons:
    IMAP server = the main storage server
    smarthost = the receiving server (configured as MX)

    First setupa an SSH tunnel between the hosts according to instructions given
    above in this document.

    Next, you need to configure postfix on the smarthost to relay emails through
    the tunnel:
       
       One quick-and-dirty example to try it out is:
       - /etc/postfix/main.cf
         relay_domains  = fripost.org
         transport_maps = hash:/etc/postfix/transport
       - /etc/postfix/transport
         fripost.org smtp:localhost:1917
       - sudo postmap hash:/etc/postfix/transport

** Necessary stuff to fix for security

*** Firewall rules
TODO: Add nice rules.

** Ideas for improved security
*** Increased rate of backups when the IMAP server goes down 
*** Bacula for backups
    Also has tripwire-like capabilities.
*** Some kind of IDS
*** Monitoring



* NEED TO KNOW FOR SERVER ADMINS

** Use etckeeper

   We keep /etc in a git repository using the tool etckeeper.

   This means that every time you make changes to any files in /etc, you are expected to commit them using a descriptive commit message.  Please add a signature (initials or your username) since all commits will be made as root.

   $ etckeeper commit "This is an example change that might fix the issues we have done.  -- Signature"

   If you do not commit your changes, the next system upgrade will fail and whoever makes the upgrade will have to commit your changes for you.  They may have to guess as to why you made your changes.  Please do not put your co-administrators in this uncomfortable position.

   It is also possible to use simple git commands in /etc, e.g. `git log'.  `etckeeper' has the benefit of keeping track of file permissions, which git by itself will not.