December 2009
November 2009
October 2009
September 2009
June 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
July 2008
June 2008
October 2007
September 2007
Nagios is a systems monitor that uses a variety of clients, pings, and port scans to check up on systems. Configuring Nagios can seem like daunting task at first glance because Nagios is so extensive. It's not hard as I'll show you.
I'm assuming that you've already downloaded, compiled, and installed Nagios and Nagios Plugins. If you haven't, consult this guide.
After installation, the first file you'll want to edit is main nagios.cfg, /usr/local/nagios/etc/nagios.cfg. You will add these three lines:
cfg_file=/usr/local/nagios/etc/hosts.cfg cfg_file=/usr/local/nagios/etc/hostgroups.cfg cfg_file=/usr/local/nagios/etc/services.cfg
If you have a large amount of hosts, you may want to look into using the cfg_dir directive, with which you can specify a directory in which all files should be parsed as configuration files.
We'll now want to create /usr/local/nagios/etc/hostgroups.cfg. Hostgroups are logical groupings of hosts. They affect how hosts are displayed on the monitor's views and can also be used to attach monitoring for services common to the group. Let's create a few.
define hostgroup{
hostgroup_name linux-servers
alias Linux Servers
}
define hostgroup{
hostgroup_name windows-servers
alias Windows Servers
}
define hostgroup{
hostgroup_name web-servers
alias Web Servers
}
The hostgroup_name is what will be used to reference the hostgroup in the configuration. The alias is what appears on the web interface.
We have some hostsgroups define so we can attach templates to them. Edit /usr/local/nagios/etc/objects/templates.cfg. This file holds template definitions that we will use shortly to define hosts. Find linux-server in this file. It will looks like this:
define host{
name linux-server ; The name of this host template
use generic-host ; This template inherits other values from the generic-host template
check_period 24x7 ; By default, Linux hosts are checked round the clock
check_interval 5 ; Actively check the host every 5 minutes
retry_interval 1 ; Schedule host check retries at 1 minute intervals
max_check_attempts 10 ; Check each Linux host 10 times (max)
check_command check-host-alive ; Default command to check Linux hosts
notification_interval 120 ; Resend notifications every 2 hours
notification_options d,u,r ; Only send notifications for specific host states
contact_groups admins ; Notifications get sent to the admins by default
register 0 ; DONT REGISTER THIS DEFINITION - ITS NOT A REAL HOST, JUST A TEMPLATE!
}
Add this line within the curly braces:
hostgroups linux-servers
Every host we make using the linux-server template, will be a member of the linux-servers hostgroup. Find the windows-servers host and add a hostgroup windows-servers line to it as well.
We're ready to define hosts now.
Open /usr/local/nagios/etc/hosts.cfg and add these few lines:
define host{
use linux-server
host_name www.tylerlesmann.com
hostgroups web-servers
}
define host{
use windows-server
host_name nonexistentwindowsbox.tylerlesmann.com
address 192.168.0.125
}
With use, we tell Nagios to implement a specific template when creating a host. The host_name is the host_name of the system. Nagios will do DNS lookups if no address is defined. You can define addition hostgroups the machine should belong to here or you can do it in the hostgroups.cfg with the members directive. You'll probably want to define your own hosts here instead of using my example hosts.
We almost have something useful. The last step is defining services and attaching them to hosts and hostgroups.
Create the /usr/local/nagios/etc/services.cfg and add these lines:
define service{
use generic-service
hostgroup_name linux-servers
service_description SSH
check_command check_ssh
}
define service{
use generic-service
hostgroup_name web-servers
service_description HTTP
check_command check_http
}
define service{
use generic-service
hostgroup_name windows-servers
service_description RDP
check_command check_tcp!3389
}
The hostgroup linux-servers is attached to the service SSH and Nagios will use check_ssh to monitor the service. This service will use the generic-service template, which defines items like timeouts. You may not be running ssh on the default port. You can tell check_ssh to use another port by giving it a -p argument, like so check_command check_ssh!-p 12345. Commands are defined in /usr/local/nagios/etc/objects/commands.cfg and the documentation for the plugins used in these commands is documented in man pages and on the Nagios Plugins site. You should be able to understand the rest of the service, as they don't vary much from SSH.
You have something functional. Before you go reloading the Nagios service, use this command to check your configuration syntax:
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
The hardest part of Nagios is that it can be time consuming to define all the hosts and host-specific services to monitor. Using templates and hostgroups will save you hours.
This was a nice little learning exercise of my skills, so I'd like to share it. This parses the main dovecot log and the rawlogs for each mailbox to generate a HTML report of which host/ip has done what. The actions are still raw IMAP, but are pretty understandable.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 | #!/usr/bin/env python import cgi import datetime import glob import os import re import socket import sys HOMEDIRSPATH = '/home' MAILLOG = '/var/log/mail.log' # Regex for mail.log TIMESTAMP_RE = re.compile('.*(\d\d:\d\d:\d\d)') RIP_RE = re.compile('.*rip=(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})') MB_RE = re.compile('.*user=<(\w+?)>') # Regex for dovecot.rawlog RAWLOG_RES = [ re.compile('\w+? CREATE "', re.I), # Folder Created re.compile('\w+? DELETE "', re.I), # Folder Deleted re.compile('\w+? RENAME "', re.I), # Folder Moved/Renamed re.compile('\w+? APPEND "', re.I), # Mail Added re.compile('\w+? UID STORE.*DELETED', re.I), # Mail Deleted ] RAWLOG_SELECT_RE = re.compile('\w+? SELECT "', re.I) # Folder selected RAWLOG_COPY_RE = re.compile('\w+? UID COPY', re.I) # Folder Copied/Being Moved RAWLOG_STRIP_RE = re.compile('\w+? (.*)') # Remove the action id HTML_HEADER = """<?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>%s</title> <style type="text/css"> .mb { background-color: #BDB; margin: 0px 0px 40px 0px; padding: 5px; } .ip { background-color: #CEC; margin: 10px; padding: 5px; } .log { background-color: #DFD; margin: 10px; padding: 5px; } span { font-size: small; } </style> </head> <body> """ HTML_FOOTER = """ </body> </html> """ class parsedc: def __init__(self, day=None, maillog=MAILLOG, homedirs=HOMEDIRSPATH): if day is None: # Set to yesterday by default self.day = datetime.date.today() - datetime.timedelta(days=1) else: self.day = day self.maillog = maillog self.homedirs = homedirs self.results = {} self.current_mb = '' self.current_ip = '' def feed(self): f = open(self.maillog, 'r') s = f.read() f.close() dayts = self.day.strftime('%Y%m%d') timed = self.parsetimes(s, self.day) mailboxes = timed.keys() mailboxes.sort() for mb in mailboxes: self.current_mb = mb if not mb in self.results: self.results[mb] = {} os.chdir(os.path.join(self.homedirs, mb, 'dovecot.rawlog')) offset = 0 last = '' for rec in timed[mb]: time = rec[0] self.current_ip = ip = rec[1] if not ip in self.results[mb]: self.results[mb][ip] = [] if time == last: offset += 1 else: offset = 0 last = time logs = glob.glob('-'.join([dayts, time, '*.in'])) try: f = open(logs[offset]) except IndexError: continue # dovecot may not have made a rawlog self.parserawlog(f) f.close() def parsetimes(self, s, dt): monthday = self.day.strftime('%c')[4:10] times = {} for line in s.split('\n'): if line.startswith(monthday): time = rip = mb = '' m = TIMESTAMP_RE.match(line) if m: time = m.group(1).replace(':', '') m = RIP_RE.match(line) if m: rip = m.group(1) m = MB_RE.match(line) if m: mb = m.group(1) if time and rip and mb: if not mb in times: times[mb] = [] times[mb].append((time, rip)) return times def parserawlog(self, f): lastselect = '' for line in f.readlines(): for p in RAWLOG_RES: if p.match(line): self.results[self.current_mb][self.current_ip].append(line) continue if RAWLOG_COPY_RE.match(line): self.results[self.current_mb][self.current_ip].extend([lastselect, line]) if RAWLOG_SELECT_RE.match(line): lastselect = line def print_report(self): mailboxes = self.results.keys() mailboxes.sort() print HTML_HEADER % self.day.isoformat() for mb in mailboxes: print '<div class="mb"><span>', mb, '</span>' ips = self.results[mb].keys() ips.sort() for ip in ips: if self.results[mb][ip]: try: host, aliases, addrs = socket.gethostbyaddr(ip) except socket.herror: host = None print '<div class="ip"><span>' if not host is None: print '(%s)' % host print ip print '</span>' print '<div class="log"><span>' for line in self.results[mb][ip]: m = RAWLOG_STRIP_RE.match(line.strip()) print '', '', cgi.escape(m.group(1)), '<br />' print '</span></div>' print '</div>' print '</div>' print HTML_FOOTER if __name__ == '__main__': pdc = parsedc() pdc.feed() pdc.print_report() |
Install the kerberos client packages
apt-get install krb5-user libpam-krb5
Copy the /etc/krb5.conf from the server. You should double-check the kdc and admin_server lines.
Edit the pam configuration to tell linux to ask kerberos for authentication. There are four files, /etc/pam.d/common-{account,auth,password,session}.
Keep a session logged in as root until you verify that you can still login after making these changes!
# /etc/pam.d/common-account - authorization settings common to all services account sufficient pam_unix.so account sufficient pam_krb5.so account required pam_deny.so # /etc/pam.d/common-auth - authentication settings common to all services auth sufficient pam_unix.so nullok_secure auth sufficient pam_krb5.so use_first_pass auth required pam_deny.so # /etc/pam.d/common-password - password-related modules common to all services password sufficient pam_unix.so nullok obscure min=4 max=8 md5 password sufficient pam_krb5.so use_first_pass password required pam_deny.so # /etc/pam.d/common-session - session-related modules common to all services session optional pam_unix.so session optional pam_krb5.so
You should now be able to authenticate using kerberos. Remember that you will still need create accounts, i.e. useradd, before you will be able to login.
Important note: Make sure that the machine can resolve its hostname to an IP address. This is as simple as adding an entry to /etc/hosts.
Caching Kerberos credentials for offline logins Having kerberos is a nice way to centralize your user passwords, but what if you have laptops that you would like to log onto away from your network? You have to setup credential caching. This is achieved through PAM and the pam_ccreds module. On a Fedora, or a Red Hat derivative, system, you only need to edit one file, /etc/pam.d/system-auth-ac. This file exists on any PAM aware system. Here is mine in its entirety.
auth required pam_env.so auth sufficient pam_unix.so nullok try_first_pass auth requisite pam_succeed_if.so uid >= 500 quiet auth [default=ignore success=1 service_err=reset] pam_krb5.so use_first_pass auth [default=die success=done] pam_ccreds.so action=validate use_first_pass auth sufficient pam_ccreds.so action=store use_first_pass auth required pam_deny.so account required pam_unix.so broken_shadow account sufficient pam_localuser.so account sufficient pam_succeed_if.so uid < 500 quiet account [default=bad success=ok user_unknown=ignore] pam_krb5.so account required pam_permit.so password requisite pam_cracklib.so try_first_pass retry=3 password sufficient pam_unix.so sha512 shadow nullok try_first_pass use_authtok password sufficient pam_krb5.so use_authtok password required pam_deny.so session optional pam_keyinit.so revoke session required pam_limits.so session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid session required pam_unix.so session optional pam_krb5.so
I've highlighted the lines that do the work. It took me a while to make this. I've read tons of tutorials, as you probably have, to get this to work. In the end, I ended up mastering PAM configuration since none of the tutorial worked properly. I'm going to explain each line so that you will understand what is happening.
auth [default=ignore success=1 service_err=reset] pam_krb5.so use_first_pass
This line is the most complicated. The default=ignore tells PAM, if this module fails, like cannot reach kerberos server or password does not match, go on to the next step. The success=1 tells PAM if the module succeeds, a correct password verified by the kerberos server, skip one step. The service_err=reset, tells PAM to try again if the server reachable but down for some reason. The pam_krb5.so is the name of the module. Finally, the use_first_pass tells PAM to use the password that was first given to PAM during this run through and do not ask for a new one.
auth [default=die success=done] pam_ccreds.so action=validate use_first_pass
If kerberos was unavailable, PAM will run this line. The default=die means, if this module fails, everything fails and don't give the user access. The success=done means the auth portion of PAM is complete if this succeeds. pam_ccreds.so is the module name. The module, pam_ccreds, takes arguments and the action=validate has pam_ccreds check to see if the given password matches what is cached in /var/cache/.security.db.
auth sufficient pam_ccreds.so action=store use_first_pass
If the user entered a password that was verified by the kerberos server, PAM will apply this rule. The sufficient tells PAM if this module succeeds, the auth portion is complete. Otherwise, continue through the rest of the auth portion. The action=store, if you haven't guessed, stores the encrypted password in /var/cache/.security.db.
Now the system will cache kerberos credentials!
One gotcha on SELinux enabled systems, I have not been able to get this to work in enforcing mode. PAM is horrible restricted. I cannot find an appropriate file context to apply to /var/cache/.security.db that will allow pam_ccreds to write to it. I recommend using Permissive mode until I find a solution. This should not be too big a deal because you should only be using this on laptops, not critical systems.
One more gotcha. For some reason I have not discovered yet, xscreensaver cannot use ccreds properly. If you lock your machine, then you will not be able to log in without setting up a local password. I'm working on a solution for this.
One of the things I do is give away older computers with Ubuntu on them. I've given away quite a few and the installs can take a while, plus I like to install flash and dvd decryption for the new users. I learned about preseed from this post. It didn't have all the information I needed to create this, so that is why I'm sharing my experience.
Preseed is Debian's/Ubuntu's answer to unattended installation. It will do everything for you, from partitioning to setting up users. You just need to create a text file, make it available, and tell the installer where it is. First, here is my preseed for ubuntu hardy.
### Network configuration
d-i netcfg/choose_interface select eth0
d-i netcfg/get_hostname string ubuntu
### Mirror settings
d-i mirror/country string enter information manually
d-i mirror/http/hostname string ftp.deathcat.dci
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string
### Partitioning
d-i partman-auto/init_automatically_partition
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string regular
d-i partman-auto/purge_lvm_from_device boolean true
d-i partman-auto/expert_recipe string boot-root :: 100 50 100 ext3 \
$primary{ } \
$bootable{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext3 } \
mountpoint{ /boot } \
. \
512 512 200% linux-swap \
method{ swap } \
format{ } \
. \
512 10000 1024 reiserfs \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ reiserfs } \
mountpoint{ / } \
. \
512 10000 1024 reiserfs \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ reiserfs } \
mountpoint{ /tmp } \
. \
1000 10000 2000 reiserfs \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ reiserfs } \
mountpoint{ /var } \
. \
3000 10000 4000 reiserfs \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ reiserfs } \
mountpoint{ /usr } \
. \
100 10000 100000000 reiserfs \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ reiserfs } \
mountpoint{ /home } \
.
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition \
select Finish partitioning and write changes to disk
d-i partman/confirm boolean true
### Clock and time zone setup
d-i clock-setup/utc boolean true
d-i time/zone string America/Denver
### Installation setup
d-i prebaseconfig/reboot_in_progress note
d-i debconf/priority select critical
debconf debconf/priority select critical
base-config base-config/intro note
base-config base-config/login note
# The user's name and login.
passwd passwd/user-fullname string Ubuntu
passwd passwd/username string ubuntu
# And their password, but use caution!
passwd passwd/user-password ubuntu ubuntu
passwd passwd/user-password-again ubuntu ubuntu
# install desktop + standard packages
tasksel tasksel/first multiselect ubuntu-standard, ubuntu-desktop
d-i finish-install/reboot_in_progress note
# post install scripts
d-i preseed/late_command string \
cd /target; \
wget ftp://ftp.deathcat.dci/pub/ubuntu/scripts/post-install; \
chmod +x ./post-install; \
chroot ./ ./post-install; \
rm -f ./post-install
A lot of stuff is in there as you can see. Let's look at the part under Network Configuration. The first line, d-i netcfg/choose_interface select eth0, tells the installer to use eth0 for networking during the install. This does absolute nothing during a PXE boot install, which is irratating if you are installing on to laptops with a wireless card. The next line, d-i netcfg/get_hostname string ubuntu, tells the install to give the new install the hostname ubuntu.
Now, to the Mirror settings section. This section tells the installer I want to manually set the location of the installer packages to http://ftp.deathcat.dci/ubuntu and not to use a proxy.
The Partitioning portion is the most perplexing. It was for me anyway. I like to set up several partitions, for /boot, swap, /, /var/, /usr, and /home. I also like to use reiserfs for speed. The first two lines tell the installer to automatically wipe /dev/sda and prepare to partition it. The next line tells it to use regular partitions opposed to lvm and after that is installer is told to not warn when deleting an lvm. Now, for the meat and potatoes of the partition. This one line (Yes, that is one line and must be so to function) defines how the installer should do the actual partitioning.
d-i partman-auto/expert_recipe string boot-root :: 100 50 100 ext3 \
$primary{ } \
$bootable{ } \
method{ format } \
format{ } \
use_filesystem{ } \
filesystem{ ext3 } \
mountpoint{ /boot } \
.
The first thing is leave the 100 50 100 ext3 on the first line. Don't attempt to place the backslash after the :: and continue on the next. It doesn't work. I wish it did. Anyway, the first 100 says to give this partition a minimal size of 100MB. The 50 is the priority. The lower it is, the more likely it will be filled to maximum size, which is the second 100. Ext3 is the partition type to mount this partition as. $primary{ } ensures this partition will be a primary partition and $bootable{ } will set the boot flag. method{ format } tells it to format this partition and not preserve the filesystem as does the next line. Preseed is redundant, isn't it? Use_filesystem{ } tells it to auto detect the filesystem for mounting this time. Filesystem{ ext3 } says use ext3 for formating. Finally, mountpoint{ /boot } tells the installer to mount this to /boot. The "." terminates the configuration of the one partition. Read the rest of that line for more examples.
The three lines after that one big one, tell the installer to write the partitions to disk, format, and not ask questions.
The Clock and time zone setup section sets up the hard clock to use UTC and the timezone to that of Denver, Colorado.
The Installation Setup tells the system to first reboot automatically after install. It also tells the system to crash if debconf doesn't like the rest of the file.
The user's name and login portion is self-explanatory. You'll note that the password ubuntu is written four times, twice on two lines. Why it is required to type it more than once, I don't know. Preseed is redundant!
The next section tells the installer to install the ubuntu-standard and ubuntu-desktop packages and their dependencies. It also tells the installer to reboot after install...again.
At the end of the preseed, I have the installer download a file and execute it in a chroot of the new install. This is that file with comments detailing what's going on.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | #!/bin/sh SCRIPTURL=ftp://ftp.deathcat.dci/pub/ubuntu/scripts cd /tmp # Setup auto login UBUNTUUSER=$( getent passwd 1000 | cut -d : -f 1 ) wget "$SCRIPTURL/gdm.conf-custom" sed "s/AutomaticLogin=ubuntu/AutomaticLogin=$UBUNTUUSER/g" \ gdm.conf-custom > gdm.conf-custom2 mv /etc/gdm/gdm.conf-custom /etc/gdm/gdm.conf-custom.orig cp gdm.conf-custom2 /etc/gdm/gdm.conf-custom rm -f gdm.conf-custom gdm.conf-custom2 # Setup ubuntu sources wget "$SCRIPTURL/sources.list" rm -f /etc/apt/sources.list cp sources.list /etc/apt/sources.list rm -f sources.list # Get updates apt-get update apt-get -y upgrade # Install flash, vlc, and dvd playing software apt-get -y install \ gstreamer0.10-plugins-ugly-multiverse \ gstreamer0.10-plugins-bad-multiverse \ gstreamer0.10-plugins-bad \ gstreamer0.10-plugins-ugly \ gstreamer0.10-ffmpeg \ libxine1-ffmpeg \ libdvdread3 \ flashplugin-nonfree \ vlc # Setup DVD decryption /usr/share/doc/libdvdread3/install-css.sh exit 0 |
And that's it. One more note before I finish. To make this entirely hand-free, you will have to past a few arguments at boot: The locale, keymap, network interface to use, and the location of the preseed file. I do this in my pxelinux.cfg/default.
label ubuntu kernel ubuntu/hardy/i386/linux append vga=normal initrd=ubuntu/hardy/i386/initrd.gz \ locale=en_US.UTF-8 debian-installer/keymap=us \ netcfg/wireless_wep= netcfg/choose_interface=eth0 \ netcfg/get_hostname=ubuntu \ preseed/url=ftp://ftp.deathcat.dci/pub/linux/ubuntu/preseed/easy.cfg --