android (3)
ant (2)
beautifulsoup (1)
debian (1)
decorators (1)
django (9)
dovecot (1)
encryption (1)
fix (4)
gotcha (2)
hobo (1)
htmlparser (1)
imaplib (2)
java (1)
json (2)
kerberos (2)
linux (7)
lxml (5)
markdown (4)
mechanize (6)
multiprocessing (1)
mysql (2)
nagios (2)
new_features (3)
open_source (5)
optparse (2)
parsing (1)
perl (2)
postgres (1)
preseed (1)
pxe (4)
pyqt4 (1)
python (41)
raid (1)
rails (1)
red_hat (1)
reportlab (4)
request_tracker (2)
rt (2)
ruby (1)
scala (1)
screen_scraping (7)
shell_scripting (8)
soap (1)
solaris (3)
sql (2)
sqlalchemy (2)
tips_and_tricks (1)
twitter (2)
ubuntu (1)
vmware (2)
windows (1)
zimbra (2)

If you are like me, then you don't do any more coding than you have to. In Django, we have generic views which give us a lot of helpful things, like extra_content and pagination. We can copy the code and write our own views, but that isn't in good style or use of time. That's where decorators come in. We can write our decorator and then wrap the generic view with it to add functionality.

On, I have a few decorators. One of them provides filtering Entry objects by the user who authored them.

def filtering_by_user(f):
    def _dec(request, queryset, *args, **kwargs):
        nkwargs = kwargs
        username = None

        if 'username' in nkwargs:
            username = nkwargs.pop('username')
        if not username:
            username = request.REQUEST.get('byuser')
        if username:
            queryset = queryset.filter(user__username=username)

        return f(request, queryset, *args, **nkwargs)
    return _dec

The _dec funtion takes all the arguments passed to the decorated generic view. On line 3, we make a copy of the keyword arguments of the function. You have to do this or the decorator will not be able to modify the keyword arguments of the function we are decorating.

On lines 6 and 7, we first check to see if a keyword argument username has been passed to the decorated view. If there is, we are going to store it and pop it out of the nkwargs dictionary. If we don't remove username from the keyword arguments, the generic view will fail. Generic views don't take username as an argument.

On the next two lines, we check if we have a username stored yet. If we don't, then we are going to attempt to pull it from the GET and POST values.

Lines 10 and 11 will filter the queryset, if we have obtained a username.

On line 13, we call the view we are decorating with the modified keyword arguments and queryset. After that, we are simply returning execution to the caller of the decorator.

That's great! Now, how do we use this decorator? In your, you simply import the decorator and place the generic view in the arguments like so:

from django.views.generic.list_detail import object_list
from decorators import filtering_by_user

entry_list = filtering_by_user(object_list)

You can even apply multiple decorators.

entry_list = sorting(filtering_by_user(filtering_by_cat(object_list)))
Posted by Tyler Lesmann on July 29, 2008 at 5:56 and commented on 1 time
Tagged as: decorators django python

For the longest time, I've been using markdown in the Django template engine. This is the dumb way of implementing markdown in Django. It's slow to render and it puts too much programming into the template system, which is against Django philosophy.

The smart way to add markdown functionality is by extending the save method of your models. On this little blog for instance, I have an Entry model. This is what the save method looks like.

def save(self):
    self.body = markdown.markdown(self.body_markdown, safe_mode='escape')
    super(Entry, self).save()

I have two fields that hold the different bodies. The body_markdown is the body of the Entry with markdown markup and the body field is the html version. No extra programming is required. Whenever an Entry is saved, a new html body is created using markdown. Then, the models.Model save method is called and the model is written to the database.

Now, instead of markdown parsing every page load as it would have through templates, it does it once per save. The only downside of this approach is you are using up more than twice as much space for each Entry. Disk space is cheaper than cycles, memory, and your visitors' patience though.

Posted by Tyler Lesmann on July 25, 2008 at 5:58
Tagged as: django markdown python

If you are like me, then you want to use network booting for everything, especially installs. PXE booting the VMware installer with pxelinux is much easier than I thought it would be. First, copy the entire contents of the installer CD to your tftp directory. I copied mine to /tftpboot/esxi/3.5.

Now, open your pxelinux.cfg/default. Add this entry, changing the path to each of the components to coincide with where you placed the files within your tftp directory.

label esx35i
  kernel esxi/3.5/mboot.c32
  append esxi/3.5/vmkernel.gz --- esxi/3.5/binmod.tgz --- esxi/3.5/ienviron.tgz --- \
  esxi/3.5/cim.tgz --- esxi/3.5/oem.tgz --- esxi/3.5/license.tgz --- esxi/3.5/install.tgz

That's all you have to do.

You may also want to look at the post on PXE booting and kickstarting the ESX installer.

Posted by Tyler Lesmann on July 15, 2008 at 12:04 and commented on 2 times
Tagged as: pxe vmware

I've been studying Solaris 10 and 11 recently because I'm aiming to acquire a SCSA. It is not surprising that the most popular flavor of commercial Unix is losing ground.

Getting Solaris

One should not have to register/login to download Solaris or packages for Solaris. Get with the times, Sun. Solaris is supposed to be open source. That's means everyone should get easy access, no questions/logins. I'm talking about anonymous ftp, rsync, and bittorrent. What is more important? Solaris's statistics or relevance.

Package Management

It's not completely horrible. It does figure out dependencies. It's essentially just the equivalent of rpm or dpkg though. There is nothing like yum or apt-get on Solaris 10 or 11. This is a problem. Installing and updating software is a chore, especially installing Solaris Companion software, like vim on Solaris 10. The solution to this is easy. Sun needs to integrate the IPS system from OpenSolaris into their commercial versions.

The Shell

Why are Solaris administrators still using plain Bourne shell by default? Sun, be nice to your sysadmins and give them tab completion and command history by default. Bash is plenty mature to be the default shell. I cannot see any serious admin using Bourne, if they don't absolutely have to. Sun should see this too.

Posted by Tyler Lesmann on July 13, 2008 at 17:30
Tagged as: solaris

Caching Kerberos credentials for offline logins Having kerberos is a nice way to centralize your user passwords, but what if you have laptops that you would like to log onto away from your network? You have to setup credential caching. This is achieved through PAM and the pam_ccreds module. On a Fedora, or a Red Hat derivative, system, you only need to edit one file, /etc/pam.d/system-auth-ac. This file exists on any PAM aware system. Here is mine in its entirety.

auth required
auth sufficient nullok try_first_pass
auth requisite uid >= 500 quiet
auth [default=ignore success=1 service_err=reset] use_first_pass
auth [default=die success=done] action=validate use_first_pass
auth sufficient action=store use_first_pass
auth required

account required broken_shadow
account sufficient
account sufficient uid < 500 quiet
account [default=bad success=ok user_unknown=ignore]
account required

password requisite try_first_pass retry=3
password sufficient sha512 shadow nullok try_first_pass use_authtok
password sufficient use_authtok
password required

session optional revoke
session required
session [success=1 default=ignore] service in crond quiet use_uid
session required
session optional

I've highlighted the lines that do the work. It took me a while to make this. I've read tons of tutorials, as you probably have, to get this to work. In the end, I ended up mastering PAM configuration since none of the tutorial worked properly. I'm going to explain each line so that you will understand what is happening.

auth [default=ignore success=1 service_err=reset] use_first_pass

This line is the most complicated. The default=ignore tells PAM, if this module fails, like cannot reach kerberos server or password does not match, go on to the next step. The success=1 tells PAM if the module succeeds, a correct password verified by the kerberos server, skip one step. The service_err=reset, tells PAM to try again if the server reachable but down for some reason. The is the name of the module. Finally, the use_first_pass tells PAM to use the password that was first given to PAM during this run through and do not ask for a new one.

auth [default=die success=done] action=validate use_first_pass

If kerberos was unavailable, PAM will run this line. The default=die means, if this module fails, everything fails and don't give the user access. The success=done means the auth portion of PAM is complete if this succeeds. is the module name. The module, pam_ccreds, takes arguments and the action=validate has pam_ccreds check to see if the given password matches what is cached in /var/cache/.security.db.

auth sufficient action=store use_first_pass

If the user entered a password that was verified by the kerberos server, PAM will apply this rule. The sufficient tells PAM if this module succeeds, the auth portion is complete. Otherwise, continue through the rest of the auth portion. The action=store, if you haven't guessed, stores the encrypted password in /var/cache/.security.db.

Now the system will cache kerberos credentials!

One gotcha on SELinux enabled systems, I have not been able to get this to work in enforcing mode. PAM is horrible restricted. I cannot find an appropriate file context to apply to /var/cache/.security.db that will allow pam_ccreds to write to it. I recommend using Permissive mode until I find a solution. This should not be too big a deal because you should only be using this on laptops, not critical systems.

One more gotcha. For some reason I have not discovered yet, xscreensaver cannot use ccreds properly. If you lock your machine, then you will not be able to log in without setting up a local password. I'm working on a solution for this.

Posted by Tyler Lesmann on July 13, 2008 at 17:26
Tagged as: kerberos linux

One of the things I do is give away older computers with Ubuntu on them. I've given away quite a few and the installs can take a while, plus I like to install flash and dvd decryption for the new users. I learned about preseed from this post. It didn't have all the information I needed to create this, so that is why I'm sharing my experience.

Preseed is Debian's/Ubuntu's answer to unattended installation. It will do everything for you, from partitioning to setting up users. You just need to create a text file, make it available, and tell the installer where it is. First, here is my preseed for ubuntu hardy.

### Network configuration
d-i netcfg/choose_interface select eth0
d-i netcfg/get_hostname string ubuntu

### Mirror settings
d-i mirror/country string enter information manually
d-i mirror/http/hostname string ftp.deathcat.dci
d-i mirror/http/directory string /ubuntu
d-i mirror/http/proxy string

### Partitioning
d-i partman-auto/init_automatically_partition
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string regular
d-i partman-auto/purge_lvm_from_device boolean true
d-i partman-auto/expert_recipe string boot-root :: 100 50 100 ext3 \
    $primary{ } \
    $bootable{ } \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ ext3 } \
    mountpoint{ /boot } \
    . \
    512 512 200% linux-swap \
    method{ swap } \
    format{ } \
    . \
    512 10000 1024 reiserfs \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ reiserfs } \
    mountpoint{ / } \
    . \
    512 10000 1024 reiserfs \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ reiserfs } \
    mountpoint{ /tmp } \
    . \
    1000 10000 2000 reiserfs \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ reiserfs } \
    mountpoint{ /var } \
    . \
    3000 10000 4000 reiserfs \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ reiserfs } \
    mountpoint{ /usr } \
    . \
    100 10000 100000000 reiserfs \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ reiserfs } \
    mountpoint{ /home } \
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition \
    select Finish partitioning and write changes to disk
d-i partman/confirm boolean true

### Clock and time zone setup
d-i clock-setup/utc boolean true
d-i time/zone string America/Denver

### Installation setup
d-i prebaseconfig/reboot_in_progress note

d-i debconf/priority select critical
debconf debconf/priority select critical

base-config base-config/intro note
base-config base-config/login note

# The user's name and login.
passwd passwd/user-fullname string Ubuntu
passwd passwd/username string ubuntu
# And their password, but use caution!
passwd passwd/user-password ubuntu ubuntu
passwd passwd/user-password-again ubuntu ubuntu

# install desktop + standard packages
tasksel tasksel/first multiselect ubuntu-standard, ubuntu-desktop
d-i finish-install/reboot_in_progress note

# post install scripts
d-i preseed/late_command string \
    cd /target; \
    wget ftp://ftp.deathcat.dci/pub/ubuntu/scripts/post-install; \
    chmod +x ./post-install; \
    chroot ./ ./post-install; \
    rm -f ./post-install

A lot of stuff is in there as you can see. Let's look at the part under Network Configuration. The first line, d-i netcfg/choose_interface select eth0, tells the installer to use eth0 for networking during the install. This does absolute nothing during a PXE boot install, which is irratating if you are installing on to laptops with a wireless card. The next line, d-i netcfg/get_hostname string ubuntu, tells the install to give the new install the hostname ubuntu.

Now, to the Mirror settings section. This section tells the installer I want to manually set the location of the installer packages to http://ftp.deathcat.dci/ubuntu and not to use a proxy.

The Partitioning portion is the most perplexing. It was for me anyway. I like to set up several partitions, for /boot, swap, /, /var/, /usr, and /home. I also like to use reiserfs for speed. The first two lines tell the installer to automatically wipe /dev/sda and prepare to partition it. The next line tells it to use regular partitions opposed to lvm and after that is installer is told to not warn when deleting an lvm. Now, for the meat and potatoes of the partition. This one line (Yes, that is one line and must be so to function) defines how the installer should do the actual partitioning.

d-i partman-auto/expert_recipe string boot-root :: 100 50 100 ext3 \
    $primary{ } \
    $bootable{ } \
    method{ format } \
    format{ } \
    use_filesystem{ } \
    filesystem{ ext3 } \
    mountpoint{ /boot } \

The first thing is leave the 100 50 100 ext3 on the first line. Don't attempt to place the backslash after the :: and continue on the next. It doesn't work. I wish it did. Anyway, the first 100 says to give this partition a minimal size of 100MB. The 50 is the priority. The lower it is, the more likely it will be filled to maximum size, which is the second 100. Ext3 is the partition type to mount this partition as. $primary{ } ensures this partition will be a primary partition and $bootable{ } will set the boot flag. method{ format } tells it to format this partition and not preserve the filesystem as does the next line. Preseed is redundant, isn't it? Use_filesystem{ } tells it to auto detect the filesystem for mounting this time. Filesystem{ ext3 } says use ext3 for formating. Finally, mountpoint{ /boot } tells the installer to mount this to /boot. The "." terminates the configuration of the one partition. Read the rest of that line for more examples.

The three lines after that one big one, tell the installer to write the partitions to disk, format, and not ask questions.

The Clock and time zone setup section sets up the hard clock to use UTC and the timezone to that of Denver, Colorado.

The Installation Setup tells the system to first reboot automatically after install. It also tells the system to crash if debconf doesn't like the rest of the file.

The user's name and login portion is self-explanatory. You'll note that the password ubuntu is written four times, twice on two lines. Why it is required to type it more than once, I don't know. Preseed is redundant!

The next section tells the installer to install the ubuntu-standard and ubuntu-desktop packages and their dependencies. It also tells the installer to reboot after install...again.

At the end of the preseed, I have the installer download a file and execute it in a chroot of the new install. This is that file with comments detailing what's going on.



cd /tmp

# Setup auto login
UBUNTUUSER=$( getent passwd 1000 | cut -d : -f 1 )
wget "$SCRIPTURL/gdm.conf-custom"
sed "s/AutomaticLogin=ubuntu/AutomaticLogin=$UBUNTUUSER/g" \
    gdm.conf-custom > gdm.conf-custom2
mv /etc/gdm/gdm.conf-custom /etc/gdm/gdm.conf-custom.orig
cp gdm.conf-custom2 /etc/gdm/gdm.conf-custom
rm -f gdm.conf-custom gdm.conf-custom2

# Setup ubuntu sources
wget "$SCRIPTURL/sources.list"
rm -f /etc/apt/sources.list
cp sources.list /etc/apt/sources.list
rm -f sources.list

# Get updates
apt-get update
apt-get -y upgrade

# Install flash, vlc, and dvd playing software
apt-get -y install \
    gstreamer0.10-plugins-ugly-multiverse \
    gstreamer0.10-plugins-bad-multiverse \
    gstreamer0.10-plugins-bad \
    gstreamer0.10-plugins-ugly \
    gstreamer0.10-ffmpeg \
    libxine1-ffmpeg \
    libdvdread3 \
    flashplugin-nonfree \

# Setup DVD decryption

exit 0

And that's it. One more note before I finish. To make this entirely hand-free, you will have to past a few arguments at boot: The locale, keymap, network interface to use, and the location of the preseed file. I do this in my pxelinux.cfg/default.

label ubuntu
  kernel ubuntu/hardy/i386/linux
  append vga=normal initrd=ubuntu/hardy/i386/initrd.gz \
  locale=en_US.UTF-8 debian-installer/keymap=us \
  netcfg/wireless_wep= netcfg/choose_interface=eth0 \
  netcfg/get_hostname=ubuntu \
  preseed/url=ftp://ftp.deathcat.dci/pub/linux/ubuntu/preseed/easy.cfg --
Posted by Tyler Lesmann on July 6, 2008 at 12:58 and commented on 8 times
Tagged as: linux preseed pxe ubuntu

Pxelinux can do a lot of things, but loading the solaris installer is not one of them. For that, it is easier to use pxegrub. What can you do if you still want to use pxelinux as your primary network bootloader? You can easily make an entry in your pxelinux.cfg/default for pxegrub!

First of all, here is what my /tftpboot directory without the many linux, bsd, and solaris boot files:


If you are paying attention, you'll notice that pxegrub's filename is different than normal. You have to end the filename with a .0. If you don't, then pxelinux will not load it properly, which caused me a bit of confusion. Also, you will need to place pxegrub's configuration, menu.lst, in /boot/grub at the root of your tftp. Pxegrub looks there by default.

If you have everything set as so, you will be able to load pxegrub from pxelinux with the following clause:

label g
  kernel pxegrub.0

It's really that easy.

Posted by Tyler Lesmann on July 6, 2008 at 0:40
Tagged as: pxe