Archive
Tags
android (3)
ant (2)
beautifulsoup (1)
debian (1)
decorators (1)
django (9)
dovecot (1)
encryption (1)
fix (4)
gotcha (2)
hobo (1)
htmlparser (1)
imaplib (2)
java (1)
json (2)
kerberos (2)
linux (7)
lxml (5)
markdown (4)
mechanize (6)
multiprocessing (1)
mysql (2)
nagios (2)
new_features (3)
open_source (5)
optparse (2)
parsing (1)
perl (2)
postgres (1)
preseed (1)
pxe (4)
pyqt4 (1)
python (41)
raid (1)
rails (1)
red_hat (1)
reportlab (4)
request_tracker (2)
rt (2)
ruby (1)
scala (1)
screen_scraping (7)
shell_scripting (8)
soap (1)
solaris (3)
sql (2)
sqlalchemy (2)
tips_and_tricks (1)
twitter (2)
ubuntu (1)
vmware (2)
windows (1)
zimbra (2)

The management of my current place of business wants to have high priority tickets updated on a regular basis. RT doesn't have anything built in to accomplish this, which is where the RT API comes in handy. Below is my script, which is based on Tim Bishop's rt-escalate. It will send email to a set of addresses whenever a ticket of a given priority has not been updated within a specific threshold.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
#!/usr/bin/env perl                                 

my $criticalPriority = 3; # Minimum priority to check
my $criticalOffset = 2 * 60 * 60; # Time since update
my $fromAddr = 'root@example.com'; # Who mail should appear to come from
my $trackpath = "/tmp"; # Where to place tracking files                 
my @toAddrs = (); # You shouldn't change this.  Use command-line arguments.
my @queues; # You shouldn't change this.  Use command-line arguments.      
my $is_test = 0;                                                           
my $show_help = 0;                                                         

# Location of RT3's libs                                                   
use lib ("/opt/rt3/lib", "/opt/rt3/local/lib");                            

use strict;                                                                
use warnings;                                                              
use Getopt::Long;                                                          

GetOptions(                                                                
    'queue=s' => \@queues,                                                 
    'to=s' => \@toAddrs,                                                   
    'threshold=i' => \$criticalOffset,                                     
    'priority=i' => \$criticalPriority,                                    
    'test' => \$is_test,                                                   
    'help' => \$show_help,                                                 
);                                                                         

if ($show_help){                                                           
    print "\n$0 --threshold seconds --priority number [options]...         
        --threshold         # Number of seconds before next update         
        --priority          # Priority to check                            
        --to                # Who should get mailed, you can specify multiple
        --queue queuename   # Searches all by default, you can specify multiple
        --test              # Don't send mail, just print message              
        --help              # Show this screen                                 
        \n";                                                                   
    exit 1;                                                                    
}                                                                              


# Pull in the RT stuff                                                         
package RT;                                                                    
use RT::Interface::CLI qw(CleanEnv);                                           
use Mail::Mailer;                                                              
use POSIX qw(floor);                                                           
use Time::Piece;                                                               

# Clean our the environment                                                    
CleanEnv();                                                                    

# Load the RT configuration                                                    
RT::LoadConfig();                                                              

# Initialise RT                                                                
RT::Init();                                                                    

# If no queues given, get all enabled queues                                   
if(!@queues) {                                                                 
    my $queues = new RT::Queues($RT::SystemUser);                              
    $queues->LimitToEnabled();                                                 
    foreach my $queue (@{$queues->ItemsArrayRef()}) {                          
        push @queues, $queue->Name;                                            
    }                                                                          
}                                                                              

#Load tracking hash for repeat notification avoidance                          
my %th; # tracking hash                                                        
my @idlist; # used to strip tickets that are closed                            
my $trackfile = "$trackpath/NotifyTimed.$criticalPriority";                    
open(FH, "<$trackfile");                                                       
while(<FH>){                                                                   
    my @parts = split;                                                         
    $th{$parts[0]} = $parts[1];                                                
}                                                                              
close(FH);                                                                     

foreach my $queuename (@queues) {                                              
    my $queue = new RT::Queue($RT::SystemUser);                                
    $queue->Load($queuename);                                                  

    # Get hold of new, open, and stalled tickets only                          
    my $tickets = new RT::Tickets($RT::SystemUser);                            
    $tickets->LimitStatus(VALUE => 'open');                                    
    $tickets->LimitStatus(VALUE => 'new');                                     
    $tickets->LimitStatus(VALUE => 'stalled');                                 
    $tickets->LimitPriority(OPERATOR => '=', VALUE => $criticalPriority);      
    $tickets->LimitQueue(VALUE => $queue->Id);                                 

    while (my $ticket = $tickets->Next) {                                      
        my $ticketdt = Time::Piece->strptime($ticket->LastUpdated, "%Y-%m-%d %H:%M:%S");
        my $now = localtime;                                                            
        my $dt = $now - $criticalOffset;                                                
        my $will_notify = 0;                                                            

        if ($dt->epoch > $ticketdt->epoch){                                             
            my $minutes = floor(($now->epoch - $ticketdt->epoch) / 60);                 
            my $owner = $ticket->OwnerObj;                                              

            # Keep track of ids to strip old ones                                       
            push(@idlist, $ticket->Id);                                                 

            # Only notify once                                                          
            if (exists $th{$ticket->Id}){                                               
                if ($th{$ticket->Id} < $ticketdt->epoch){                               
                    $th{$ticket->Id} = $ticketdt->epoch;                                
                    $will_notify = 1;                                                   
                }                                                                       
            } else {                                                                    
                $th{$ticket->Id} = $ticketdt->epoch;                                    
                $will_notify = 1;                                                       
            }                                                                           

            # Prepare report                                                            
            my $mailmsg = '<html>                                                       
              <body>                                                                    
                <p>Ticket <a href="https://rt.wbsconnect.com/Ticket/Display.html?id='   
                .$ticket->Id.'">'.$ticket->Id.'</a> needs to be updated.</p>            
                <p>Subject: '.$ticket->Subject.'</p>                                    
                <p>Owner: '.$owner->RealName.'</p>                                      
                <p>Status: '.$ticket->Status.'</p>                                      
                <p>Priority: '.$ticket->Priority.'</p>                                  
                <p>Minutes since last update: '.$minutes.'</p>                          
              </body>                                                                   
            </html>';                                                                   

            # Send email, if this isn't a repeat or test                                
            next if not $will_notify;                                                   
            if ($is_test){
                print $mailmsg;
            } else {
                foreach (@toAddrs){
                    my $mailer = Mail::Mailer->new;
                    $mailer->open({
                        'From'    => $fromAddr,
                        'To'      => $_,
                        'Subject' => "Update Required:  Priority "
                            .$ticket->Priority." ticket "
                            .$ticket->Id." needs to be updated",
                        'MIME-Version' => "1.0",
                        'Content-Type' => "text/html",

                    });
                    print $mailer $mailmsg;
                    $mailer->close();
                }
            }
        }
    }
}

# Strip old ids from tracking hash
my %nth; # New tracking hash
foreach(@idlist){
    $nth{$_} = $th{$_};
}

# Write tracking hash to file for future use
open(FH, ">$trackfile");
while((my $key, my $value) = each(%nth)){
    print FH $key, " ", $value, "\n";
}
close(FH);

# Disconnect before we finish off
$RT::Handle->Disconnect();
exit 0;

Usage is as such, assuming the script is named NotifyTimed.pl.

NotifyTimed.pl --threshold seconds --priority number [options]...
        --threshold         # Number of seconds before next update
        --priority          # Priority to check
        --to                # Who should get mailed, you can specify multiple
        --queue queuename   # Searches all by default, you can specify multiple
        --test              # Don't send mail, just print message
        --help              # Show this screen

To have this be useful, you'll want to set up a cron job. This was developed and tested on RT 3.8.1. It should work fine on many older and newer RT versions.

Posted by Tyler Lesmann on December 30, 2008 at 12:06
Tagged as: perl request_tracker rt

I finally got around to integrating Django's comments into this blog.

Posted by Tyler Lesmann on December 20, 2008 at 11:38 and commented on 2 times
Tagged as: new_features

This last week I was putting together a password management application for our group at work. The biggest obstacle was adding encryption to the password data in the database. At first I thought, "I'll use Django's password field." Django uses one-way hashes like all good authentication systems though. I solved this by making my own custom field, extended from django's CharField.

The first step was learning how to do cryptography in python. I found PyCrypto. I does both ciphers and hashes in various algorithms. I chose AES for my algorithm.

AES has a few requirements. The key must be 16, 24, or 32 characters and the length of the string to be encrypted must be a multiple of 16. The key isn't bad, but the string length is a pain. I made a little wrapper to simplify this, so a string of any length my be passed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#!/usr/bin/env python
import string
from random import choice
from Crypto.Cipher import AES

EOD = '`%EofD%`' # This should be something that will not occur in strings

def genstring(length=16, chars=string.printable):
    return ''.join([choice(chars) for i in range(length)])

def encrypt(key, s):
    obj = AES.new(key)
    datalength = len(s) + len(EOD)
    if datalength < 16:
        saltlength = 16 - datalength
    else:
        saltlength = 16 - datalength % 16
    ss = ''.join([s, EOD, genstring(saltlength)])
    return obj.encrypt(ss)

def decrypt(key, s):
    obj = AES.new(key)
    ss = obj.decrypt(s)
    return ss.split(EOD)[0]

if __name__ == '__main__':
    for i in xrange(8, 20):
        s = genstring(i)
        key = genstring(32)
        print 'The key is', key
        print 'The string is', s, i
        cipher  = encrypt(key, s)
        print 'The encrypted string is', cipher
        print 'This decrypted string is', decrypt(key, cipher)

If you execute this, it will generate a key and a string, plus encrypt and decrypt the string. This gives binary data. Binary data and databases don't mix though. Django 1.0 turns everything into unicode, which further complicates things. The trick is to translate the binary into something Django and databases can use easily, like base64. We do this with my new favorite module, binascii. Here's an example of its use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#!/usr/bin/env python
import binascii
import qaes # This is the wrapper for PyCrypto

key = "32 character key can be anything"

s = "Sensitive Data"
print "Unencrypted data:", s

es = qaes.encrypt(key, s)
print "Encrypted binary:", es

esb64 = binascii.b2a_base64(es)
print "Encrypted base64:", esb64

esbin = binascii.a2b_base64(esb64)
print "Back to binary:", esbin

ds = qaes.decrypt(key, esbin)
print "Decrypted data", ds

If you run this, you'll see a line like, Encrypted base64: vF7nMm7N1EhalQ/4gtQhaxEHeCgY2dOsNf2rA7tQZW8=. Everytime you run it though, you'll get a different string because of the random salt on the end.

All that's left now is our custom field. I based this on CharField, but you could use a TextField here too.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import binascii
import qaes
from django.db import models
from django.conf import settings

class AESEncryptedField(models.CharField):
    def save_form_data(self, instance, data):
        setattr(instance, self.name,
            binascii.b2a_base64(qaes.encrypt(settings.AESKEY, data)))
    def value_from_object(self, obj):
        return qaes.decrypt(settings.AESKEY,
            binascii.a2b_base64(getattr(obj, self.attname)))

This requires the user to have a AESKEY set in the settings.py of their project. After that, it is transparent and works like any other CharField!

Posted by Tyler Lesmann on December 19, 2008 at 11:47 and commented on 3 times
Tagged as: django encryption python

Google has put a lot of work into their Android platform, but their non-Eclipse development instructions could use some work. Just trying to build their Hello, Android fails if you follow their notes to the letter. You'll get this error or similar:

[tlesmann@kimiko HelloAndroid]$ ant
Buildfile: build.xml

dirs:
     [echo] Creating output directories if needed...

resource-src:
     [echo] Generating R.java / Manifest.java from the resources...

aidl:
     [echo] Compiling aidl files into Java classes...

compile:
    [javac] Compiling 2 source files to /home/lethal/src/android/HelloAndroid/bin/classes
    [javac] Compliance level '1.4' is incompatible with target level '1.5'. A compliance level '1.5' or better is required

BUILD FAILED
/home/tlesmann/src/android/HelloAndroid/build.xml:140: Compile failed; see the compiler error output for details.

Total time: 1 second

I was completely at a loss with this error and all the pages I found on Google were worthless. I'm hoping to change that with this post. The problem is that Sun's javac has a compliance level of 1.4 by default. This can be changed with a command-line argument of -1.5. To get ant to use this argument when compiling, you have to edit the build.xml like so:

<!-- Compile this project's .java files into .class files. -->
<target name="compile" depends="dirs, resource-src, aidl">
    <javac encoding="ascii" target="1.5" debug="true" extdirs=""
            srcdir="."
            destdir="${outdir-classes}"
            bootclasspath="${android-jar}">
        <compilerarg value="-1.5"/>
        <classpath>
            <fileset dir="${external-libs}" includes="*.jar"/>
        </classpath>
     </javac>
</target>

Add the line, <compilerarg value="-1.5"/>, within the javac element and ant will properly build the Hello, Android application. I'm surprised that the activitycreator program does not do this for you as Google advises you to use Sun's JDK.

I hope I've helped you avoid the frustration I suffered solving this silly issue.

NOTE: You no longer have to do this with the release of the 1.5 SDK! Just install Sun's Java and it will build without a problem.

Posted by Tyler Lesmann on December 6, 2008 at 7:08 and commented on 3 times
Tagged as: android ant fix gotcha java