android (3)
ant (2)
beautifulsoup (1)
debian (1)
decorators (1)
django (9)
dovecot (1)
encryption (1)
fix (4)
gotcha (2)
hobo (1)
htmlparser (1)
imaplib (2)
java (1)
json (2)
kerberos (2)
linux (7)
lxml (5)
markdown (4)
mechanize (6)
multiprocessing (1)
mysql (2)
nagios (2)
new_features (3)
open_source (5)
optparse (2)
parsing (1)
perl (2)
postgres (1)
preseed (1)
pxe (4)
pyqt4 (1)
python (41)
raid (1)
rails (1)
red_hat (1)
reportlab (4)
request_tracker (2)
rt (2)
ruby (1)
scala (1)
screen_scraping (7)
shell_scripting (8)
soap (1)
solaris (3)
sql (2)
sqlalchemy (2)
tips_and_tricks (1)
twitter (2)
ubuntu (1)
vmware (2)
windows (1)
zimbra (2)

The most fundamental of abilities in shell scripts is moving about the file system.

import os
contents = os.listdir(os.getcwd())

To change directories, we use os.chdir with an argument of our destination. To get a listing of the contents of a directory, os.listdir is the command to use. It takes one argument, which is the directory that holds the contents. If we want to know our current location on the filesystem, then os.getcwd will tell us the path.

Instead of moving around, you may want to do the same thing on the files and folders of a tree of directories, a.k.a. a recursive operation. In this next scenario, I have a hypothetical module called mp32vorbis that has a convert function, which converts a specified mp3 file into an ogg vorbis file.

import os
import mp32vorbis

for root, dirs, files in os.walk(os.getcwd()):
    for dir in dirs:
        print "Processing", os.path.join(root, dir)
    for file in files:
        if file.endswith('.mp3'):
            print "Converting", file

As you can see, recursive operations are quick and easy with os.walk. It is a generator that yields each directory, root, plus a collection of directories and files within it. You can process each separate directory and file by looping through them. You may have noticed os.path.join. This is a simple function that connects its arguments with the directory separator of the host OS, which is backslashes for Windows and forward slashes for everything else.

Posted by Tyler Lesmann on January 15, 2009 at 5:41 and commented on 2 times
Tagged as: python shell_scripting

At times, you will have to get and set environment variables to get the desired effect in your scripts. The os module offers this function through a dictionary like object called environ.

import os
homedir = os.environ['HOME']
if not '/opt/mysuite/bin' in os.environ['PATH']:
    os.environ['PATH'] += ':/opt/mysuite/bin'
os.system('mysuiteapp') # Executable in /opt/mysuite/bin
Posted by Tyler Lesmann on January 14, 2009 at 5:41
Tagged as: python shell_scripting

So you are running programs from python now. Perhaps you want to check that a file is readable and executable before trusting it to run? The os module comes to the rescue once again!

import os
filename = 'bin/someprogram'
if not os.path.exists(filename):
    print filename, 'does not exist!'
    raise SystemExit, 2
if not os.access(filename, os.R_OK):
    print filename, 'is not readable!'
    raise SystemExit, 3
if not os.access(filename, os.X_OK):
    print filename, 'is not executable!'
    raise SystemExit, 4

We would like to know that the file exists. This is determined with os.path.exists. Is the file readable and executable? We can find out with os.access and a couple of constants from the os module, os.R_OK and os.X_OK. We could also test if a file is writeable with os.access. We just use the os.W_OK constant. Simple, yes?

Would you like to know the times of creation, modification, and access? What about file size?

import datetime
import os
filename = 'somedir/somefile'
ctime = datetime.datetime.fromtimestamp(os.path.getctime(filename))
mtime = datetime.datetime.fromtimestamp(os.path.getmtime(filename))
atime = datetime.datetime.fromtimestamp(os.path.getatime(filename))
sizeinbytes = os.path.getsize(filename)

It is pretty straight forward. You might be wondering about why I am using datetime.datetime.fromtimestamp here. The os.path.getXtime functions all spit out POSIX timestamps. In python, it is more useful to have these be python datetimes. The datetime.datetime.fromtimestamp function produces a datetime object from a given timestamp.

Posted by Tyler Lesmann on January 13, 2009 at 5:47
Tagged as: python shell_scripting

Like shell scripts, python may not be able to do everything you want by itself. You may need to execute an external program to get the job done.

NOTE: This is the old way to call external programs and still valid for python versions 2.3 and earlier. However, if you are using 2.4 or later, then use subprocess. The documentation for subprocess is very flushed out so I will not be expanding on it here.

Python's os module offers loads of ways to accomplish this. I will cover only the ones you are to use when replacing shell scripts.

import os
return_status = os.system('ping -c 5 localhost')

The os.system function is the least useful. I will execute a command as specified and return the return status of the program. This is especially useless on operating systems with inconsistent return statuses, i.g. Windows. Any output from the program will be sent to the standard locations, stdout and stderr.

import os
output = os.popen('ping -c 5 localhost')
print # Streams are file-like

Perhaps you might want to capture the program's output for later. The os.popen function returns a stream of the output from the program. These streams are handled just like files in python. Use the read and readline methods to access the output. With popen, the function returns immediately, so you can run multiple external programs at once.

import os
stdin, stdout = os.popen2('sed s/p/q/g')
stdin.writelines('Replace the ps with qs please')
print stdout.readline() # prints Reqlace qs with qs qlease

What if you want to give the program a little input? The os.popen2 function lets you do just that. It returns a file stream to both the stdin and stdout of the program. Just write your input, close the stdin stream, and read the output. There are more popen functions. The os.popen3 returns file streams to stdin, stdout, and stderr. os.popen4 returns two file streams, the stdin and the merged stdout/stderr.

Posted by Tyler Lesmann on January 12, 2009 at 5:51
Tagged as: python shell_scripting

You might need to make folders and link files from time to time. The os module is used for both of these tasks.

import os
os.mkdir('newdir') # like mkdir
os.makedirs('notexistent/bogus/newdir') # like mkdir -p

os.mkdir and os.makedirs offer little in the way of surprises. os.mkdir works like mkdir and os.makedirs works like mkdir -p, creating any intermediate directories it needs to create the specified directory.

import os'file', 'newhardlink') # think ln
os.symlink('fileordir', 'newsymlink') # think ln -s creates hard links to files just like ln. os.symlink creates soft links in the way that ln -s does. I love how python works in such a predictable fashion.

Posted by Tyler Lesmann on January 11, 2009 at 5:48
Tagged as: python shell_scripting