Archive
Tags
android (3)
ant (2)
beautifulsoup (1)
debian (1)
decorators (1)
django (9)
dovecot (1)
encryption (1)
fix (4)
gotcha (2)
hobo (1)
htmlparser (1)
imaplib (2)
java (1)
json (2)
kerberos (2)
linux (7)
lxml (5)
markdown (4)
mechanize (6)
multiprocessing (1)
mysql (2)
nagios (2)
new_features (3)
open_source (5)
optparse (2)
parsing (1)
perl (2)
postgres (1)
preseed (1)
pxe (4)
pyqt4 (1)
python (41)
raid (1)
rails (1)
red_hat (1)
reportlab (4)
request_tracker (2)
rt (2)
ruby (1)
scala (1)
screen_scraping (7)
shell_scripting (8)
soap (1)
solaris (3)
sql (2)
sqlalchemy (2)
tips_and_tricks (1)
twitter (2)
ubuntu (1)
vmware (2)
windows (1)
zimbra (2)

Want to have fun? Try migrating an existing web application between different database technologies! With Django and SQLAlchemy, it actually isn't that hard! I used the following procedure to migrate both deathcat.org and this blog to Postgres. I'm assuming your know you to use Postgres and you are doing this as a Postgres superuser. All of this assumes ident authentication for Postgres, but should be easily tweaked for other configurations.

Make a directory in your Django application to store these scripts, like scripts/. Make sure this directory resides at the same level as manage.py. Now, get the code for my SQLAlchemy table copier and put it in a new file called puller.py. Comment out the line that reads table.metadata.create_all(dengine).

Now put this in a file called migrate2pg.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/bin/bash

database=my_django_db
mysql_user=django_user
mysql_pass=django_passwd
mysql_connection_string="mysql://$mysql_user:$mysql_pass@localhost/$database?charset=utf8"
postgres_connection_string="postgres:///$database"

tables=$(echo 'show tables' | mysql -u $mysql_user -p"$mysql_pass" $database | xargs echo | cut -d ' ' -f 2-)

echo $tables

echo "Dropping old postgres database, if any"
dropdb $database

echo "Creating new database"
createdb $database

echo "Setting up Django schema"
../manage.py syncdb --noinput

echo "Removing initial data"
echo 'DELETE FROM auth_permission' | psql $database
echo 'DELETE FROM django_content_type' | psql $database

echo "Importing data from MySQL"
python puller.py \
    -f $mysql_connection_string \
    -t $postgres_connection_string \
    $tables

echo "Fixing sequences"
for table in $tables
do
    echo Fixing "${table}'s sequence"
    echo "select setval('${table}_id_seq', max(id)) from ${table};" | psql $database
done

Tweak the variables at the top as necessary for your case. Run a bash migrate2pg.sh and read the messages. One error you will see is a during the Fixing sequences phase when the script attempts to fix django_session_id_seq sequence. Ignore this error.

The final part is to give permissions or ownership to the user who will be accessing the data. I'm assuming you can do this, but if you are using Postgres ident authentication and apache, then here's a helpful script for you.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#!/bin/bash

database='my_django_db'

echo "Granting apache rights to ${database}"
echo "GRANT ALL ON DATABASE ${database} TO apache;" | psql $database

tables=$(echo '\dt' | psql $database | awk -F '|' '/table/ {print $2}')
sequences=$(echo '\ds' | psql $database | awk -F '|' '/sequence/ {print $2}')

echo "Tables:" $tables
echo

echo "Sequences:" $sequences
echo

tablesql=$(for table in $tables; do echo "ALTER TABLE $table OWNER TO apache;"; done)
seqsql=$(for seq in $sequences; do echo "ALTER TABLE $seq OWNER TO apache;"; done)

echo "Table Alteration SQL:" $tablesql
echo
echo "Sequence Alteration SQL:" $seqsql
echo

echo $tablesql $seqsql | psql $database
Posted by Tyler Lesmann on September 4, 2009 at 16:47 and commented on 3 times
Tagged as: django mysql postgres python sql sqlalchemy

Waylan Limberg tells me python-markdown will be releasing version 2.0 at the end of this week. This release breaks my extension and all others written for 1.7. My extension obsoleted within a less than a week of release. That would not do, so I have updated the code to work on 2.0! It's still under LGPL.

You can get it here or clone the Mercurial repository like so:

hg clone http://code.tylerlesmann.com/mdx_video2

Use and install are exactly the same as with the earlier version. Adding new services has changed and I will get to documenting that soon.

Posted by Tyler Lesmann on April 2, 2009 at 15:20 and commented on 6 times
Tagged as: django markdown open_source python

I'm a big fan of markdown, especially the fact that it can be extended so easily. I wanted to give the users of DeathCat Inc. the ability to embed video from popular services in their posts with only an URL, so I wrote up a new extension. To see it in action, go here. This code is licensed under LGPL.

You can get it here.

It is installable as many other python modules...

python setup.py install

..., but mdx_video.py only has to be in your PYTHON_PATH. With Django, for instance, you can place it in the same directory as settings.py.

Using my extension is like using any other extension for markdown.

>>> import markdown
>>> s = "http://www.youtube.com/watch?v=F8qwxzQar2g"
>>> markdown.markdown(s, ['video'], safe_mode='escape')

Remember to set safe_mode to escape if you are passing untrusted users' input through markdown, extension or not. This extension supports arguments for setting the dimension of the video. By default, the dimensions are what the specific service gives for their example embed. If you don't like this size, then you can change it like so:

>>> markdown.markdown(s, ['video(youtube_width=720, youtube_height=400)'], safe_mode='escape')

If you want to integrate markdown and this extension with Django, then I recommend looking at this post.

This extension supports the following services:

  • Blip.tv
  • Dailymotion
  • Metacafe
  • Veoh
  • Vimeo
  • Yahoo! video
  • Youtube

NOTE: Blip.tv works a little differently than the others because there is no way to construct a working object with the player URL. Instead of the URL to the Blip.tv page, you will use the URL to the flv file, like http://blip.tv/file/get/Pycon-DjangoOnJython531.flv for example. This is located in Files and Links section of Blip.tv.

Adding extra services is easy. Note: This portion is relevant to the extension for python-markdown 1.7. The first part is defining what URL for the video service should look like. You do this in the extendMarkdown method of VideoExtension.

# This regular expression looks for a youtube URL that do not start with parenthesis.
# It does this to avoid eating regular markdown links.
YOUTUBE_RE = r'([^(]|^)http://www\.youtube\.com/watch\?\S*v=(?P<youtubeid>[A-Za-z0-9_-]+)\S*'
# Here we plug our expression into the bit that builds the video embed html.
# We define this shortly.
YOUTUBE_PATTERN = Youtube(YOUTUBE_RE)
# The next two lines allow control of markdown through instances of this class.
YOUTUBE_PATTERN.md = md
YOUTUBE_PATTERN.ext = self
# This registers everything with markdown, so the code will be executed.
md.inlinePatterns.append(YOUTUBE_PATTERN)

Next, we get to build the HTML. We need to add a new subclass of markdown.BasePattern to the module.

class Youtube(markdown.BasePattern):
    def handleMatch(self, m, doc):
        url = 'http://www.youtube.com/v/%s' % m.group('youtubeid')
        width = self.ext.getConfig('youtube_width')
        height = self.ext.getConfig('youtube_height')
        return FlashObject(doc, url, width, height)

For the most part, building the HTML is easy. I have defined a flash_object function that builds an object element that work in most cases. You only need to feed it your minidom instance, doc, an url, and width/height, both as strings. You will notice that I am using self.ext.getConfig to assign my width and height. These are the extension arguments. You will want to use these too. To do so, add a new key to the self.config dictionary of __init__ in VideoExtension.

'youtube_width': ['640', 'Width for Youtube videos'],
'youtube_height': ['385', 'Height for Youtube videos'],

The first part is the default value and the second bit is a description of the argument. If you need to define more HTML, then I suggest taking look at the Yahoo class.

Posted by Tyler Lesmann on April 2, 2009 at 8:16 and commented on 10 times
Tagged as: django markdown open_source python

There is this really nice plugin for WordPress called sociable that makes it easy to make links to various social bookmarking sites. There did not seem to be anything similar in Django, so I made a clone. It does not do everything that the sociable plugin does yet, but it is incredibly flexible. This code is licensed under LGPL.

You can get the latest version here or you can get the whole repository with mercurial:

hg clone http://code.tylerlesmann.com/django-sociable

You will probably want the sociable images also. You can get them from here.

Install is like many other python packages:

python setup.py install

Using django-sociable is easy. After install, edit your settings.py. You will need to specify a MEDIA_URL, if you have not already. Now add SOCIABLE_IMAGE_PATH, which is the path relative to the MEDIA_URL to find the sociable images. The last thing to add is put sociable into your INSTALLED_APPS.

The next step is defining a Site because django-sociable uses this to construct your submission link. You do not want it pointing to example.com. The easiest way to do this is from the Django admin site.

You are ready to use the django-sociable templatetags in your templates. Here's an example of its use from this very blog:

<div>
{% load sociable_tags %}
{% get_sociable Reddit Digg as sociable_links with url=object.get_absolute_url title=object.headline %}
{% for link in sociable_links %}
    <a href="{{ link.link }}">
        <img alt="{{ link.site }}" title="{{ link.site }}" src="{{ link.image }}" />
    </a>
{% endfor %}
</div>

Load sociable_tags. Use the get_sociable tag to get the information you need to build the links. Specify the sites to target, then the variable to map the information to, and finally the arguments. The only required argument is url, which should be a context variable containing link to submit. You can also submit title, excerpt, and source. Use the list of information from get_sociable in whatever way you like. Each element of the list has three attributes. The site attribute is the name of the target site. The link contains the full submission link. The image is the absolute path to the image corresponding to the site.

Here is a full list of the acceptable target sites:

BarraPunto blinkbits BlinkList BlogMemes blogmarks Blogsvine blogtercimlap Faves Book.mark.hu Bumpzee co.mments connotea del.icio.us De.lirio.us Digg DotNetKicks DZone eKudos email Facebook Fark feedmelinks Furl Fleck GeenRedactie Google Gwar Haohao HealthRanker Hemidemi IndianPad Internetmedia kick.ie Kirtsy laaik.it Leonaut LinkArena LinkaGoGo LinkedIn Linkter Live Ma.gnolia Meneame MisterWong MisterWong.DE Mixx muti MyShare MySpace N4G Netvibes NewsVine Netvouz NuJIJ Ping.fm PlugIM Pownce ppnow Print Propeller Ratimarks Rec6 Reddit SalesMarks Scoopeo scuttle Segnalo Shadows Simpy Slashdot Smarking Socialogs Spurl SphereIt Sphinn StumbleUpon Symbaloo Taggly Technorati TailRank ThisNext Tipd Tumblr TwitThis Upnews Webnews.de Webride Wikio Wists Wykop Xerpi YahooBuzz YahooMyWeb YCombinator Yigg

Posted by Tyler Lesmann on March 9, 2009 at 18:16 and commented on 9 times
Tagged as: django new_features open_source python

In my earlier post, I walked you through the process I took to adapt this blog, with its existing data, to include tagging under the hood. In this post, I will cover the process of making the tags useful to your visitors.

The first thing to do is make a view to filter your model, or queryset, by tag. You have it easy because django-tagging includes a view built on top of django.views.generic.list_detail.object_list. You can add a tag filtered list as easily as adding a line like this to the urlpatterns in your urls.py:

(r'^tag/(?P<tag>[^/]+)/$', 'tagging.views.tagged_object_list',
    dict(queryset_or_model=MyModel)),

Going to http://yourdjangosite/tag/python/ would now bring up a list of MyModel instances featuring a python tag, assuming you have a myapp/mymodel_list.html template written up. How will your users navigate to this view? On this blog, I first have a collection of links under each post. These are simple to make. The code is something like this in the template:

{% load tagging_tags %}
<div>
    Tagged as:
    {% for tag in object.tags %}
        <a href="/tag/{{ tag }}/">{{ tag }}</a>
    {% endfor %}
</div>

On this blog, I also have a nice list of tags on the right with the count of posts bearing each tag. Here is the code for this:

{% load tagging_tags %}
<div class="header">Tags</div>
<div class="indent">
{% tags_for_model blog.Entry as entry_tags with counts %}
{% for entry_tag in entry_tags %}
    <div>
        <a href="/tag/{{ entry_tag }}/">{{ entry_tag }}</a>
        <span class="tag_count">({{ entry_tag.count }})</span></div>
{% endfor %}
</div>

That is just too easy, but maybe you want a tag cloud, like the one on my other site. Here is the code for the tag cloud from that very site:

{% load tagging_tags %}
<div class="tagbox">
    <div class="header">Filter by Tag:</div>
    <div class="group">
        {% tag_cloud_for_model deltad.Entry as entry_tags with steps=10 min_count=1 distribution=log %}
        {% for tag in entry_tags %}
            <a class="tag{{ tag.font_size }}" href="/tag/{{ tag }}/">{{ tag }}</a>
        {% endfor %}
    </div>
</div>

The tag_cloud_for_model tag gives me a entry_tags object with 10 levels of size distributed in logarithmic order as long as there is at least one entry bearing the tag. If you look carefully, you will notice that I am using the tag.font_size to classify my anchors. You can do whatever you want with this number. I just like to keep all appearance information in CSS.

Posted by Tyler Lesmann on March 9, 2009 at 8:17 and commented on 5 times
Tagged as: django python