Beardy Geek

Google Calls for Link Spam Reports

BeardyGeek — Thu, 04 Mar 2010 10:34:45 +0000

In Matt Cutts latest blog post, he calls for users to report any sites they deem ’spammy’. On the report form, this includes anything that is considered:

Hidden text or links
Misleading or repeated words
Page does not match Google’s description
Cloaked page
Deceptive redirects
Doorway pages
Duplicate site or pages

Matt says that Google have been working on new algorithms to detect link spam, and it seems that through this manual reporting process, Google users will be helping to ‘train’ the system to be more effective.

It doesn’t say yet what action will be taken upon finding such a page, and whether Google will allow webmasters to make amends or defend themselves. So look out for tales of woe from people removed from Google’s index for spurious reasons.

Django Tips – Unique Date Querysets

BeardyGeek — Sun, 15 Feb 2009 18:50:26 +0000

I swear this particular feature of querysets was made especially for the archive listing of a blog. You know, the month and year listing in the side bar for all your previous posts. This solves that problem exactly.

Let’s say you have a BlogPost model, and a post_date field within said model. You want a list on the sidebar of all the unique month and year combinations. Here’s what you do:


queryset = BlogPost.objects.dates('post_date', 'month')

And that’s it. You’ll now have a query set full of datetime objects with all your unique months and years.

To get unique years only, substitute ‘year’ for ‘month’, and for unique day, month and year, substitute ‘day’ for ‘month’.

From Wordpress to Django – Part 2

BeardyGeek — Tue, 27 Jan 2009 18:59:08 +0000

View Part 1
Beardy Geek Git Hub Repository

Welcome back. In part 2 I’ll be getting to the meat of the issue, which is retrieving the data from an existing Wordpress blog, and feeding the data into my own models.

Models

I’ve decided to start the models from scratch, rather than trying to copy the way Wordpress has laid them out. It should keep things simpler that way. I’ll be using the contrib comments system for the post comments, and I’ll also be creating 3 other models, Category, Tag, and Post. Here’s the code for the models:


from django.db import models
from django.contrib.auth.models import User
from datetime import datetime

POST_STATUS = (
               ('P', 'Published'),
               ('U', 'Unpublished'),
)

class Tag(models.Model):
    text = models.CharField(max_length=75)
    slug = models.CharField(max_length=75)

    def __unicode__(self):
        return self.text

class Category(models.Model):
    text = models.CharField(max_length=75)
    slug = models.CharField(max_length=75)

    def __unicode__(self):
        return self.text

    class Meta:
        verbose_name_plural = "categories"

class Post(models.Model):
    title = models.CharField(max_length=75)
    slug = models.CharField(max_length=75)
    content = models.TextField()
    author = models.ForeignKey(User)
    post_date = models.DateTimeField(default=datetime.now)
    status = models.CharField(max_length=1, choices=POST_STATUS)
    categories = models.ManyToManyField(Category)
    tags = models.ManyToManyField(Tag)

    def __unicode__(self):
        return self.title

Wordpress Export

Just a quick note: I am using Wordpress Version 2.6.1. Things may be different in other versions, but the code here works on this version. I’ll go through the code so you should see where the problem is if it doesn’t work with your version.

OK, if you didn’t already know, you can export all your Wordpress data into an xml file. Go to your dashboard, click ‘Manage’, and then ‘Export’. Select which authors to restrict (if any) and hit the download button. You should now have the data in an xml file.

XML File Editing

When I first tried to parse this file, I got an error as one of the namespaces used in the document is undeclared. To rectify this, you need to open up the file in a text editor (not an xml editor, you may get the same parse error). Near the top of the file, you should see something like this:

The missing namespace is ‘excerpt’, and we need to add this. It doesn’t matter what value you give it. I did this:

Save the file and we’re ready to start parsing the data.

XML Parsing

To parse the XML I have used ElementTree. This is included in Python 2.5, but if you’re using an earlier version, you can get ElementTree from Effbot.org.

I won’t include all the code here, just some snippets as examples. To view the full source, please check out the BeardyGeek Github Repository.

First we need to load our xml file.


tree = ElementTree.parse('c:/wordpress.xml')

Then find the top level under which our data resides, which is the ‘channel’ tag.


chan = tree.find('channel')

Now I want to create some shortcut variables for the namespaces that we will use when finding tags.


wp_ns = '{http://wordpress.org/export/1.0/}'
content_ns = '{http://purl.org/rss/1.0/modules/content/}'

Now we can get all the category, tag and item(post) entries:


cats = chan.findall('{http://wordpress.org/export/1.0/}category')
tags = chan.findall('{http://wordpress.org/export/1.0/}tag')
items = chan.findall('item')

This will give us lists of all the elements for those three items.

Finding and Saving Data

I’ll give an example of saving the data using the category tag.


for cat in cats:
        c = Category(text=cat.find(wp_ns + 'cat_name').text,
                 slug=cat.find(wp_ns + 'category_nicename').text)
        c.save()

The tag data is the same as the above.

The item data is a bit more complex. If you look at the xml you’ve exported, you’ll see that the item data includes both posts and pages. But it also gives all previous revisions of each post, which will include any drafts saved whilst writing a post. So we need to find all those with a status of ‘publish’ and a page type of ‘post’. We’ll deal with the ‘page’ data another time, using Flatpages.


if item.find(wp_ns + 'status').text == 'publish' and
               item.find(wp_ns + 'post_type').text == 'post':
            i = Post(title=item.find('title').text,
                     slug=item.find(wp_ns + 'post_name').text,
                     content=item.find(content_ns + 'encoded').text, author=u,
                     post_date=item.find(wp_ns + 'post_date').text,
                     status='P')
            i.save()

The ‘u’ (value for author) is a User object I create earlier in the code that I’ve used as the default author of each post (see source).

Post Categories

Now we have to find out which categories and tags this post has. Within each ‘item’ we have ‘category’ data. A bit confusingly this ‘category’ data also includes the tags, and to discover that you need to look at the ‘domain’ attribute to see which it is. Plus we only need the category with the ‘nicename’ in it (slugified).


post_cats = item.findall('category')

for pc in post_cats:
     #check for attributes
     if pc.get('nicename'):
         if pc.attrib['domain'] == 'category':
              c2 = Category.objects.get(slug=pc.attrib['nicename'])
              i.categories.add(c2)
         elif pc.attrib['domain'] == 'tag':
              t2 = Tag.objects.get(slug=pc.attrib['nicename'])
              i.tags.add(t2)

Comments

The last section deals with comments. I am using the django.contrib.comments module for this.



comments = item.findall(wp_ns + 'comment')
for comm in comments:

    if not comm.find(wp_ns + 'comment_author_email').text:

        comm_email = ''

    else:

        comm_email = comm.find(wp_ns + 'comment_author_email').text
        if not comm.find(wp_ns + 'comment_author_url').text:

            comm_url = ''

        else:

            comm_url = comm.find(wp_ns + 'comment_author_url').text
        db_comm = Comment(comment=comm.find(wp_ns + 'comment_content').text,

            ip_address=comm.find(wp_ns + 'comment_author_IP').text,

            object_pk=i.id, submit_date=comm.find(wp_ns + 'comment_date').text,

            user_email=comm_email,

            user_name=comm.find(wp_ns + 'comment_author').text[:50],

            user_url=comm_url,

            content_type=ct, site=site)

        db_comm.save()





Conclusion
Well that wraps it up for this post. I'll cover extracting the data for Flatpages in the next post, but this should give you enough to get started. You can see how to extract the required data from the xml document, so if you want to extend the models beyond what I have, you shouldn't have any problems.  Again, check out the fully code, plus the other file changes (url.py etc) at the Beardy Geek Git Hub Repository. Have fun.

From Wordpress to Django – Part 1

BeardyGeek — Wed, 14 Jan 2009 18:55:00 +0000

Now don’t get me wrong, there’s nothing wrong with Wordpress. It’s just that I like to play with stuff, so I thought it would be fun to create a blog in Django, copy all my Wordpress posts across, and add at least some of the functionality that Wordpress has built in.

Open Source

Throughout this process, so that you can follow along, I will be storing the code in a GitHub repository. Click here to view the BeardyGeek repository

A Few Considerations

There are a few things to think about before getting started:

Data Conversion – getting the data from Wordpress into my Django models. Do I want to keep the same database structure as Wordpress, or create my own? If I create my own, how will I get the data across?
Services – how much do I try and implement? The main ones are comments, tags, pings, anti-spam.

I’m sure more will come to mind as I go through this

Conclusion

Well, I’d better stop jabbering and get on with it then! Keep an eye on the blog, or watch the Git repo for updates. I’ll get done whatever my current workload allows.

How to Call a .Net Webservice using Python

BeardyGeek — Thu, 01 Jan 2009 12:51:46 +0000

My day job involves working with software that automatically creates webservices on the .Net platform. Up until now I have used C# to create web applications to use these webservices.

But it would be nice to have the flexibility to use another programming language to create a solutions, and consume the webservices from there.

So, here’s a short tutorial on calling your .net webservice using Python.

Pre-requisites

I’m currently using Python 2.5, so I can’t speak for other versions
You will need the ElementSoap package from effbot.org. You can get it from here
You’re also going to need a .Net webservice with which to test this out. If you’re reading this tutorial, you probably already have one.

SOAP

Firstly we need to look at the SOAP examples on your .Net webservice. Go your webservice’s asmx page, and then click on one of your webservices to view the details. You will see some example SOAP requests and responses. The one we’re interested in here is SOAP 1.1. Mine looks like this:


POST /lookserver/webservices.asmx HTTP/1.1
Host: localhost
Content-Type: text/xml; charset=utf-8
Content-Length: length
SOAPAction: "http://tempuri.org/GetVersion"



  
    
  


HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8
Content-Length: length



  
    
      string

As we can see here, I have a webservice called GetVersion, the namespace is http://tempuri.org, and the SOAPAction is http://tempuri.org/GetVersion. Note these down for your webservice.

Into the Python

Fire up an interactive session, and we’ll go through calling this service.

Firstly we need to import the ElementSOAP library:

from elementsoap import ElementSOAP as ES

If you get an error message, then you haven’t installed ElementSoap properly.

Next we create a SoapRequest:

sr = ES.SoapRequest("{http://tempuri.org/}GetVersion")

And now we initialize a SoapService object:

serv = ES.SoapService("http://localhost/lookserver/webservices.asmx")

Obviously put in your own url for your webservice.

Now we call the webservice, using the SOAPAction value:

result = serv.call("http://tempuri.org/GetVersion", sr)

The result is an xml element which should contain the response from your webservice. When I type result I get:

If you look at the xml at the top of the article, you’ll see that the result returns inside a GetVersionResponse tag. To get the value of the result we type:

result.find("{http://tempuri.org/}GetVersionResult").text

This finds the appropriate tag, and returns the text.

Conclusion

I hope this has helped. When I searched, I found that the other tutorials were either confusing, or out of date. Take a look at the various libraries at effbot.org, especially ElementTree for parsing XML, very useful. Enjoy!

Django and Ajax – using Prototype and Scriptaculous

BeardyGeek — Fri, 12 Dec 2008 18:53:00 +0000

I started adding a sprinkling of Ajax in one of my Django projects, so I thought I’d share a few snippets that might help you add some Ajax goodness into your app.

To handle the Ajax callbacks, I’ll be using Prototype, a lightweight Ajax library. And for a couple of effects, like fade-in, I’ll be using Scriptaculous.

Ajax Voting

I wanted to add a Digg-like voter to my app, where you have the number of votes showing with a vote button, and when you click on the vote button, the number disappears, then fades back in with an increase in the vote. The new vote will be added to the database. I also wanted to only allow each user to only vote on each item once (for obvious reasons!)

Overview

Here’s how it hangs together. The user presses the vote button. This calls back to a url, which has a view, which retrieves and/or stores data, which then renders the output to a template, which is then inserted by magic under the button with the new stored value.

Now on with some code.

Urls

First we need to define a url for the callback.

url(r'^callback/vote/(?P\d+)/$', 'vote', name='callback_vote'),

Models

For the purposes of this post, I’ll create 2 models. The first is an item model, where we can store an item description, and the number of the votes that item has:


class Item(models.Model):
    item_desc = models.CharField(max_length=200)
    votes = models.IntegerField()

Now we can create a Voter model, which stores who voted for which item, so we can prevent multiple voting for each user:


from django.contrib.auth.models import User

class Voter(models.Model):
    voter = models.ForeignKey(User)
    item = models.ForeignKey(Item)

Script Tags

Now we need to import the Javascript for Prototype and Scriptaculous. We can either do this in our base.html, which means the javascript will be loaded on every page, or we could add a block into the header, and just load the javascript on a per page basis. I have also stored the scripts in my MEDIA_URL.

You’ll see the load=effects option after the scriptaculous url. This is so it doesn’t load all of the modules. You can if you want, but for this example we only need the effect module.

Prototype Script

For the prototype script to work, and update our vote number, we need to have a page element to update. You could have a div or a span, but you need to give the element an id and a name.

Now for the script.

I’m assuming we’ve called the span or div id ‘item’. The $(‘item’) is a short cut to get a reference to the element. Firstly we call the function with the item id as a parameter. Then we set the display to ‘none’ before doing the callback. The callback is done, we update the div element using result.update, and the final line uses the Scriptaculous ‘appear()’ function, which fades the div back in.

Don’t forget to add an onclick() or similar event to what ever you want to use to trigger the callback. In my case I used the tick image.

I used n as a context for my item on the view for this page, in case you’re wondering what the {{ n.id }} is for.

Callback Code

Now we have the client side script, we need to take that item id and do something with it, then return the result for display.

First we need to create the vote view.


def vote(request, item_id):
    i = Item.objects.get(pk=item_id)
    #see if user has voted
    voter = Voter.objects.filter(voter=request.user, item=i)

    if not voter:
        v = Voter(voter=request.user, item=i)
        v.save()
        i.votes += 1
        i.save()

    return render_to_response('item/callback_vote.html', {'votes':i.votes})

We get our item object first, then check the Voter model to see if the current logged in user has already voted for this item. If they haven’t, then we create a Voter entry, and increase the number of votes by one. We then return the number of votes in the context.

Callback Template

Now we create a simple template to display the vote view.


{{ votes }}

Conclusion

And there we have it. That’s the basics of Ajax in Django. There’s a lot more to Scriptaculous than just the fade and appear, so check it out.

Creating User Profiles with Image Thumbnails in Django

BeardyGeek — Thu, 20 Nov 2008 18:48:18 +0000

I am currently working on a django application that will include a profile page for each user. Like many web apps, it will have the ability to upload a profile picture, and have this picture shrunk to a suitable thumbnail size.

To complicate matters, I want the photo uploaded to a directory which will be named after the user.

I’m fairly new to Django, so it took some time to piece this all together, but someone out there may have had the same problems and I hope this post helps.

Prerequisites

You must have the Python Imaging Library installed. You can get this from the PIL homepage.

User Profiles

The first thing we need to do is create a model for our user profile.

from django.db import models
from django.contrib.auth.models import User

class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)
    nickname = models.CharField(max_length=50, blank=True, null=True)
    photo = models.ImageField(upload_to='profile_pics', blank=True, null=True)
    thumbnail = models.ImageField(upload_to='profile_thumb', blank=True, null=True,
          editable=False)

I’ve made the upload_to attributes static for the time being, but we will be changing this later.

User Profile Setup

So that Django knows which model to use for you user profile, you have to add a setting to your settings file.


AUTH_PROFILE_MODULE = 'appname.UserProfile'

Make sure that this uses your app name, not your project name. I’ve used UserProfile for the model, but you can call it whatever you like.

Dynamic Upload Directory

To make the upload_to attribute dynamic based on the current user, we have to write a small function that returns the appropriately formed path. My thanks to Josh Ourisman and his post for this part.

Create a file called files.py and save it in your application directory.


def get_profile_path(instance, filename)
    dir = "%s/profile_pics/%s" % (instance.user, filename)
    return dir

This will make the profile picture save in the directory ‘username/profile_pics/filename’. You can move these around if you want a different directory structure. The MEDIA_ROOT value from your settings file will be prepended to this path automatically, so make sure you set that.

Amend the Model

Now back to our model, where we call our new function from the upload_to attribute.


from projname.appname.files import get_profile_path

class UserProfile(models.Model):
    user = models.ForeignKey(User, unique=True)
    nickname = models.CharField(max_length=50, blank=True, null=True)
    photo = models.ImageField(upload_to='get_profile_path', blank=True, null=True)
    thumbnail = models.ImageField(upload_to='profile_thumb', blank=True, null=True,
         editable=False)

We will leave the thumbnail upload_to field, as we will be dealing with that one manually.

Creating the View

I wanted to just create one view that would handle creating a new profile, and displaying and updating an existing profile. First I’ll create a form to display in the view.


from django import forms
from django.forms import ModelForm
from projname.appname.models import UserProfile

class UserProfileForm(ModelForm):
    class Meta:
        model = UserProfile
        exclude = ('user')

Now to the view


from django.shortcuts import render_to_response
from django.template import RequestContext
from famblog.blog.forms import UserProfileForm
from famblog.blog.models import UserProfile

from django.contrib.auth.decorators import login_required

@login_required
def profile(request):
    try:
        myprofile = request.user.get_profile()
    except:
        up = UserProfile(user=request.user)
        up.save()
        myprofile = request.user.get_profile()

    if request.method == 'POST':
        f = UserProfileForm(request.POST, request.FILES, instance=myprofile)
        if f.is_valid():
            f.save()
    else:
        f = UserProfileForm(instance=myprofile)

    return render_to_response('appname/profile.html', {'f':f, 'profile':myprofile},
        context_instance = RequestContext(request))

This view will try and retrieve the user profile based on the current logged in user. If the user profile doesn’t exist, then an exception is thrown. If this happens, we create a profile with just the username and call get_profile() again. We then deal either with a POST event, or we bind the retrieved profile instance to the form ready for updating by the user.

If you don’t want to create thumbnails of your images, you can leave it here. The images will have been loaded

Thumbnail Creation

To create the thumbnail, we now need to override the save def in our model.


def save(self, force_insert=False, force_update=False):
        #get mtime stats from file
        thumb_update = False

        if self.thumbnail:
            statinfo1 = os.stat(self.photo.path)
            statinfo2 = os.stat(self.thumbnail.path)
            if statinfo1 > statinfo2:
                thumb_update = True

        if self.photo and not self.thumbnail or thumb_update:
            from PIL import Image

            THUMB_SIZE = (150, 150)

            #self.thumbnail = self.photo

            image = Image.open(self.photo.path)

            if image.mode not in ('L', 'RGB'):
                image = image.convert('RGB')

            image.thumbnail(THUMB_SIZE, Image.ANTIALIAS)
            (head, tail) = os.path.split(self.photo.path)
            (a, b) = os.path.split(self.photo.name)

            if not os.path.isdir(head + '/thumbs'):
                os.mkdir(head + '/thumbs')

            image.save(head + '\\thumbs\\' + tail)

            self.thumbnail = a + '/thumbs/' + b

        super(UserProfile, self).save()

First we set a thumb_update var to false. Then, if the thumbnail exists, we check to see if the timestamp on the original image is newer than the existing thumbnail. If we didn’t check for this, then when we changed the image, we wouldn’t know whether or not to create the thumbnail.

The rest of the code deals with using the Imaging Library to resize the image. It also creates a ‘thumbs’ directory if one doesn’t already exist, saves the image, and updates the ‘thumbnail’ field in the model.

Conclusion

I hope this has been helpful. Like I said, I haven’t been using Django or Python for more than a few weeks, so if anyone has any suggestions for improving this code, then please comment.

Enjoy!

Django Template Auto-completion in Eclipse

BeardyGeek — Tue, 11 Nov 2008 13:56:28 +0000

I’ve just recently started using Django and I wanted to set up an IDE for some coding niceties.

I already use Eclipse for PHP development, so I installed the PyDev plugin for Eclipse.

Now I get autocomplete in the py files, but when writing the Django html templates, there is no autocomplete for the various template tags.

So here’s a quick guide to adding your own tags in Eclipse.

In Eclipse, go to Window->Preferences
Go down to Web and XML.
Then to HTML files and finally Templates
You will see a list of all the various HTML templates there.
Click on New.
Give your tag a name. It can’t be one that’s already taken, but I found I didn’t have any issues. If you want to keep them all together you could start your tag name with ‘DJ’.
Select HTML Tag from the drop down.
Give your tag a description.
Type in the the tag pattern. If you want the full tag on autocomplete, you might type in {% block ${cursor} %}, otherwise you could start with block, which means you would need to type in the {% and then activate the autocomplete in the HTML editor. The ${cursor} places the cursor where you want it after the tag has been inserted.
Click Apply and OK.
Test this out in the HTML editor. Press CTRL and Space. You should get a list of available tags. Type b (or what ever the first character is for you tag name), and you’ll see your block tag with the description next to it.
Press enter, and your tag will appear.

Hopefully this will make your Django coding a little easier.

Web 3.0 – How to put Semantic Tags in your Blog

BeardyGeek — Sat, 01 Mar 2008 12:00:05 +0000

I read the other day how Yahoo was going to start supporting semantic web standards.

The semantic web is something that is often associated with Web 3.0 (if that’s what you want to call it!), and will allow greater data exchange, better and more targeted search, and generally open things up for a super mash-up. Take a look at Wikipedia’s definition of Web 3.0 for more info.

One of the formats that Yahoo will initially support is hAtom, which is a microformat especially for blog entries.

So, I thought I’d take a look at it and try and implement it in the BeardyGeek blog.

First Steps

To add an hAtom feed to your blog, all you need to do is add some class names to what are probably existing divs or spans in your blog template.

For example, if you have a div around your entire content that looks like this:

you just need to add an extra class tag called hfeed:

To have more than one class, just separate them with a space.

There are 3 more main tags that you need.

hentry – the outer tag for each blog post
entry-title - the tag for the post title
entry-content – the tag for your post content

And that’s it. Easy as pie!

How I changed my Wordpress Theme

Ok, here’s what I changed in my theme.

Header page – At or near the bottom is a div tag –

– I changed that to

Single Post - I added the hentry tag to

so it now read

. I then added entry-title to the

so it read
. And I also add entry-content to
so it read
.

Main index template – pretty much the same as the single post page, as the tags are all the same.

Testing

To test the feed, I downloaded Greasemonkey plugin for Firefox. I then downloaded the RSS Panel X script for Greasemonkey, which allows you to view a page’s feeds in a small JavaScript window. This is ideal for testing in this case as it supports hAtom.

Once you have these installed and running (tip: install the plugin first, then the script installs automatically), just load up your blog page. If all went according to plan, you should see your hAtom feed along with your RSS feed.

In fact, you can test it out here, as I have my hAtom feed up and running!

Good luck, let me know how you get on.