I work at Tangent Labs, a digital agency, writing applications in python and django. I spend most of my free time hacking. I run commandlinefu.com and am the author of the e-commerce framework django-oscar amongst other things. I used to be a mathematician; I have a PhD in Mathematics from the University of Nottingham and an associated interest in cryptic crosswords, chess and playing devil's advocate. David is a DZone MVB and is not an employee of DZone and has posted 25 posts at DZone. You can read more from them at their website. View Full User Profile

purl - Immutable URL Objects for Python

04.25.2012
| 3040 views |
  • submit to reddit

Working with URLs in Python feels clunky when it should be pleasant. In urlparse and urllib, the standard library has all the functionality you need, but the code you have to write is often cumbersome and unclear.

For instance, here's a typical test method that makes an assertion about a query parameter:

import urlparse

def test_url_has_correct_query_parameter(self):
    url = get_url_from_somewhere()
    parse_result = urlparse.parseurl(url)
    query_params = urlparse.parse_qs(parse_result.query)
    self.assertEqual('testing', query_params['q'][0])

Not terrible, but could be more concise. I would prefer something like:

from somelibrary import URL

def test_url_has_correct_query_parameter(self):
    url = URL.from_string(get_url_from_somewhere())
    self.assertEqual('testing', url.query_param('q'))

Further, when working with webservices, you often need to build URLs programmatically but it just isn't easy enough in python. You often end up using string formatting:

import urllib

URL_TEMPLATE = 'https://github.com/%s?w=%s'
def get_github_url(username):
    return URL_TEMPLATE % (urllib.quote(username), '0')

A preferable API might look something like:

from somelibrary import URL

BASE_URL = URL.from_string('https://github.com/')
def get_github_url(username):
    return BASE_URL.path_segment(0, username).query_param('w', 0)

This is a toy example, the problem is much worse when building more complicated URLs.

purl

So I wrote a utility class to scratch this itch. It's a simple immutable URL class that uses jQuery-style overloading of the attribute methods to be both accessors and mutators.

Install with:

pip install purl

Construct URL instances as follows:

from purl import URL

# Explicit constructor
u = URL(scheme='https', host='www.google.com', path='/search', query='q=testing')

# Use factory class-method
u = URL.from_string('https://www.google.com/search?q=testing')

# Chain mutator methods (which each return a new instance)
u = URL().scheme('https').host('www.google.com').path('search').query_param('q', 'testing')

# Combine to suit your use-case
u = URL.from_string('https://www.google.com').path('search') \
                                             .query_param('q', 'testing')

There's a full range of inspection methods:

# Simple attributes
u.scheme()      # 'https'
u.host()        # 'www.google.com'
u.domain()      # 'www.google.com' - alias of host
u.port()        # None (only returns something if explicitly set)
u.path()        # '/search'
u.query()       # 'q=testing'
u.fragment()    # 'q=testing'

# Convenience methods for inspecing path, query and host
u.path_segment(0)                   # 'search'
u.path_segments()                   # ('search',)
u.query_param('q')                  # 'testing'
u.query_param('q', as_list=True)    # ['testing']
u.query_param('lang', default='GB') # 'GB'
u.query_params()                    # {'q': 'testing'}
u.has_query_param('q')              # True
u.subdomains()                       # ['www', 'google', 'com']
u.subdomain(0)                       # 'www'

Each accessor method is overloaded to be a mutator method too, similar to the jQuery API. Since the URL class is immutable, any mutation will return a new URL instance.

u = URL.from_string('https://github.com/codeinthehole')

# Access
u.path_segment(0) # returns 'codeinthehole'

# Mutate (creates a new instance)
new_url = u.path_segment(0, 'tangentlabs') # returns new URL object

Here's a fancier example of building a URL:

u = URL().scheme('https')\
         .host('github.com')`\
         .path_segment(0, 'codeinthehole')\
         .path_segment(1, 'purl')\
print u.as_string()

# returns 'https://github.com/codeinthehole/purl'

Source and further details on Github.

Alternatives

There are a couple of URL classes already for python - however neither had the exact API I was looking for.

  • mxURL - Part of the 'eGenix.com mx Base Distribution', this has quite a comprehensie API. It comes bundles with other utility modules with the 'egenix-mx-base' package.
  • URLObject - There's nothing wrong with this implementation - it's very similar to my one above. The API's not quite to my tastes but that's purely subjective thing.

Discussion

There is a discussion of this post on /r/Python.

Published at DZone with permission of David Winterbottom, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Tags: