Simon Willison's Weblog: libxml2

How to install lxml python module on mac os 10.5 (leopard)

2008-12-15T00:05:21+00:00

How to install lxml python module on mac os 10.5 (leopard)

Instructions that work! Finally, I can find out what all the fuss is about.

Tags: leopard, libxml2, lxml, macos, python, xml

lxml.cssselect

2007-09-24T23:57:17+00:00

lxml includes an implementation of CSS 3 selectors, which compiles them to XPath expressions. Should be a useful tool for parsing Microformats from Python.

Via Ian Bicking

Tags: css, css3, libxml2, lxml, microformats, python, selectors, xpath

lxml

2005-04-13T17:04:25+00:00

lxml

A Pythonic wrapper for libxml2.

Via Martijn Faassen

Tags: libxml2, python, xml

PHP 5 Release Candidate 1

2004-03-19T01:27:51+00:00

I haven't blogged much about PHP in a while because I've been up to my nose in mod_python and loving every minute of it. This news is just too important to miss: PHP 5 Release Candidate 1 has been released, bringing the first production-ready release tantilisingly close. While I doubt PHP 5 will tempt me back it's definitely an exciting upgrade - my biggest complaint with PHP 4 is the brain-dead object model which defaults to copying whole objects rather than passing references, and this is one of the many things addressed by PHP 5. The new libxml2 powered XML features sound really powerful, and SQLite as an on-board database should be ideal for knocking out small stand-alone applications without needing to set up a mySQL database for them.

I may well throw a copy on my Mac over the weekend and try out the changes since version 4.3.

Tags: libxml2, php, python, sqlite, xml

XML.com: Lightweight XML Search Servers [Jan. 21, 2004]

2004-01-26T14:51:54+00:00

XML.com: Lightweight XML Search Servers [Jan. 21, 2004]

More fun with Python and libxml2

Via Daily Python-URL

Tags: libxml2, python, xml

Using XPath to mine XHTML

2003-10-21T05:31:23+00:00

This morning, I finally decided to install libxml2 and see what all the fuss was about, in particular with respect to XPath. What followed is best described as an enlightening experience.

XPath is a beautifully elegant way of adressing "nodes" within an XML document. XPath expressions look a little like file paths, for example:

/first/second: Match any <second> elements that occur inside a <first> element that is the root element of the document
//second: Match all <second> elements irrespective of their place in the document
//second[@hi]: Match all <second> elements with a 'hi' attribute
//second[@hi="there"]: Match all <second> elements with a 'hi' attribute that equals "there"

A full XPath tutorial is available.

The Python libxml2 bindings make running XPath expressions incredibly simple. Here's some code that extracts the titles of all of the entries on my Kansas blog from the site's RSS feed:

>>> import libxml2
>>> import urllib
>>> rss = libxml2.parseDoc(
      urllib.urlopen('http://www.a-year-in-kansas.com/syndicate/').read())
>>> rss.xpathEval('//item/title')
[<xmlNode (title) object at 0xb4b260>, <xmlNode (title) object at 0xa99968>, 
<xmlNode (title) object at 0x10dce68>]
>>> [node.content for node in rss.xpathEval('//item/title')]
['Music and Brunch', 'House hunting', 'Arrival']
>>>

Why is this so exciting? I've been saying for over a year that XHTML is an ideal format for storing pieces of content in a database or content management system. Serving content to browsers as HTML 4 makes perfect sense, but storing your actual content as XML gives you the ability to process that content in the future using XML tools.

So far, the best example of a powerful tool for manipulating this stored XML has been XSLT. XSLT has its fans, but is also often criticised as being unintuitive and having a steep learning curve. XPath is a far better example of a powerful, easy to use tool that can be brought to bare on XHTML content.

Enough talk, here's an example of what I mean. The following code snippet creates a Python dictionary of all of the acronyms currently visible on the front page of my blog, mapping their shortened version to the expanded text (extracted from the title attribute):


>>> blog = libxml2.parseDoc(
    urllib.urlopen('http://simon.incutio.com/').read())
>>> ctxt = blog.xpathNewContext()
>>> ctxt.xpathRegisterNs('xhtml', 'http://www.w3.org/1999/xhtml')
0
>>> acronyms = dict([(a.content, a.prop('title')) 
    for a in ctxt.xpathEval('//xhtml:acronym')])
>>> for acronym, fulltext in acronyms.items():
	print acronym, ':', fulltext


DHTML : Dynamic HyperText Markup Language
URL : Universal Republic of Love
HTML : HyperText Markup Language
SIG : Special Interest Group
PHP : PHP: Hypertext Preprocessor
CSS : Cascading Style Sheets
>>>

The above code is slightly more complicated than the first example, as using XPath with a document that uses XML namespaces requires some extra work to register the namespace with the XPath parser. Still, it's a pretty short piece of code considering what it does.

For an example of how powerful XPath can be on a much larger scale, take a look at Sam Ruby's XPath enabled blog search feature.

Tags: libxml2, python, xhtml, xml, xpath, xslt