Archive for the ‘Development’ Category

The Guardian Lite android app

Monday, July 12th, 2010

Guardian Lite, an unofficial android reader for the Guardian newspaper, is now available on the Android Market (search for ‘Guardian’).

The application aims to encourage wide ranging browsing through the Guardian’s extensive collection of sections, tags and contributors as well as providing the fastest possible syncing for off line viewing.

Features

Online viewing

Articles are available for viewing immediately if a network connection is available. No pre syncing is required to begin browsing content.

Top stories

The current main stories from the network front of the Guardian website.

Contributors and tag links

Navigate through the Guardian’s current content with the tag and contributor links attached to each article. Discover relevant blogs and contributors for each section.

Favourite tags

Mark sections, tags and contributors for quick reference and offline syncing.
The favourites screen contains a custom news feed based on your favourites collection.

Offline syncing

Download your favourite article sets in the background for offline viewing.
Considerable attention has been given to making the syncing process as efficient as possible.

Syncing speeds of up to 100 articles per minute should be possible over a Wifi connection.

The application aggressively caches articles viewed while online for up to 24 hours. These articles can still be browsed when no network connection is available.

Guardian Lite is designed to run on any android 1.5 or higher device.

How does it work

Guardian Lite makes use of the Guardian’s excellent RSS feeds and Content API. The article content is sourced from the RSS feeds, while the Content API provides information about available sections and tags.

Information from these two data sources is collated in a Google appengine hosted proxy. The proxy formats and compresses the content for efficient delivery to user’s handsets over limited bandwidth connections.

To enable fast performance on low end devices, the application uses flat files and vanilla Java serialization for local storage.

Thanks to

Many thanks to those who assisted with device testing, including @gdb_, @robertomango, @bruntonspall, @brunns and Alan Lord.

New Zealand Government feeds

Wednesday, March 17th, 2010

NZ Govt feeds is an unofficial, experimental aggregation of New Zealand Government news items.

The list of source feeds is drawn from Mike Riverdale’s collection of Government feeds.

The site software was originally developed to drive the wellynews local news feed.

The system provides a number of outputs which might be of interest to developers:

- Content can be tagged.
Users may add tags to individual resources. These tagging votes are combined with automatic tagging to arrange news items into different output feeds.

- Every tag has an RSS feed.
For example, the Auckland tag page has it’s own RSS feed.

- Tags and publishers can be combined.
ie. Health and Wellington related items, or everything the Electricity Commission has said about Auckland.

- All feeds are also available in JSON format
See the API instructions for details.

The site still contains some references to it’s original Wellington related usage; these will be removed if this turns out to be of use to people.

Any enquires should be directed to enquires (at) wellington.gen.nz.

Multihoming a PHP application with Apache2::PerlSections

Sunday, November 15th, 2009

Squadlist is a simple Mysql / PHP application which is used by a number of UK rowing clubs. Each instance of the application is hosted on a separate Apache virtual host and uses a separate Mysql database.

Unmodified, each installation requires a separate copy of the webapp and a separate Apache virtual host configuration file. Maintaining separate copies in this way would be complicated. Generally, I don’t do complicated, so needed a way to reduce the duplication.

Extracting the database configuration into the Apache virtual host

The application contains a service class which holds the database connection creditials for the given instance:

The first problem is that this file needs to be localised for each instance, requiring a separate copy of the application to be installed and maintained for each new instance.

We’re going to fix that by defining each instances unique database connection settings in the apache environment for each given vhost

Then in PHP, we can assess the apache environment to retrieve the database credentials specific to this instance.

Because our PHP app is completely stateless, this is a safe approach and we’re now down to one copy of the application files.

Automatically generate the apache vhosts

Next problem is maintaining the apache vhosts.

Adding a new install requires a new vhost file to be hand crafted and installed. Upgrading the system requires the DocumentRoot to be updated in each file.

Enter Apache2::PerlSections which lets you execute Perl code in your apache config files. By placing the credentials for each instance into a mysql database, we can use a Perl DBI script to automatically generate the required vhosts on apache startup.

‘instances’ is the Mysql table containing our credentials. The connection details for this instance are in the root owned apache conf file so this does not expose the access details for the instance databases.

Note how $PerlConfig lets you print conf directly into Apache. To reiterate to those who haven’t seen this approach before – that Perl code is in the Apache conf files and is actually been executed inside Apache on startup to generate the vhosts.

Adding a new instance of the application now consists of adding a row to ‘instances’, setting up the DNS and restarting apache.

Sceptics may well ask what drugs was I on I wrote this? I’m happy to answer; it was Oseltamivir

Wellynews tips – custom RSS feeds

Sunday, November 15th, 2009

A number of people take the main wellynews RSS feed at http://wellington.gen.nz/rss. However, this is just one of many feeds which the system automatically creates.

Tag feeds

All tags have there own RSS feed. You just need to substitute the tagname into the url:

ie. http://wellington.gen.nz/transport/rss

This feed contains news items tagged as transport, or any of the transport sub tags (such as airport and parking). It also includes all news items published by organisations tagged with transport (ie. Snapper card).

A full list of available tags is shown here.

Combinations

If you want an RSS feed about, say, zombie outbreaks but only on the waterfront then you can use a combination feed.

ie. http://wellington.gen.nz/zombies+waterfront/rss

Note two tag names separated by a plus sign in the url. Interestingly enough, zombies and waterfront is really a valid combination and will return content.

Publisher combinations

If you’re looking for what a given publisher has said about a given tag, you can extract that using a publisher combination.

For example, mentions of Newtown made by Wellington City Council.

ie. http://wellington.gen.nz/wellington-city-council+newtown/rss

A list of available publishers is shown here.

This url scheme is inspired by the excellent implementation on the Guardian newspaper’s site.

WCN hosted content

Monday, June 8th, 2009

Wellington Community Net (WCN) is a free web hosting service for Wellington community groups. WCN has been funded by the City Council for a number of years now, but that funding is now under review.

WCN hosts 600 odd community groups, meaning that alot of the content that Search Wellington indexes is probably hosted on WCN. I wanted to find out how much.

All WCN sites are physically hosted on the same machine. Taking a dump of all valid websites from Search Wellington, and filtering out the who’s DNS resolves to the WCN server, should identify WCN hosted sites.

The Search Wellington API can then be used to tag each of these sites with a hidden ‘wcnhosted’ tag. The following Perl script checks the DNS for the uses an API call to apply a tag to wcn hosted sites.

All of the news items and feeds published by these sites are automatically pulled into this tag, giving:

http://wellington.gen.nz/wcnhosted

Which shows that approximately 340 of 9000 odd indexed, news items originated from WCN.

This is a low measurement, as Search Wellington chooses not to list all groups on WCN and only includes news items with a permanent url, but I think shows that the loss of WCN would leave quite a significant hole in the local webspace.

Guardian Hackday project live

Wednesday, January 21st, 2009

The Afganistan casualties feed developed during Guardian Hackday is now live at a permanent location.

The contents of the MOD confirmed fatalities page is available as an RSS feed or can be followed on twitter:

RSS feed: http://cambridge.wellington.gen.nz/feeds/afghanistan
Twitter: @ukcasualties

Blessed are the users…

Saturday, January 19th, 2008

… for they’re cleverer than you think.

Two examples of user’s exceeding expectations this week.

1) Submitting availability from a cellphone

Squadlist is as web based system which allows rowers to share their availability for training sessions.

"POST /rowers_availability.php HTTP/1.1" 200 8381 "-" "Nokia6300/2.0 (05.00) Profile/MIDP-2.0 Configuration/CLDC-1.1"

That looks like somebody in Wales using a cellphone to confirm their availability for an outing.
In the two years I’ve been involved in maintaining this system the idea that it might be accessed from a cellphone never occured to me.

In hindsight of course that makes perfect sense; the information in the system is most useful at 7am in the morning when everyone is standing around outside a boat shed; not when everyone is in the office in front of a PC.

2) Search Wellington tagging

This new submission for the Wellington City Chorus is correctly tagged with 4 relevant tags; even though the submission form is complete rubbish.

The tag selection form is the quickest, simplest implementation possible; a multiple select box.

It’s terrible for a number of reasons. Firstly, the user has to scroll through a massive list of options looking for possible matches. Secondly, to select more than one tag the user has to know who to perform a multiple select. This involves holding down a button somewhere in the bottom left of the keyboard and carefully clicking on a selection; then having to scroll to your next selection and clicking again (without been able to seen the effect on your first selection).

When this was implemented it was primararly for my own use; I expected no one would ever submit more than one tag through that interface.

In constrast, we’ve NEVER had a submission come through that form which didn’t have multiple, correct selections.

It’s a Bug!

Sunday, June 10th, 2007

The spider which was crawling around the side of the laptop screen this afternoon.

A real bug

It’s been far too warm to be working indoors recently, so most of my writing has been happening under the oak trees by the river or around Richmond Green. It’s definately a better working environment than the flat (and almost as good as the trains); creepy crawlies not withstanding.