Postcodes have become the new bugbear of the UK opendata campaigners, its something everything knows and uses as their default spatial reference to home or work. As such, its the reference that is essential to make many location enabled websites work, we almost unconciously look to enter our postcode in a little searchbox when we see a map.
I really, really don’t want to see a massive debate rage in the comments about “ooh, it should be free”, yes we all agree, but it won’t happen. The Postcode Address File is a commercial asset to the Royal Mail, who are a quasi-private company, its their IP and funnily enough they ain’t going to give it away for free.
Ernest the Troll
Ernest Marples, was a valiant effort but as someone put it, was a “blatant trolling exercise” on the Royal Mail. Raising the flag, saying we’re making your copyrighted data freely available for good purposes, come and get us if you want a veritable hurricane of bad publicity.
Predictably, the Royal Mail responded with legal threats and poor old Ernest was exiled to Monaco. In the even more predictable furore that followed, we had the futile petition to the Prime Minister, and an awful lot of ‘outraged of Oxford’ posts on the OSM and data.gov.uk mailing lists.
We Need a Practical Solution
Putting together something that’s usable and won’t get you sued seems like a top priorty to me. It turns out a Masters in GIS was useful after all. The Office of National Statistics aggregates the UK’s census and national statistics to a hierarchical spatial referencing system called Output Areas, like this one which includes my postcode:
From conversations with people from the ONS, this is free from copyright, confirmed here:
Boundaries for Output Areas (OAs), Lower Layer Super Output Areas (LSOAs), Middle Layer Super Output Areas (MSOAs) and Travel to Work Areas (TTWAs) are available free of charge to all users.
The supply of boundaries for all other geographies requires recipients to be licensed for use of OS Boundary-Line™ before we are able to supply them.
ONS Website, 20th October 2009
Output Areas are free of Ordnance Survey contamination and available to all without licence. Win. You can request a copy here.
What about Postcodes?
The ONS also make available postcode to Output Area lookup files, correct as of 2004. Free of copyright. Big win:
Conditions of Supply
There are no restrictions on the use of the information contained on this CD.
“OA to higher areas lookup readme file final.doc” – OA to Higher Area & Postcode to OA Lookup Files (Mar 2004)
What does this mean?
We can reference any postcode, free of copyright, to a local area. While it doesn’t georefence to an exact street, it does reference to an area that is perfectly usable for websites such as PlanningAlerts.com and JobCentreProPlus.com. The ONS states that each OA corresponds to about 125 households, and there are 175,434 OAs in England. Another example:
What’s Next?
We had a good hacking session and I expect @simonw will be releasing something very useful, very soon. So, free postcodes for all then.
What is the relationship between these boundaries and Wards, is there any kind of match/overlap?
is there a mapping between WTWA/OA’s and Wards, or is that another arbitrary overlay?
Thanks.
Wow, nice work! You’re right that that resolution (125 households) is plenty good enough for many (most?) purposes.
Frankie
There’s a rather large problem there…
“The ONS also make available postcode to Output Area lookup files, correct as of 2004.”
That’s nearly 6 years out of date. While old postcodes haven’t disappeared – lots of new ones have been created. My new office in W2 opened in January – there are still GPS receivers and mapping tools which can’t find its correct location.
What you’ve got is a very clever hack around the situation, but it fails for anyone living or working in an area built in the last 6 years.
Postcodes need to be free. We own the Royal Mail – we would get more value from having a free, regularly updated, canonical geo-spatial reference system for the UK than we would from selling the PAF.
T
Could you confirm the nature of free.
Free of charge is not the same as free of copyright.
There is crown copyright on the boundaries. No Ordnance Survey copyright, but Crown copyright nonetheless. One would also assume that any derivative databases would also be under the same copyright?
You still need to be careful. The Postcode to Output Area lookup table is way out of date. The All Fields Postcode Directory (which I presume you refer to) is now superceded by the National Statistics Postcode Directory (NSPD)http://bit.ly/3wBHGD which is a GridLink product that requires a licence to use.
“Users should be aware that both Ordnance Survey and Royal Mail royalties now apply when using the NSPD”
Yes, you can use ONS data free of charge, for internal business use. But you can’t publish this data on the internet, nor give it to anyone else (all users have to apply for a Click-Use licence).
The main problem is that the postcode data is very out-of-date, so won’t contain new postcodes. You could probably guess their locations for most purposes, though.
Anyone making this generally available to the public, like Ernest Marples, should probably expect to hear from Royal Mail, or Ordnance Survey, or both.
“Free as in beer” is not the same as “free as in speech”, sadly.
Hopefully the UK will eventually decide to follow the USA, and make postcode and basic map data truly free for anyone to use. If they don’t then OS will soon start losing serious business to OpenStreetMap and the companies that are building their businesses around it.
It’s a useful solution but in the countryside Output areas are not so small. An output area can stretch across 4 miles or more.
It would be better if Postcomm could ensure that the Royal Mail stuck to the Legislation and charged only a reasonable cost to protect it’s IP. There doesn’t seem to be any mention of using it as a commercial asset.
PaulG: http://www.statistics.gov.uk/geography/census_geog.asp#oa has the details about Output Areas – basically, OAs are groups of postcodes, and are split along ward boundaries. The OA->higher ward data will include wards, but they will be statistical wards (e.g. some tiny ones merged to protect confidentiality and so on).
Anthony Cartmell – you certainly can republish click-use licenced data on the internet (I quote “publishing the Material in any medium. This includes featuring the Material on websites which can be accessed via the internet or via an internal electronic network or on an Intranet”). You can (following section) allow end users to download it for personal use, and ask them to get their own licence for further (that’s exactly what we do for e.g. the TheyWorkForYou API).
I can’t see any issue with using the ONS data in this way, caveats about age of data notwithstanding.
Coincidentally I’ve been looking at this info myself (and also having a conversation with ONS about it), and I’m not sure this is a solution. On the CD with this info on (see also http://www.statistics.gov.uk/census2001/terms_and_conditions.asp), it states that some of explains that OA boundaries are part-owned by OS and as such there are some restrictions on use. (i.e. usual OS-license contamination).
There seems no way of working out which boundaries are OS-contaminated, so I couldn’t see how this was a solution (though would be delighted to be proved wrong).
I am curious how we reconcile the messages from the different bits of the same organisation about the boundary data (note that ‘www.ons.gov.uk’ and ‘www.statistics.gov.uk’ both appear to be for the same organisation which is the ‘Office for National Statistics’.
This one, that Chris pointed to that makes us happy:-
http://www.ons.gov.uk/about-statistics/geography/products/boundaries/index.html
And this one which CountCulture highlighted makes us unhappy:-
http://www.statistics.gov.uk/census2001/terms_and_conditions.asp
To be clear, we are not wanting to use the census data which the 2nd one refers to in part, we only want to use the OA boundaries and the post codes relations associated with them. However… the 2nd link does make it clear that the boundaries are covered in some parts by OS (c) in contradiction to the other one.
I will be interested to see how this pans out and too see if there is enough ‘wriggle room’ to get something online as Chris suggests.
@Matthew Somerville – You can’t publish the OA geographical data, as this is also covered by OS “derived data” licensing rules, as clearly specified on the data CD* (it’s not so easy to find on the ONS website at the moment). You may only use the geographical data for internal business use, and only where the data doesn’t form a significant part of any resulting product. I’m pretty sure that this means that using Census data to geocode postcodes for public consumption would break the OS licence agreement that all users of Census data have to agree to.
*On my CD “OA boundaries for England and Wales in MID/MIF and Shapefile format”, the licence in question is found in a file called OS_OA_licence.doc in the root directory of the CD.
There isn’t distinct correlation between OAs and postcodes, or at least, the lookup isn’t 1-1. But I’d say wider than that, the opening up of postcodes should really be geared around, what do you want to use them for? Do you want to use Postcodes as a point Unit (WC1A 4WW) to define a particular location? Or Postcode Sectors as a polygon? The latter are produced by Geoplan. You may want to use their data for marketing purposes. How you convert either into an open source model may need some thought, but first of all you need to understand why you’re doing it.