Data tools - reviews

Data visualisation & mapping tools are something I've been curious about for a long time but never quite had a few hours to devote to tracking them all down and playing with them. Well, someone else has tracked them all down, so I thought I'd spend a fun few hours (my definition of fun is not your definition of fun, I appreciate that), playing with them, reviewing them and putting the results here, if only so in 3 months time when I need a tool, I can come back to this and hopefully know which one to use without getting horribly confused.

Chartle

Purports to offer simplicity, uniquity and interactivity. I've got some arguments with this. You can't make claims like that and not expect me to be harsh. The welcome splash screen is an abomination, but we're not judging on design here....no wait, a website splash screen for a website helping to me design lovely charts and graphs shouldn't make my eyes bleed. -1 point. Also on the front screen is the warning of things to come - a) it's in Beta and b) improved import from Excel. So another -1 point because already I've got a sneaking suspicion the word simplicity is nothing but marketing speak. Click on Create your own and another window spawns. Click your type of graphic - pie, bar, line etc.

I'm not going to do a walkthrough for every tool, but things I noticed in producing the graphic below: Import means pointing something at a spreadsheet on my desktop and hitting Import, not copying & pasting, it's easy to forget to click the Import button, you have to hand type the Title which in this day & age seems terribly retro, on importing the data, the axis retain the descriptors from the example - you're not asked to change them. To change them, you click on Special. I don't class being able to name my axis from your default assignation Special. I call it Mandatory. And confusing as I distinctly remember clicking 'first row are headers' in the Import tab. Each key tap dynamically updates so people who hate Googles live searching are going to detest this, as are those of us on netbooks.

The biggest thing this falls down on, however, is that I can't specify or change any colours on the actual chart. My whole chart is blue. I might not like blue. I might want to have each bar in different colours, or group bars visually for some reason. I can't.


So there you have it - it took very little time to do but there's some serious tweaking to be done on this site before I'll go back and use it in anger. Once it's launched, however, I will be going back and reviewing again.

ColorBrewer

Load it up. Go on. What do you see? A map of something which looks vaguely like the US down to County level and a world of pain. Click on the How to use page and all becomes clear - it's colour diagnostics for your infographics to ensure the colour differentiation is great enough between colours sat next to each other that the infographic is of value and not a mass of 'oh my eyes'. Genius in other words. It's got print friendly and colour blind friendly options, you can overlay roads and you can pull all the suggested colour schemes out in RGB, CMYK or Hex. You can also Export your colours to Excel.

This is a simple tool, and quite a specific one, but it's done well with a good selection of options but without overcomplicating things. Will definitly be using it.

Dundas Data Visualisation

Paid for so not reviewed as I'm a casual user at this stage. Someone else I'm sure will review it - flag below if you come across one. Included here only for completeness.

Exhibit

Confusion reigned because the link from the Digital Research Tools Wiki goes to a landing page and not the tool itself. Once you've followed the yellow brick road, you get to Exhibit. Unfortunately, you get to a page showing you what Exhibit can do but absolutely no clue on how to begin creating such lovely looking things. And since I suspect that working it out is going to take days and not hours, it's not something for the casual user so there the review ends.

Flare

I've deliberately linked to the tutorial page, within which it explains a working flash development environment. Not suitable for a casual user again, going to take more than a few hours to work it out. Again.


Geocommons

Incredibly interesting looking web based GIS tool by the looks of it - however I tried to search for layers to add to my map and found difficulties finding anything UK based as there doesn't seem to bea filter, so tried exiting the add layer dialogue which promptly caused an adobe shockwave plugin crash. Which is where my patience ended, I'm afraid, as I suspect a lot of others would.

This is the point where I started to lose the will to live so I scanned down the list and picked a tool at random - the Internet Community Text Analyser. It beautifully sums up a number of issues which I will summarise at the bottom of this post.

The ICTA is a good summary of my morning so far. I wanted to analyse my Twitter stream on Loulouk, including @replies. The whole lot. ICTA says it can deal with RSS feeds. So I tried initially using Twitter's raw offering of rss and validated it in Google Reader. Because of the introduction of oath, it stopped displaying anything after November 2009. Okay, fine. So I do a bit of investigating and find Free My Feed. I enter the RSS from Twitter, my username, Loulouk, and my password. For the first 4 attempts it said no Feed found. In writing this, it's generated a freed feed which will not require oath. So I've gone to ICTA and entered that web address that Free my Feed gave me, entered a name for it and hit Import and Save. It parses - no error. Wonderful. Except it says 20 records. Yep, it's trying to tell me I've tweeted 20 times from my account. I've also got a webpage full of white space, with a few lines of text at the top but we'll ignore that for the moment.  So on clicking Next, what do I find after a page where I'm supposed to remove something but I'm not quite sure what?

Nothing. Zeroes everywhere. No explanation that I should wait or click on the Analyze button then wait (because it appears as I should as after a few seconds things start spinning and loading, though I'm not entirely sure what). On loading, it tells me # of names found is 1. Which is impossible because on the previous screen, inside those 20 messages it clearly showed more than one name. So what does it mean when it says name? Click on the ? and you get:

This is how many unique personal names that ICTA found in this dataset.

By clicking on this number, you can review all names found by ICTA, add or delete names as necessary.

 And this, this is why I am coming to detest arbitrary lists of useful tools. Half of them don't work. Realy desperately do not work. Of the half which do and are out of Beta mode, 90% of them require you to be a 'developer' or have a damn good understanding of GIS - Open Heat Maps was another tool I had a quick play with - I had the csv on my desktop of the location of all the Boris Bike stations lying around so I tried to use that. This is what I got:






Yes, apparently there is a Boris Bike station in the middle of the Atlantic. Handy. Do you know why it's there? Because the data which TLA give out on the docking station locations includes co-ordinates and the co-ordinates are in Eastings and Northings which means they are a type of co-ordinate projection called Universal Transverse Mercator. There are, quite literally, hundreds of different projection systems. The ones you will be familiar with are porobably the one which gives co-ordinates in degrees, minutes and seconds, which is referred to as Latitude and Longitude and finally the Ordnance Survey term of reference - the British National Grid which uses a sequence of letters and numbers to enable you to pinpoint your exact position.

I don't know what projection, what type of system, Open Heat Maps is using. There is no option in the process offered to change the projection. And this is why there are dots in places on the map which frankly would incur such a monumental penalty in being over the return time limit that it doesn't bear thinking about.

So what I have learned today?

A) A projection standard is going to become an issue far quicker than I had thought as more and more people without geography degrees (me) or GIS degrees (me) try and play around with data to make it mean something, say something, represent something.
B) Beta means beta means beta. Don't expect it to work. It's a nice surprise then when it actually does.
C) People compose lists on a regular basis which point to tools which they clearly have not tested in any way shape or form. Once you know this and accept this, frustration disappears.
D) Data standards are also going to be a massive issue outside of co-ordinate data. But the issues shown above in co-ordinate data are being replicated across swathes of the UK and US as people rush to publish their data without first questioning the validity of it, the usefulness of it, the integrity of it or how someone will take it and compare it to anything else without metadata standards which are adhered to.

Now if you'll excuse me, I'm off to try and retrieve some of my enthusiasm for examining, visualising, interrogating and spatially mapping data. It's waned somewhat.

Tracing a road around the world

I've always loved maps. There's a 'joke' about a certain kind of kid swallowing the dictionary. I wasn't ever that kid. Nope, I was the kid who read the atlas instead. We actually had one, which attentive readers will perhaps understand was something of a win.

Atlases, and a love of maps, all kinds of maps, playing with maps, drawing maps, interacting with maps is something that I used to be a little bit shy about admitting. No more. The world has changed, or perhaps rather the circles I move within have changed, I'm not sure. But regardless, the simple pleasure of getting a system to place a marker in the right place, then colour it depending on pre-determined requirements still fills me with glee. It will always fill me with glee.

So, I guess we start at the beginning again, bearing in mind that the best holiday I ever had was navigating off a Michelin map through the Pyrenees as my partner got arm ache from driving around all the hairpins, that I can comfortably navigate people through the centre of London with the aid of an A-Z, in our house we don't use Tom Tom, we use Lou Lou and that I don't ever have to turn the map the right way round to orientate myself. I'm not boasting here, simply pre-empting the inevitable 'but you're a girl, girls can't read maps/can't navigate/can't read signs/can't read maps without turning them around'. This one can, just so we're clear. I am not alone in this, just so we're clear. They're not pre-requisites to being able to map data onto a map, just so we're clear. But loving maps, so intensely, understanding their power but also their restrictions? That really helps, I think.

There are two kinds of maps in the world. One comes as a photograph, a picture, a jpeg. We call them raster images - they are simply images and nothing more. They cannot be asked questions of, you can't search them, if you zoom into them, the points on the map ( the distance between your house and the local pub, for example) will move, but not in relation to each other to any kind of scale. Flat, 2 dimensional map. They've got their uses, of course they do, they're great as a print out on some water resilient paper to take out on the hill with you. They're great for printing out and taking into London with you on a sight seeing trip.

But. You can't move anything, change anything, search for anything, update anything. It's static. A snapshot in time of the way things were, because the second you printed it, it's out of date. History.

The second type of map, we call vector. What it actually means is, 3 dimensional. Sometimes quite literally with the help of some funky tools and as this lovely map of Hong Kong shows.But more often than not, it's 3 dimensional in an entirely different way in that it's searchable, scaleable and interactive. You can drop pins on it, move around the map and the pin will stay in the right place - over the top of the house you dropped it on.  You can search by street name and you can plot a path on it - well draw one actually, you don't have to plot anything at all.

There are 2 kinds of mapping tool The first kind are the ones like Google, Bing and the previously linked to 3D Hong Kong map. They're useful to a point, but the point quickly becomes limiting. You can search it, and it will take you to the street you were looking for. But it will be a rough approximation of a street. Put the satellite layer over the top and you'll see what I mean - they're ever so slightly out of sync.

Next, try dropping a pin somewhere on a Google map with the zoom level not zoomed all the way in. Now zoom all the way in. Pin isn't quite on the corner of that junction that you dropped it on any more. It's moved. Fine if you trust that people are capable of spotting a bright yellow bin full of grit from 20 feet away. Not good if that pin represents the something you need to be accurate. Next, search for Witton Park on Google maps. Zoom in to the 2nd from last setting. Note that Witton is spelt as Whitton - right next to each other. One of those spellings is right. One is wrong. Do you know who I can report that to? No one.

Finally, embedding Google maps is a complete nightmare. If you have more than 30 things or so to map, then it will trip over to page 2. And then page 3. Which is fine if you're viewing a map in Google and you realise that that's what happening (some people wont and will think, for example, that you've only mapped the grit bins in Darwen and ignored Blackburn completely). When you come to embed the map into a webpage, the fact that there is more than one page of information you've mapped isn't mentioned. In order to get all the points to display in your embedded map, you have to go to Google Maps, hit the RSS button, get the RSS url of your points and chuck it back into Google Maps again. Then you are graced with a map which you can embed which will show everyone all your points.

I could go on. I think you get the point. You might ask why on earth anyone ever uses Google Maps. I'll explain that at the bottom but over the next few paragraphs, I suspect it will become clear.

The second kind of map is Ordnance Survey built and oh boy is it accurate. Need to know where a set of steps is or how wide a pavement is? OS mapping can tell you. Until April 1st this year (2010) the maps were acknowledged to be so accurate that people paid a lot of money to access that data. Witton gets spelt the right way. If it weren't I'd know exactly who to speak to to get it corrected. They not only provide line maps, they also provide maps with door numbers on to help you orientate yourself (or check your co-ordinates are correct). There is still a bit of a zoom problem, in that if you draw on the map at a certain zoom level, it will move slightly when you zoom in, say from 5km to 1km. But under 1km where you switch to a more detailed map, it's accurate. Points dropped don't wander off. It's accurate, it's reliable.

More interestingly, it can also be fed into Geographical Information Systems (GIS). GIS is a loose term for the set of tools which allows you to place massive amounts of points onto OS mapping, quickly and reasonably simply. Most people have done a degree in these systems and are can make them dance. I've taught myself almost entirely and so am a little behind. To add insult to injury, GIS systems are enormously expensive and so I don't have access to one any more because it's not part of my job. Which is where Open Space comes in. Open Space allows you to do similar things to what you historically could only do with an expensive GIS tool.

So, why, I hear you ask, have I mapped our grit bins and gritting routes using Google?

Ordnance Survey have made a big big deal of their Open Space tools. This 'path' map is an amazing example of what can be done - entirely for free. Except. There's always an except, isn't there. In the bottom left hand corner, you will note a little column which depending on the time of day you view it, will either be green or red. Hover over it with your mouse and you will see "XX% of daily map tile limit used". Open Space, when publicising their wonderful new tools, seem to forget to mention this. You get 150 'views' a day unless you pay for more. We'd go over that in about 3-4 hours, I'd guess. What happens after you exceed that 150 views? Well, we don't know but we're assuming that no information is served at all. Which isn't going to look very good now, is it?

And we're skint. Utterly and completely broke, as a Council, with £48 million of savings to make before 31st March 2011. So I didn't map the routes/bins in anything but Google, because Google was free. Which is why they can afford to ignore complaints of spelling. Why inaccuracies can be forgiven. Why display errors meaning suddenly my screen shows 10 markers all on top of each other practically, for no reason whatsoever, must be simply ignored.

It's free.

So, in true Brit style, we make do and mend. In the meantime, I've bought myself a book and will hopefully be teaching myself how to code so I can make Google behave a little better, so I can code around some of the things which are causing a problem. But (I may be wrong here, but I don't think I am) even then, once you start to code and hack about with Google in anything more complicated than what we've done already, Google control access to their maps with something called an API key. And once again, imposes limits for page views.

So what's the answer?

I don't know. This is where I hope someone else picks it up and passes it on - back to me. Can anyone help? Is there an open source way of displaying a map, inside a box, with a zoom in and out tool, a search tool and is accurate, which I can embed in our web pages and will cost me nothing, be entirely reliable and not melt my brain in the process of trying to work it out?

So. To summarise. There might only be 2 different kinds of map. And only 2 different ways of seeing those maps and interacting with 1 kind of those 2 different kinds of map. But when you start to delve a little further, it's really rather more complicated than it might first appear. And free is free for a reason.

GIS is to #opendata as.......

GIS (Geographic Information System) is to open data as Microsoft Word is to a bunch of incohesive words and letters. Or at least, when you've been working with GIS for a bit, that's how it looks. A slightly distorted view of course, because GIS wasn't involved in the 15 minute dalliance with some data I danced the other night, but nevertheless, data and GIS is intrinsically linked for me and so I thought I'd try and explain why - with a diagram!

Look ma, no hands! Now, as regular readers will know, I'm not very good at diagrams. Or data visualisations or mash ups or whatever the cool and funky kids are calling them these days - I'm not a cool and funky kid either so I wouldn't know. But this is a pitiful attempt at an explanation, nevertheless because letters are boring and pictures are shiny.

Open data is the end of the story as I hope I've made clear here. Operations is the start. Operations is what generates all the data to go out into the open. Operations means childrens services, it means street scene or refuse collection and street cleansing with some park maintenance in there as well depending on what you're calling it this year, it's the actual day to day stuff like how many fly tips we collected or how many parking tickets we issued or how many library books we lent. It's our bread and butter, it's what we do, it paints a picture in numbers of the service we provide to everyone on a day to day basis.

The data from all that operational activity currently goes in one direction but eventually, one day, will go in two as shown here. At the moment, it goes into management information and in a lot of cases it gets fed, if appropriate and in most cases it is, into the GIS server and processed by our GIS software to make easy to understand visual representations of the data which our Managers can use to make informed management decisions quickly and easily, because the data is not 64,000 rows of ascii (raw letters), but instead thematic mapping, showing them where their hotspots and notspots are, where they need to focus more resource and where there was a problem 12 months ago but now isn't and so they can move resource. It's done more often than 12 monthly, but for the purpose of this, we'll call it 12 months.

The appropriateness of the information going into GIS software is generally whether it's spatial. Spatial means, relational, means does it have a latitude or longitude on it, does it have GPS data attached to it, will mapping it spatially make sense for the data to mean something. In a lot of cases, well actually, in most cases it does, from school catchment areas crossed with deprivation indices crossed with academic achievement levels to a thousand and one other 'mash ups' as they're now called.

Research and intelligence and policy actually have a two way relationship with GIS. They pull data out of the datasets parked there by ops and they crunch it, create 'mash ups' and provide it to Directors and Heads of Service to inform them. They do trend analysis and many other complicated and funky things. Policy take this crunched data too and they build our strategic advice on it. They tell people in words of one syllable (and yes sometimes more) where we were, where we are and where we will be if things continue as they are, but also where we will be if they do not. They don't hold crystal balls, they assure me, but I'm not so sure. GIS and much other non-spatial data is their bread and butter, I think (someone will correct me if I am wrong here, I'm sure).

The datasets which drive all this, will one day go straight onto data.gov.uk once signed off too. INSPIRE says they will. Please read the link, if you've got this far, you need to know about the existence of this Directive.

Which brings me to the other side of open data and the things in data.gov.uk which will sit next to the Operations generated stuff. The cost of satisying Freedom of Information requests was quoted at me by someone at a conference recently from their Authority and I will not post here what it was but it was enough to make my jaw drop. If we are transparent where it comes to information, if we are open, the assumption is that we will actually never receive FOI's ever again, because it will all be easily found, therefore cutting out the middlemen of the poor Administrators and Officers tasked with fulfilling these requests and instead leaving them to do their main job roles. But for the moment, perhaps it would be a nice interim policy for people to put the results of FOI's onto the web automatically, in the assumption that if one person wants to request the information, then perhaps a second will too? It's obviously of interest to someone, right?

And then there's spending data. Generated by Operations but kept by Finance. This is left to last, because Government have almost wrapped this one up. All spend over £500 will be published by local government in January 2011. All NHS PCT spend over £25,000 is already online.

Here ends the tour of data within local government. If you got this far, you know as much as I do, almost. Which in the interests of openness, is exactly the way I believe it should be.