The following post is entirely Hadley Beeman's (@hadleybeeman) fault. So much is, actually, when it comes to opendata and linked data - just Google her and you'll find it why. For the US orientated among you, Hadley is essentially our national lead on opendata in the UK, she's an American and she yay's. In public. Unashamedly. And that makes her just awesome because us Brits, frankly, we do not yay enough.
Anyway.
This tweet kicked off a waterfall of thoughts and feelings:
"Soundthing: Ambient music selection: just walking into a pub with your phone can bend the pub's playlist towards your Spotify prefs. #gsxsw"
The tag is also worth investigating - gsxsw was curated by Rewired State (Emma Mulqueeny) and held at the Guardian newspaper offices this weekend - it was a hack day with a bit of a difference. Hack days might be a Brit thing also - if so, Google it, there's some amazing stuff going from idea to actuality here at the moment under these banners.
So, the tweet led me to respond with the idea that not only would the pubs sound system bend to me and my mates Spotified taste in music, but that this would somewhere be recorded for future questioning - that somehow I would land in a new city and know where to find like minded music lovers because I could see on a map layer or something similar all the pubs which linked with my Spotify preferences by more than x percent, where I set the percentage. So, for example, and purely hypothetically (ahem) if I hated 80's music with a passion most people only reserved for politicians, I could avoid pubs which played any 80's music at all. Or, if I was ambivalent to ambient but absolutely loved techno, I could set the filter percentages accordingly and go and find my tribe. Or if I was feeling kinda brave and adventurous, I could change the settings randomly with a button and land somewhere entirely new and unlikely.
But, I thought, what if you took that further? What if you took that out of a pub environment, that linked data which told me where I should be depending on my musical preferences, and as Hadley suggested with food, you applied it to something else?
I'm going to segue a little and tell you a story, based purely on an experience I had yesterday. It's true and it's relevant, so bare with me a little if this comes across as self indulgent.
I've got quite severe tendonitis which flares up every now and again. It means my tendon contracts which then pulls on the muscle above it which dislikes it, which then has a knock on effect upwards again. Riding my bike fixes it, I've not ridden my bike for a bit thanks to snow and torrential rain. You try mountain biking in snow on tech Northern trails sometime. Anyway.
We went out for the day and parked up in Manchester, in a car park we don't normally use. The car park is essentially one of those 'this used to have something built on it (in this case Boddingtons Brewery), it now doesn't, someone decided to make money off the land by parking cars on it'. My other half led us towards a sneaky exit he knew which wasn't the official exit. Just by the exit out onto the street was a short but incredibly steep upward incline. My calf muscles combined have lost about 1/2 inch of flexibility. Lots of unladylike swearing ensued. For the next 30 minutes walking was faintly hideous. It eventually stopped when I relented and dived into a handy second hand bookshop to let them calm down a little and stop contracting.
It could have been avoided, those 30 minutes of pain, and I'll tell you how. Augmented linked data. Somewhere else out there, I have no doubt, is someone else with slightly broken calf muscles who's probably done exactly the same thing and was discomforted enough to want to ensure someone else didn't suffer the same ouchiness. Currently, there is no way for her to do that. Currently, there is no way for me to do that. I can't tell anyone. Putting it on a forum would be pointless, it would get lost in the noise. Putting it an email to someone would get me nowhere. I now know something about the environment in a car park in Manchester would could save someone else some pain and I can't share.
In each of our heads is data. We know things. Some of these things are small, so infinitesimally small that we might never think of sharing them or that they might be of worth to someone else, and some of those things are very big things. Some of those big things are only big to us and some of those big things would make the world a better place if we could share them with everyone.
Imagine, then, a way of sharing, if you wanted to, everything you ever knew about obstacles or opportunities. A world where on your GPS there was a layer you could switch on or off which linked in to the local councils database in the area you were passing through and told you where the major potholes were or into the local police database so you could know as a motorcyclist where the accident blackspots and blind corners were. Or, even, which gate you needed to slow down for as you passed it because the cows always came across the road for milking at 4:45pm and as a result the road would be slippy and possibly treacherous for the next few hours after.
Imagine a world where local knowledge was at your fingertips. Don't park down that side street because anyone who does gets broken into - or if you must remove the GPS suction circle from your windscreen because the only people who get broken into are the people who forget. Imagine a world where you knew which restaurants and pubs were genuinely child friendly because other parents had indicated it so and you could see this in real time when driving down a street without having to get your smartphone out of your pocket, because your GPS told you? No more relying on the restaurant owners claims to be child friendly when his definition includes a high chair but nothing else.
But most of all, to me, imagine a world where you could see issues before they arose. Where people with sensory overload problems could reference to other people with similar issues without ever needing to know their name, but only by linking with someone just because they have a similar intolerance, and following where they've been and had problems, but more importantly where they've been and not had any problems at all.
A world where I'd never have walked up the slope because someone with the same issue as me had a way of indicating somehow that people with x issue should use the gate on the flat at the other end, should walk the long way around, should not even think about attempting if in a wheelchair.
In my world, in my future, I believe these things will happen. I believe everyone will see through the same eyes I do - through eyes which see data layered across and streamed through reality in real time. One where my world is shaped by others experiences who are like me. One where there are no potholes to break your suspension and no steep inclines to break your muscles.
Don't tell me tech can't change the world for the better. Don't tell me maps don't matter and visualisation is pointless. Don't tell me these are pipe dreams. If no one dreams, reality will never be changed. Yes, thinking small allows people to JFDI. But sometimes, just sometimes, someone has got to think BIG.
Showing posts with label mapping. Show all posts
Showing posts with label mapping. Show all posts
Sunday, 13 February 2011
Wednesday, 29 December 2010
Data tools - reviews
Data visualisation & mapping tools are something I've been curious about for a long time but never quite had a few hours to devote to tracking them all down and playing with them. Well, someone else has tracked them all down, so I thought I'd spend a fun few hours (my definition of fun is not your definition of fun, I appreciate that), playing with them, reviewing them and putting the results here, if only so in 3 months time when I need a tool, I can come back to this and hopefully know which one to use without getting horribly confused.
Chartle
Purports to offer simplicity, uniquity and interactivity. I've got some arguments with this. You can't make claims like that and not expect me to be harsh. The welcome splash screen is an abomination, but we're not judging on design here....no wait, a website splash screen for a website helping to me design lovely charts and graphs shouldn't make my eyes bleed. -1 point. Also on the front screen is the warning of things to come - a) it's in Beta and b) improved import from Excel. So another -1 point because already I've got a sneaking suspicion the word simplicity is nothing but marketing speak. Click on Create your own and another window spawns. Click your type of graphic - pie, bar, line etc.
I'm not going to do a walkthrough for every tool, but things I noticed in producing the graphic below: Import means pointing something at a spreadsheet on my desktop and hitting Import, not copying & pasting, it's easy to forget to click the Import button, you have to hand type the Title which in this day & age seems terribly retro, on importing the data, the axis retain the descriptors from the example - you're not asked to change them. To change them, you click on Special. I don't class being able to name my axis from your default assignation Special. I call it Mandatory. And confusing as I distinctly remember clicking 'first row are headers' in the Import tab. Each key tap dynamically updates so people who hate Googles live searching are going to detest this, as are those of us on netbooks.
The biggest thing this falls down on, however, is that I can't specify or change any colours on the actual chart. My whole chart is blue. I might not like blue. I might want to have each bar in different colours, or group bars visually for some reason. I can't.
So there you have it - it took very little time to do but there's some serious tweaking to be done on this site before I'll go back and use it in anger. Once it's launched, however, I will be going back and reviewing again.
ColorBrewer
Load it up. Go on. What do you see? A map of something which looks vaguely like the US down to County level and a world of pain. Click on the How to use page and all becomes clear - it's colour diagnostics for your infographics to ensure the colour differentiation is great enough between colours sat next to each other that the infographic is of value and not a mass of 'oh my eyes'. Genius in other words. It's got print friendly and colour blind friendly options, you can overlay roads and you can pull all the suggested colour schemes out in RGB, CMYK or Hex. You can also Export your colours to Excel.
This is a simple tool, and quite a specific one, but it's done well with a good selection of options but without overcomplicating things. Will definitly be using it.
Dundas Data Visualisation
Paid for so not reviewed as I'm a casual user at this stage. Someone else I'm sure will review it - flag below if you come across one. Included here only for completeness.
Exhibit
Confusion reigned because the link from the Digital Research Tools Wiki goes to a landing page and not the tool itself. Once you've followed the yellow brick road, you get to Exhibit. Unfortunately, you get to a page showing you what Exhibit can do but absolutely no clue on how to begin creating such lovely looking things. And since I suspect that working it out is going to take days and not hours, it's not something for the casual user so there the review ends.
Flare
I've deliberately linked to the tutorial page, within which it explains a working flash development environment. Not suitable for a casual user again, going to take more than a few hours to work it out. Again.
Geocommons
Incredibly interesting looking web based GIS tool by the looks of it - however I tried to search for layers to add to my map and found difficulties finding anything UK based as there doesn't seem to bea filter, so tried exiting the add layer dialogue which promptly caused an adobe shockwave plugin crash. Which is where my patience ended, I'm afraid, as I suspect a lot of others would.
This is the point where I started to lose the will to live so I scanned down the list and picked a tool at random - the Internet Community Text Analyser. It beautifully sums up a number of issues which I will summarise at the bottom of this post.
The ICTA is a good summary of my morning so far. I wanted to analyse my Twitter stream on Loulouk, including @replies. The whole lot. ICTA says it can deal with RSS feeds. So I tried initially using Twitter's raw offering of rss and validated it in Google Reader. Because of the introduction of oath, it stopped displaying anything after November 2009. Okay, fine. So I do a bit of investigating and find Free My Feed. I enter the RSS from Twitter, my username, Loulouk, and my password. For the first 4 attempts it said no Feed found. In writing this, it's generated a freed feed which will not require oath. So I've gone to ICTA and entered that web address that Free my Feed gave me, entered a name for it and hit Import and Save. It parses - no error. Wonderful. Except it says 20 records. Yep, it's trying to tell me I've tweeted 20 times from my account. I've also got a webpage full of white space, with a few lines of text at the top but we'll ignore that for the moment. So on clicking Next, what do I find after a page where I'm supposed to remove something but I'm not quite sure what?
Nothing. Zeroes everywhere. No explanation that I should wait or click on the Analyze button then wait (because it appears as I should as after a few seconds things start spinning and loading, though I'm not entirely sure what). On loading, it tells me # of names found is 1. Which is impossible because on the previous screen, inside those 20 messages it clearly showed more than one name. So what does it mean when it says name? Click on the ? and you get:
Yes, apparently there is a Boris Bike station in the middle of the Atlantic. Handy. Do you know why it's there? Because the data which TLA give out on the docking station locations includes co-ordinates and the co-ordinates are in Eastings and Northings which means they are a type of co-ordinate projection called Universal Transverse Mercator. There are, quite literally, hundreds of different projection systems. The ones you will be familiar with are porobably the one which gives co-ordinates in degrees, minutes and seconds, which is referred to as Latitude and Longitude and finally the Ordnance Survey term of reference - the British National Grid which uses a sequence of letters and numbers to enable you to pinpoint your exact position.
I don't know what projection, what type of system, Open Heat Maps is using. There is no option in the process offered to change the projection. And this is why there are dots in places on the map which frankly would incur such a monumental penalty in being over the return time limit that it doesn't bear thinking about.
So what I have learned today?
A) A projection standard is going to become an issue far quicker than I had thought as more and more people without geography degrees (me) or GIS degrees (me) try and play around with data to make it mean something, say something, represent something.
B) Beta means beta means beta. Don't expect it to work. It's a nice surprise then when it actually does.
C) People compose lists on a regular basis which point to tools which they clearly have not tested in any way shape or form. Once you know this and accept this, frustration disappears.
D) Data standards are also going to be a massive issue outside of co-ordinate data. But the issues shown above in co-ordinate data are being replicated across swathes of the UK and US as people rush to publish their data without first questioning the validity of it, the usefulness of it, the integrity of it or how someone will take it and compare it to anything else without metadata standards which are adhered to.
Now if you'll excuse me, I'm off to try and retrieve some of my enthusiasm for examining, visualising, interrogating and spatially mapping data. It's waned somewhat.
Chartle
Purports to offer simplicity, uniquity and interactivity. I've got some arguments with this. You can't make claims like that and not expect me to be harsh. The welcome splash screen is an abomination, but we're not judging on design here....no wait, a website splash screen for a website helping to me design lovely charts and graphs shouldn't make my eyes bleed. -1 point. Also on the front screen is the warning of things to come - a) it's in Beta and b) improved import from Excel. So another -1 point because already I've got a sneaking suspicion the word simplicity is nothing but marketing speak. Click on Create your own and another window spawns. Click your type of graphic - pie, bar, line etc.
I'm not going to do a walkthrough for every tool, but things I noticed in producing the graphic below: Import means pointing something at a spreadsheet on my desktop and hitting Import, not copying & pasting, it's easy to forget to click the Import button, you have to hand type the Title which in this day & age seems terribly retro, on importing the data, the axis retain the descriptors from the example - you're not asked to change them. To change them, you click on Special. I don't class being able to name my axis from your default assignation Special. I call it Mandatory. And confusing as I distinctly remember clicking 'first row are headers' in the Import tab. Each key tap dynamically updates so people who hate Googles live searching are going to detest this, as are those of us on netbooks.
The biggest thing this falls down on, however, is that I can't specify or change any colours on the actual chart. My whole chart is blue. I might not like blue. I might want to have each bar in different colours, or group bars visually for some reason. I can't.
So there you have it - it took very little time to do but there's some serious tweaking to be done on this site before I'll go back and use it in anger. Once it's launched, however, I will be going back and reviewing again.
ColorBrewer
Load it up. Go on. What do you see? A map of something which looks vaguely like the US down to County level and a world of pain. Click on the How to use page and all becomes clear - it's colour diagnostics for your infographics to ensure the colour differentiation is great enough between colours sat next to each other that the infographic is of value and not a mass of 'oh my eyes'. Genius in other words. It's got print friendly and colour blind friendly options, you can overlay roads and you can pull all the suggested colour schemes out in RGB, CMYK or Hex. You can also Export your colours to Excel.
This is a simple tool, and quite a specific one, but it's done well with a good selection of options but without overcomplicating things. Will definitly be using it.
Dundas Data Visualisation
Paid for so not reviewed as I'm a casual user at this stage. Someone else I'm sure will review it - flag below if you come across one. Included here only for completeness.
Exhibit
Confusion reigned because the link from the Digital Research Tools Wiki goes to a landing page and not the tool itself. Once you've followed the yellow brick road, you get to Exhibit. Unfortunately, you get to a page showing you what Exhibit can do but absolutely no clue on how to begin creating such lovely looking things. And since I suspect that working it out is going to take days and not hours, it's not something for the casual user so there the review ends.
Flare
I've deliberately linked to the tutorial page, within which it explains a working flash development environment. Not suitable for a casual user again, going to take more than a few hours to work it out. Again.
Geocommons
Incredibly interesting looking web based GIS tool by the looks of it - however I tried to search for layers to add to my map and found difficulties finding anything UK based as there doesn't seem to bea filter, so tried exiting the add layer dialogue which promptly caused an adobe shockwave plugin crash. Which is where my patience ended, I'm afraid, as I suspect a lot of others would.
This is the point where I started to lose the will to live so I scanned down the list and picked a tool at random - the Internet Community Text Analyser. It beautifully sums up a number of issues which I will summarise at the bottom of this post.
The ICTA is a good summary of my morning so far. I wanted to analyse my Twitter stream on Loulouk, including @replies. The whole lot. ICTA says it can deal with RSS feeds. So I tried initially using Twitter's raw offering of rss and validated it in Google Reader. Because of the introduction of oath, it stopped displaying anything after November 2009. Okay, fine. So I do a bit of investigating and find Free My Feed. I enter the RSS from Twitter, my username, Loulouk, and my password. For the first 4 attempts it said no Feed found. In writing this, it's generated a freed feed which will not require oath. So I've gone to ICTA and entered that web address that Free my Feed gave me, entered a name for it and hit Import and Save. It parses - no error. Wonderful. Except it says 20 records. Yep, it's trying to tell me I've tweeted 20 times from my account. I've also got a webpage full of white space, with a few lines of text at the top but we'll ignore that for the moment. So on clicking Next, what do I find after a page where I'm supposed to remove something but I'm not quite sure what?
Nothing. Zeroes everywhere. No explanation that I should wait or click on the Analyze button then wait (because it appears as I should as after a few seconds things start spinning and loading, though I'm not entirely sure what). On loading, it tells me # of names found is 1. Which is impossible because on the previous screen, inside those 20 messages it clearly showed more than one name. So what does it mean when it says name? Click on the ? and you get:
And this, this is why I am coming to detest arbitrary lists of useful tools. Half of them don't work. Realy desperately do not work. Of the half which do and are out of Beta mode, 90% of them require you to be a 'developer' or have a damn good understanding of GIS - Open Heat Maps was another tool I had a quick play with - I had the csv on my desktop of the location of all the Boris Bike stations lying around so I tried to use that. This is what I got:
This is how many unique personal names that ICTA found in this dataset.
By clicking on this number, you can review all names found by ICTA, add or delete names as necessary.
Yes, apparently there is a Boris Bike station in the middle of the Atlantic. Handy. Do you know why it's there? Because the data which TLA give out on the docking station locations includes co-ordinates and the co-ordinates are in Eastings and Northings which means they are a type of co-ordinate projection called Universal Transverse Mercator. There are, quite literally, hundreds of different projection systems. The ones you will be familiar with are porobably the one which gives co-ordinates in degrees, minutes and seconds, which is referred to as Latitude and Longitude and finally the Ordnance Survey term of reference - the British National Grid which uses a sequence of letters and numbers to enable you to pinpoint your exact position.
I don't know what projection, what type of system, Open Heat Maps is using. There is no option in the process offered to change the projection. And this is why there are dots in places on the map which frankly would incur such a monumental penalty in being over the return time limit that it doesn't bear thinking about.
So what I have learned today?
A) A projection standard is going to become an issue far quicker than I had thought as more and more people without geography degrees (me) or GIS degrees (me) try and play around with data to make it mean something, say something, represent something.
B) Beta means beta means beta. Don't expect it to work. It's a nice surprise then when it actually does.
C) People compose lists on a regular basis which point to tools which they clearly have not tested in any way shape or form. Once you know this and accept this, frustration disappears.
D) Data standards are also going to be a massive issue outside of co-ordinate data. But the issues shown above in co-ordinate data are being replicated across swathes of the UK and US as people rush to publish their data without first questioning the validity of it, the usefulness of it, the integrity of it or how someone will take it and compare it to anything else without metadata standards which are adhered to.
Now if you'll excuse me, I'm off to try and retrieve some of my enthusiasm for examining, visualising, interrogating and spatially mapping data. It's waned somewhat.
Saturday, 18 December 2010
Head in the clouds
Delicious is a sobering reminder of why dumping all your data into someone elses hands can be a very bad thing indeed.
Somewhere in my list of posts I've written and never published is one titled exactly like this one, written in February this year. This is a rehash of that, in some ways, but things have moved on a long way, both in where I work and what I do for a living, but also in the digital landscape, so I can't simply hit publish.
'The cloud' for those unfamiliar with the sometimes ridiculous names geeks give to technologies, is a bit hard to define, but essentially, when you save your information, documents, maps or data of any kind to somewhere which is not either in your house, or on your companies network, it's saved to the cloud. For example, gmail is email in the cloud. All the emails you send and receive are stored far away somewhere in America. If gmail broke, you wouldn't be able to access your email. Google documents is cloud computing, so is Flickr, so are Bing or Google maps. And so is Delicious, the place where a large amount of people saved their favourite web pages to, in the form of bookmarks.
Yahoo! bought Delicious. Now, it transpires, it is in on their hit list, or rather their 'sunset' list. Details are emerging that this might mean it's on it's asset sell list, but there's no guarantees anyone will buy it. So, at the flick of a switch in a remote server farm somewhere over in America, someone will cut my connection to my bookmark list.
I have no comeback. No legal grounds to demand the switch is flicked back. I can export my bookmarks to somewhere else, another service, but those services don't seem to be quite the same. I put them with delicious for a reason, it was a good service. But I assume a loss making service, and so it's on the switch off list.
That post I wrote back in Feb? It asked why we were all so quick to trust our data to the cloud, and this was one of the reasons. There were many others. Do a search on Bing and on Google for the same road you live on and zoom out a little and you will see the discrepancies in road curves and levels of detail. Try and write a complicated document in Google docs in Word, with annotations and footnotes and see what happens when you pull it back down off the cloud again. Check the terms of use for Google maps and understand that anything you map with them, anything at all, they claim intellectual property rights to. Fine if you don't care about such things but for those of us who are trying to explain internally in our organisations that Google might be free but it's free for a reason, it's a really big deal.
So what's the solution? I'm not sure there is one. Ultimately, if you use a service, and it's free, and you don't pay for it, you've no rights. If you use a service which nabs IPR off you, you've no rights either. If a hurricane hits the server farm your data is stored on, what can you do? It's all about risk analysis. What's the liklihood of the server farm your cloud data is sitting on getting hit by a terorrist attack, a natural disaster or a company takeover? If your data is that important to you, I would humbly suggest that these are the questions which you should be asking yourself. To be social and to share is a wonderful thing, but perhaps there is a limit to what can and should be put in the cloud? Maybe we should all be keeping regular back ups?
I don't have the answers, but I think it's important someone asks the questions. We all assume that Yahoo and Google will be around forever. Some of us assume that they're 'geek' companies, trying to provide tools for the geeks to use and use well and that if those tools are used, the companies will somehow feel some responsibility to maintain them. This is not the case. They are companies, floated companies, with shareholders and the bottom line will always be, profit to return to those shareholders.
The cloud is not fluffy. The cloud is for making money. Social networking, also, is there to ultimately make money. Companies do not make money from goodwill. People don't sit around and chat all day to be friendly. People do it to make money. One eye always on profit.
Don't be deceived by the fluffy name. It's not fluffy at all, and clouds can disappear in seconds.
Somewhere in my list of posts I've written and never published is one titled exactly like this one, written in February this year. This is a rehash of that, in some ways, but things have moved on a long way, both in where I work and what I do for a living, but also in the digital landscape, so I can't simply hit publish.
'The cloud' for those unfamiliar with the sometimes ridiculous names geeks give to technologies, is a bit hard to define, but essentially, when you save your information, documents, maps or data of any kind to somewhere which is not either in your house, or on your companies network, it's saved to the cloud. For example, gmail is email in the cloud. All the emails you send and receive are stored far away somewhere in America. If gmail broke, you wouldn't be able to access your email. Google documents is cloud computing, so is Flickr, so are Bing or Google maps. And so is Delicious, the place where a large amount of people saved their favourite web pages to, in the form of bookmarks.
Yahoo! bought Delicious. Now, it transpires, it is in on their hit list, or rather their 'sunset' list. Details are emerging that this might mean it's on it's asset sell list, but there's no guarantees anyone will buy it. So, at the flick of a switch in a remote server farm somewhere over in America, someone will cut my connection to my bookmark list.
I have no comeback. No legal grounds to demand the switch is flicked back. I can export my bookmarks to somewhere else, another service, but those services don't seem to be quite the same. I put them with delicious for a reason, it was a good service. But I assume a loss making service, and so it's on the switch off list.
That post I wrote back in Feb? It asked why we were all so quick to trust our data to the cloud, and this was one of the reasons. There were many others. Do a search on Bing and on Google for the same road you live on and zoom out a little and you will see the discrepancies in road curves and levels of detail. Try and write a complicated document in Google docs in Word, with annotations and footnotes and see what happens when you pull it back down off the cloud again. Check the terms of use for Google maps and understand that anything you map with them, anything at all, they claim intellectual property rights to. Fine if you don't care about such things but for those of us who are trying to explain internally in our organisations that Google might be free but it's free for a reason, it's a really big deal.
So what's the solution? I'm not sure there is one. Ultimately, if you use a service, and it's free, and you don't pay for it, you've no rights. If you use a service which nabs IPR off you, you've no rights either. If a hurricane hits the server farm your data is stored on, what can you do? It's all about risk analysis. What's the liklihood of the server farm your cloud data is sitting on getting hit by a terorrist attack, a natural disaster or a company takeover? If your data is that important to you, I would humbly suggest that these are the questions which you should be asking yourself. To be social and to share is a wonderful thing, but perhaps there is a limit to what can and should be put in the cloud? Maybe we should all be keeping regular back ups?
I don't have the answers, but I think it's important someone asks the questions. We all assume that Yahoo and Google will be around forever. Some of us assume that they're 'geek' companies, trying to provide tools for the geeks to use and use well and that if those tools are used, the companies will somehow feel some responsibility to maintain them. This is not the case. They are companies, floated companies, with shareholders and the bottom line will always be, profit to return to those shareholders.
The cloud is not fluffy. The cloud is for making money. Social networking, also, is there to ultimately make money. Companies do not make money from goodwill. People don't sit around and chat all day to be friendly. People do it to make money. One eye always on profit.
Don't be deceived by the fluffy name. It's not fluffy at all, and clouds can disappear in seconds.
Wednesday, 8 December 2010
Tracing a road around the world
I've always loved maps. There's a 'joke' about a certain kind of kid swallowing the dictionary. I wasn't ever that kid. Nope, I was the kid who read the atlas instead. We actually had one, which attentive readers will perhaps understand was something of a win.
Atlases, and a love of maps, all kinds of maps, playing with maps, drawing maps, interacting with maps is something that I used to be a little bit shy about admitting. No more. The world has changed, or perhaps rather the circles I move within have changed, I'm not sure. But regardless, the simple pleasure of getting a system to place a marker in the right place, then colour it depending on pre-determined requirements still fills me with glee. It will always fill me with glee.
So, I guess we start at the beginning again, bearing in mind that the best holiday I ever had was navigating off a Michelin map through the Pyrenees as my partner got arm ache from driving around all the hairpins, that I can comfortably navigate people through the centre of London with the aid of an A-Z, in our house we don't use Tom Tom, we use Lou Lou and that I don't ever have to turn the map the right way round to orientate myself. I'm not boasting here, simply pre-empting the inevitable 'but you're a girl, girls can't read maps/can't navigate/can't read signs/can't read maps without turning them around'. This one can, just so we're clear. I am not alone in this, just so we're clear. They're not pre-requisites to being able to map data onto a map, just so we're clear. But loving maps, so intensely, understanding their power but also their restrictions? That really helps, I think.
There are two kinds of maps in the world. One comes as a photograph, a picture, a jpeg. We call them raster images - they are simply images and nothing more. They cannot be asked questions of, you can't search them, if you zoom into them, the points on the map ( the distance between your house and the local pub, for example) will move, but not in relation to each other to any kind of scale. Flat, 2 dimensional map. They've got their uses, of course they do, they're great as a print out on some water resilient paper to take out on the hill with you. They're great for printing out and taking into London with you on a sight seeing trip.
But. You can't move anything, change anything, search for anything, update anything. It's static. A snapshot in time of the way things were, because the second you printed it, it's out of date. History.
The second type of map, we call vector. What it actually means is, 3 dimensional. Sometimes quite literally with the help of some funky tools and as this lovely map of Hong Kong shows.But more often than not, it's 3 dimensional in an entirely different way in that it's searchable, scaleable and interactive. You can drop pins on it, move around the map and the pin will stay in the right place - over the top of the house you dropped it on. You can search by street name and you can plot a path on it - well draw one actually, you don't have to plot anything at all.
There are 2 kinds of mapping tool The first kind are the ones like Google, Bing and the previously linked to 3D Hong Kong map. They're useful to a point, but the point quickly becomes limiting. You can search it, and it will take you to the street you were looking for. But it will be a rough approximation of a street. Put the satellite layer over the top and you'll see what I mean - they're ever so slightly out of sync.
Next, try dropping a pin somewhere on a Google map with the zoom level not zoomed all the way in. Now zoom all the way in. Pin isn't quite on the corner of that junction that you dropped it on any more. It's moved. Fine if you trust that people are capable of spotting a bright yellow bin full of grit from 20 feet away. Not good if that pin represents the something you need to be accurate. Next, search for Witton Park on Google maps. Zoom in to the 2nd from last setting. Note that Witton is spelt as Whitton - right next to each other. One of those spellings is right. One is wrong. Do you know who I can report that to? No one.
Finally, embedding Google maps is a complete nightmare. If you have more than 30 things or so to map, then it will trip over to page 2. And then page 3. Which is fine if you're viewing a map in Google and you realise that that's what happening (some people wont and will think, for example, that you've only mapped the grit bins in Darwen and ignored Blackburn completely). When you come to embed the map into a webpage, the fact that there is more than one page of information you've mapped isn't mentioned. In order to get all the points to display in your embedded map, you have to go to Google Maps, hit the RSS button, get the RSS url of your points and chuck it back into Google Maps again. Then you are graced with a map which you can embed which will show everyone all your points.
I could go on. I think you get the point. You might ask why on earth anyone ever uses Google Maps. I'll explain that at the bottom but over the next few paragraphs, I suspect it will become clear.
The second kind of map is Ordnance Survey built and oh boy is it accurate. Need to know where a set of steps is or how wide a pavement is? OS mapping can tell you. Until April 1st this year (2010) the maps were acknowledged to be so accurate that people paid a lot of money to access that data. Witton gets spelt the right way. If it weren't I'd know exactly who to speak to to get it corrected. They not only provide line maps, they also provide maps with door numbers on to help you orientate yourself (or check your co-ordinates are correct). There is still a bit of a zoom problem, in that if you draw on the map at a certain zoom level, it will move slightly when you zoom in, say from 5km to 1km. But under 1km where you switch to a more detailed map, it's accurate. Points dropped don't wander off. It's accurate, it's reliable.
More interestingly, it can also be fed into Geographical Information Systems (GIS). GIS is a loose term for the set of tools which allows you to place massive amounts of points onto OS mapping, quickly and reasonably simply. Most people have done a degree in these systems and are can make them dance. I've taught myself almost entirely and so am a little behind. To add insult to injury, GIS systems are enormously expensive and so I don't have access to one any more because it's not part of my job. Which is where Open Space comes in. Open Space allows you to do similar things to what you historically could only do with an expensive GIS tool.
So, why, I hear you ask, have I mapped our grit bins and gritting routes using Google?
Ordnance Survey have made a big big deal of their Open Space tools. This 'path' map is an amazing example of what can be done - entirely for free. Except. There's always an except, isn't there. In the bottom left hand corner, you will note a little column which depending on the time of day you view it, will either be green or red. Hover over it with your mouse and you will see "XX% of daily map tile limit used". Open Space, when publicising their wonderful new tools, seem to forget to mention this. You get 150 'views' a day unless you pay for more. We'd go over that in about 3-4 hours, I'd guess. What happens after you exceed that 150 views? Well, we don't know but we're assuming that no information is served at all. Which isn't going to look very good now, is it?
And we're skint. Utterly and completely broke, as a Council, with £48 million of savings to make before 31st March 2011. So I didn't map the routes/bins in anything but Google, because Google was free. Which is why they can afford to ignore complaints of spelling. Why inaccuracies can be forgiven. Why display errors meaning suddenly my screen shows 10 markers all on top of each other practically, for no reason whatsoever, must be simply ignored.
It's free.
So, in true Brit style, we make do and mend. In the meantime, I've bought myself a book and will hopefully be teaching myself how to code so I can make Google behave a little better, so I can code around some of the things which are causing a problem. But (I may be wrong here, but I don't think I am) even then, once you start to code and hack about with Google in anything more complicated than what we've done already, Google control access to their maps with something called an API key. And once again, imposes limits for page views.
So what's the answer?
I don't know. This is where I hope someone else picks it up and passes it on - back to me. Can anyone help? Is there an open source way of displaying a map, inside a box, with a zoom in and out tool, a search tool and is accurate, which I can embed in our web pages and will cost me nothing, be entirely reliable and not melt my brain in the process of trying to work it out?
So. To summarise. There might only be 2 different kinds of map. And only 2 different ways of seeing those maps and interacting with 1 kind of those 2 different kinds of map. But when you start to delve a little further, it's really rather more complicated than it might first appear. And free is free for a reason.
Atlases, and a love of maps, all kinds of maps, playing with maps, drawing maps, interacting with maps is something that I used to be a little bit shy about admitting. No more. The world has changed, or perhaps rather the circles I move within have changed, I'm not sure. But regardless, the simple pleasure of getting a system to place a marker in the right place, then colour it depending on pre-determined requirements still fills me with glee. It will always fill me with glee.
So, I guess we start at the beginning again, bearing in mind that the best holiday I ever had was navigating off a Michelin map through the Pyrenees as my partner got arm ache from driving around all the hairpins, that I can comfortably navigate people through the centre of London with the aid of an A-Z, in our house we don't use Tom Tom, we use Lou Lou and that I don't ever have to turn the map the right way round to orientate myself. I'm not boasting here, simply pre-empting the inevitable 'but you're a girl, girls can't read maps/can't navigate/can't read signs/can't read maps without turning them around'. This one can, just so we're clear. I am not alone in this, just so we're clear. They're not pre-requisites to being able to map data onto a map, just so we're clear. But loving maps, so intensely, understanding their power but also their restrictions? That really helps, I think.
There are two kinds of maps in the world. One comes as a photograph, a picture, a jpeg. We call them raster images - they are simply images and nothing more. They cannot be asked questions of, you can't search them, if you zoom into them, the points on the map ( the distance between your house and the local pub, for example) will move, but not in relation to each other to any kind of scale. Flat, 2 dimensional map. They've got their uses, of course they do, they're great as a print out on some water resilient paper to take out on the hill with you. They're great for printing out and taking into London with you on a sight seeing trip.
But. You can't move anything, change anything, search for anything, update anything. It's static. A snapshot in time of the way things were, because the second you printed it, it's out of date. History.
The second type of map, we call vector. What it actually means is, 3 dimensional. Sometimes quite literally with the help of some funky tools and as this lovely map of Hong Kong shows.But more often than not, it's 3 dimensional in an entirely different way in that it's searchable, scaleable and interactive. You can drop pins on it, move around the map and the pin will stay in the right place - over the top of the house you dropped it on. You can search by street name and you can plot a path on it - well draw one actually, you don't have to plot anything at all.
There are 2 kinds of mapping tool The first kind are the ones like Google, Bing and the previously linked to 3D Hong Kong map. They're useful to a point, but the point quickly becomes limiting. You can search it, and it will take you to the street you were looking for. But it will be a rough approximation of a street. Put the satellite layer over the top and you'll see what I mean - they're ever so slightly out of sync.
Next, try dropping a pin somewhere on a Google map with the zoom level not zoomed all the way in. Now zoom all the way in. Pin isn't quite on the corner of that junction that you dropped it on any more. It's moved. Fine if you trust that people are capable of spotting a bright yellow bin full of grit from 20 feet away. Not good if that pin represents the something you need to be accurate. Next, search for Witton Park on Google maps. Zoom in to the 2nd from last setting. Note that Witton is spelt as Whitton - right next to each other. One of those spellings is right. One is wrong. Do you know who I can report that to? No one.
Finally, embedding Google maps is a complete nightmare. If you have more than 30 things or so to map, then it will trip over to page 2. And then page 3. Which is fine if you're viewing a map in Google and you realise that that's what happening (some people wont and will think, for example, that you've only mapped the grit bins in Darwen and ignored Blackburn completely). When you come to embed the map into a webpage, the fact that there is more than one page of information you've mapped isn't mentioned. In order to get all the points to display in your embedded map, you have to go to Google Maps, hit the RSS button, get the RSS url of your points and chuck it back into Google Maps again. Then you are graced with a map which you can embed which will show everyone all your points.
I could go on. I think you get the point. You might ask why on earth anyone ever uses Google Maps. I'll explain that at the bottom but over the next few paragraphs, I suspect it will become clear.
The second kind of map is Ordnance Survey built and oh boy is it accurate. Need to know where a set of steps is or how wide a pavement is? OS mapping can tell you. Until April 1st this year (2010) the maps were acknowledged to be so accurate that people paid a lot of money to access that data. Witton gets spelt the right way. If it weren't I'd know exactly who to speak to to get it corrected. They not only provide line maps, they also provide maps with door numbers on to help you orientate yourself (or check your co-ordinates are correct). There is still a bit of a zoom problem, in that if you draw on the map at a certain zoom level, it will move slightly when you zoom in, say from 5km to 1km. But under 1km where you switch to a more detailed map, it's accurate. Points dropped don't wander off. It's accurate, it's reliable.
More interestingly, it can also be fed into Geographical Information Systems (GIS). GIS is a loose term for the set of tools which allows you to place massive amounts of points onto OS mapping, quickly and reasonably simply. Most people have done a degree in these systems and are can make them dance. I've taught myself almost entirely and so am a little behind. To add insult to injury, GIS systems are enormously expensive and so I don't have access to one any more because it's not part of my job. Which is where Open Space comes in. Open Space allows you to do similar things to what you historically could only do with an expensive GIS tool.
So, why, I hear you ask, have I mapped our grit bins and gritting routes using Google?
Ordnance Survey have made a big big deal of their Open Space tools. This 'path' map is an amazing example of what can be done - entirely for free. Except. There's always an except, isn't there. In the bottom left hand corner, you will note a little column which depending on the time of day you view it, will either be green or red. Hover over it with your mouse and you will see "XX% of daily map tile limit used". Open Space, when publicising their wonderful new tools, seem to forget to mention this. You get 150 'views' a day unless you pay for more. We'd go over that in about 3-4 hours, I'd guess. What happens after you exceed that 150 views? Well, we don't know but we're assuming that no information is served at all. Which isn't going to look very good now, is it?
And we're skint. Utterly and completely broke, as a Council, with £48 million of savings to make before 31st March 2011. So I didn't map the routes/bins in anything but Google, because Google was free. Which is why they can afford to ignore complaints of spelling. Why inaccuracies can be forgiven. Why display errors meaning suddenly my screen shows 10 markers all on top of each other practically, for no reason whatsoever, must be simply ignored.
It's free.
So, in true Brit style, we make do and mend. In the meantime, I've bought myself a book and will hopefully be teaching myself how to code so I can make Google behave a little better, so I can code around some of the things which are causing a problem. But (I may be wrong here, but I don't think I am) even then, once you start to code and hack about with Google in anything more complicated than what we've done already, Google control access to their maps with something called an API key. And once again, imposes limits for page views.
So what's the answer?
I don't know. This is where I hope someone else picks it up and passes it on - back to me. Can anyone help? Is there an open source way of displaying a map, inside a box, with a zoom in and out tool, a search tool and is accurate, which I can embed in our web pages and will cost me nothing, be entirely reliable and not melt my brain in the process of trying to work it out?
So. To summarise. There might only be 2 different kinds of map. And only 2 different ways of seeing those maps and interacting with 1 kind of those 2 different kinds of map. But when you start to delve a little further, it's really rather more complicated than it might first appear. And free is free for a reason.
Wednesday, 3 November 2010
GIS is to #opendata as.......
GIS (Geographic Information System) is to open data as Microsoft Word is to a bunch of incohesive words and letters. Or at least, when you've been working with GIS for a bit, that's how it looks. A slightly distorted view of course, because GIS wasn't involved in the 15 minute dalliance with some data I danced the other night, but nevertheless, data and GIS is intrinsically linked for me and so I thought I'd try and explain why - with a diagram!
Look ma, no hands! Now, as regular readers will know, I'm not very good at diagrams. Or data visualisations or mash ups or whatever the cool and funky kids are calling them these days - I'm not a cool and funky kid either so I wouldn't know. But this is a pitiful attempt at an explanation, nevertheless because letters are boring and pictures are shiny.
Open data is the end of the story as I hope I've made clear here. Operations is the start. Operations is what generates all the data to go out into the open. Operations means childrens services, it means street scene or refuse collection and street cleansing with some park maintenance in there as well depending on what you're calling it this year, it's the actual day to day stuff like how many fly tips we collected or how many parking tickets we issued or how many library books we lent. It's our bread and butter, it's what we do, it paints a picture in numbers of the service we provide to everyone on a day to day basis.
The data from all that operational activity currently goes in one direction but eventually, one day, will go in two as shown here. At the moment, it goes into management information and in a lot of cases it gets fed, if appropriate and in most cases it is, into the GIS server and processed by our GIS software to make easy to understand visual representations of the data which our Managers can use to make informed management decisions quickly and easily, because the data is not 64,000 rows of ascii (raw letters), but instead thematic mapping, showing them where their hotspots and notspots are, where they need to focus more resource and where there was a problem 12 months ago but now isn't and so they can move resource. It's done more often than 12 monthly, but for the purpose of this, we'll call it 12 months.
The appropriateness of the information going into GIS software is generally whether it's spatial. Spatial means, relational, means does it have a latitude or longitude on it, does it have GPS data attached to it, will mapping it spatially make sense for the data to mean something. In a lot of cases, well actually, in most cases it does, from school catchment areas crossed with deprivation indices crossed with academic achievement levels to a thousand and one other 'mash ups' as they're now called.
Research and intelligence and policy actually have a two way relationship with GIS. They pull data out of the datasets parked there by ops and they crunch it, create 'mash ups' and provide it to Directors and Heads of Service to inform them. They do trend analysis and many other complicated and funky things. Policy take this crunched data too and they build our strategic advice on it. They tell people in words of one syllable (and yes sometimes more) where we were, where we are and where we will be if things continue as they are, but also where we will be if they do not. They don't hold crystal balls, they assure me, but I'm not so sure. GIS and much other non-spatial data is their bread and butter, I think (someone will correct me if I am wrong here, I'm sure).
The datasets which drive all this, will one day go straight onto data.gov.uk once signed off too. INSPIRE says they will. Please read the link, if you've got this far, you need to know about the existence of this Directive.
Which brings me to the other side of open data and the things in data.gov.uk which will sit next to the Operations generated stuff. The cost of satisying Freedom of Information requests was quoted at me by someone at a conference recently from their Authority and I will not post here what it was but it was enough to make my jaw drop. If we are transparent where it comes to information, if we are open, the assumption is that we will actually never receive FOI's ever again, because it will all be easily found, therefore cutting out the middlemen of the poor Administrators and Officers tasked with fulfilling these requests and instead leaving them to do their main job roles. But for the moment, perhaps it would be a nice interim policy for people to put the results of FOI's onto the web automatically, in the assumption that if one person wants to request the information, then perhaps a second will too? It's obviously of interest to someone, right?
And then there's spending data. Generated by Operations but kept by Finance. This is left to last, because Government have almost wrapped this one up. All spend over £500 will be published by local government in January 2011. All NHS PCT spend over £25,000 is already online.
Here ends the tour of data within local government. If you got this far, you know as much as I do, almost. Which in the interests of openness, is exactly the way I believe it should be.
Look ma, no hands! Now, as regular readers will know, I'm not very good at diagrams. Or data visualisations or mash ups or whatever the cool and funky kids are calling them these days - I'm not a cool and funky kid either so I wouldn't know. But this is a pitiful attempt at an explanation, nevertheless because letters are boring and pictures are shiny.
Open data is the end of the story as I hope I've made clear here. Operations is the start. Operations is what generates all the data to go out into the open. Operations means childrens services, it means street scene or refuse collection and street cleansing with some park maintenance in there as well depending on what you're calling it this year, it's the actual day to day stuff like how many fly tips we collected or how many parking tickets we issued or how many library books we lent. It's our bread and butter, it's what we do, it paints a picture in numbers of the service we provide to everyone on a day to day basis.
The data from all that operational activity currently goes in one direction but eventually, one day, will go in two as shown here. At the moment, it goes into management information and in a lot of cases it gets fed, if appropriate and in most cases it is, into the GIS server and processed by our GIS software to make easy to understand visual representations of the data which our Managers can use to make informed management decisions quickly and easily, because the data is not 64,000 rows of ascii (raw letters), but instead thematic mapping, showing them where their hotspots and notspots are, where they need to focus more resource and where there was a problem 12 months ago but now isn't and so they can move resource. It's done more often than 12 monthly, but for the purpose of this, we'll call it 12 months.
The appropriateness of the information going into GIS software is generally whether it's spatial. Spatial means, relational, means does it have a latitude or longitude on it, does it have GPS data attached to it, will mapping it spatially make sense for the data to mean something. In a lot of cases, well actually, in most cases it does, from school catchment areas crossed with deprivation indices crossed with academic achievement levels to a thousand and one other 'mash ups' as they're now called.
Research and intelligence and policy actually have a two way relationship with GIS. They pull data out of the datasets parked there by ops and they crunch it, create 'mash ups' and provide it to Directors and Heads of Service to inform them. They do trend analysis and many other complicated and funky things. Policy take this crunched data too and they build our strategic advice on it. They tell people in words of one syllable (and yes sometimes more) where we were, where we are and where we will be if things continue as they are, but also where we will be if they do not. They don't hold crystal balls, they assure me, but I'm not so sure. GIS and much other non-spatial data is their bread and butter, I think (someone will correct me if I am wrong here, I'm sure).
The datasets which drive all this, will one day go straight onto data.gov.uk once signed off too. INSPIRE says they will. Please read the link, if you've got this far, you need to know about the existence of this Directive.
Which brings me to the other side of open data and the things in data.gov.uk which will sit next to the Operations generated stuff. The cost of satisying Freedom of Information requests was quoted at me by someone at a conference recently from their Authority and I will not post here what it was but it was enough to make my jaw drop. If we are transparent where it comes to information, if we are open, the assumption is that we will actually never receive FOI's ever again, because it will all be easily found, therefore cutting out the middlemen of the poor Administrators and Officers tasked with fulfilling these requests and instead leaving them to do their main job roles. But for the moment, perhaps it would be a nice interim policy for people to put the results of FOI's onto the web automatically, in the assumption that if one person wants to request the information, then perhaps a second will too? It's obviously of interest to someone, right?
And then there's spending data. Generated by Operations but kept by Finance. This is left to last, because Government have almost wrapped this one up. All spend over £500 will be published by local government in January 2011. All NHS PCT spend over £25,000 is already online.
Here ends the tour of data within local government. If you got this far, you know as much as I do, almost. Which in the interests of openness, is exactly the way I believe it should be.
Subscribe to:
Posts (Atom)