Wednesday, 10 October 2012

'But I haven't got time' and other assorted whinings

I'm getting really bored of hearing the 'but we haven't got time to do digital' refrain. So I thought I'd tell a little story of something which happened to me on Sunday which demonstrates the economic impact of not doing digital to a company I assume most of you will have heard of: See tickets.

The players


There are three main protagonists in this tale, but a cast, quite literally, of thousands. The first is e-festivals. You might not have heard of them but in a world where every festival going has a discussion forum of its own, there is one festival which does not have a forum and thus e-festivals fills a customer driven hole. If you want to know anything at all about Glastonbury Festival, you go to the e-festivals forum, not the Glastonbury Festival official website because despite having Twitter streams and Facebook pages, online forums are a space they've decided not to enter for some reason. This may be a choice they're currently regretting.

The second protagonist is Glastonbury Festival itself. For ease of reference we'll use Emily Eavis as the reference person as she was very visible over last weekend in various places and thus self nominates as the face of Glastonbury Festival (she's the founders daughter).

The final player is See tickets. Owned by Vivendi, trading since 2002 in its current for - they're in the same space as Ticketmaster.

Pride before a fall


On Saturday Emily Eavis spoke to NME about the impending yearly disaster that had historically been Glastonbury Festival tickets going on sale. She said:

"We’ve got a new system which we’re trying out, and we’ve had assurance from See Tickets that it is going to work well and be much more efficient. All I can say is that we’ve done all that we can to try and make it as quick as possible and not have people hanging on phone lines for hours."

And we all know where this is going, don't we?

Sunday morning dawned. I, and what estimates mark as 1,999,999 other people hauled ourselves out of bed, consumed a lot of coffee, and tried to connect to glastonbury.seetickets.com which was the link which had been circulated by official sources prior to sales commencing.

Early warning systems


It was immediately quite clear something was wrong. Normally, once you get to the point where you're seeing webpages and entering details and pressing a submit button on a webpage something quite neat has happened behind the scenes of the webpage. You've got what's called a session ID and it tracks you through the process from end to end to ensure that no matter how many other people are trying to connect to the servers in the form of trying to get webpages to load, you will always be able to get to the end of your session once you've started it.

No.

Those lucky enough to get just one of the 20 or more webpage tabs to connect (in our case across two laptops, a phone and an ipad) were finding that entering registration codes and postcodes of those wishing to pay for tickets were clicking on Submit and being thrown back out to the 'page cannot be displayed' error message. At this point I sent a message to @glastofest on Twitter mentioning that their new improved system seemed to actually be worse than the last one. I didn't get a reply but I did get a lot of people replying who were in the same boat.

Eventually I decided to abandon clicking on reload buttons endlessly and popped over to e-festivals. This is where it gets really interesting.

It's not what you know, it's who you know


This comment popped up with quite a baffling message saying that editing your host file with a set of numbers (an IP address), a space and then glastonbury.seetickets.com meant you could get through to See tickets. This comment quite rightly pointed out that the numbers, if you checked the address behind it, landed you up at a Virgin Media private account. Not See tickets.

Note the timestamps on the comments. At this point it's 9:26am. At this point, had you been an employee of See tickets and you'd had your wits about you and done a bit of the old digital stakeholder mapping, the alarm bells would be seriously seriously ringing at this point.

They weren't watching.

And the comments started rolling in from people who'd mysteriously suddenly acquired tickets super quickly, once they'd amended their host file. Lots of comments from lots of people buying lots and lots of tickets on multiple machines, interspersed with a few slightly more cautious people. At 9:58 someone states that the IP number version of the web address (194.168.202.201) was now pointing at glastonbury.seetickets.com and not the private Virgin media account. My other half in the midst of this checked the same IP address and got told it belonged to a school as had someone else.

By this point suspicions were rising as to what had happened and what was causing the problem in a few techies minds. Meanwhile, over on Twitter:

9:39 am

followed by this at 10:11 am
Magically, at 10:12 am exactly, my ipad paid for itself by suddenly displaying the registration page, and then immediately the payment page and the the ticket purchase confirmation page, one after the other, in the space of 3 minutes.

So what actually happened that a number version of a web address that variously pointed at Virgin and a school actually seemed to send you straight to the Glastonbury purchase bit of See tickets?

The technical bit


Someone mistyped a number. This next bit's technical, you can skip it if you want. When you set up servers, the web has to know how to find them and IP's are those identifiers. They're the digital equivalent of your house no and postcode. Unique, they prevent mix ups. Now you don't ever see those numbers when you go to web pages, and you don't have to remember the numbers of your favourite websites either. You type in words instead. This is because there is a system called DNS which matches the unique numbers to the word versions of the addresses. You type the words, press enter or click on load, your browser disappears off, goes looks at the DNS, and similar to a telephone directory, finds the entry for that word based address, matches it to the correlating numbers and sends the request to the numbers listed next to the word address in the directory.

Someone has to tell the directory which numbers match which addresses specially when registering a new address lots of people are going to be looking for. Like glastonbury.seetickets.com and matching it to 194.168.202.201, and other addresses besides, as because so many people would be visiting that address they decided to send the requests to lots of different numeric addresses to process the requests faster. Except they mistyped one. They registered 192.168.202.201. Which is wrong. It's the equivalent of someone mistyping you're and your however, in that 192.168 usually denotes a home network. That's why it went to a private Virgin Media account.

Damage done


Once fixed, effectively See doubled the amount of requests for webpages they could handle. Unsurprisingly, this tweet appeared shortly after at 10:24 am.
Yep, it'd taken a full 84 minutes to get to the point where they'd sold half their 137,000 or so tickets despite the previous tweets about high demand. What's even more telling is how quickly they sold the other half of the 137,000 tickets available. This tweet appeared at 10:44 am:
68,000 tickets in 20 minutes as opposed to the 80 minutes the other 68,000 had taken.

Look at what you could have won


I'm reasonably sure someone at See tickets has uttered the immortal words 'but I haven't got time' when it comes to monitoring feedback on their services on social media. I'm sure there are days when it must look like a thankless task wading through thousands of tweets moaning about how people didn't get Take That or Kylie Minogue tickets. Lets face it, there's little value in that kind of monitoring, you're not learning anything new. But if you close the door and turn your back, dismissing all the content on social media as irrelevant you can find yourself in a rather unfortunate situation. One where your reputation is in tatters, one person has made a fundamental mistake which is now impacting on your SLA which has been agreed, which the figurehead of the biggest festival in the world has gone on record as trusting in, which has resulted in a well respected public figure ending up with serious egg on their face and which, one expects, might put future years agreements of sole ticket provision with aforementioned festival into serious jeopardy.

I've not got to how the customers feel yet. For a flavour, see their Facebook page where over 2,000 people are expressing their problems. Interestingly, and I haven't read the whole comment thread, but a fair portion of it, the problems with the DNS don't seem to have permeated over to that bit of the internet. So even if See did have a social media monitoring strategy for Twitter and Facebook, they'd still have no warning of something being wrong at their end. 

So what's the lesson?

Being outside the conversation, being ignorant of the conversations, it can cost you money. Real cash in terms of missed SLA's or a stupid judgement on direction or trend which could have been avoided instead of needing to be unpicked and unraveled a year later when that decision is shown to be stupid. Real reputational terms in that you can turn around disasters before they become ones using the early warning system which is the social media estate. But perhaps more fundamentally, it can cost you the trust you have with your customers, when you've invested thousands into a new shiny system, convinced your customer that the new shiny system will deliver and then left them with egg on their face when the system not only doesn't perform, but it's clear the customers knew it wasn't performing before you did. That's not a happy situation for anyone to be in.

So next time someone is trying to help you with your social media presence and you feel like uttering those words 'I haven't got time', have a quick think.

What's the risk and economic implications wrapped up with deciding not to make time?





Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)