Archive | News

19 July 2012 ~ 0 Comments

O2 outage fallout

News

Following our exhausting O2 outage liveblog last week we thought we’d summarise what happened and take a look at some of the fallout.

What happened?

About 1pm on Wednesday 11th July many O2 users lost signal. The blackout affected all services – 2G, 3G, data and texts. At the time, no explanation was given for the outage, the Network Status page crashed and it took a long time for O2 to acknowledge the problem. The outage seemed to affect people somewhat randomly – one person could be fine but another users in the same room using the same phone might have no service whatsoever. O2 were unable to provide any advice or a timeline for restoring service. In total, the outage lasted about 24 hours.

Who was affected?

O2 have about 23 million customers in the UK. The network estimated that about a third of customers were affected which equates to almost 8 million users. In addition, as they are virtual mobile networks that piggyback off the O2 signal, Giffgaff and Tesco Mobile customers were also affected. The issue caused signal loss to all customers regardless of whether they were business or retail customers or whether they were on PAYG or contract. Even celebrities such as Jemima Khan, Huw Edwards and Kym Marsh were frustrated as their phones were affected.

In addition, several vital services went down because of the outage. Some bike hire stations in London use O2 for processing payments. According to Transport for London, about 100 bike hire stations and 1000s of bikes were affected.

Even worse, troubled private contractor G4S is not only incompetent and morally corrupt but they use
O2 in their electronic tag systems for monitoring criminals on curfew. Unknown numbers of convicts on probation may have broken their curfews as the O2 outage compromised the systems that are used to prevent breaches. The Ministry of Justice admitted that about 250 offenders were affected by the outage.

When was everything fixed?

We kept everyone up-to-date as things progressed and monitored user forums to get the latest on people’s experiences getting signal back again. The first users reported a restoration of their mobile service at about half past midnight on the 12th July. This continued for several users over the next couple of hours.

However, many people still couldn’t get a connection and remained without signal well into the late morning. Just before 8am it became clear that :2g: services had mostly been restored and by turning off 3G, many users could get their service back. By this point, 19 hours after the initial problem, a significant proportion of users were back online. It took two more hours for O2 to officially declared that all 2G services (voice and data) were restored but 3G was still down. Even so, the blackout wasn’t completely over as 3G services were still down and a huge number of customers still couldn’t get their phones to work at all.

Just before 11am O2 gave their first estimate for a timescale on the fix. They said things should be back up and running by the afternoon (not very precise, we know). It wasn’t until 13:40 before O2 claimed that all 3G services were back up and running again. In total, the outage had lasted about 24 hours.

What did they do to fix it and keep people informed?

It’s not very clear what was required to fix the problem as O2 have been far from open. They also haven’t given any explanation for why it took so long. What we do know is that O2 had hundreds of engineers working through the night to fix the problem.

In fact, their communication throughout was very poor. It took a long time to acknowledge the problem in the first place, the Twitter account didn’t update sometimes for 10 hours at a time and the status page was inaccessible for hours at the time. When they did release updates, they were labelled with incorrect timestamps to mislead people into thinking they had been posted earlier, questions about compensation were routinely ignored and it took over 20 hours for an estimate for how long things would take to fix to be announced. In the meantime, the outage was extensively covered by all the major media outlets.

Even though the outage was eventually rectified, it took an entire day. Perhaps the worst aspect of the customer communication was the fact that at no point was the fault explained and it took until about 3pm on the following day for any apology to be issued.

Giffgaff were much better than O2 with almost hourly updates posted on their user forums. However, they also wouldn’t give any timescale for a fix or attempt offer an explanation of what went wrong. Users also had to wait until 10am the day after the outage began for a proper apology.

What caused it?

O2 have still refused to be anything but extremely vague and what went wrong, where and why. All they would say is that there was a network issue. Giffgaff went somewhat further by saying that the problem was “an issue within part of our core network that is preventing some mobile phones from successfully connecting.”

The actual issue was a problem with a part of the network called the Home Location Register (HLR). This allows users to connect to the network by consulting something called a Centralized User Database (CUDB). We don’t know exactly what the problem was but a issue with the CUDB and HLR prevented some users from making and maintaining a connection with the cell towers as they could not be authenticated correctly. This explains why the outage wasn’t localised at all and affected all types of user. The fact that two people in the same room on the same handset could be affected differently also makes sense with this explanation.

Stay tuned as we’re planning a post in the next few days going into a bit more detail about what caused the outage.

Can I get any compensation?

O2 (and Giffgaff) were initially incredibly coy about the subject of compensation. At the very least, millions of people have lost service that they have paid for and the forums are awash with people who were left stranded or lost £1000s worth of work due to the outage. However, the networks will probably do their best to weasel out of their responsibility toward their users and during the outage neither network responded to press enquiries or confirmed they will accept liability and pay compensation.

Early on Giffgaff said they had “no information on compensation” and further queries were ignored. Previous issues with Giffgaff outages and instances of overcharging have resulted in Giffgaff avoiding paying out any compensation at all. All they have done in the past is make a small donation to charity instead of refunding any of their customers. However, after the outage, a blog post admitted that they are discussing and considering “providing minutes, money off or similar as an apology for the outage”.

O2 have been incredibly careful to avoid saying they will pay any compensation and their terms and conditions have been cleverly written so they are absolved of any responsibility. Ofcom rules probably won’t force they to pay anything out and they haven’t ruled out reimbursing any customers. Ronan Dunne, O2‘s UK CEO wouldn’t make any statement about compensation but did hint that there might be some sort of non-monetary gift to affected customers.

In the end, both Giffgaff and O2 decided against offering proper compensation. Instead, Giffgaff and O2 PAYG customers will get 10% extra credit on a top-up. O2 customers can top-up any time in September (the distant date is perhaps a cynical attempt to keep customers on the network for another month?) while Giffgaff customers have only from the 19th until the 25th of July. If you are an O2 contract customer, you’ll get 10% off your July bill instead. This equates to just a few quid. Both the O2 discounts only apply to those affected but all O2 customers will also get a £10 voucher to spend in-store.

While this is better than nothing, many people will be furious that a pound here or there does nothing to make up for the inconvenience and damages caused by the avoidable outage. Furthermore, it will probably be pointed out that offering bonus credit doesn’t exactly cost the networks much as they get heavily-discounted commercial rates and that products in O2 stores are already heavily overpriced.

Even so, don’t let this discourage you – Which? has encourage the network to do the right thing. In any case, if you feel you deserve proper compensation, ask your network and take it to the Ombudsman if necessary. If you want to get out of an O2 contract, you might also be able to use this outage to your benefit if you feel that O2 have broken its terms.

Questions?

Does that sum things up for you? Were you affected by the outage? How has it affected your opinion of your network? And is there anything else you’d like to know from us?

Continue Reading

Tags: , , ,

13 July 2012 ~ 1 Comment

4G delayed again?

News

Things aren’t looking good for high-speed mobile users in the UK. Once again, Ofcom‘s mismanagement has result in us Brits lagging behind Asia, the States and the rest of Europe as it’s not looking like the full 4G roll out won’t happen until summer or autumn next year. 4G networks are already running strong in America and even in countries such as Saudi Arabia and Sri Lanka but it’s looking like we won’t see them on our shores until July 2013 at the earliest. The race to 4G hasn’t even started properly yet.

With mobile phones becoming much more than just phones and with individuals and companies relying on mobile internet more than ever this is only going to harm British business and doesn’t make anyone look good. Mobile growth has already been hindered by the crippling data costs Ofcom allows networks to charge us and now the delay in Ofcom auctioning the 4G spectrum is going to set things back even further.

With the release and auction of the spectrum likely to make millions for the government they obviously want to take their time and do it properly. But issues with other wireless transmission technologies (such as Freeview TV) and squabbling between the various networks has complicated things. Culture Secretary Jeremy Hunt somehow still has his job but is certainly dragging his feet and allowing Ofcom to waste valuable time. We’ve already complained that Ofcom aren’t being forced to use the spectrum auction for good.

O2 have recently been complaining that the difficulty of just getting the spectrum sorted out so the mobile networks can get on with it has been vastly exaggerated. They have already spent over £1 billion preparing for 4G and are eager to get started. However, Ofcom have previously criticised O2‘s conduct about the release of 4G frequencies. Whatever the true situation is, it looks like delays have got the better of things again. And the sad conclusion is we still have at least another year to wait.

Why do you think 4G is taking so long in the UK? Will you use it when it is ready? And do you think it’s fair to blame Ofcom?

Continue Reading

Tags: , , ,

11 July 2012 ~ 16 Comments

O2 outage liveblog

News

We’re getting reports that O2 is suffering a partial network outage. Giffgaff and Tesco Mobile are also supposedly affected although our staff are currently not having any problems. We’ll update from across various sources as the situation progresses.

Update 17:01

Giffgaff have confirmed they are also affected by the outage:

We are currently experiencing an outage for voice, text and data (2G and 3G). We are investigating the issue and hope to have it up and running as soon as possible.

All users will experience intermittent voice and data for the duration. This will only affect active SIMs, so this won’t stop new SIMs from activating.

Update 17:15

O2 have just tweeted the following:

Update 17:25

Seems the issue has been going since at least 01:30pm according to the O2 network status checker:

We continue to manage the problem on our network that is affecting some of our customers. Customers may still be having difficulty making and receiving calls, sending texts or using data.We are sorry for any inconvenience. Our engineers are dealing with the problem as a priority and we hope to restore full service as soon as possible

Update 18:01

The issue does not seem to be localised as we are getting reports of service loss from across the country.

Giffgaff have made another official update:

Update 19:10

Hi guys, at the moment there is no new news, but the tech team are working to resolve this as quickly as possible. It will not affect ports or sim activations.

Update 20:03

Another update from the Giffgaff forums:

We have no new news at the moment and no ETA, but please be assured that the tech team are working to resolve this as fast as possible. As soon as we have any new information I will update this thread.

Update 21:02

Another totally useless update from Giffgaff. No news from O2 in the last 4 hours.

Hi guys, unfortunately we’ve heard no new news from the tech team but are assured they are working hard to resolve this as quickly as possible.

Update 21:17

Another tweet from O2. Still not very enlightening 🙁

https://twitter.com/O2/status/223147999000408064

[edit: 21:30 – the link to this tweet is now broken as O2 have deleted it.]

Update 21:21

This has hit all the major UK news networks. A round-up of some of the stories:
BBC – O2 mobile users suffer network problems across UK
The Guardian – O2 network crash leaves mobile users without service
‎The Telegraph – O2 customers suffer nationwide network failure
The Register – O2, GiffGaff network goes titsup for unlucky punters
The Verge – O2 experiencing major outage in the UK affecting data, calls, and texts
Daily Mail – O2 hit by nationwide network failure that left ‘millions of users’ unable to make calls or text this afternoon

Update 21:27

O2 just helpfully made an exact copy of their last tweet 🙁

Update 21:35

http://status.o2.co.uk/ now appears to be down.

Update 21:37

The Guardian are now reporting that the O2 webpage has crashed.

Update 21:57

The outage thread on the Giffgaff community forums has now hit 136 pages – that’s over 1360 posts. There’s still no real information forthcoming unfortunately.

Update 22:02

Some people on Giffgaff forums are saying only O2 should be to blame but others say they are equally as responsible as it’s not out their control that they use O2 who aren’t exactly known to be very reliable. What do you think?

Update 22:03

Another hastily-written non-update from Hazel on the Giffgaff forums:

Good Evening, unfortunately we;ve received no news from the tech team working on this issue, but we do know that they are working to resolve this as quickly as possible and that they will work until it fixed,

Update 22:13

No more news yet. Instead, a shameless plug: if you’ve come here to follow this liveblog, you might want to visit our super mobile price comparison table to weigh up your other options. We gratefully receive Facebook likes and retweets too 🙂

Update 22:15

Are you on O2, Tesco Mobile or Giffgaff? Have you been affecting? Are things getting better or worse? We want to know! Please respond in the comments below.

Update 22:19

It appears the problem is with SIM cards registering with O2 masts. Once you’re connected, you should keep your signal okay there are just issues with that initial connection. This explains why one phone might be okay but another in the same city/room might have no signal.

Update 22:24

Can anyone get http://status.o2.co.uk/ to resolve right now?

Update 22:27

People are getting a little frustrated on the Giffgaff forums. “theboy” posted the following:

GG Educators – can you stop posting rather pointless replies.

By now, O2 would have at least a Plan A, B and C in progress to resolve this. Can you find out what they are, and post something useful?

That would be more encouraging that the rather mindless “we haven’t heard anything, but we know they are working on it”

They are certainly being more communicative than O2 or Tesco but that isn’t saying much.

Update 22:30

O2 are on BBC News on TV – they are saying that there is a problem with their core network processes. Sounds serious.

Update 22:35

An O2 spokesperson says that at least “thousands” of users are affected. Judging by the outrage on forums and Twitter, it would seem as though that’s a low estimate given that there are about 23 million O2 customers in the UK. It’s even affecting Jersey right now.

Update 22:45

Apparently O2 think it’s “harsh” that their customers are saying they suck regarding this outage.

Some might say it’s harsh that they still haven’t explained what’s causing it, what they are doing to fix it and when they expect it to be sorted let alone that an unknown number of their customers are completely without service.

Update 22:55

Just noticed that O2‘s Live Status Checker is back up (for now at least). They added an update timestamped 19:00 saying that they “continue to manage the problem on our network that is affecting some of our customers”. Still no clarification of how they are “managing” it or how long they will “manage” it for.

Update 23:02

Another hourly update from Giffgaff. Unfortunately it doesn’t say much about the outage.

Due to a huge increase in traffic on our website tonight we are experiencing some extra delays with top-ups, purchasing goodybags and timeout errors for logging in and ‘mygiffgaff’. These issues will subside by 23:30.

The issue is not based on geography. Some members still have connections at the same location as others who do not. The problem is an issue within part of our core network that is preventing some mobile phones from successfully connecting.

The tech team are working hard to resolve this as quickly as possible.

Update 23:10

O2 claim to have 200 engineers dedicated to network performance. They don’t mention how many are working right now or whether any of them have any news about the outage.

Update 23:20

O2 status update datestamped 23:00:

Our engineers and vendorscontinuing [sic] to deal with the problem as a priority and we hope to restore full service as soon as possible

Update 23:39

A Giffgaff user writes the following:

The fault is that some customers’ phones cannot authenticate because the HLR (these hold the account details) is unreachable.

We can’t comment on the accuracy of this information but it seems reasonable. There’s definitely some problem within the O2 backbone it seems.

Update 23:45

Doesn’t look like there will be any news for a while so take a gander at our favourite tweets about the outage so far – what are yours?


https://twitter.com/1OMIG/status/223184574363082755

Update 23:50

Can anyone confirm whether switching on and off again fixes the problem?

Update 23:54

http://status.o2.co.uk/ seems down again here. Anyone else?

Update 23:59

It seems that it does work intermittently but seems to be experiencing symptoms of a DDoS. Not great!

Update 00:00

Midnight update! Still no solution on the horizon. There are now over 92 news articles about this issue. And thanks to Hayley who confirmed that switching it on and off won’t help. Hopefully switching it on and off won’t break phones that are currently working either then.

Update 00:01

Giffgaff’s most recent update from hardworking Hazel:

The delay issues with goodybags and top ups, logging in and timeout errors has now stabilised and cleared up.

The latest on this issue is that the engineers are continuing to work throughout the night to resolve a fix.

Update 00:13

If your phone is currently working, turning it off and on again shouldn’t cause you to lose signal so don’t worry about that.

Update 00:15

We’re hearing more and more rumours that the cause of the issue is a problem with O2‘s HLR (Home Location Register). This is a network-wide system that deals with connecting subscribers to cell towers. The HLR is usually made up of several components including a gateway/frontend, backend servers as well as the database they get their information from.

It’s not yet clear where exactly the current problem lies or what’s causing it but O2 documents confirm that the HLR includes information on supplementary services, authentication credentials and subscriber profile information such as details about APN (Access Point Names) which provide data access. For example, when you try to use the internet on your phone, O2 checks with the HLR to see whether you have permission to use the APN which provides web access. If the HLR is not working properly you won’t be able to get online.

Orange very recently had an issue with their new HLR set-up in France which took over 12 hours to resolve. Vodafone have also had issues in the past. O2 (nor any of the MVNOs that use it) have yet to officially confirm that the HLR is causing the issue.

Update 00:36

Anthony from Giffgaff says:

We have no imformation on compensation currently as all efforts are being put towards resolving this issue primarily. We are waiting for further information at the moment so please keep checking back here for updates. Apologies fo any incovenience caused. [sic]

Update 00:37

Another quick-sharp update from Giffgaff. Might it almost be sorted?

Crossing fingers, but I’ve seen ten members in the last 15 minutes say their services has come back.

Update 00:44

We’re getting more and more reports that service is now being restored. If yours is now fixed, please let us know.

Update 00:58

The latest official Giffgaff update bodes well 🙂

No official update from the engineers as of yet but we’re seeing more and more reports of service returning for members. I’ll update you as soon as we get any information,

Update 01:05

Here’s the latest O2 update. No mention of an imminent fix yet.

We can confirm that the problem with our mobile service is due to a fault with one of our network systems, which has meant some mobile phone numbers are not registering correctly on our network. As a result, some customers are having difficulty making or receiving calls, sending texts or using data. We, and our central supplier, have deployed all possible resources and are working through the night to restore service as soon as possible. We apologise again to customers affected and will provide further updates as soon as we can

Update 01:59

Looks like the end is nigh at last 🙂 Here’s what Giffgaff have to say:

Thank you for your continuing patience through-out the night. The engineers are aware of the cause and are working hard to resolve this as quickly as possible. We’re witnessing more and more member having their services returned. Thank you again for your understanding and sorry for any inconvinence

Update 03:15

No hourly update from Giffgaff it seems. Is anyone still waiting for service to be restored?

Update 03:31

Seems it’s not completely over. An update from O2 dated 03:25:

We’re still working with our support teams and vendors trying to restore service as quickly as possible. We apologise for any inconvenience caused whiile this recovery work is in progress. We’ll provide further updates as soon as we can.

Update 04:02

Here’s the latest from Giffgaff Towers:

We don’t have any new information for you now but justto reassure you that the tecn team are working on this and that improvements are being seen in the community

Update 05:05

Another bleary-eyed hourly update has come out from Giffgaff:

Thank you for your patience during this issue and what must be a frustrating time. Just to assure you that the engineers are still working to resolve this as quickly as possible. We apologise for any inconvenience caused whiile this recovery work is in progress and we’ll provide further updates as soon as we can. If you have any questions please don’t hestitate to ask.

Update 05:50

O2 have published another pretty uninformative update on their status page. This is what it says:

05:40 – We’re continuing to drive the recovery of this issue with our support teams and vendors. We’re working to restore service as quickly as possible. Unfortunately we’re unable to confirm any timescales for service restoration at this time. We’ll let you know as soon as we can. We apologise once again for any inconvenience this incident is causing.

Update 05:50

YAWN! Here’s the current situation at Giffgaff:

The latest on this issue is that the engineers are still working furiously to resolve this as soon as possible. As we can see, a lot of members are now reporting being fixed which is great. We’ve no ETA at the moment for a final resolution but will update you when we receive any new information.

Please be assured that the tech team will work until this is resolved.

Update 07:02

Another Giffgaff update for you:

Just to pre-warn you, we are seeing the same side-effect as last night happening again this morning, whereby certain services are affected due to the high traffic and knock on effect on this issue. At this moment in time, I can’t tell you all the features that are effected, but slow top ups, goodybags, trouble logging in and time out issues on mygiffgaff may be experienced. Our Operations Manager is currently investigating and the Educator on morning shift will update you with more details after 8am.

Update 07:44

Plenty of people are reporting that they still have issues on the Giffgaff and O2 forums. If anything, they are saying it’s been going off and on all night but now it’s actually getting worse not better 🙁

Update 07:55

Many users are reporting that using just 2G can fix things. Does it work for you guys?

Update 08:15

Two quick back-to-back updates about the situation from the Giffgaff forums to share with you with confirmation about the 2G-only trick:

Many members are stating that if you turn your 3G off then your calls and texts will work better. I cannot officially confirm this, however it’s worth a try.

I am chasing for updates this morning, so I will get these out to you as soon as possible. Unfortunately we are still not able to give a timescale for this issue. Engineers are still working hard to get it fixed.

Sadly, it seems this problem is still not completely fixed over 19 hours later 🙁 We’ll keep the blog update running until everyone’s back up and running.

Update 08:31

The first tweets from O2 in about 10 hours are starting to roll in. If you’ve been staying up for them, you’ll be disappointed as they are more useless than ever. They are pretty much spamming @replies ignoring he actual question and saying some variation on “Our engineers are doing their level best to resolve the current network issue. Apologies for the inconvenience caused.” Gee, thanks guys – that’s soooo helpful!

If they are not doing that, they are dodging the question of compensation.

Update 08:36

Another misleadingly timestamped update from O2 status:

08:00 – We can confirm that our 2G network service has now been restored. Customers who were affected should now be able to make and receive calls.

Our 3G service is starting to restore and customers should expect to see a gradual return of data services as the day progresses.

Customers affected may wish to try switching their mobile phones off and on as service returns.

We are sorry again for the inconvenience this has caused some customers and can provide reassurance that we continue to deploy all possible resources, and will do so until full service is restored.

Update 08:46

Dunno who they’ve roped in this early in the morning but the O2 Twitter account is getting increasingly embarrassing:

They still don’t know the different between “bear” and “bare” and keep pushing the useless timescale “soon” when obviously that’s not what they’ve been letting their engineers get away with telling them all night long.

Update 08:48

Looks like we’re almost there! A 08:35 update from Giffgaff that was amended a few minutes later!

I can now confirm that 2G should now be working again and 3G is slowly coming back online. Text and calls should now be working and data should come back on line soon.

Please try resetting your handsets and being patient while remaining services come back online.

We’ll continue updating the blog as we hear more 🙂

Update 08:48

The outage has now made the FT and well as dozens more news agencies as it stretched into its second day. Also, for those wondering about compensation, O2 haven’t ruled it out yet although they are probably not obliged under Ofcom rules as they are unlikely to be technically in breach of contract. When BlackBerry users had a four day outage last October, all users got was £63 worth of crappy apps.

If you’re on Giffgaff, don’t hold out for anything either. They have a very poor record of compensating customers when they get things wrong. They did give a donation to charity for their last big cock up though.

Update 09:57

Official confirmation from O2 that 2G should be working fine. Turn off your 3G if you’re still having problems.

Wonder why it took them so much longer than Giffgaff…

Update 10:49

If you’re still having problems, O2 claim it should be sorted this afternoon. Thank goodness they finally managed to extract some sort of ETA from their engineers (no matter how vague). Is it really acceptable to have to wait over 20 hours before you get any indication of a timescale?

Update 11:00

An update from Giffgaff just came out implying that the issue is largely resolved:

Knock on issues:

Ports:
We are currently emailing members who have had their ports in and out affected by this issue. We are hoping to process a fix for these members later this afternoon, but we have no formal ETA for resolution.

Replacement SIMs:
These are pending, customers who request their handset to be barred will have their request accepted, but this will be queued until the service is available before it completes. Customers who have a SIM to replace their lost and stolen SIM will be able to add the 6 digit code, but again this will not be completed until the service is back up and the queue has gone through.

We are asking for a list of those affected so that we can send an email out to let affected customers know.

Update 11:40

Another update from Giffgaff clarifying that it’s only 2G services that are fine. Good on them for keeping everyone informed 🙂

We’ve seen that a number of you still have problems with voice services and believe that although the O2 2G network is running, the 3G network (which can also be used to connect the voice calls) is being restored during the day

So we are suggesting a couple of things
– if your phone supports choosing 2G rather than 3G, or turning off 3G then try this
– turning on and off will re-associate the phone to the network and may result in picking up elements of newly restored network or 2G for voice calls

We now have an ETA for the 3G data network. We fully expect it to be back up and running this afternoon.

Update 13:40

A promising update from O2:

Is anyone still having issues?

Update 14:16

More related to that last Giffgaff announcement.

We’ve just had confirmation that the 3G network is gradually being restored and we’re fully on schedule to get all 3G restored this afternoon.

All 3G should now be restored. Hurray! Please try a reboot of your handset to get it working.

We are now strongly advise members to not request to port in or out of the service for the time being as this will cause both the number that you are porting to and from to lose service while the system is down.

Update 14:25

And another quick post from Giffgaff HQ:

Knock on issues:

Replacement SIM activations:
We are fixing the issues pending with Sim Activations and this should be completed by 4pm. Please retry after 4pm and the activation code should work.

Update 14:51

Giffgaff have updated their blog post about the outage. Even though the post is by the company’s CEO, Mike Fairman, he’s not really giving out much information. Still nobody admitting that the issue was due to an error in the HLR servers or an explanation of the actual problem. And not exactly a reassuring apology – seems they are still more interested in palming off responsibility to O2 🙁

Update 15:00

Readers in London have let us know that the outage affected the bike hire scheme as the rental stations use the O2 network to process payment. A TfL spokesperson said that about 100 of the 570 hire stations were affected.

Update 15:33

The first proper apology from O2 almost 24 hours after the outage began:

Ronan, what took you so long? What have you been doing for the last 26 hours?

Update 15:37

O2/Telefónica UK CEO Ronan Dunne (see last update) has just appeared on Sky News to apologise. In an interview with poison-monger Kay Burley, he finally admitted that the issue was with the network’s HLR. He did not explain why this was the first time he’d mentioned it even though we first reported almost 16 hours ago and later expanded upon in a full explanation. He claimed to still not know the root cause of the problems with the HLR which led to the outage. Why weren’t we told this from O2 earlier?

Ronan Dunne also explained that the issue affected about a third of their customers which equates to about 8 million users – that’s a lot! He refused to make any firm statement on compensation. In an interview littered with clichés, he asked people to contact them about compensation and admitted that he was “embarrassed” by the outage.

Update 15:41

Hooray! Looks like we’re almost done here 🙂 Giffgaff have the following update:

All core phone services (calls, texts, 2G and 3G) should now be working. Please try resetting your handsets to get these working.

The ongoing issues are being fixed slowly. The throttling issue with unprovisioned SIMs explained in this service thread are solved so they are being fixed now. The will be able to be activated tomorrow

Ports and replacement SIMs work on the same systems so these will be addressed once the SIM issue has been fixed. We do not have an ETA for this, but we will be able to provide more information on it at 7pm today.

Update 17:16

Another note from the Giffgaff forums:

We’ve just been told that all work has completed and 3G services should now be restored for everyone.

Please reboot your handset now if you are still seeing 2G-only (making sure you’ve switched 3G back on of course – your phone may list it as “Mobile data” or WCDMA) and report back here if you’re still seeing this problem.

Listing what area you’re in may help too as there may be planned maintenance where you live.

Update 17:23

We’re glad everyone is back and their phones are fully connected. The latest from The Guardian seems to suggest that the problem is related to an upgrade designed to prepare the network for the increased volume of the 2012 London Olympics. It seems that the technicians in charge of the network-wide roll-out had failed to test it properly. Oh dear.

Earlier the BBC reported that a principal analyst at Ovum suspected that the HLR was at fault. It was also pointed out that for both 2G and 3G networks to lose service, two separate systems must have been affected. The CEO of Comtek also claimed that “O2 may have taken its eye off the ball – ignoring the maintenance of its existing service in the rush to deliver 4G“.

O2 still refuse to explain the problem to their paying customers.

Update 17:50

Sounds like O2 are ruling out actual monetary refunds in a vague way:

Update 20:45

Giffgaff have a bit more news:

A quick update from me to give you some information about port-ins and SIMs.

Some people are already seeing their port-ins complete, however this will not be the case for everyone. We are still looking at a time of tomorrow noon for these, but it’s very likely that they will be completed much earlier.

Please do not make new port-in or out requests as they may not process properly; we will update you when it’s advisable to do this.

If you reported your SIM lost or stolen either yesterday or today (11/07 & 12/07) then these will be dispatched tomorrow, so please factor in this as a potential delay if you’re waiting.

Thank you for your support, understanding, and continued patience.

Update 13th July – 08:24

Here’s what looks like the final update from Giffgaff:

I’ve just spoken with our CTO and everything should be resolved now: Core services, ports, unprovisioned SIMs etc. Anything that was affected by the outage.

If you experienced a loss of 3G data please try re-entering your internet resettings in your phone again and if that does not work please contact the agents. This should be treated as though it is an unrelated issue.

Update 13th July – 10:23

That’s it for us 🙂 It’s now time to wrap up the liveblog and get some well deserved rest (hopefully avoiding caffeine psychosis). Now everything seems sorted, we’ll leave you with this.

Continue Reading

Tags: , , , ,
« First« 73 74 75 76 77 78 79 »Last »