Mobile Network
Comparison

Use this comparison site to pick the best and cheapest mobile phone network in the UK

11 July 2012 • 16 Comments by Jon M

O2 outage liveblog

News

We’re getting reports that O2 is suffering a partial network outage. Giffgaff and Tesco Mobile are also supposedly affected although our staff are currently not having any problems. We’ll update from across various sources as the situation progresses.

Update 17:01

Giffgaff have confirmed they are also affected by the outage:

We are currently experiencing an outage for voice, text and data (2G and 3G). We are investigating the issue and hope to have it up and running as soon as possible.

All users will experience intermittent voice and data for the duration. This will only affect active SIMs, so this won’t stop new SIMs from activating.

Update 17:15

O2 have just tweeted the following:

Update 17:25

Seems the issue has been going since at least 01:30pm according to the O2 network status checker:

We continue to manage the problem on our network that is affecting some of our customers. Customers may still be having difficulty making and receiving calls, sending texts or using data.We are sorry for any inconvenience. Our engineers are dealing with the problem as a priority and we hope to restore full service as soon as possible

Update 18:01

The issue does not seem to be localised as we are getting reports of service loss from across the country.

Giffgaff have made another official update:

Update 19:10

Hi guys, at the moment there is no new news, but the tech team are working to resolve this as quickly as possible. It will not affect ports or sim activations.

Update 20:03

Another update from the Giffgaff forums:

We have no new news at the moment and no ETA, but please be assured that the tech team are working to resolve this as fast as possible. As soon as we have any new information I will update this thread.

Update 21:02

Another totally useless update from Giffgaff. No news from O2 in the last 4 hours.

Hi guys, unfortunately we’ve heard no new news from the tech team but are assured they are working hard to resolve this as quickly as possible.

Update 21:17

Another tweet from O2. Still not very enlightening 🙁

https://twitter.com/O2/status/223147999000408064

[edit: 21:30 – the link to this tweet is now broken as O2 have deleted it.]

Update 21:21

This has hit all the major UK news networks. A round-up of some of the stories:
BBC – O2 mobile users suffer network problems across UK
The Guardian – O2 network crash leaves mobile users without service
‎The Telegraph – O2 customers suffer nationwide network failure
The Register – O2, GiffGaff network goes titsup for unlucky punters
The Verge – O2 experiencing major outage in the UK affecting data, calls, and texts
Daily Mail – O2 hit by nationwide network failure that left ‘millions of users’ unable to make calls or text this afternoon

Update 21:27

O2 just helpfully made an exact copy of their last tweet 🙁

Update 21:35

http://status.o2.co.uk/ now appears to be down.

Update 21:37

The Guardian are now reporting that the O2 webpage has crashed.

Update 21:57

The outage thread on the Giffgaff community forums has now hit 136 pages – that’s over 1360 posts. There’s still no real information forthcoming unfortunately.

Update 22:02

Some people on Giffgaff forums are saying only O2 should be to blame but others say they are equally as responsible as it’s not out their control that they use O2 who aren’t exactly known to be very reliable. What do you think?

Update 22:03

Another hastily-written non-update from Hazel on the Giffgaff forums:

Good Evening, unfortunately we;ve received no news from the tech team working on this issue, but we do know that they are working to resolve this as quickly as possible and that they will work until it fixed,

Update 22:13

No more news yet. Instead, a shameless plug: if you’ve come here to follow this liveblog, you might want to visit our super mobile price comparison table to weigh up your other options. We gratefully receive Facebook likes and retweets too 🙂

Update 22:15

Are you on O2, Tesco Mobile or Giffgaff? Have you been affecting? Are things getting better or worse? We want to know! Please respond in the comments below.

Update 22:19

It appears the problem is with SIM cards registering with O2 masts. Once you’re connected, you should keep your signal okay there are just issues with that initial connection. This explains why one phone might be okay but another in the same city/room might have no signal.

Update 22:24

Can anyone get http://status.o2.co.uk/ to resolve right now?

Update 22:27

People are getting a little frustrated on the Giffgaff forums. “theboy” posted the following:

GG Educators – can you stop posting rather pointless replies.

By now, O2 would have at least a Plan A, B and C in progress to resolve this. Can you find out what they are, and post something useful?

That would be more encouraging that the rather mindless “we haven’t heard anything, but we know they are working on it”

They are certainly being more communicative than O2 or Tesco but that isn’t saying much.

Update 22:30

O2 are on BBC News on TV – they are saying that there is a problem with their core network processes. Sounds serious.

Update 22:35

An O2 spokesperson says that at least “thousands” of users are affected. Judging by the outrage on forums and Twitter, it would seem as though that’s a low estimate given that there are about 23 million O2 customers in the UK. It’s even affecting Jersey right now.

Update 22:45

Apparently O2 think it’s “harsh” that their customers are saying they suck regarding this outage.

Some might say it’s harsh that they still haven’t explained what’s causing it, what they are doing to fix it and when they expect it to be sorted let alone that an unknown number of their customers are completely without service.

Update 22:55

Just noticed that O2‘s Live Status Checker is back up (for now at least). They added an update timestamped 19:00 saying that they “continue to manage the problem on our network that is affecting some of our customers”. Still no clarification of how they are “managing” it or how long they will “manage” it for.

Update 23:02

Another hourly update from Giffgaff. Unfortunately it doesn’t say much about the outage.

Due to a huge increase in traffic on our website tonight we are experiencing some extra delays with top-ups, purchasing goodybags and timeout errors for logging in and ‘mygiffgaff’. These issues will subside by 23:30.

The issue is not based on geography. Some members still have connections at the same location as others who do not. The problem is an issue within part of our core network that is preventing some mobile phones from successfully connecting.

The tech team are working hard to resolve this as quickly as possible.

Update 23:10

O2 claim to have 200 engineers dedicated to network performance. They don’t mention how many are working right now or whether any of them have any news about the outage.

Update 23:20

O2 status update datestamped 23:00:

Our engineers and vendorscontinuing [sic] to deal with the problem as a priority and we hope to restore full service as soon as possible

Update 23:39

A Giffgaff user writes the following:

The fault is that some customers’ phones cannot authenticate because the HLR (these hold the account details) is unreachable.

We can’t comment on the accuracy of this information but it seems reasonable. There’s definitely some problem within the O2 backbone it seems.

Update 23:45

Doesn’t look like there will be any news for a while so take a gander at our favourite tweets about the outage so far – what are yours?


https://twitter.com/1OMIG/status/223184574363082755

Update 23:50

Can anyone confirm whether switching on and off again fixes the problem?

Update 23:54

http://status.o2.co.uk/ seems down again here. Anyone else?

Update 23:59

It seems that it does work intermittently but seems to be experiencing symptoms of a DDoS. Not great!

Update 00:00

Midnight update! Still no solution on the horizon. There are now over 92 news articles about this issue. And thanks to Hayley who confirmed that switching it on and off won’t help. Hopefully switching it on and off won’t break phones that are currently working either then.

Update 00:01

Giffgaff’s most recent update from hardworking Hazel:

The delay issues with goodybags and top ups, logging in and timeout errors has now stabilised and cleared up.

The latest on this issue is that the engineers are continuing to work throughout the night to resolve a fix.

Update 00:13

If your phone is currently working, turning it off and on again shouldn’t cause you to lose signal so don’t worry about that.

Update 00:15

We’re hearing more and more rumours that the cause of the issue is a problem with O2‘s HLR (Home Location Register). This is a network-wide system that deals with connecting subscribers to cell towers. The HLR is usually made up of several components including a gateway/frontend, backend servers as well as the database they get their information from.

It’s not yet clear where exactly the current problem lies or what’s causing it but O2 documents confirm that the HLR includes information on supplementary services, authentication credentials and subscriber profile information such as details about APN (Access Point Names) which provide data access. For example, when you try to use the internet on your phone, O2 checks with the HLR to see whether you have permission to use the APN which provides web access. If the HLR is not working properly you won’t be able to get online.

Orange very recently had an issue with their new HLR set-up in France which took over 12 hours to resolve. Vodafone have also had issues in the past. O2 (nor any of the MVNOs that use it) have yet to officially confirm that the HLR is causing the issue.

Update 00:36

Anthony from Giffgaff says:

We have no imformation on compensation currently as all efforts are being put towards resolving this issue primarily. We are waiting for further information at the moment so please keep checking back here for updates. Apologies fo any incovenience caused. [sic]

Update 00:37

Another quick-sharp update from Giffgaff. Might it almost be sorted?

Crossing fingers, but I’ve seen ten members in the last 15 minutes say their services has come back.

Update 00:44

We’re getting more and more reports that service is now being restored. If yours is now fixed, please let us know.

Update 00:58

The latest official Giffgaff update bodes well 🙂

No official update from the engineers as of yet but we’re seeing more and more reports of service returning for members. I’ll update you as soon as we get any information,

Update 01:05

Here’s the latest O2 update. No mention of an imminent fix yet.

We can confirm that the problem with our mobile service is due to a fault with one of our network systems, which has meant some mobile phone numbers are not registering correctly on our network. As a result, some customers are having difficulty making or receiving calls, sending texts or using data. We, and our central supplier, have deployed all possible resources and are working through the night to restore service as soon as possible. We apologise again to customers affected and will provide further updates as soon as we can

Update 01:59

Looks like the end is nigh at last 🙂 Here’s what Giffgaff have to say:

Thank you for your continuing patience through-out the night. The engineers are aware of the cause and are working hard to resolve this as quickly as possible. We’re witnessing more and more member having their services returned. Thank you again for your understanding and sorry for any inconvinence

Update 03:15

No hourly update from Giffgaff it seems. Is anyone still waiting for service to be restored?

Update 03:31

Seems it’s not completely over. An update from O2 dated 03:25:

We’re still working with our support teams and vendors trying to restore service as quickly as possible. We apologise for any inconvenience caused whiile this recovery work is in progress. We’ll provide further updates as soon as we can.

Update 04:02

Here’s the latest from Giffgaff Towers:

We don’t have any new information for you now but justto reassure you that the tecn team are working on this and that improvements are being seen in the community

Update 05:05

Another bleary-eyed hourly update has come out from Giffgaff:

Thank you for your patience during this issue and what must be a frustrating time. Just to assure you that the engineers are still working to resolve this as quickly as possible. We apologise for any inconvenience caused whiile this recovery work is in progress and we’ll provide further updates as soon as we can. If you have any questions please don’t hestitate to ask.

Update 05:50

O2 have published another pretty uninformative update on their status page. This is what it says:

05:40 – We’re continuing to drive the recovery of this issue with our support teams and vendors. We’re working to restore service as quickly as possible. Unfortunately we’re unable to confirm any timescales for service restoration at this time. We’ll let you know as soon as we can. We apologise once again for any inconvenience this incident is causing.

Update 05:50

YAWN! Here’s the current situation at Giffgaff:

The latest on this issue is that the engineers are still working furiously to resolve this as soon as possible. As we can see, a lot of members are now reporting being fixed which is great. We’ve no ETA at the moment for a final resolution but will update you when we receive any new information.

Please be assured that the tech team will work until this is resolved.

Update 07:02

Another Giffgaff update for you:

Just to pre-warn you, we are seeing the same side-effect as last night happening again this morning, whereby certain services are affected due to the high traffic and knock on effect on this issue. At this moment in time, I can’t tell you all the features that are effected, but slow top ups, goodybags, trouble logging in and time out issues on mygiffgaff may be experienced. Our Operations Manager is currently investigating and the Educator on morning shift will update you with more details after 8am.

Update 07:44

Plenty of people are reporting that they still have issues on the Giffgaff and O2 forums. If anything, they are saying it’s been going off and on all night but now it’s actually getting worse not better 🙁

Update 07:55

Many users are reporting that using just 2G can fix things. Does it work for you guys?

Update 08:15

Two quick back-to-back updates about the situation from the Giffgaff forums to share with you with confirmation about the 2G-only trick:

Many members are stating that if you turn your 3G off then your calls and texts will work better. I cannot officially confirm this, however it’s worth a try.

I am chasing for updates this morning, so I will get these out to you as soon as possible. Unfortunately we are still not able to give a timescale for this issue. Engineers are still working hard to get it fixed.

Sadly, it seems this problem is still not completely fixed over 19 hours later 🙁 We’ll keep the blog update running until everyone’s back up and running.

Update 08:31

The first tweets from O2 in about 10 hours are starting to roll in. If you’ve been staying up for them, you’ll be disappointed as they are more useless than ever. They are pretty much spamming @replies ignoring he actual question and saying some variation on “Our engineers are doing their level best to resolve the current network issue. Apologies for the inconvenience caused.” Gee, thanks guys – that’s soooo helpful!

If they are not doing that, they are dodging the question of compensation.

Update 08:36

Another misleadingly timestamped update from O2 status:

08:00 – We can confirm that our 2G network service has now been restored. Customers who were affected should now be able to make and receive calls.

Our 3G service is starting to restore and customers should expect to see a gradual return of data services as the day progresses.

Customers affected may wish to try switching their mobile phones off and on as service returns.

We are sorry again for the inconvenience this has caused some customers and can provide reassurance that we continue to deploy all possible resources, and will do so until full service is restored.

Update 08:46

Dunno who they’ve roped in this early in the morning but the O2 Twitter account is getting increasingly embarrassing:

They still don’t know the different between “bear” and “bare” and keep pushing the useless timescale “soon” when obviously that’s not what they’ve been letting their engineers get away with telling them all night long.

Update 08:48

Looks like we’re almost there! A 08:35 update from Giffgaff that was amended a few minutes later!

I can now confirm that 2G should now be working again and 3G is slowly coming back online. Text and calls should now be working and data should come back on line soon.

Please try resetting your handsets and being patient while remaining services come back online.

We’ll continue updating the blog as we hear more 🙂

Update 08:48

The outage has now made the FT and well as dozens more news agencies as it stretched into its second day. Also, for those wondering about compensation, O2 haven’t ruled it out yet although they are probably not obliged under Ofcom rules as they are unlikely to be technically in breach of contract. When BlackBerry users had a four day outage last October, all users got was £63 worth of crappy apps.

If you’re on Giffgaff, don’t hold out for anything either. They have a very poor record of compensating customers when they get things wrong. They did give a donation to charity for their last big cock up though.

Update 09:57

Official confirmation from O2 that 2G should be working fine. Turn off your 3G if you’re still having problems.

Wonder why it took them so much longer than Giffgaff…

Update 10:49

If you’re still having problems, O2 claim it should be sorted this afternoon. Thank goodness they finally managed to extract some sort of ETA from their engineers (no matter how vague). Is it really acceptable to have to wait over 20 hours before you get any indication of a timescale?

Update 11:00

An update from Giffgaff just came out implying that the issue is largely resolved:

Knock on issues:

Ports:
We are currently emailing members who have had their ports in and out affected by this issue. We are hoping to process a fix for these members later this afternoon, but we have no formal ETA for resolution.

Replacement SIMs:
These are pending, customers who request their handset to be barred will have their request accepted, but this will be queued until the service is available before it completes. Customers who have a SIM to replace their lost and stolen SIM will be able to add the 6 digit code, but again this will not be completed until the service is back up and the queue has gone through.

We are asking for a list of those affected so that we can send an email out to let affected customers know.

Update 11:40

Another update from Giffgaff clarifying that it’s only 2G services that are fine. Good on them for keeping everyone informed 🙂

We’ve seen that a number of you still have problems with voice services and believe that although the O2 2G network is running, the 3G network (which can also be used to connect the voice calls) is being restored during the day

So we are suggesting a couple of things
– if your phone supports choosing 2G rather than 3G, or turning off 3G then try this
– turning on and off will re-associate the phone to the network and may result in picking up elements of newly restored network or 2G for voice calls

We now have an ETA for the 3G data network. We fully expect it to be back up and running this afternoon.

Update 13:40

A promising update from O2:

Is anyone still having issues?

Update 14:16

More related to that last Giffgaff announcement.

We’ve just had confirmation that the 3G network is gradually being restored and we’re fully on schedule to get all 3G restored this afternoon.

All 3G should now be restored. Hurray! Please try a reboot of your handset to get it working.

We are now strongly advise members to not request to port in or out of the service for the time being as this will cause both the number that you are porting to and from to lose service while the system is down.

Update 14:25

And another quick post from Giffgaff HQ:

Knock on issues:

Replacement SIM activations:
We are fixing the issues pending with Sim Activations and this should be completed by 4pm. Please retry after 4pm and the activation code should work.

Update 14:51

Giffgaff have updated their blog post about the outage. Even though the post is by the company’s CEO, Mike Fairman, he’s not really giving out much information. Still nobody admitting that the issue was due to an error in the HLR servers or an explanation of the actual problem. And not exactly a reassuring apology – seems they are still more interested in palming off responsibility to O2 🙁

Update 15:00

Readers in London have let us know that the outage affected the bike hire scheme as the rental stations use the O2 network to process payment. A TfL spokesperson said that about 100 of the 570 hire stations were affected.

Update 15:33

The first proper apology from O2 almost 24 hours after the outage began:

Ronan, what took you so long? What have you been doing for the last 26 hours?

Update 15:37

O2/Telefónica UK CEO Ronan Dunne (see last update) has just appeared on Sky News to apologise. In an interview with poison-monger Kay Burley, he finally admitted that the issue was with the network’s HLR. He did not explain why this was the first time he’d mentioned it even though we first reported almost 16 hours ago and later expanded upon in a full explanation. He claimed to still not know the root cause of the problems with the HLR which led to the outage. Why weren’t we told this from O2 earlier?

Ronan Dunne also explained that the issue affected about a third of their customers which equates to about 8 million users – that’s a lot! He refused to make any firm statement on compensation. In an interview littered with clichés, he asked people to contact them about compensation and admitted that he was “embarrassed” by the outage.

Update 15:41

Hooray! Looks like we’re almost done here 🙂 Giffgaff have the following update:

All core phone services (calls, texts, 2G and 3G) should now be working. Please try resetting your handsets to get these working.

The ongoing issues are being fixed slowly. The throttling issue with unprovisioned SIMs explained in this service thread are solved so they are being fixed now. The will be able to be activated tomorrow

Ports and replacement SIMs work on the same systems so these will be addressed once the SIM issue has been fixed. We do not have an ETA for this, but we will be able to provide more information on it at 7pm today.

Update 17:16

Another note from the Giffgaff forums:

We’ve just been told that all work has completed and 3G services should now be restored for everyone.

Please reboot your handset now if you are still seeing 2G-only (making sure you’ve switched 3G back on of course – your phone may list it as “Mobile data” or WCDMA) and report back here if you’re still seeing this problem.

Listing what area you’re in may help too as there may be planned maintenance where you live.

Update 17:23

We’re glad everyone is back and their phones are fully connected. The latest from The Guardian seems to suggest that the problem is related to an upgrade designed to prepare the network for the increased volume of the 2012 London Olympics. It seems that the technicians in charge of the network-wide roll-out had failed to test it properly. Oh dear.

Earlier the BBC reported that a principal analyst at Ovum suspected that the HLR was at fault. It was also pointed out that for both 2G and 3G networks to lose service, two separate systems must have been affected. The CEO of Comtek also claimed that “O2 may have taken its eye off the ball – ignoring the maintenance of its existing service in the rush to deliver 4G“.

O2 still refuse to explain the problem to their paying customers.

Update 17:50

Sounds like O2 are ruling out actual monetary refunds in a vague way:

Update 20:45

Giffgaff have a bit more news:

A quick update from me to give you some information about port-ins and SIMs.

Some people are already seeing their port-ins complete, however this will not be the case for everyone. We are still looking at a time of tomorrow noon for these, but it’s very likely that they will be completed much earlier.

Please do not make new port-in or out requests as they may not process properly; we will update you when it’s advisable to do this.

If you reported your SIM lost or stolen either yesterday or today (11/07 & 12/07) then these will be dispatched tomorrow, so please factor in this as a potential delay if you’re waiting.

Thank you for your support, understanding, and continued patience.

Update 13th July – 08:24

Here’s what looks like the final update from Giffgaff:

I’ve just spoken with our CTO and everything should be resolved now: Core services, ports, unprovisioned SIMs etc. Anything that was affected by the outage.

If you experienced a loss of 3G data please try re-entering your internet resettings in your phone again and if that does not work please contact the agents. This should be treated as though it is an unrelated issue.

Update 13th July – 10:23

That’s it for us 🙂 It’s now time to wrap up the liveblog and get some well deserved rest (hopefully avoiding caffeine psychosis). Now everything seems sorted, we’ll leave you with this.

Tags: , , , ,

16 Responses to “O2 outage liveblog”