23 April 2013 ~ 56 Comments

Giffgaff outage liveblog

News

We’re getting reports that Giffgaff is suffering a partial network outage. It’s not yet clear whether O2 and Tesco Mobile are also affected. We’ll update from across various sources as the situation progresses.

2G and 3G calls – grn_pix.jpg Available
2G and 3G texts – grn_pix.jpg Available
2G and 3G data – grn_pix.jpg Available
Emergency calls – grn_pix.jpg Available

Update 22:53

It’s 100% official now:

That’s it for us, then. Thanks for reading. We’ll be back soon with an in-depth blog post on exactly what went wrong and how come it took so long to fix as soon as we get confirmation from Giffgaff. We’re not holding out hope but if there’s any form of compensation or goodwill gesture offered, we’ll be sure to have the full lowdown too.

For now: Goodnight and sleep well, safe in the knowledge that your Giffgaff phone is working again 🙂

Update 22:44

Thanks to everyone who came here to read updates and left comments or tweets with their experience. While you’re here, please check out our articles and especially our comprehensive comparison table of all PAYG mobile networks. We’d also be really grateful if you could rate the networks we’ve reviewed (especially this one), subscribe to our newsletter or consider retweeeting some of our articles or linking to some of our pages.

Official Update 22:37

Just wanted to add to Will T’s update. Thank you all for remaining patient with us while we resolved this. I know it was very frustrating to lose your service and I can assure you a full review is now going on into how this can be avoided. I am sure there will be some future updates around this.

Update 22:36

Woohoo! What a relief! After almost four hours we finally managed to make our first call with our Giffgaff SIM. Over on Twitter it seems that service is back up and running. Thanks for coming by to follow our updates.

Official Update 22:35

Good news – we’re back. Calls, texts and data all appear to be working well.

Thanks so much for following along over the last few hours, everyone. We’ll be keeping an eye on the service for a little while to make sure it’s stable – if you’re not receiving service please make sure that you reset your phone and try again.

Official Update 22:25

Good news – after some sterling work by the engineers, we’re back. Calls, texts and data all appear to be working well.

Thanks so much for following along over the last few hours, everyone. We’ll be keeping an eye on the service for a little while to make sure it’s stable – if you’re not receiving service please make sure that you reset your phone and try again.

Update 22:24

At last one other news source has picked up this story. Over at TechEYE.net they have a quickly-written article about the downtime.

Update 22:23

We’re getting some reports that calls are working any. Can any of you confirm?

Official Update 22:22

Thanks so much to everyone who’s reporting their data service status – please keep it coming. Make sure that you’re restarting your handset before trying again as this will force a new connection to the network and give the most up to date result.

Official Update 22:10

We’ve been working hard to bring service back up and we believe we may have restored data service for those of you that were experiencing difficulty. Would those of you that were having trouble please ensure your settings are correct, restart your handset, and confirm whether this is the case?

Update 22:05

So we’ve hit the 3-hour mark now and there’s still no end in sight. However, there is some reason to be cheerful. At least it seems that mobile data is back up and running for most people. And finally it appears that Giffgaff have identified the issue with their network. They just don’t know how long it will take to fix.

Hopefully we’ll have some more positive updates to bring you over the next hour or so…

Official Update 21:53

Just a quick one – the exact issue with the hardware that failed has been identified and the engineers are working in earnest to correct it. Thanks again and we’ll be updating once we have further updates.

Update 21:52

It appears data is back where we are. Anyone else?

Update 21:49

For those still worried about rumours that the network will be down until 30th April, Giffgaff have officially denied this twice now. However, they have yet to give any other ETA for a fix nor any indication whether it’s more likely to be minutes, hours, days or weeks…

Official Update 21:44

Thanks again for your patience. We’re talking with the tech team and it looks like they’re making good progress on this. We’ll update with more as soon as we’re able.

Update 21:41

Still no mention of the outage from any major blog or news source – seems we’re the only ones coving it. Even though Giffgaff has got 100,000s of users now, it’s not yet deemed important enough to write about.

Anyone want to take bets for how long this will last in total?

Update 21:37

If you’re bored with waiting to get your network service back, here’s some things to while away the time:

Got anything funny or interesting you want to share with people while we wait for a more substantial update from Giffgaff?

Update 21:34

Well that was useless…

So it seems we don’t really know anything other than nobody can say how long this outage will last and that it’s something to do with the voice call hardware that is, for some reason, affecting texts and data too.

On the plus side we have got Twitter updates working again even if Giffgaff aren’t providing much info on their Twitter account:

Official Update 21:30

We have no further updates at this time. Unfortunately we don’t have an ETA for a fix yet. As soon as we do we will let you all know.

Official Update 21:29

All we know at the moment is, it was a problem with piece of hardware that manages voice calls. Any other information we find out will be updated on the thread.

Official Update 21:27

Thanks again so much for the patience you’re showing with us on this. We know how important it is to be connected and we’re working as hard as we can to make sure we can bring the system back online as soon as possible.

Update 21:18

We haven’t been able to get data to actually work at all. And nobody on Twitter seems to either. Is data working for anyone at all? Is it really “intermittent”?

Official Update 21:15

Thanks for your comments about your varying degress [sic] of data service – I’ve added this as intermittent at the top of the thread as it’s a more accurate representation of the service currently.

I also want to assure you that we’re still working hard to make sure that this is resolved as soon as we can.

Update 21:13

We’re getting unconfirmed reports that Tesco Mobile is unaffected (although O2 is definitely fine. Anyone with a Tesco Mobile SIM able to verify this for us?

Update 21:11

It’s just past 9pm and the issue has been going on for over two hours now. The community thread just hit 100 pages! Still no sign as to when it will be fixed though 🙁

Lots of people saying that data definitely is not working in opposition to Giffgaff staffer t_will. Only a few are saying it is still working.

Official Update 21:01

zobia22 just responded to a post where someone claimed that the outage will continue until 30th April:

We have no idea how long calls and texts will be down for. As soon as we do have a time frame or any more information, we will update this thread.

Update 20:57

So Giffgaff are saying that data should still be working. No luck here though. What about you guys?

And anyone else managed to receive any texts in the last couple of hours?

Official Update 20:54

I’ve just heard from the tech team and they’re looking at a couple of different options right now. The issue appears to be with a piece of hardware that manages voice calls, which we’ll be fixing shortly.

After investigation, the reports that data is affected appear to be incorrect, so I’ve marked this as available. If you’re having problems with your data allowance please make sure that your settings are correct and that you have an active allowance.

Official Update 20:50

We are updating the thread as soon as we get any information. Once we have more details on what is causing the issue, we will update the thread.

Update 20:50

Could the end be nigh?! We just got a text into our text Giffgaff phone. It was sent at 20:48 and arrived by 20:49. If texts can be received maybe they can be sent too? We’re trying our best to test this hypothesis out… Let us know if you have any luck too…

Update 20:47

Some black humour in this difficult time 🙂

Official Update 20:42

Seems they are trying to deflect the blame. But no hard information about what exactly is causing the fault.

I just want to jump in to reassure you all we are doing everything within our powers to get this resolved as soon as possible. We are working with our third parties and we will update as and when we know more.

I am very sorry for the inconvenience this is causing you.

Update 20:39

An update on the data issue. We’ve been in contact with several other Giffgaff users as well as running our own tests – it seems that the phone will still say it’s connected to data but won’t be able to load any pages or send/receive any data. So data seems to be down everywhere now according to what we’re hearing. Is it working for any of you?

Official Update 20:35

I’ve noticed a few people concerned about this, so wanted to make clear that emergency calls to 999 should still work while we’re experiencing this difficulty, so if you are experiencing an emergency that would require it you should still be able to get assistance.

Update 20:32

Giffgaff users are getting ncreasingly irate. This is from hogweed on the forums:

Well time for the giffgaff boss to resign. All his plans to make sure this didn’t happen again have ben [sic] shown to be incompetent lies.

It is fair to mention that Giffgaff have had more than their fair share of outages. Usually it’s been O2‘s fault but at the moment it seems this problem rests squarely on their shoulders.

The forum thread is now up to 42 pages and is being updated faster than we can read it….

Official Update 20:31

Just wanted to thank you for all your comments so far. We know how much people rely on their mobile phones and we’re working as quickly and as well as possible to ensure that service is restored as soon as we’re able.

Update 20:29

The good news is that the Giffgaff community forum seems to be standing up to the strain at the moment. The bad news is that there’s still no word on the cause or duration of this outage. Even worse, we haven’t had our dinner yet 🙁

Official Update 20:20

It appears that this issue may also affect data service intermittently. I’ll update the title to reflect this, so if you see anyone posting about losing their data service please direct them to this thread.

We’re still working on the issue and will update you once we have plans for a resolution.

So there you have it, data is officially affected too.

Update 20:19

The Giffgaff community thread (when you can get it to load) is now up to 20 pages…

User therosey1973 (I mean “member”) just reported that she couldn’t get a taxi so had to walk 3 miles home…

Please help spread the word by linking to this page if you have a website or by tweeting a link so that people can keep up-to-date with the outage.

Update 20:17

We’re getting reports that data is now down too. Our own tests are not yet conclusive but we strangely got redirected to the URL http://tablet.o2.co.uk/ when trying to visit Google. We suspect data will be fully down soon too but still have no idea what is causing the outage. Giffgaff haven’t given any clues at all. What experience are you getting?

Update 20:16

No updates of any substance from Giffgaff recently. We still can’t access the community post and their Twitter account is just trying to reassure indiviudals rather than discuss the problem and its solution.

Update 20:13

Sorry if your comment isn’t approved immediately – they are coming in quicker than we can publish them at the moment! Don’t stop though…

Update 20:11

Can anyone positively confirm that O2 and Tesco Mobile are unaffected by this issue?

Update 20:10

Seems tweets aren’t embedding properly. We’re working on fixing it but for the time being just visit the URLs.

Update 20:07

Giffgaff have officially confirmed that the outage is nationwide.

They have also said that they are unable/unwilling to give an estimate for how long the downtime will last.

Update 20:05

Our tests and comments from users seem to show that data is still working. But what about the speeds? Are you seeing any difference compared to what you usually get?

Update 19:59

Giffgaff just confirmed that calls and texts are down. Are we to infer from this that data is still fine. Does yours work? Let us know…

Update 19:53

Update from Giffgaff:

We’re really hoping they will be able to give an estimate for the restoration of service soon.

Update 19:49

We just ran some tests and it’s still definitely not working. Even triggering Airplane mode and back again or doing a hard reboot of the phone does resolve the issue.

Update 19:46

The Giffgaff community is struggling under the strain of all the customers without service. It keeps going down making it harder for people to get information. Our server is taking a beating too but we’ll do our best to keep it going for you guys…

Update 19:43

Just rejigged this post so that the latest updates are at the top and so you don’t need to keep scrolling down to check if anything new has been posted 🙂

Update 19:40

Offical word from Giffgaff:

Update 19:37

Nothing new to report from Giffgaff yet unfortunately. But from what people have been tweeting us and commenting here, this definitely seems to be nationwide. Reports coming in of the network being down from all over the UK. Keep them coming so we can see if there are any improvements anywhere over the course of this issues…

Update 19:30

Just a friendly reminder – please help us keep track of the outage by commenting below with your location and which services are currently unavailable. Cheers 🙂

Update 19:29

There are currently a torrent of tweets about the outage. Please refer people to this post for the latest updates and use the hashtag #giffgaffoutage to keep track of the problem.

Update 19:25

If you are on Giffgaff and your network is currently down, please comment below with your location and the services that aren’t working. We’re currently trying to get a picture of whether this outage is nationwide or localised. We have already had emails from people in Sussex, West Mids, Glasgow and London about their phones not working.

Update 19:22

Some forum threads from users about this issue: 1, 2, 3, 4.

But still nothing on the official Service Updates board.

Update 19:19

Twitter is currently awash with reports that people can neither text nor call. There’s still no official word from Giffgaff acknowledging the outage.

Continue Reading

Tags: , ,

11 July 2012 ~ 16 Comments

O2 outage liveblog

News

We’re getting reports that O2 is suffering a partial network outage. Giffgaff and Tesco Mobile are also supposedly affected although our staff are currently not having any problems. We’ll update from across various sources as the situation progresses.

Update 17:01

Giffgaff have confirmed they are also affected by the outage:

We are currently experiencing an outage for voice, text and data (2G and 3G). We are investigating the issue and hope to have it up and running as soon as possible.

All users will experience intermittent voice and data for the duration. This will only affect active SIMs, so this won’t stop new SIMs from activating.

Update 17:15

O2 have just tweeted the following:

Update 17:25

Seems the issue has been going since at least 01:30pm according to the O2 network status checker:

We continue to manage the problem on our network that is affecting some of our customers. Customers may still be having difficulty making and receiving calls, sending texts or using data.We are sorry for any inconvenience. Our engineers are dealing with the problem as a priority and we hope to restore full service as soon as possible

Update 18:01

The issue does not seem to be localised as we are getting reports of service loss from across the country.

Giffgaff have made another official update:

Update 19:10

Hi guys, at the moment there is no new news, but the tech team are working to resolve this as quickly as possible. It will not affect ports or sim activations.

Update 20:03

Another update from the Giffgaff forums:

We have no new news at the moment and no ETA, but please be assured that the tech team are working to resolve this as fast as possible. As soon as we have any new information I will update this thread.

Update 21:02

Another totally useless update from Giffgaff. No news from O2 in the last 4 hours.

Hi guys, unfortunately we’ve heard no new news from the tech team but are assured they are working hard to resolve this as quickly as possible.

Update 21:17

Another tweet from O2. Still not very enlightening 🙁

https://twitter.com/O2/status/223147999000408064

[edit: 21:30 – the link to this tweet is now broken as O2 have deleted it.]

Update 21:21

This has hit all the major UK news networks. A round-up of some of the stories:
BBC – O2 mobile users suffer network problems across UK
The Guardian – O2 network crash leaves mobile users without service
‎The Telegraph – O2 customers suffer nationwide network failure
The Register – O2, GiffGaff network goes titsup for unlucky punters
The Verge – O2 experiencing major outage in the UK affecting data, calls, and texts
Daily Mail – O2 hit by nationwide network failure that left ‘millions of users’ unable to make calls or text this afternoon

Update 21:27

O2 just helpfully made an exact copy of their last tweet 🙁

Update 21:35

http://status.o2.co.uk/ now appears to be down.

Update 21:37

The Guardian are now reporting that the O2 webpage has crashed.

Update 21:57

The outage thread on the Giffgaff community forums has now hit 136 pages – that’s over 1360 posts. There’s still no real information forthcoming unfortunately.

Update 22:02

Some people on Giffgaff forums are saying only O2 should be to blame but others say they are equally as responsible as it’s not out their control that they use O2 who aren’t exactly known to be very reliable. What do you think?

Update 22:03

Another hastily-written non-update from Hazel on the Giffgaff forums:

Good Evening, unfortunately we;ve received no news from the tech team working on this issue, but we do know that they are working to resolve this as quickly as possible and that they will work until it fixed,

Update 22:13

No more news yet. Instead, a shameless plug: if you’ve come here to follow this liveblog, you might want to visit our super mobile price comparison table to weigh up your other options. We gratefully receive Facebook likes and retweets too 🙂

Update 22:15

Are you on O2, Tesco Mobile or Giffgaff? Have you been affecting? Are things getting better or worse? We want to know! Please respond in the comments below.

Update 22:19

It appears the problem is with SIM cards registering with O2 masts. Once you’re connected, you should keep your signal okay there are just issues with that initial connection. This explains why one phone might be okay but another in the same city/room might have no signal.

Update 22:24

Can anyone get http://status.o2.co.uk/ to resolve right now?

Update 22:27

People are getting a little frustrated on the Giffgaff forums. “theboy” posted the following:

GG Educators – can you stop posting rather pointless replies.

By now, O2 would have at least a Plan A, B and C in progress to resolve this. Can you find out what they are, and post something useful?

That would be more encouraging that the rather mindless “we haven’t heard anything, but we know they are working on it”

They are certainly being more communicative than O2 or Tesco but that isn’t saying much.

Update 22:30

O2 are on BBC News on TV – they are saying that there is a problem with their core network processes. Sounds serious.

Update 22:35

An O2 spokesperson says that at least “thousands” of users are affected. Judging by the outrage on forums and Twitter, it would seem as though that’s a low estimate given that there are about 23 million O2 customers in the UK. It’s even affecting Jersey right now.

Update 22:45

Apparently O2 think it’s “harsh” that their customers are saying they suck regarding this outage.

Some might say it’s harsh that they still haven’t explained what’s causing it, what they are doing to fix it and when they expect it to be sorted let alone that an unknown number of their customers are completely without service.

Update 22:55

Just noticed that O2‘s Live Status Checker is back up (for now at least). They added an update timestamped 19:00 saying that they “continue to manage the problem on our network that is affecting some of our customers”. Still no clarification of how they are “managing” it or how long they will “manage” it for.

Update 23:02

Another hourly update from Giffgaff. Unfortunately it doesn’t say much about the outage.

Due to a huge increase in traffic on our website tonight we are experiencing some extra delays with top-ups, purchasing goodybags and timeout errors for logging in and ‘mygiffgaff’. These issues will subside by 23:30.

The issue is not based on geography. Some members still have connections at the same location as others who do not. The problem is an issue within part of our core network that is preventing some mobile phones from successfully connecting.

The tech team are working hard to resolve this as quickly as possible.

Update 23:10

O2 claim to have 200 engineers dedicated to network performance. They don’t mention how many are working right now or whether any of them have any news about the outage.

Update 23:20

O2 status update datestamped 23:00:

Our engineers and vendorscontinuing [sic] to deal with the problem as a priority and we hope to restore full service as soon as possible

Update 23:39

A Giffgaff user writes the following:

The fault is that some customers’ phones cannot authenticate because the HLR (these hold the account details) is unreachable.

We can’t comment on the accuracy of this information but it seems reasonable. There’s definitely some problem within the O2 backbone it seems.

Update 23:45

Doesn’t look like there will be any news for a while so take a gander at our favourite tweets about the outage so far – what are yours?


https://twitter.com/1OMIG/status/223184574363082755

Update 23:50

Can anyone confirm whether switching on and off again fixes the problem?

Update 23:54

http://status.o2.co.uk/ seems down again here. Anyone else?

Update 23:59

It seems that it does work intermittently but seems to be experiencing symptoms of a DDoS. Not great!

Update 00:00

Midnight update! Still no solution on the horizon. There are now over 92 news articles about this issue. And thanks to Hayley who confirmed that switching it on and off won’t help. Hopefully switching it on and off won’t break phones that are currently working either then.

Update 00:01

Giffgaff’s most recent update from hardworking Hazel:

The delay issues with goodybags and top ups, logging in and timeout errors has now stabilised and cleared up.

The latest on this issue is that the engineers are continuing to work throughout the night to resolve a fix.

Update 00:13

If your phone is currently working, turning it off and on again shouldn’t cause you to lose signal so don’t worry about that.

Update 00:15

We’re hearing more and more rumours that the cause of the issue is a problem with O2‘s HLR (Home Location Register). This is a network-wide system that deals with connecting subscribers to cell towers. The HLR is usually made up of several components including a gateway/frontend, backend servers as well as the database they get their information from.

It’s not yet clear where exactly the current problem lies or what’s causing it but O2 documents confirm that the HLR includes information on supplementary services, authentication credentials and subscriber profile information such as details about APN (Access Point Names) which provide data access. For example, when you try to use the internet on your phone, O2 checks with the HLR to see whether you have permission to use the APN which provides web access. If the HLR is not working properly you won’t be able to get online.

Orange very recently had an issue with their new HLR set-up in France which took over 12 hours to resolve. Vodafone have also had issues in the past. O2 (nor any of the MVNOs that use it) have yet to officially confirm that the HLR is causing the issue.

Update 00:36

Anthony from Giffgaff says:

We have no imformation on compensation currently as all efforts are being put towards resolving this issue primarily. We are waiting for further information at the moment so please keep checking back here for updates. Apologies fo any incovenience caused. [sic]

Update 00:37

Another quick-sharp update from Giffgaff. Might it almost be sorted?

Crossing fingers, but I’ve seen ten members in the last 15 minutes say their services has come back.

Update 00:44

We’re getting more and more reports that service is now being restored. If yours is now fixed, please let us know.

Update 00:58

The latest official Giffgaff update bodes well 🙂

No official update from the engineers as of yet but we’re seeing more and more reports of service returning for members. I’ll update you as soon as we get any information,

Update 01:05

Here’s the latest O2 update. No mention of an imminent fix yet.

We can confirm that the problem with our mobile service is due to a fault with one of our network systems, which has meant some mobile phone numbers are not registering correctly on our network. As a result, some customers are having difficulty making or receiving calls, sending texts or using data. We, and our central supplier, have deployed all possible resources and are working through the night to restore service as soon as possible. We apologise again to customers affected and will provide further updates as soon as we can

Update 01:59

Looks like the end is nigh at last 🙂 Here’s what Giffgaff have to say:

Thank you for your continuing patience through-out the night. The engineers are aware of the cause and are working hard to resolve this as quickly as possible. We’re witnessing more and more member having their services returned. Thank you again for your understanding and sorry for any inconvinence

Update 03:15

No hourly update from Giffgaff it seems. Is anyone still waiting for service to be restored?

Update 03:31

Seems it’s not completely over. An update from O2 dated 03:25:

We’re still working with our support teams and vendors trying to restore service as quickly as possible. We apologise for any inconvenience caused whiile this recovery work is in progress. We’ll provide further updates as soon as we can.

Update 04:02

Here’s the latest from Giffgaff Towers:

We don’t have any new information for you now but justto reassure you that the tecn team are working on this and that improvements are being seen in the community

Update 05:05

Another bleary-eyed hourly update has come out from Giffgaff:

Thank you for your patience during this issue and what must be a frustrating time. Just to assure you that the engineers are still working to resolve this as quickly as possible. We apologise for any inconvenience caused whiile this recovery work is in progress and we’ll provide further updates as soon as we can. If you have any questions please don’t hestitate to ask.

Update 05:50

O2 have published another pretty uninformative update on their status page. This is what it says:

05:40 – We’re continuing to drive the recovery of this issue with our support teams and vendors. We’re working to restore service as quickly as possible. Unfortunately we’re unable to confirm any timescales for service restoration at this time. We’ll let you know as soon as we can. We apologise once again for any inconvenience this incident is causing.

Update 05:50

YAWN! Here’s the current situation at Giffgaff:

The latest on this issue is that the engineers are still working furiously to resolve this as soon as possible. As we can see, a lot of members are now reporting being fixed which is great. We’ve no ETA at the moment for a final resolution but will update you when we receive any new information.

Please be assured that the tech team will work until this is resolved.

Update 07:02

Another Giffgaff update for you:

Just to pre-warn you, we are seeing the same side-effect as last night happening again this morning, whereby certain services are affected due to the high traffic and knock on effect on this issue. At this moment in time, I can’t tell you all the features that are effected, but slow top ups, goodybags, trouble logging in and time out issues on mygiffgaff may be experienced. Our Operations Manager is currently investigating and the Educator on morning shift will update you with more details after 8am.

Update 07:44

Plenty of people are reporting that they still have issues on the Giffgaff and O2 forums. If anything, they are saying it’s been going off and on all night but now it’s actually getting worse not better 🙁

Update 07:55

Many users are reporting that using just 2G can fix things. Does it work for you guys?

Update 08:15

Two quick back-to-back updates about the situation from the Giffgaff forums to share with you with confirmation about the 2G-only trick:

Many members are stating that if you turn your 3G off then your calls and texts will work better. I cannot officially confirm this, however it’s worth a try.

I am chasing for updates this morning, so I will get these out to you as soon as possible. Unfortunately we are still not able to give a timescale for this issue. Engineers are still working hard to get it fixed.

Sadly, it seems this problem is still not completely fixed over 19 hours later 🙁 We’ll keep the blog update running until everyone’s back up and running.

Update 08:31

The first tweets from O2 in about 10 hours are starting to roll in. If you’ve been staying up for them, you’ll be disappointed as they are more useless than ever. They are pretty much spamming @replies ignoring he actual question and saying some variation on “Our engineers are doing their level best to resolve the current network issue. Apologies for the inconvenience caused.” Gee, thanks guys – that’s soooo helpful!

If they are not doing that, they are dodging the question of compensation.

Update 08:36

Another misleadingly timestamped update from O2 status:

08:00 – We can confirm that our 2G network service has now been restored. Customers who were affected should now be able to make and receive calls.

Our 3G service is starting to restore and customers should expect to see a gradual return of data services as the day progresses.

Customers affected may wish to try switching their mobile phones off and on as service returns.

We are sorry again for the inconvenience this has caused some customers and can provide reassurance that we continue to deploy all possible resources, and will do so until full service is restored.

Update 08:46

Dunno who they’ve roped in this early in the morning but the O2 Twitter account is getting increasingly embarrassing:

They still don’t know the different between “bear” and “bare” and keep pushing the useless timescale “soon” when obviously that’s not what they’ve been letting their engineers get away with telling them all night long.

Update 08:48

Looks like we’re almost there! A 08:35 update from Giffgaff that was amended a few minutes later!

I can now confirm that 2G should now be working again and 3G is slowly coming back online. Text and calls should now be working and data should come back on line soon.

Please try resetting your handsets and being patient while remaining services come back online.

We’ll continue updating the blog as we hear more 🙂

Update 08:48

The outage has now made the FT and well as dozens more news agencies as it stretched into its second day. Also, for those wondering about compensation, O2 haven’t ruled it out yet although they are probably not obliged under Ofcom rules as they are unlikely to be technically in breach of contract. When BlackBerry users had a four day outage last October, all users got was £63 worth of crappy apps.

If you’re on Giffgaff, don’t hold out for anything either. They have a very poor record of compensating customers when they get things wrong. They did give a donation to charity for their last big cock up though.

Update 09:57

Official confirmation from O2 that 2G should be working fine. Turn off your 3G if you’re still having problems.

Wonder why it took them so much longer than Giffgaff…

Update 10:49

If you’re still having problems, O2 claim it should be sorted this afternoon. Thank goodness they finally managed to extract some sort of ETA from their engineers (no matter how vague). Is it really acceptable to have to wait over 20 hours before you get any indication of a timescale?

Update 11:00

An update from Giffgaff just came out implying that the issue is largely resolved:

Knock on issues:

Ports:
We are currently emailing members who have had their ports in and out affected by this issue. We are hoping to process a fix for these members later this afternoon, but we have no formal ETA for resolution.

Replacement SIMs:
These are pending, customers who request their handset to be barred will have their request accepted, but this will be queued until the service is available before it completes. Customers who have a SIM to replace their lost and stolen SIM will be able to add the 6 digit code, but again this will not be completed until the service is back up and the queue has gone through.

We are asking for a list of those affected so that we can send an email out to let affected customers know.

Update 11:40

Another update from Giffgaff clarifying that it’s only 2G services that are fine. Good on them for keeping everyone informed 🙂

We’ve seen that a number of you still have problems with voice services and believe that although the O2 2G network is running, the 3G network (which can also be used to connect the voice calls) is being restored during the day

So we are suggesting a couple of things
– if your phone supports choosing 2G rather than 3G, or turning off 3G then try this
– turning on and off will re-associate the phone to the network and may result in picking up elements of newly restored network or 2G for voice calls

We now have an ETA for the 3G data network. We fully expect it to be back up and running this afternoon.

Update 13:40

A promising update from O2:

Is anyone still having issues?

Update 14:16

More related to that last Giffgaff announcement.

We’ve just had confirmation that the 3G network is gradually being restored and we’re fully on schedule to get all 3G restored this afternoon.

All 3G should now be restored. Hurray! Please try a reboot of your handset to get it working.

We are now strongly advise members to not request to port in or out of the service for the time being as this will cause both the number that you are porting to and from to lose service while the system is down.

Update 14:25

And another quick post from Giffgaff HQ:

Knock on issues:

Replacement SIM activations:
We are fixing the issues pending with Sim Activations and this should be completed by 4pm. Please retry after 4pm and the activation code should work.

Update 14:51

Giffgaff have updated their blog post about the outage. Even though the post is by the company’s CEO, Mike Fairman, he’s not really giving out much information. Still nobody admitting that the issue was due to an error in the HLR servers or an explanation of the actual problem. And not exactly a reassuring apology – seems they are still more interested in palming off responsibility to O2 🙁

Update 15:00

Readers in London have let us know that the outage affected the bike hire scheme as the rental stations use the O2 network to process payment. A TfL spokesperson said that about 100 of the 570 hire stations were affected.

Update 15:33

The first proper apology from O2 almost 24 hours after the outage began:

Ronan, what took you so long? What have you been doing for the last 26 hours?

Update 15:37

O2/Telefónica UK CEO Ronan Dunne (see last update) has just appeared on Sky News to apologise. In an interview with poison-monger Kay Burley, he finally admitted that the issue was with the network’s HLR. He did not explain why this was the first time he’d mentioned it even though we first reported almost 16 hours ago and later expanded upon in a full explanation. He claimed to still not know the root cause of the problems with the HLR which led to the outage. Why weren’t we told this from O2 earlier?

Ronan Dunne also explained that the issue affected about a third of their customers which equates to about 8 million users – that’s a lot! He refused to make any firm statement on compensation. In an interview littered with clichés, he asked people to contact them about compensation and admitted that he was “embarrassed” by the outage.

Update 15:41

Hooray! Looks like we’re almost done here 🙂 Giffgaff have the following update:

All core phone services (calls, texts, 2G and 3G) should now be working. Please try resetting your handsets to get these working.

The ongoing issues are being fixed slowly. The throttling issue with unprovisioned SIMs explained in this service thread are solved so they are being fixed now. The will be able to be activated tomorrow

Ports and replacement SIMs work on the same systems so these will be addressed once the SIM issue has been fixed. We do not have an ETA for this, but we will be able to provide more information on it at 7pm today.

Update 17:16

Another note from the Giffgaff forums:

We’ve just been told that all work has completed and 3G services should now be restored for everyone.

Please reboot your handset now if you are still seeing 2G-only (making sure you’ve switched 3G back on of course – your phone may list it as “Mobile data” or WCDMA) and report back here if you’re still seeing this problem.

Listing what area you’re in may help too as there may be planned maintenance where you live.

Update 17:23

We’re glad everyone is back and their phones are fully connected. The latest from The Guardian seems to suggest that the problem is related to an upgrade designed to prepare the network for the increased volume of the 2012 London Olympics. It seems that the technicians in charge of the network-wide roll-out had failed to test it properly. Oh dear.

Earlier the BBC reported that a principal analyst at Ovum suspected that the HLR was at fault. It was also pointed out that for both 2G and 3G networks to lose service, two separate systems must have been affected. The CEO of Comtek also claimed that “O2 may have taken its eye off the ball – ignoring the maintenance of its existing service in the rush to deliver 4G“.

O2 still refuse to explain the problem to their paying customers.

Update 17:50

Sounds like O2 are ruling out actual monetary refunds in a vague way:

Update 20:45

Giffgaff have a bit more news:

A quick update from me to give you some information about port-ins and SIMs.

Some people are already seeing their port-ins complete, however this will not be the case for everyone. We are still looking at a time of tomorrow noon for these, but it’s very likely that they will be completed much earlier.

Please do not make new port-in or out requests as they may not process properly; we will update you when it’s advisable to do this.

If you reported your SIM lost or stolen either yesterday or today (11/07 & 12/07) then these will be dispatched tomorrow, so please factor in this as a potential delay if you’re waiting.

Thank you for your support, understanding, and continued patience.

Update 13th July – 08:24

Here’s what looks like the final update from Giffgaff:

I’ve just spoken with our CTO and everything should be resolved now: Core services, ports, unprovisioned SIMs etc. Anything that was affected by the outage.

If you experienced a loss of 3G data please try re-entering your internet resettings in your phone again and if that does not work please contact the agents. This should be treated as though it is an unrelated issue.

Update 13th July – 10:23

That’s it for us 🙂 It’s now time to wrap up the liveblog and get some well deserved rest (hopefully avoiding caffeine psychosis). Now everything seems sorted, we’ll leave you with this.

Continue Reading

Tags: , , , ,