O2 outage fallout
Following our exhausting O2 outage liveblog last week we thought we’d summarise what happened and take a look at some of the fallout.
What happened?
About 1pm on Wednesday 11th July many O2 users lost signal. The blackout affected all services – 2G, 3G, data and texts. At the time, no explanation was given for the outage, the Network Status page crashed and it took a long time for O2 to acknowledge the problem. The outage seemed to affect people somewhat randomly – one person could be fine but another users in the same room using the same phone might have no service whatsoever. O2 were unable to provide any advice or a timeline for restoring service. In total, the outage lasted about 24 hours.
Who was affected?
O2 have about 23 million customers in the UK. The network estimated that about a third of customers were affected which equates to almost 8 million users. In addition, as they are virtual mobile networks that piggyback off the O2 signal, Giffgaff and Tesco Mobile customers were also affected. The issue caused signal loss to all customers regardless of whether they were business or retail customers or whether they were on PAYG or contract. Even celebrities such as Jemima Khan, Huw Edwards and Kym Marsh were frustrated as their phones were affected.
In addition, several vital services went down because of the outage. Some bike hire stations in London use O2 for processing payments. According to Transport for London, about 100 bike hire stations and 1000s of bikes were affected.
Even worse, troubled private contractor G4S is not only incompetent and morally corrupt but they use
O2 in their electronic tag systems for monitoring criminals on curfew. Unknown numbers of convicts on probation may have broken their curfews as the O2 outage compromised the systems that are used to prevent breaches. The Ministry of Justice admitted that about 250 offenders were affected by the outage.
When was everything fixed?
We kept everyone up-to-date as things progressed and monitored user forums to get the latest on people’s experiences getting signal back again. The first users reported a restoration of their mobile service at about half past midnight on the 12th July. This continued for several users over the next couple of hours.
However, many people still couldn’t get a connection and remained without signal well into the late morning. Just before 8am it became clear that :2g: services had mostly been restored and by turning off 3G, many users could get their service back. By this point, 19 hours after the initial problem, a significant proportion of users were back online. It took two more hours for O2 to officially declared that all 2G services (voice and data) were restored but 3G was still down. Even so, the blackout wasn’t completely over as 3G services were still down and a huge number of customers still couldn’t get their phones to work at all.
Just before 11am O2 gave their first estimate for a timescale on the fix. They said things should be back up and running by the afternoon (not very precise, we know). It wasn’t until 13:40 before O2 claimed that all 3G services were back up and running again. In total, the outage had lasted about 24 hours.
What did they do to fix it and keep people informed?
It’s not very clear what was required to fix the problem as O2 have been far from open. They also haven’t given any explanation for why it took so long. What we do know is that O2 had hundreds of engineers working through the night to fix the problem.
In fact, their communication throughout was very poor. It took a long time to acknowledge the problem in the first place, the Twitter account didn’t update sometimes for 10 hours at a time and the status page was inaccessible for hours at the time. When they did release updates, they were labelled with incorrect timestamps to mislead people into thinking they had been posted earlier, questions about compensation were routinely ignored and it took over 20 hours for an estimate for how long things would take to fix to be announced. In the meantime, the outage was extensively covered by all the major media outlets.
Even though the outage was eventually rectified, it took an entire day. Perhaps the worst aspect of the customer communication was the fact that at no point was the fault explained and it took until about 3pm on the following day for any apology to be issued.
Giffgaff were much better than O2 with almost hourly updates posted on their user forums. However, they also wouldn’t give any timescale for a fix or attempt offer an explanation of what went wrong. Users also had to wait until 10am the day after the outage began for a proper apology.
What caused it?
O2 have still refused to be anything but extremely vague and what went wrong, where and why. All they would say is that there was a network issue. Giffgaff went somewhat further by saying that the problem was “an issue within part of our core network that is preventing some mobile phones from successfully connecting.”
The actual issue was a problem with a part of the network called the Home Location Register (HLR). This allows users to connect to the network by consulting something called a Centralized User Database (CUDB). We don’t know exactly what the problem was but a issue with the CUDB and HLR prevented some users from making and maintaining a connection with the cell towers as they could not be authenticated correctly. This explains why the outage wasn’t localised at all and affected all types of user. The fact that two people in the same room on the same handset could be affected differently also makes sense with this explanation.
Stay tuned as we’re planning a post in the next few days going into a bit more detail about what caused the outage.
Can I get any compensation?
O2 (and Giffgaff) were initially incredibly coy about the subject of compensation. At the very least, millions of people have lost service that they have paid for and the forums are awash with people who were left stranded or lost £1000s worth of work due to the outage. However, the networks will probably do their best to weasel out of their responsibility toward their users and during the outage neither network responded to press enquiries or confirmed they will accept liability and pay compensation.
Early on Giffgaff said they had “no information on compensation” and further queries were ignored. Previous issues with Giffgaff outages and instances of overcharging have resulted in Giffgaff avoiding paying out any compensation at all. All they have done in the past is make a small donation to charity instead of refunding any of their customers. However, after the outage, a blog post admitted that they are discussing and considering “providing minutes, money off or similar as an apology for the outage”.
O2 have been incredibly careful to avoid saying they will pay any compensation and their terms and conditions have been cleverly written so they are absolved of any responsibility. Ofcom rules probably won’t force they to pay anything out and they haven’t ruled out reimbursing any customers. Ronan Dunne, O2‘s UK CEO wouldn’t make any statement about compensation but did hint that there might be some sort of non-monetary gift to affected customers.
In the end, both Giffgaff and O2 decided against offering proper compensation. Instead, Giffgaff and O2 PAYG customers will get 10% extra credit on a top-up. O2 customers can top-up any time in September (the distant date is perhaps a cynical attempt to keep customers on the network for another month?) while Giffgaff customers have only from the 19th until the 25th of July. If you are an O2 contract customer, you’ll get 10% off your July bill instead. This equates to just a few quid. Both the O2 discounts only apply to those affected but all O2 customers will also get a £10 voucher to spend in-store.
While this is better than nothing, many people will be furious that a pound here or there does nothing to make up for the inconvenience and damages caused by the avoidable outage. Furthermore, it will probably be pointed out that offering bonus credit doesn’t exactly cost the networks much as they get heavily-discounted commercial rates and that products in O2 stores are already heavily overpriced.
Even so, don’t let this discourage you – Which? has encourage the network to do the right thing. In any case, if you feel you deserve proper compensation, ask your network and take it to the Ombudsman if necessary. If you want to get out of an O2 contract, you might also be able to use this outage to your benefit if you feel that O2 have broken its terms.
Questions?
Does that sum things up for you? Were you affected by the outage? How has it affected your opinion of your network? And is there anything else you’d like to know from us?
