More Information on Blackberry outage...

 

Following my earlier blog post this morning - Business and trade press including Bloomberg, WNBC and PC Advisor have reported all day on the disruption caused to millions of BlackBerry users when a system failure disabled push email for eleven hours last night. According to RIM officials, all halted e mails will now be sent on in stages so as not to overload devices.

BlackBerry Service Disrupted; No Timeframe for Restoration

Gwen Ackerman, Bloomberg, April 18, 2007

https://www.bloomberg.com/apps/news?pid=20601082&sid=a_r1Lp6zBi_0&refer=canada

April 18 (Bloomberg) -- Research In Motion Ltd.'s BlackBerry service experienced disruptions in North America and other parts of the world, affecting an unspecified number of the company's 8 million subscribers. The company hasn't provided a timeframe for when the service might be restored.

``Please be advised that we are currently experiencing a service interruption that is causing delays in sending and receiving messages,'' said a recording on a customer service number in Vancouver, Canada. The company ``will provide updates as soon as they become available.'' Officials at the Waterloo, Ontario-based company could not be reached for comment.

The disruption may be limited to North America, with some problems in other parts of the world for customers who receive their e-mails through North American servers. Mobile-phone companies in Asia and Europe said they haven't experienced problems. Research In Motion had about 8 million subscribers at the end of the fiscal fourth quarter ended in February.

Before today, shares of Research In Motion had gained 2.7 percent in Nasdaq Stock Market trading.

`Paralyzing'

General Motors Corp. spokesman Chris Preus, who is based in Detroit, was one of several U.S.-based participants on a press trip in China who had intermittent or blacked out BlackBerry coverage today.

The BlackBerry service is ``the only way I have to communicate,'' said Preuss, who was in Liuzhou, China, for a press trip of a General Motors joint venture factory. ``When I lose it, it's paralyzing.''

Blackberry users helpless after outage

John Blau, PC Advisor, April 18, 2007

https://www.pcadvisor.co.uk/news/index.cfm?newsid=9040

Research In Motion's Blackberry service down

The BlackBerry wireless email service from Research In Motion has suffered a widespread outage starting in the US last night.

Customers on the BlackBerry Forums discussion board complained of having no service, while callers to the BlackBerry US technical support line were still greeted with the following message early Wednesday morning in the US: "We are currently experiencing a service interruption that is causing delays in sending or receiving messages. We apologise for the inconvenience and will provide updates as soon as they become available."

New York television news channel NewsChannel4 reported on Tuesday night that the problem affected "all users in the Western hemisphere." Postings to BlackBerry Forums indicated that the problem may be limited to North America, however.

"Officials with RIM said they are trying to reset the system and told NewsChannel4 that they are concerned that the backlog of data, which will rush through when it comes back on line, could cause a bigger problem," the news station reported on its website.

RIM officials advised people who use Blackberry as a major way of communications to make back-up plans, the TV station reported.

A trouble ticket from T-Mobile said customers in all markets were experiencing "intermittent issues when sending and receiving e-mail from BlackBerry devices," according to one of the forum postings. It said the problem affected all markets and all geographies.

RIM didn't immediately return calls for comment.

NewsChannel4 Exclusive: Massive System Failure Affecting Blackberry Users

WNBC.com, April 18, 2007

https://www.wnbc.com/news/12339359/detail.html?taf=ny

NEW YORK -- NewsChannel 4 learned of a massive system failure that affected all blackberry users in the western hemisphere late Tuesday.

The RIM Company, which stands for Research In Motion, developed blackberry technology and said its infrastructure failed around 8 p.m. Tuesday and was until about 7 a.m. Wenesday.

E-mails were not being pushed to portable blackberry devices.

Officials with RIM said they tried to reset the system and they were concerned that the backlog of data, which could cause a bigger problem as it rushes through now that the system appears to be online.

RIM officials said messages would be sent out in stages so the system does not crash.

RIM officials recommend all who depend on their blackberry as a major way of communication should make some back-up plans in case more problems occur Wednesday morning.

For the latest on this issue, stay with NewsChannel 4 and WNBC.com.

Now this isn't the first time the RIM NOC has been down but prior incidents have just affected Blackberry users on particular operator networks. From what I've managed to pull together.  

• The failure covered the entire North American region, as indicated by RIM officials in media stories. The main NOC in Waterloo appears to have been the problem.
• The failure started about 8PM EST last night, and the outage was fixed "around 4am". There is still a backlog of traffic running through the NOC.  
• RIM **does** have failover built into its NOC, so if a Waterloo outage occurs, traffic should be able to run through the UK NOC. Obviously this didn't happenRIM admits that obviously this feature did not work.   

RIM play very heavily on the supposed benefits of a NOC based model introducing reliability however these are obviously completely eroded when you lose a service for such a long time and impact so many people.