Lounge Network New Zealand
Lounge Network News and Network Status


RSS Subscribe News & Network Status:
RSS Subscribe MAJOR INCIDENT REPORT AT THE ORCON DATA CENTRE
Posted on 26/06/12 04:05PM

Below is Orcons MIR for the unscheduled outage that occurred yesterday that knocked out our websites and also some of our clients. Please note that this was a Datacentre issue with our NZ providers Orcon.

WHEN DID IT START 25/06/2012 16:47

WHEN DID IT END 25/06/2012 18:33


WHAT HAPPENED?
Majority of services on Orcon housed servers were inaccessible

WHO DID IT AFFECT?
Orcon housed server (everything apart from cloud hosting).

HOW DID WE FIX IT?
Hard reset of Orcon core firewall (tyrant.orcon.net.nz).
Due to no out of band management access, we have called the Datacom remote hands to reboot the machine.

WHAT CAUSED IT?
Restarting the BGP daemon on tyrant.Orcon.net..nz caused it to kernel panic.


WAS THIS CAUSED BY WORK BEING UNDERTAKEN? IF UNSCHEDULED DESCRIBE WORK. IF SCHEDULED INCLUDE CR#.
The BGP was restarted to attempt to fix BGP issues on Orcon network. There are BGP inconsistencies that need to be addressed, as there are routing problems which started since early morning of 25/06/2012. The specific issue which resulted in restart of BGP on tyrant.orcon.net.nz (which is not resolved yet) is where the cloud firewall cannot access its database due to routing issue on one of the core routers.


WHAT ARE WE GOING TO DO TO STOP THIS HAPPENING IN THE FUTURE?
Review the need for BGP for local routing inside of Orcon network, and replace BGP with static routes for local network.
Retire Orcon infrastructure.




If you would like to receive server announcements and latest news, please subscribe to our new RSS Newsfeedss below. Most of the latest browsers such as Firefox , Internet Explorer and Opera now have inbuilt RSS newsfeed readers, as does most good email software.



All prices exclude GST unless otherwise stated. Errors and omissions excepted.
© Copyright Lounge Network, a division of Design Lounge Ltd 2001-2018