At about 4:30pm Eastern today (Sunday), we noticed an issue with BayRing Communications, the New Hampshire-based phone company where our NH data center infrastructure is domiciled.
The issue BayRing have told us is due to a street power outage impacting all of the former Pease Air Foce base.
As of this initial writing (5:20pm Sunday) BayRing is working to restore power to their core Internet routers.
For our Zimbra hosting farm clients, we maintain a secondary MX server in Denver, Colorado, unaffected by this outage, so no legitimate inbound email will be lost during this (nor any other) outage.
You can always check our network status page for realtime updates: http://status.reliablenetworks.com
UPDATE: 5:39pm Sunday – Service has been restored. As soon as we have a documented RFO, we will post it.
UPDATE: 8:32pm Sunday – BayRing has posted an update on their web site. We have been told by phone that Internet traffic from and to our data center at 77 Aviation drive is normally routed through another BayRing building at 11 Manchester, also at Pease. It was the 11 Manchester facility which experienced the power event. 77 Aviation traffic has been rerouted. Again, as soon as we have a documented RFO, we will post it.
UPDATE: 11:50am Monday – BayRing has had another power event starting around 10:45am. Currently one of the two power feeds to one of our racks is dark. From the logs it appears power from both of our BayRing power feeds (fed by separate BayRing UPSs) was lost, but only briefly on one of the feeds. Several of our cloud server hosts rebooted; several did not. SAN storage did not lose power. Since we do not have clarity from BayRing that the second power feed will not go down, we are for the moment keeping I/O-sensitive servers shut down to preserve data integrity. Servers whose applications tolerate what the backup vendors call “crash consistent” backups as safe are being brought back online.
UPDATE: 2:05pm Monday – This morning’s power interruption was caused by failover from generator power back to the UPSs. All but one of our racks now has full redundant power restored; that one rack has only one active power feed and BayRing are locating the break in power for us. As we have a fully redundant networking stack and all of our servers/SANs have redundant power supplies, we are fully operational and most client servers are back online. Many servers required disk checks on reboot. We have been told by BayRing they will produce an RFO tomorrow.
UPDATE: 8:30pm Monday – Full power has been restored to all of our racks and all systems are fully back on line. BayRing have reiterated they expect to issue an RFO on Tuesday and we will distribute that to clients directly as soon as we receive it. Thank you all for your patience as BayRing and we worked to recover from this power event.
L. Mark Stone
General Manager, Managed and Private/Hybrid Cloud Services
A Division of OTT Communications
The information provided in this blog is intended for informational and educational purposes only. The views expressed herein are those of Mr. Stone and do not necessarily reflect those of Reliable Networks, OTT Communications or Otelco Inc. The contents of this site are not intended as advice for any purpose and are subject to change without notice. We make no warranties of any kind regarding the accuracy or completeness of any information on this site, and we make no representations regarding whether such information is up-to-date or applicable to any particular situation. All copyrights are reserved by Mr. Stone. Any portion of the material on this site may be used for personal or educational purposes provided appropriate attribution is given to Mr. Stone and this blog.