All services

resolved

Description

We have received notification from our datacentre providers that they are currently experiencing a power issue and that the failover to generators has not happened as expected. This affects all 34SP.com services.

We will provide updates here as we receive them from our providers.


Updates

3rd June 2021, at 3:02PM

[resolved] At 13:40 on 19th May, mains power to our datacentre tripped. At this point, electrical load was picked up by the building's UPS systems, which are designed to provide power until generators start up. This only takes a couple of minutes but their battery capacity is sufficient for about 25 minutes of service. The generators started and 3 of the 4 switches that change between mains and generator power made the change. The fourth did not, and has been identified as the cause of the mains power trip due to a physical fault. As a result, any equipment powered through this switch lost power at 14:03.

At 14:09, with the cause of the trip identified, mains power was restored and our systems began booting up.

A few systems then required manual intervention to bring back into service, which we began immediately. The majority of these are by design to ensure that service is restored in a controlled fashion. Two servers did not come back up as expected and required on-site intervention to restore service, affecting around 100 customers. This was completed by 15:26 and full service for all customers was restored at that point.

Our datacentre provider's inspections have shown that the remaining 3 power switches are not vulnerable to the same fault and that it cannot repeat on the switch that failed.

We will be looking at opportunities to increase our reliability but at this point have not identified any.

21st May 2021, at 10:55AM

[resolved] We have received an update from our datacentre providers regarding this weeks loss of power. Their investigations continue, but it is currently believed that the fault lies in the equipment which moves electrical load from mains power to generator power, and which failed to do so when mains power failed on Wednesday. This equipment is currently being removed from service so that a full investigation can be conducted, with the final change to a replacement taking place on Monday. They assess that there is no risk during these works.

We expect to receive a further update next week once these investigations are complete and will post a final update at that time.

19th May 2021, at 4:49PM

[resolved] All services are now restored.

19th May 2021, at 3:31PM

[unresolved] The majority of services - including email - have now been restored. Systems continue to work through the startup of all hosted sites and we expect this to be complete shortly.

19th May 2021, at 2:27PM

[unresolved] Power appears to have been restored and we are working to bring services back online. We do not have any explanation for the fault yet.