34SP.com System Status History

Description

Update at 13:31:25(BST): The new core switches have been acting as gateways & routing quite flawlessly for several days now.

At this time we will mark this status post as resolved, but any customers with issues they suspect are related to this work are encouraged to contact support via e-mail.

---

Update at 01:28:16(BST): The routing issue between sites (Manchester and London) was resolved at around midnight. This was not planned and a full investigation into the cause of the outage to those routes will be carried-out in due course to improve resilience and prevent the same fault from occurring in the future.

Further apologies are required for customers sitting behind the access switch that failed to take the changes required for the new core switches.

This was eventually diagnosed as a faulty switch and was swapped for another model. Because of this event, multiple Reseller, Personal & VPS hosting customers will have seen some extended downtime.

All issues do now appear to be resolved. Our monitoring systems are no longer reporting any outages and randomly-tested websites are working.

Now that this upgrade is complete, it is intended that any future upgrades will be much less disruptive, due to the improved network design. More flexibility and better performance are also key benefits to the new hardware.

Any customer still experiencing issues this morning should contact support to raise a ticket regarding this.

---

Update at 23:50:21(BST): We've noticed some further errors with the routing changes that are now affecting certain routes in/out of our network. We're working on this as a priority as it is affecting some customers in other areas.

---

Update at 23:44:40(BST): Network upgrades were completed by 23:33, with a small routing oversight that caused further delay until 23:42.

We have one access switch that is resisting re-configuration. We should have this resolved shortly.

---

Update at 22:46:22(BST): Beginning in approx. 15 minutes.

---

As a reminder to customers, the planned maintenance window for upgrades to our core network will be beginning at 23:00 (BST) and finishing at most 60 minutes later, 00:00.

Any further updates will be posted to this status page. It is hosted externally and will be unaffected by any outage.

Start Time

8th September 2011, at 9:48PM

Description

Update at 12:00:32(BST): The issue has now been resolved with this server.GlobalSign certificates are now being ordered as usual. Thank you for your continues patience during this time.


As per the GlobalSign official statement below:

On Sep 5th 2011 the individual/group previously confirmed to have hacked several Comodo resellers, claimed responsibility for the recent DigiNotar hack. In his message posted on Pastebin, he also referred to having access to 4 further high profile Certificate Authorities, and named GlobalSign as one of the 4.

GlobalSign takes this claim very seriously and is currently investigating. As a responsible CA, we have decided to temporarily cease issuance of all Certificates until the investigation is complete. We will post updates as frequently as possible.

We apologise for any inconvenience.

Start Time

7th September 2011, at 9:20AM

Description

We would like to apologise for the lack of phone support during the evening of September 6th. Phone service will resume again at 8am on September 7th

Start Time

6th September 2011, at 7:21PM

Description

Update at 14:48:36(BST): Update at 14:48:10(BST): The issue has now been resolved with this server.

---

The personal hosting server max.34sp.com is currently unavailable. We are investigating this and will post an update as soon as we have any more information to give.

Start Time

6th September 2011, at 2:33PM

Description

Update at 10:19:06(BST): The issue has now been resolved with this server.The 4PSA SpamGuardian and CleanServer systems have now been re-added to this server, Please let us know if you find any issues with the migrated domains.

---

Update at 11:43:32(BST): IPs have now been moved over, we are still doing some minor post migration work, but most sites should be back up and running.

If you are experiencing any issues with your site since the move please do get in contact with us.

---

Update at 11:09:57(BST): Ip reconfiguration to move the ips to the new server is now in progress, we expect around 30minutes from this point.

---

Update at 10:38:25(BST): All webservices are now going offline for a sync up of files and databases.

---

Update at 10:12:56(BST): All mailservice on server6 have now been stopped for a sync up of the email on the server.

---

The server6.xenserve.com is undergoing migration to new hardware today as per the email sent to all clients.

Try to avoid making any changes on any websites now to minimize loss of data when transferred to the new server.

We will update this post with progress throughout the day

Start Time

6th September 2011, at 8:11AM

Description

Update at 11:56:14(BST): Business5 has been operating happily for over a few hours now, with no further signs of disruption. Load levels appear normal.

If any customer is experiencing any lingering issues, please contact support.

---

Update at 10:39:10(BST): Business5 is now recovering; all services should be responding, but may be a little slow whilst users refresh their websites & inboxes.

We'll continue to monitor Business5 for a short while.

---

At present Business5 is experiencing a VERY high level of load across its services, which are being caused by certain malicious activities.

We're working to restore service as soon as possible. We appreciate your patience at this time.

Start Time

1st September 2011, at 10:22AM

Description

Update at 08:10:51(BST): Update at 19:14:18(BST): This migration was completed several hours ago, please do let us know if there are any issues with your sites from this move.

---

Update at 13:12:54(BST): All services are now stopped on the server for a final sync of the databases and files before we move the ips across.

---

Update at 12:32:18(BST):
Mail services have now been taken offline to resync the mail across to the new server.

---

FTP service has now been disabled on this server for an resync of files.

---

The server8.xenserve.com is undergoing migration to new hardware today as per the email sent to all clients.

Try to avoid making any changes on any websites now to minimize loss of data when transferred to the new server.

We will update this post with progress throughout the day

Start Time

31st August 2011, at 11:57AM

Description

Update at 10:24:11(BST): The issue affecting Globalsign and 34SP.com is now resolved.

---

Update at 11:47:51(BST): GlobalSign are aware of an issue with their verification systems at present. This is affecting Firefox users. Their twitter feed is here:

http://twitter.com/#!/globalsign/status/108839143517327360

This only affects logging into the 34SP.com control panel.

Start Time

31st August 2011, at 11:42AM

Description

Update at 10:00:59(BST): The issue has now been resolved with this server.This issue was resolved around an hour ago, an issue with the firewall setup within each domain had caused it to block all external connections, we have increased limits on the firewall connection tracking systems to avoid this issue re-occurring.

At the same time a kernel update was also applied to the server as well.

---

Sites hosted on businesshost1 are currently inaccessible. Engineers are working to resolve this issue.

Start Time

30th August 2011, at 8:19AM

Description

Update at 10:26:29(BST): The outage was caused by a "rare failure" in the UPS systems. Despite this hardware being designed with N+1 redundancy, the loss of two units resulted in a 20ms dip in power to our racks within that room.

We are confident that our providers have rectified this fault and will be working to mitigate any recurrence of this particular scenario in the future.

---

Update at 15:26:12(BST): All services should now have been returned for some time.

We're awaiting further explanation from the facilities provider as to the nature of the outage.

---

Only one floor of the Manchester facility (Synergy House) was affected.

We have confirmation from UKGrid of the failure (other ISPs/hosting providers have also experienced the same problem).

Most VPS' & hosting services are now running again as they should be. Customers with unmanaged Dedicated Servers/VPS' that have had a loss of power are advised to ensure the consistency of any databases they may have.

All shared hosting services have had their databases checked accordingly.

Any customers with lingering issues are urged to e-mail support as the telephone lines are currently very busy via support@34sp.com.

---

We have experienced a power failure to one of the cabinets in Manchester.

Affected machines have automatically restarted as the power was returned, but it will take a short while for virtual machines and some other services to come back online.

More information as we find out.

Start Time

24th August 2011, at 2:36PM