34SP.com System Status History

Description

Update at 09:36:50(BST): Phone lines are now open again.

---

Update at 08:51:04(BST): Due to the rioting in Manchester, it is going to be particularly difficult for staff members to reach the 34SP.com office.

Given the short notice, it is not possible to provide an acceptable notice for telephone support via remote workers.

With this in mind, please be advised that telephone support will not be available throughout Wednesday the 10th of August, until further notice.

All customers are urged to contact the support team via the usual e-mail address: support@34sp.com

Start Time

9th August 2011, at 11:36PM

Description

Update at 23:33:44(BST): Prohost12 now appears to be running acceptably once again, outage marked as resolved.

---

Update at 11:04:35(BST): The affected website has now been re-located on to fresh hardware. Cached requests via Prohost12 are being redirected. At this time, other customers utilising Prohost12 should not have any further difficulties with their websites.

---

Update at 10:38:06(BST): We are in the process of moving the website in question another server, with more capacity.

Unfortunately, as this publicity was not anticipated by the owner of the website, we will have significant DNS caching issues to endure.

DNS records have already been updated and all efforts will be made to minimise the effects of the caching.

---

Following last night's rioting in the UK, one particularly popular website has been hosted on our basic shared hosting service and this is completely flattening the server, Prohost12.

Of course this service was never designed for such site loads and we have had no choice but to disable the site to ensure that others sharing this service are able to use their websites.

We will be contacting the owner of this website shortly to arrange for a better hosting solution.

34SP.com apologises for any inconvenience caused by this surge in traffic.

Start Time

9th August 2011, at 9:19AM

Description

Update at 23:34:39(BST): No lasting issues have been experienced in the week since the maintenance, therefore this outage is now marked as resolved.

---

Update at 00:27:42(BST): All maintenance has now been completed successfully.

Any customers that believe they are experiencing any sort of un-planned or extended outage are encouraged to contact support@34sp.com as soon as possible.

---

Update at 23:51:45(BST): Switches 10 & 11 completed.

Switches 12, 13 & 15 will be upgraded shortly.

---

Update at 23:32:30(BST): Switches 8 & 9 are now done.

Switches 10 & 11 will be upgraded shortly.

---

Update at 23:11:50(BST): VPS1 is now back up and running.

The first of the access switch upgrades will be starting shortly.

---

Update at 22:44:14(BST): VPS1 will be going down shortly.

---

Update at 22:14:42(BST): The reseller server xenserve11 has now been re-located and is back up and running successfully.

At 22:45, the VPS host VPS1 will be downed to have its RAM replaced.

---

Update at 21:57:13(BST): In approximately 3 minutes time, xenserve11 (server11.xenserve.com) will be shutdown, re-located and rebooted.

Incidentally this is the first time that the machine has been offline in 825 days! Almost a shame to have to move it.

---

Customers are advised that scheduled maintenance tasks will be starting shortly:

22:00 - xenserve11 re-location

22:45 - VPS1 RAM replacement

23:15 - Access switch firmware upgrades

Updates will be posted to reflect current events.

Start Time

2nd August 2011, at 9:40PM

Description

Update at 16:22:01(BST): The issue has now been resolved with this server.This matter was resolved around 2.30PM

---

Ted is experiencing a serious level of load at present, which is causing a denial of service.

Attempts to recover the server remotely have failed thus far. We'll try rebooting the machine now.

Start Time

2nd August 2011, at 2:22PM

Description

Update at 16:33:11(BST): This migration is now being marked as finished, if however anyone is having any issues since the migration please do contact us and we will aid in looking into the problems for you.

---

Update at 16:22:18(BST): This migration has now been completed, we are doing some small amount of maintenance work, but do let us know if there are any issues on any sites.

---

Update at 15:16:28(BST): The main webservices will now be going offline for the final stage of this migration, we expect the downtime to be around 1-2hrs

---

Update at 14:43:38(BST): FTP services are now stopped for a initial sync of files.

---

Update at 14:19:14(BST): Mail services have been stopped on this server for the syncing of the mail

---

The deepthought.34sp.com domains are undergoing migration to a different server as per the email sent to all clients.

Try to avoid making any changes on any websites now to minimize loss of data when transferred to the new server.

We will update this post with progress throughout the day.

Start Time

2nd August 2011, at 8:07AM

Description

Update at 14:21:50(BST): We haven't seen a resurgence of this issue lately and no specific issue was identified since the last episode.

---

Update at 12:31:55(BST): Mostly back under control now. We're working on identifying the cause of the outage.

---

We're investigating the cause of the outage affecting xenserve16 at this point in time.

Start Time

30th July 2011, at 12:16PM

Description

Update at 19:16:42(BST): The issue has now been resolved with this server.Update at 19:11:13(BST): We identified an exploited user account that allowed a rogue process to cause excessive load. This has been eliminated and the server has been rebooted (19.07).

Update at 18:42:11(BST): We are presently investigating an issue of severe load on the bill.34sp.com server. Services are offline at this time.

Start Time

20th July 2011, at 6:39PM

Description

Update at 21:43:02(BST): The UPS backup for the primary core switch is preventing a remote reboot this evening. We'll leave the secondary switch running things (it's doing a fine job, being identical in configuration) and we should hopefully be able to resolve this completely at another point in the near future.

---

Update at 21:28:22(BST): Switchover OK. We'll monitor this for a short while before rebooting the primary switch.

---

Due to a bug in the firmware, we have unfortunately lost the management interface of our primary core switch in Manchester.

Whilst we've left it a few days without making any changes (everything else is working fine) we do need to reboot the switch to regain control of it.

The vendor has qualified the bug as credible and will be releasing a fix in a later update.

However, until then we must force fail-over to the secondary switch. This should not incur any loss of service, given that the configuration is naturally redundant in the case of a sudden failure.

Any customers that do experience significant trouble are encouraged to contact support@34sp.com at once.

Start Time

19th July 2011, at 9:01PM

Description

Update at 12:00:20(BST): No major problems reported after this migration from users so this status is closed - do however let us know if you do notice any issues.

---

Update at 12:31:17(BST): All sites should now be active again on the new hardware - let us know if there are any issues.

---

Update at 11:00:40(BST): Access to email and websites have now been disabled for the final sync operations, we expect the downtime from this point to be around 1-2hrs

---

Update at 10:27:10(BST): Incoming email services are now disabled.

---

Update at 10:13:10(BST): FTP service has now been disabled on this server for an resync of files.

---

The server5.xenserve.com is undergoing migration to new hardware today as per the email sent to all clients.

Try to avoid making any changes on any websites now to minimize loss of data when transferred to the new server.

We will update this post with progress throughout the day.

Start Time

19th July 2011, at 7:41AM

Description

Update at 22:36:19(BST): The issue has now been resolved with this server. This work has now been carried out.

---

Prohost12 is having some security updates applied to it, it will be rebooted very shortly.

The downtime should only be a few minutes.

Start Time

12th July 2011, at 10:29PM