34SP.com System Status History

Description

Update at 10:00:59(BST): The issue has now been resolved with this server.This issue was resolved around an hour ago, an issue with the firewall setup within each domain had caused it to block all external connections, we have increased limits on the firewall connection tracking systems to avoid this issue re-occurring.

At the same time a kernel update was also applied to the server as well.

---

Sites hosted on businesshost1 are currently inaccessible. Engineers are working to resolve this issue.

Start Time

30th August 2011, at 8:19AM

Description

Update at 10:26:29(BST): The outage was caused by a "rare failure" in the UPS systems. Despite this hardware being designed with N+1 redundancy, the loss of two units resulted in a 20ms dip in power to our racks within that room.

We are confident that our providers have rectified this fault and will be working to mitigate any recurrence of this particular scenario in the future.

---

Update at 15:26:12(BST): All services should now have been returned for some time.

We're awaiting further explanation from the facilities provider as to the nature of the outage.

---

Only one floor of the Manchester facility (Synergy House) was affected.

We have confirmation from UKGrid of the failure (other ISPs/hosting providers have also experienced the same problem).

Most VPS' & hosting services are now running again as they should be. Customers with unmanaged Dedicated Servers/VPS' that have had a loss of power are advised to ensure the consistency of any databases they may have.

All shared hosting services have had their databases checked accordingly.

Any customers with lingering issues are urged to e-mail support as the telephone lines are currently very busy via support@34sp.com.

---

We have experienced a power failure to one of the cabinets in Manchester.

Affected machines have automatically restarted as the power was returned, but it will take a short while for virtual machines and some other services to come back online.

More information as we find out.

Start Time

24th August 2011, at 2:36PM

Description

Update at 20:23:36(BST): Xenserve19 has been running well since the incident today, however please see the recent post regarding the emergency upgrade as this will require some further (planned) downtime this evening, but will hopefully put to bed these issues with load.

---

Update at 13:45:31(BST): Xenserve19 is now back-up and running. Once again we apologise for the obvious inconvenience caused and are working on a long-term solution to the re-occurring problems faced with this particular machine.

---

Update at 13:30:20(BST): The machine has experience another large peak in load and has been (cleanly) rebooted from its console.

It will restart shortly. We apologise for the inconvenience caused.

Action to prevent the on-going 'peaks' in load that are affecting this server is being discussed and there will be some notice to customers very soon.

---

Update at 07:54:20(BST):
The server has just had a further reboot to bring inline some further changes which should bring the controls from the 34SP account pages back to fully working order.

---

Update at 14:59:56(BST): 4PSA are looking to rebuild an more upto date version of there software for this release, we will keep this on monitoring status until we receive more details from them.

---

Update at 13:32:05(BST): As per the email sent out to all clients on this server, the 4PSA support team are currently logged into the server looking into this issue.

---

Server is now back online - the downtime was less than 5minutes. Please do let us know if you experience any issues with this update.

---

This server is currently undergoing a operating system update which will require a reboot, the expected downtime should be about 10minutes.

Start Time

19th August 2011, at 8:17AM

Description

Update at 09:36:50(BST): Phone lines are now open again.

---

Update at 08:51:04(BST): Due to the rioting in Manchester, it is going to be particularly difficult for staff members to reach the 34SP.com office.

Given the short notice, it is not possible to provide an acceptable notice for telephone support via remote workers.

With this in mind, please be advised that telephone support will not be available throughout Wednesday the 10th of August, until further notice.

All customers are urged to contact the support team via the usual e-mail address: support@34sp.com

Start Time

9th August 2011, at 11:36PM

Description

Update at 23:33:44(BST): Prohost12 now appears to be running acceptably once again, outage marked as resolved.

---

Update at 11:04:35(BST): The affected website has now been re-located on to fresh hardware. Cached requests via Prohost12 are being redirected. At this time, other customers utilising Prohost12 should not have any further difficulties with their websites.

---

Update at 10:38:06(BST): We are in the process of moving the website in question another server, with more capacity.

Unfortunately, as this publicity was not anticipated by the owner of the website, we will have significant DNS caching issues to endure.

DNS records have already been updated and all efforts will be made to minimise the effects of the caching.

---

Following last night's rioting in the UK, one particularly popular website has been hosted on our basic shared hosting service and this is completely flattening the server, Prohost12.

Of course this service was never designed for such site loads and we have had no choice but to disable the site to ensure that others sharing this service are able to use their websites.

We will be contacting the owner of this website shortly to arrange for a better hosting solution.

34SP.com apologises for any inconvenience caused by this surge in traffic.

Start Time

9th August 2011, at 9:19AM

Description

Update at 23:34:39(BST): No lasting issues have been experienced in the week since the maintenance, therefore this outage is now marked as resolved.

---

Update at 00:27:42(BST): All maintenance has now been completed successfully.

Any customers that believe they are experiencing any sort of un-planned or extended outage are encouraged to contact support@34sp.com as soon as possible.

---

Update at 23:51:45(BST): Switches 10 & 11 completed.

Switches 12, 13 & 15 will be upgraded shortly.

---

Update at 23:32:30(BST): Switches 8 & 9 are now done.

Switches 10 & 11 will be upgraded shortly.

---

Update at 23:11:50(BST): VPS1 is now back up and running.

The first of the access switch upgrades will be starting shortly.

---

Update at 22:44:14(BST): VPS1 will be going down shortly.

---

Update at 22:14:42(BST): The reseller server xenserve11 has now been re-located and is back up and running successfully.

At 22:45, the VPS host VPS1 will be downed to have its RAM replaced.

---

Update at 21:57:13(BST): In approximately 3 minutes time, xenserve11 (server11.xenserve.com) will be shutdown, re-located and rebooted.

Incidentally this is the first time that the machine has been offline in 825 days! Almost a shame to have to move it.

---

Customers are advised that scheduled maintenance tasks will be starting shortly:

22:00 - xenserve11 re-location

22:45 - VPS1 RAM replacement

23:15 - Access switch firmware upgrades

Updates will be posted to reflect current events.

Start Time

2nd August 2011, at 9:40PM

Description

Update at 16:22:01(BST): The issue has now been resolved with this server.This matter was resolved around 2.30PM

---

Ted is experiencing a serious level of load at present, which is causing a denial of service.

Attempts to recover the server remotely have failed thus far. We'll try rebooting the machine now.

Start Time

2nd August 2011, at 2:22PM

Description

Update at 16:33:11(BST): This migration is now being marked as finished, if however anyone is having any issues since the migration please do contact us and we will aid in looking into the problems for you.

---

Update at 16:22:18(BST): This migration has now been completed, we are doing some small amount of maintenance work, but do let us know if there are any issues on any sites.

---

Update at 15:16:28(BST): The main webservices will now be going offline for the final stage of this migration, we expect the downtime to be around 1-2hrs

---

Update at 14:43:38(BST): FTP services are now stopped for a initial sync of files.

---

Update at 14:19:14(BST): Mail services have been stopped on this server for the syncing of the mail

---

The deepthought.34sp.com domains are undergoing migration to a different server as per the email sent to all clients.

Try to avoid making any changes on any websites now to minimize loss of data when transferred to the new server.

We will update this post with progress throughout the day.

Start Time

2nd August 2011, at 8:07AM

Description

Update at 14:21:50(BST): We haven't seen a resurgence of this issue lately and no specific issue was identified since the last episode.

---

Update at 12:31:55(BST): Mostly back under control now. We're working on identifying the cause of the outage.

---

We're investigating the cause of the outage affecting xenserve16 at this point in time.

Start Time

30th July 2011, at 12:16PM

Description

Update at 19:16:42(BST): The issue has now been resolved with this server.Update at 19:11:13(BST): We identified an exploited user account that allowed a rogue process to cause excessive load. This has been eliminated and the server has been rebooted (19.07).

Update at 18:42:11(BST): We are presently investigating an issue of severe load on the bill.34sp.com server. Services are offline at this time.

Start Time

20th July 2011, at 6:39PM