34SP.com System Status History

Description

Update at 11:56:14(BST): Business5 has been operating happily for over a few hours now, with no further signs of disruption. Load levels appear normal.

If any customer is experiencing any lingering issues, please contact support.

---

Update at 10:39:10(BST): Business5 is now recovering; all services should be responding, but may be a little slow whilst users refresh their websites & inboxes.

We'll continue to monitor Business5 for a short while.

---

At present Business5 is experiencing a VERY high level of load across its services, which are being caused by certain malicious activities.

We're working to restore service as soon as possible. We appreciate your patience at this time.

Start Time

1st September 2011, at 10:22AM

Description

Update at 08:10:51(BST): Update at 19:14:18(BST): This migration was completed several hours ago, please do let us know if there are any issues with your sites from this move.

---

Update at 13:12:54(BST): All services are now stopped on the server for a final sync of the databases and files before we move the ips across.

---

Update at 12:32:18(BST):
Mail services have now been taken offline to resync the mail across to the new server.

---

FTP service has now been disabled on this server for an resync of files.

---

The server8.xenserve.com is undergoing migration to new hardware today as per the email sent to all clients.

Try to avoid making any changes on any websites now to minimize loss of data when transferred to the new server.

We will update this post with progress throughout the day

Start Time

31st August 2011, at 11:57AM

Description

Update at 10:24:11(BST): The issue affecting Globalsign and 34SP.com is now resolved.

---

Update at 11:47:51(BST): GlobalSign are aware of an issue with their verification systems at present. This is affecting Firefox users. Their twitter feed is here:

http://twitter.com/#!/globalsign/status/108839143517327360

This only affects logging into the 34SP.com control panel.

Start Time

31st August 2011, at 11:42AM

Description

Update at 10:00:59(BST): The issue has now been resolved with this server.This issue was resolved around an hour ago, an issue with the firewall setup within each domain had caused it to block all external connections, we have increased limits on the firewall connection tracking systems to avoid this issue re-occurring.

At the same time a kernel update was also applied to the server as well.

---

Sites hosted on businesshost1 are currently inaccessible. Engineers are working to resolve this issue.

Start Time

30th August 2011, at 8:19AM

Description

Update at 10:26:29(BST): The outage was caused by a "rare failure" in the UPS systems. Despite this hardware being designed with N+1 redundancy, the loss of two units resulted in a 20ms dip in power to our racks within that room.

We are confident that our providers have rectified this fault and will be working to mitigate any recurrence of this particular scenario in the future.

---

Update at 15:26:12(BST): All services should now have been returned for some time.

We're awaiting further explanation from the facilities provider as to the nature of the outage.

---

Only one floor of the Manchester facility (Synergy House) was affected.

We have confirmation from UKGrid of the failure (other ISPs/hosting providers have also experienced the same problem).

Most VPS' & hosting services are now running again as they should be. Customers with unmanaged Dedicated Servers/VPS' that have had a loss of power are advised to ensure the consistency of any databases they may have.

All shared hosting services have had their databases checked accordingly.

Any customers with lingering issues are urged to e-mail support as the telephone lines are currently very busy via support@34sp.com.

---

We have experienced a power failure to one of the cabinets in Manchester.

Affected machines have automatically restarted as the power was returned, but it will take a short while for virtual machines and some other services to come back online.

More information as we find out.

Start Time

24th August 2011, at 2:36PM

Description

Update at 20:23:36(BST): Xenserve19 has been running well since the incident today, however please see the recent post regarding the emergency upgrade as this will require some further (planned) downtime this evening, but will hopefully put to bed these issues with load.

---

Update at 13:45:31(BST): Xenserve19 is now back-up and running. Once again we apologise for the obvious inconvenience caused and are working on a long-term solution to the re-occurring problems faced with this particular machine.

---

Update at 13:30:20(BST): The machine has experience another large peak in load and has been (cleanly) rebooted from its console.

It will restart shortly. We apologise for the inconvenience caused.

Action to prevent the on-going 'peaks' in load that are affecting this server is being discussed and there will be some notice to customers very soon.

---

Update at 07:54:20(BST):
The server has just had a further reboot to bring inline some further changes which should bring the controls from the 34SP account pages back to fully working order.

---

Update at 14:59:56(BST): 4PSA are looking to rebuild an more upto date version of there software for this release, we will keep this on monitoring status until we receive more details from them.

---

Update at 13:32:05(BST): As per the email sent out to all clients on this server, the 4PSA support team are currently logged into the server looking into this issue.

---

Server is now back online - the downtime was less than 5minutes. Please do let us know if you experience any issues with this update.

---

This server is currently undergoing a operating system update which will require a reboot, the expected downtime should be about 10minutes.

Start Time

19th August 2011, at 8:17AM

Description

Update at 09:36:50(BST): Phone lines are now open again.

---

Update at 08:51:04(BST): Due to the rioting in Manchester, it is going to be particularly difficult for staff members to reach the 34SP.com office.

Given the short notice, it is not possible to provide an acceptable notice for telephone support via remote workers.

With this in mind, please be advised that telephone support will not be available throughout Wednesday the 10th of August, until further notice.

All customers are urged to contact the support team via the usual e-mail address: support@34sp.com

Start Time

9th August 2011, at 11:36PM

Description

Update at 23:33:44(BST): Prohost12 now appears to be running acceptably once again, outage marked as resolved.

---

Update at 11:04:35(BST): The affected website has now been re-located on to fresh hardware. Cached requests via Prohost12 are being redirected. At this time, other customers utilising Prohost12 should not have any further difficulties with their websites.

---

Update at 10:38:06(BST): We are in the process of moving the website in question another server, with more capacity.

Unfortunately, as this publicity was not anticipated by the owner of the website, we will have significant DNS caching issues to endure.

DNS records have already been updated and all efforts will be made to minimise the effects of the caching.

---

Following last night's rioting in the UK, one particularly popular website has been hosted on our basic shared hosting service and this is completely flattening the server, Prohost12.

Of course this service was never designed for such site loads and we have had no choice but to disable the site to ensure that others sharing this service are able to use their websites.

We will be contacting the owner of this website shortly to arrange for a better hosting solution.

34SP.com apologises for any inconvenience caused by this surge in traffic.

Start Time

9th August 2011, at 9:19AM

Description

Update at 23:34:39(BST): No lasting issues have been experienced in the week since the maintenance, therefore this outage is now marked as resolved.

---

Update at 00:27:42(BST): All maintenance has now been completed successfully.

Any customers that believe they are experiencing any sort of un-planned or extended outage are encouraged to contact support@34sp.com as soon as possible.

---

Update at 23:51:45(BST): Switches 10 & 11 completed.

Switches 12, 13 & 15 will be upgraded shortly.

---

Update at 23:32:30(BST): Switches 8 & 9 are now done.

Switches 10 & 11 will be upgraded shortly.

---

Update at 23:11:50(BST): VPS1 is now back up and running.

The first of the access switch upgrades will be starting shortly.

---

Update at 22:44:14(BST): VPS1 will be going down shortly.

---

Update at 22:14:42(BST): The reseller server xenserve11 has now been re-located and is back up and running successfully.

At 22:45, the VPS host VPS1 will be downed to have its RAM replaced.

---

Update at 21:57:13(BST): In approximately 3 minutes time, xenserve11 (server11.xenserve.com) will be shutdown, re-located and rebooted.

Incidentally this is the first time that the machine has been offline in 825 days! Almost a shame to have to move it.

---

Customers are advised that scheduled maintenance tasks will be starting shortly:

22:00 - xenserve11 re-location

22:45 - VPS1 RAM replacement

23:15 - Access switch firmware upgrades

Updates will be posted to reflect current events.

Start Time

2nd August 2011, at 9:40PM

Description

Update at 16:22:01(BST): The issue has now been resolved with this server.This matter was resolved around 2.30PM

---

Ted is experiencing a serious level of load at present, which is causing a denial of service.

Attempts to recover the server remotely have failed thus far. We'll try rebooting the machine now.

Start Time

2nd August 2011, at 2:22PM