34SP.com System Status History

Slowed network (Multiple servers)

resolved

Description

Update at 18:11:34(GMT): Update at 22:52:13(GMT): Services should now be running much smoother. However users may see one off minor glitches as we work to patch systems affected by the weeks attacks.

We are presently seeing a spike in traffic across our network which is causing slowed access to several systems (notably reseller servers 2, 6, 7, 8).

Most servers and systems should be accessible albeit with reduced speed at this time.

Start Time

12th February 2012, at 7:26PM

Resolution Time

13th February 2012, at 6:11PM

Xenserve 4 unreachable (xenserve4)

resolved

Description

Update at 19:09:49(GMT): Update at 19:15:15(GMT): Back online - investigations continuing as to why this is happening.




Update at 18:54:30(GMT): Server is currently offline again, we are continuing to investigate this.




Update at 17:44:06(GMT): The server has had a full disk check and is now back online, a database consistency check is currently running.

We will continue to monitor the server over the next 24hrs.




An automated service restart was initiated on server4 at 16.47. This failed to restart cleanly, and engineers were notified. A full reboot command was issued at 17.00 - this has so far not brought services back online. Engineers will continue to attempt to bring services back online as soon as possible, and we will update the status as the situation develops.

Start Time

12th February 2012, at 5:21PM

Resolution Time

20th February 2012, at 7:09PM

Phone Support 12th Feb (n/a)

resolved

Description

Update at 22:52:24(GMT): Update at 14:14:15(GMT): Due to unforseen circumstances, phone support for today, Sunday 12th February, will be via email only. Phone support will resume tomorrow, Monday 13th. Apologies for any inconvenience caused.

Start Time

12th February 2012, at 9:35AM

Resolution Time

12th February 2012, at 10:52PM

Network slowness (Multiple services)

resolved

Description

Update at 19:25:14(GMT): Update at 14:52:26(GMT): Whilst we continue to monitor the situation, we believe at this time that the problem has been resolved and that all services should be performing normally.

Update at 09:07:35(GMT): Engineers have identified a number of servers that are being maliciously flooded with requests. The increase in traffic has had a knock on effect to most servers in our network. Engineers are currently working on blocking IPs that initiated this attack, and searching out and replacing code on affected sites.

Update at 08:07:27(GMT): We have seen a few shorter and less serious repetitions of the same problem, and continue to solve them each time. We are looking to implement a more permanent fix as soon as possible.

Update at 22:48:21(GMT): Update at 22:39:42(GMT): Most services should now be back online and accessible at normal speeds.

Update at 22:19:35(GMT): We have started to isolate the cause of the network load and work has begun to mitigate the excess network traffic. Some users will now see services back to full operation while others may find access intermittent while work continues.

Update at 21:09:08(GMT): We now have an engineer onsite and are working on correcting the fault.

Update at 20:36:19(GMT): We presently have an engineer on site at one of our datacenters investigating this issue.

We are presently investigating slowed access to our network. All hosted services may experience access issues at this time.

Start Time

7th February 2012, at 8:27PM

Resolution Time

12th February 2012, at 7:25PM

Reseller accounts emergency maintenance (Various)

resolved

Description

Update at 20:37:28(GMT): Update at 10:13:20(GMT): We have completed this maintenance and are monitoring the situation on these servers to see that it has been improved.


Reseller accounts hosted on the following servers will be unavailable for a short period beginning very shortly.

server1.xenserve.com
server3.xenserve.com
server5.xenserve.com
server9.xenserve.com

Last night during some maintenance work on server1.xenserve.com we discovered a problem affecting the above 4 servers. It is imperative that we correct this as soon as possible as it is affecting both performance and stability of the servers. To do so will require a short period of downtime for all of these servers and we intend to do this very shortly. We apologise for the inconvenience that this will cause.

Start Time

7th February 2012, at 9:31AM

Resolution Time

7th February 2012, at 8:37PM

Prohost3 emergency DB work (prohost3.34sp.com)

resolved

Description

Update at 00:44:57(GMT): At midnight, databases on the PH3 server will be unavailable for approximately 10 minutes time, while emergency work takes place.

Start Time

6th February 2012, at 11:40PM

Resolution Time

7th February 2012, at 12:44AM

Slow access to reseller accounts on server1.xenserve.com (server1.xenserve.com)

resolved

Description

Update at 09:44:40(GMT): The issue has now been resolved with this server.server1.xenserve.com is now available again and the fault has been corrected.


Update at 09:29:31(GMT): The problem with this server has revealed a larger fault. To rectify this we are having to reboot the server now. This will cause a short period of downtime in the next few minutes. When the server comes back to service, performance will be greatly enhanced and stability improved.


Update at 21:52:37(GMT): We are now performing a full disk check on this server. Unfortunately while this is in progress, all services will be unavailable.


Our engineers are currently investigating the cause of slow performance on server1.
We will update this status as soon as we know more.

We thank you for your patience. Apologies for any inconvenience caused.

Start Time

6th February 2012, at 8:32PM

Resolution Time

7th February 2012, at 9:44AM

Network slowness (Multiple)

resolved

Description

Update at 20:47:26(GMT): The issue has now been resolved with this server.We have now tracked the cause of this fault and network traffic should be back to normal levels.

Update at 20:41:02(GMT): We are presently investigating a spike in traffic at our Manchester data center.

This is affecting several systems including (but no limited too):

business8,

xen16, 17, 18, 20, 21

VPS3, 20, 21, 22, PH10, 11, 12

The traffic is causing intermittent access slow down. Services may be slow to access while we mitigate the traffic.

Start Time

5th February 2012, at 8:39PM

Resolution Time

5th February 2012, at 8:47PM

server21.xenserve.com hardware issue (server21.xenserve.com)

resolved

Description

Update at 18:55:03(GMT): Update at 20:19:00(GMT): Server is now back online, with new memory modules in place. We will monitor the server for the next few days just incase there is any other hardware issues.



Update at 19:58:44(GMT): The server is currently offline for some memory modules to be replaced, we do not expect this to take long.



Despite rigorous testing and QA processes, this new server appears to be having some hardware issues. We are in the process of escalating this to our suppliers and hope to have a resolution soon. In the meantime, we have seen the server reboot unexpectedly.

Start Time

28th January 2012, at 1:23PM

Resolution Time

12th February 2012, at 6:55PM

Prohost11 professional websites offline (prohost11)

resolved

Description

Update at 21:28:21(GMT): Update at 20:41:33(GMT): There were further issues with prohost11 tonight, this server has now had an upgrade on the kernel and other key packages and is being monitored further.




Update at 19:45:32(GMT): The issue has now been resolved with this server.We have rebooted prohost11 after the server became non responsive.

We will update this status should it not return normally.

Apologies for any inconvenience caused.

Start Time

27th January 2012, at 7:32PM

Resolution Time

7th February 2012, at 9:28PM