34SP.com System Status History

Network slowness (Multiple services)

resolved

Description

Update at 19:25:14(GMT): Update at 14:52:26(GMT): Whilst we continue to monitor the situation, we believe at this time that the problem has been resolved and that all services should be performing normally.

Update at 09:07:35(GMT): Engineers have identified a number of servers that are being maliciously flooded with requests. The increase in traffic has had a knock on effect to most servers in our network. Engineers are currently working on blocking IPs that initiated this attack, and searching out and replacing code on affected sites.

Update at 08:07:27(GMT): We have seen a few shorter and less serious repetitions of the same problem, and continue to solve them each time. We are looking to implement a more permanent fix as soon as possible.

Update at 22:48:21(GMT): Update at 22:39:42(GMT): Most services should now be back online and accessible at normal speeds.

Update at 22:19:35(GMT): We have started to isolate the cause of the network load and work has begun to mitigate the excess network traffic. Some users will now see services back to full operation while others may find access intermittent while work continues.

Update at 21:09:08(GMT): We now have an engineer onsite and are working on correcting the fault.

Update at 20:36:19(GMT): We presently have an engineer on site at one of our datacenters investigating this issue.

We are presently investigating slowed access to our network. All hosted services may experience access issues at this time.

Start Time

7th February 2012, at 8:27PM

Resolution Time

12th February 2012, at 7:25PM

Reseller accounts emergency maintenance (Various)

resolved

Description

Update at 20:37:28(GMT): Update at 10:13:20(GMT): We have completed this maintenance and are monitoring the situation on these servers to see that it has been improved.


Reseller accounts hosted on the following servers will be unavailable for a short period beginning very shortly.

server1.xenserve.com
server3.xenserve.com
server5.xenserve.com
server9.xenserve.com

Last night during some maintenance work on server1.xenserve.com we discovered a problem affecting the above 4 servers. It is imperative that we correct this as soon as possible as it is affecting both performance and stability of the servers. To do so will require a short period of downtime for all of these servers and we intend to do this very shortly. We apologise for the inconvenience that this will cause.

Start Time

7th February 2012, at 9:31AM

Resolution Time

7th February 2012, at 8:37PM

Prohost3 emergency DB work (prohost3.34sp.com)

resolved

Description

Update at 00:44:57(GMT): At midnight, databases on the PH3 server will be unavailable for approximately 10 minutes time, while emergency work takes place.

Start Time

6th February 2012, at 11:40PM

Resolution Time

7th February 2012, at 12:44AM

Slow access to reseller accounts on server1.xenserve.com (server1.xenserve.com)

resolved

Description

Update at 09:44:40(GMT): The issue has now been resolved with this server.server1.xenserve.com is now available again and the fault has been corrected.


Update at 09:29:31(GMT): The problem with this server has revealed a larger fault. To rectify this we are having to reboot the server now. This will cause a short period of downtime in the next few minutes. When the server comes back to service, performance will be greatly enhanced and stability improved.


Update at 21:52:37(GMT): We are now performing a full disk check on this server. Unfortunately while this is in progress, all services will be unavailable.


Our engineers are currently investigating the cause of slow performance on server1.
We will update this status as soon as we know more.

We thank you for your patience. Apologies for any inconvenience caused.

Start Time

6th February 2012, at 8:32PM

Resolution Time

7th February 2012, at 9:44AM

Network slowness (Multiple)

resolved

Description

Update at 20:47:26(GMT): The issue has now been resolved with this server.We have now tracked the cause of this fault and network traffic should be back to normal levels.

Update at 20:41:02(GMT): We are presently investigating a spike in traffic at our Manchester data center.

This is affecting several systems including (but no limited too):

business8,

xen16, 17, 18, 20, 21

VPS3, 20, 21, 22, PH10, 11, 12

The traffic is causing intermittent access slow down. Services may be slow to access while we mitigate the traffic.

Start Time

5th February 2012, at 8:39PM

Resolution Time

5th February 2012, at 8:47PM

server21.xenserve.com hardware issue (server21.xenserve.com)

resolved

Description

Update at 18:55:03(GMT): Update at 20:19:00(GMT): Server is now back online, with new memory modules in place. We will monitor the server for the next few days just incase there is any other hardware issues.



Update at 19:58:44(GMT): The server is currently offline for some memory modules to be replaced, we do not expect this to take long.



Despite rigorous testing and QA processes, this new server appears to be having some hardware issues. We are in the process of escalating this to our suppliers and hope to have a resolution soon. In the meantime, we have seen the server reboot unexpectedly.

Start Time

28th January 2012, at 1:23PM

Resolution Time

12th February 2012, at 6:55PM

Prohost11 professional websites offline (prohost11)

resolved

Description

Update at 21:28:21(GMT): Update at 20:41:33(GMT): There were further issues with prohost11 tonight, this server has now had an upgrade on the kernel and other key packages and is being monitored further.




Update at 19:45:32(GMT): The issue has now been resolved with this server.We have rebooted prohost11 after the server became non responsive.

We will update this status should it not return normally.

Apologies for any inconvenience caused.

Start Time

27th January 2012, at 7:32PM

Resolution Time

7th February 2012, at 9:28PM

VPS5 Server Issues (VPS5.34sp.com)

resolved

Description

Update at 10:46:46(GMT): The issue has now been resolved with this server.


We are currently experiencing issues with our Server VPS5 which is causing all VPS servers contained to have various issues (Unresponsive emails and website issues). Our engineers are investigating into this now and will update as and when we have more information.

Start Time

20th January 2012, at 8:24AM

Resolution Time

24th January 2012, at 10:46AM

Connectivity issues with BT (n/a)

resolved

Description

Update at 23:40:38(GMT): Update at 14:33:28(GMT): We are seeing traffic back to 100% reliability. We are monitoring the situation but believe it to now be resolved.



Update at 10:47:13(GMT): We have escalated this issue with our suppliers and are waiting on a response from them.



Update at 09:48:48(GMT): Level(3) have reported that this issue is resolved but we are still seeing similar levels of packet loss to yesterday. We are following up with our providers on this.



Update at 16:28:15(GMT): We have confirmed with our providers that Level(3) are aware of the problem, but are currently unable to offer an estimated time to repair. If we are able to offer any further updates, they will be posted here.



We are currently seeing some issues with customers using BT as their ISP. After some investigation, we've found it to be related to the link between BT and Level(3), one of our connectivity providers.

We are aware that a number of our customers, and a larger number of other users have contacted BT regarding this issue and encourage anyone affected to do the same.

Start Time

18th January 2012, at 10:00AM

Resolution Time

6th February 2012, at 11:40PM

server4.xenserve.com unavailable (server4.xenserve.com)

resolved

Description

Update at 17:53:55(GMT): The issue has now been resolved with this server.The server is now available again.

This server is currently unavailable. We are investigating this and will restore the server to full service as soon as possible.

Start Time

8th January 2012, at 5:26PM

Resolution Time

8th January 2012, at 5:53PM