34SP.com System Status History

Description

Update at 15:01:18(GMT): This update is now marked as resolved.

Update at 16:37:41(GMT): Parallels have today asked if we can perform another upgrade to Plesk, to further debug some recent issues with the server.

This will not affect any e-mail functions (as before) and will begin after 18:00 GMT.

Any further information will be posted to status during that time.

---

Update at 14:33:08(GMT): Now back up.

---

Update at 14:04:53(GMT): We're unfortunately having to restart this server. However it should be back up very shortly.

---

Parallels are currently working on our ASP Business hosting server, business7.

This includes an upgrade to Plesk which may result in a few service restarts and/or some short service interruptions.

The work has unfortunately been made necessary following a bug that has affected one of our customers quite severely and could potentially affect others.

We appreciate your patience during this period.

Start Time

28th February 2011, at 9:00PM

Description

Update at 18:16:52(GMT): The machine appears to be running normally once again. We are now resolving this issue.

Update at 15:46:24(GMT): Business8 is now up and running again. We will continue to monitor this machine for a short while.

Update at 15:38:35(GMT): We have identified the fault and gained remote access to the machine (via KVM) and hope to have this resolved within the next 15-20 minutes.

We're investigating a fault with the Business hosting server, business8.34sp.com.

This machine is currently unresponsive and we're working to resolve this as quickly as possible.

Start Time

28th February 2011, at 2:49PM

Description

Update at 14:48:53(GMT): This issue is now marked as resolved.

Update at 12:10:32(GMT): We've recovered the server with too much trouble and will continue to monitor it for a short while.

---

We're seeing very high loads on server17 at process.

We are currently investigating and hope to have this resolved shortly.

Start Time

25th February 2011, at 12:00PM

Description

Update at 18:40:52(GMT): After a brief outage, the machine has now recovered from its load and appears to be functioning normally.

The Atmail server array, commonly used for Professional hosting e-mail (& others) is currently suffering from some high server loads that will be causing very slow performance.

At present this appears to be limited to one of the two front-facing webmail servers only, so will not be affected *all* customers.

We're working to resolve this as soon as possible.

Start Time

18th February 2011, at 5:52PM

Description

Update at 20:16:34(GMT): Issues should now be resolved for the most part. The Dell switching hardware that failed has been removed from use, and a workaround put in place for the time being. All services and access will be restored at this point. Moving forward we will be working with Dell to correct the faulty switching hardware before restoring this within the datacentre.

Update at 16:47:27(GMT): Our engineers have been in constant contact with their counterparts at Dell for several hours as we continue to try and isolate the issue affecting our Manchester datacentre. It currently seems that the issue is in the switching part of the network and is intermittently affecting all services within our Manchester network.

We are currently looking into an intermittent issue that is affecting availability to the servers located in the Manchester datacenter.

If your server IP address is within the following ranges:

80.82.124.x->80.82.126.x

or

46.183.8.x -> 46.183.9.x

Then your services are located in this datacenter.

We are activly looking into this issue at this time.

As soon as we have more information available we will update this post.

Start Time

15th February 2011, at 1:42PM

Description

Update at 20:14:49(GMT): Update at 12:44:34(GMT): Xenseve7 is now back up. We will continue to monitor this machine for a short while to ensure there are no lasting effects.

---

Update at 11:58:53(GMT): The Engineer is on-site and we're working to rectify the issues affecting xenserve7.

---

Update at 10:09:53(GMT): We have called-out an Engineer to assist on-site with xenserve7.

---

We are aware that server7 has become unreachable. We are currently working to restore service as soon as possible.

Start Time

14th February 2011, at 9:49AM

Description

Update at 11:30:52(GMT): Update at 12:46:18(GMT): We've updated the 'server' field to reflect the correct machine. Apologies for any confusion this may have caused.

Update at 23:53:44(GMT): Server back up and online after a power cycle, we will continue to monitor this through the weekend.

Update at 23:13:23(GMT): We are presently investigating a fault on this server.

Update at 23:30:28(GMT): Services are now back again but we are continuing to monitor this issue closely.

Update at 23:01:51(GMT): The problem has resurfaced and we are looking into this once more.

Update at 22:17:53(GMT): The issue has now been resolved with this server.

We are presently investigating a fault on this server.

Start Time

4th February 2011, at 9:35PM

Description

Update at 12:43:53(GMT): The issue has now been resolved with this server.This is now resolved.

---

Update at 10:10:38(GMT): Our earlier attempts to resolve this have failed; the Engineer is en-route again to help resolve this.

Due an SSH failure on the twiki server, the FTP lock feature has stalled. This will mean the present status of the FTP lock on your account will be fixed and unchangeable.

We presently have an engineer en route to the data center to attend to this.

Services should be corrected in the next couple of hours.

Start Time

4th February 2011, at 5:46PM

Description

Update at 09:48:51(GMT): This is now set to resolved.

---

Update at 17:42:18(GMT): At approximately 11.50 GMT today we experienced an outage on our network. After investigation we tracked this to a client hosted site. We disabled services and successfully restored full network activity 20 minutes later.

Unfortunately this did not resolve the matter. At approximately 13.30 GMT problems resurfaced. After continued investigation we identified a DDoS (distributed denial of service) attack against out network. This affect both our London and Manchester datacentres. The attack traffic overwhelmed our network and services became severely slowed or unreachable for some users.

At approximately 16.30 GMT our engineering team started to make headway in mitigating the attack traffic. Genuine traffic started to increase, and users reported an increased ability to access their services. By 17.00 GMT the effects of the attack were greatly diminished and most users had access to their services at reliable speeds. We are continuing to monitor the situation at this time.

Moving forward we are also actively looking at ways we can improve our ore network operations. If you have any questions regarding this incident please email us at support@34sp.com.

Update at 17:10:13(GMT): Traffic is improving, more and more is getting through now. Most people will see their sites as usual now but we are still working to restore full service.

Update at 16:03:18(GMT): We continue to improve the amount of traffic getting through. At this time it seems as though we are experiencing a denial of service (DoS) attack and are doing our best to mitigate the effects of this.

Update at 15:32:53(GMT): We continue to work to resolve the matter affecting our network. Some traffic is now intermittently getting through but the traffic causing the issue is still arriving at our network.

Update at 14:35:59(GMT): The earlier issue seem to have happened again we are working on resolving the issue as quickly as we can.

Update at 12:26:14(GMT): The cause of the network issues has been isolated and resolved. We will continue to closely monitor the sitaution

We are currently seeing large amounts of traffic across the 34SP networks, this is resulting in slow or no access to sites.

We are working on this issue at this time.

Start Time

3rd February 2011, at 12:07PM

Description

Update at 11:09:26(GMT): The Internet appears to be back to regular levels of health this morning -- status resolved.

---

We're aware that a large chunk of inter-ISP Internet routing traffic has disappeared into the ether this evening. This does not affect 34SP.com's network directly, but many customers will be connecting via ISPs that do peer directly via LINX.

Graphs illustrating the drop in 'normal' traffic levels can be found here.

Unfortunately there is little that we can do with regards to this issue, but hope that it was only momentary. If you experience consistent problems accessing portions of the Internet, hosted by 34SP.com or otherwise, please contact your ISP.

Start Time

2nd February 2011, at 8:57PM