34SP.com System Status History

Description

Update at 18:40:52(GMT): After a brief outage, the machine has now recovered from its load and appears to be functioning normally.

The Atmail server array, commonly used for Professional hosting e-mail (& others) is currently suffering from some high server loads that will be causing very slow performance.

At present this appears to be limited to one of the two front-facing webmail servers only, so will not be affected *all* customers.

We're working to resolve this as soon as possible.

Start Time

18th February 2011, at 5:52PM

Description

Update at 20:16:34(GMT): Issues should now be resolved for the most part. The Dell switching hardware that failed has been removed from use, and a workaround put in place for the time being. All services and access will be restored at this point. Moving forward we will be working with Dell to correct the faulty switching hardware before restoring this within the datacentre.

Update at 16:47:27(GMT): Our engineers have been in constant contact with their counterparts at Dell for several hours as we continue to try and isolate the issue affecting our Manchester datacentre. It currently seems that the issue is in the switching part of the network and is intermittently affecting all services within our Manchester network.

We are currently looking into an intermittent issue that is affecting availability to the servers located in the Manchester datacenter.

If your server IP address is within the following ranges:

80.82.124.x->80.82.126.x

or

46.183.8.x -> 46.183.9.x

Then your services are located in this datacenter.

We are activly looking into this issue at this time.

As soon as we have more information available we will update this post.

Start Time

15th February 2011, at 1:42PM

Description

Update at 20:14:49(GMT): Update at 12:44:34(GMT): Xenseve7 is now back up. We will continue to monitor this machine for a short while to ensure there are no lasting effects.

---

Update at 11:58:53(GMT): The Engineer is on-site and we're working to rectify the issues affecting xenserve7.

---

Update at 10:09:53(GMT): We have called-out an Engineer to assist on-site with xenserve7.

---

We are aware that server7 has become unreachable. We are currently working to restore service as soon as possible.

Start Time

14th February 2011, at 9:49AM

Description

Update at 11:30:52(GMT): Update at 12:46:18(GMT): We've updated the 'server' field to reflect the correct machine. Apologies for any confusion this may have caused.

Update at 23:53:44(GMT): Server back up and online after a power cycle, we will continue to monitor this through the weekend.

Update at 23:13:23(GMT): We are presently investigating a fault on this server.

Update at 23:30:28(GMT): Services are now back again but we are continuing to monitor this issue closely.

Update at 23:01:51(GMT): The problem has resurfaced and we are looking into this once more.

Update at 22:17:53(GMT): The issue has now been resolved with this server.

We are presently investigating a fault on this server.

Start Time

4th February 2011, at 9:35PM

Description

Update at 12:43:53(GMT): The issue has now been resolved with this server.This is now resolved.

---

Update at 10:10:38(GMT): Our earlier attempts to resolve this have failed; the Engineer is en-route again to help resolve this.

Due an SSH failure on the twiki server, the FTP lock feature has stalled. This will mean the present status of the FTP lock on your account will be fixed and unchangeable.

We presently have an engineer en route to the data center to attend to this.

Services should be corrected in the next couple of hours.

Start Time

4th February 2011, at 5:46PM

Description

Update at 09:48:51(GMT): This is now set to resolved.

---

Update at 17:42:18(GMT): At approximately 11.50 GMT today we experienced an outage on our network. After investigation we tracked this to a client hosted site. We disabled services and successfully restored full network activity 20 minutes later.

Unfortunately this did not resolve the matter. At approximately 13.30 GMT problems resurfaced. After continued investigation we identified a DDoS (distributed denial of service) attack against out network. This affect both our London and Manchester datacentres. The attack traffic overwhelmed our network and services became severely slowed or unreachable for some users.

At approximately 16.30 GMT our engineering team started to make headway in mitigating the attack traffic. Genuine traffic started to increase, and users reported an increased ability to access their services. By 17.00 GMT the effects of the attack were greatly diminished and most users had access to their services at reliable speeds. We are continuing to monitor the situation at this time.

Moving forward we are also actively looking at ways we can improve our ore network operations. If you have any questions regarding this incident please email us at support@34sp.com.

Update at 17:10:13(GMT): Traffic is improving, more and more is getting through now. Most people will see their sites as usual now but we are still working to restore full service.

Update at 16:03:18(GMT): We continue to improve the amount of traffic getting through. At this time it seems as though we are experiencing a denial of service (DoS) attack and are doing our best to mitigate the effects of this.

Update at 15:32:53(GMT): We continue to work to resolve the matter affecting our network. Some traffic is now intermittently getting through but the traffic causing the issue is still arriving at our network.

Update at 14:35:59(GMT): The earlier issue seem to have happened again we are working on resolving the issue as quickly as we can.

Update at 12:26:14(GMT): The cause of the network issues has been isolated and resolved. We will continue to closely monitor the sitaution

We are currently seeing large amounts of traffic across the 34SP networks, this is resulting in slow or no access to sites.

We are working on this issue at this time.

Start Time

3rd February 2011, at 12:07PM

Description

Update at 11:09:26(GMT): The Internet appears to be back to regular levels of health this morning -- status resolved.

---

We're aware that a large chunk of inter-ISP Internet routing traffic has disappeared into the ether this evening. This does not affect 34SP.com's network directly, but many customers will be connecting via ISPs that do peer directly via LINX.

Graphs illustrating the drop in 'normal' traffic levels can be found here.

Unfortunately there is little that we can do with regards to this issue, but hope that it was only momentary. If you experience consistent problems accessing portions of the Internet, hosted by 34SP.com or otherwise, please contact your ISP.

Start Time

2nd February 2011, at 8:57PM

Description

Update at 17:49:46(GMT): The issue has now been resolved with this server.We are presently rebooting the server3 server, services are presently offline.

Start Time

2nd February 2011, at 5:34PM

Description

Update at 21:12:30(GMT): The issue has now been resolved with this server.Services are all now back online and stabilising once more. We will continue to monitor the server through the night.

We are presently trying to restore services on server8. The system is under tremendous unexpected load, and several services are presently offline.

Start Time

1st February 2011, at 8:53PM

Description

Update at 17:48:00(GMT): Adam appears to be running well once again.

---

Update at 11:15:21(GMT): Adam is now back up and running and the original issue has now been resolved.

We will continue to monitor Adam for a short while to ensure the issue does not recur.

---

Adam has had to go down for an urgent reboot to solve an issue that have manifested this morning.

We hope to have the server back up and running shortly.

Start Time

28th January 2011, at 10:59AM