34SP.com System Status History

Description

Update at 11:30:52(GMT): Update at 12:46:18(GMT): We've updated the 'server' field to reflect the correct machine. Apologies for any confusion this may have caused.

Update at 23:53:44(GMT): Server back up and online after a power cycle, we will continue to monitor this through the weekend.

Update at 23:13:23(GMT): We are presently investigating a fault on this server.

Update at 23:30:28(GMT): Services are now back again but we are continuing to monitor this issue closely.

Update at 23:01:51(GMT): The problem has resurfaced and we are looking into this once more.

Update at 22:17:53(GMT): The issue has now been resolved with this server.

We are presently investigating a fault on this server.

Start Time

4th February 2011, at 9:35PM

Description

Update at 12:43:53(GMT): The issue has now been resolved with this server.This is now resolved.

---

Update at 10:10:38(GMT): Our earlier attempts to resolve this have failed; the Engineer is en-route again to help resolve this.

Due an SSH failure on the twiki server, the FTP lock feature has stalled. This will mean the present status of the FTP lock on your account will be fixed and unchangeable.

We presently have an engineer en route to the data center to attend to this.

Services should be corrected in the next couple of hours.

Start Time

4th February 2011, at 5:46PM

Description

Update at 09:48:51(GMT): This is now set to resolved.

---

Update at 17:42:18(GMT): At approximately 11.50 GMT today we experienced an outage on our network. After investigation we tracked this to a client hosted site. We disabled services and successfully restored full network activity 20 minutes later.

Unfortunately this did not resolve the matter. At approximately 13.30 GMT problems resurfaced. After continued investigation we identified a DDoS (distributed denial of service) attack against out network. This affect both our London and Manchester datacentres. The attack traffic overwhelmed our network and services became severely slowed or unreachable for some users.

At approximately 16.30 GMT our engineering team started to make headway in mitigating the attack traffic. Genuine traffic started to increase, and users reported an increased ability to access their services. By 17.00 GMT the effects of the attack were greatly diminished and most users had access to their services at reliable speeds. We are continuing to monitor the situation at this time.

Moving forward we are also actively looking at ways we can improve our ore network operations. If you have any questions regarding this incident please email us at support@34sp.com.

Update at 17:10:13(GMT): Traffic is improving, more and more is getting through now. Most people will see their sites as usual now but we are still working to restore full service.

Update at 16:03:18(GMT): We continue to improve the amount of traffic getting through. At this time it seems as though we are experiencing a denial of service (DoS) attack and are doing our best to mitigate the effects of this.

Update at 15:32:53(GMT): We continue to work to resolve the matter affecting our network. Some traffic is now intermittently getting through but the traffic causing the issue is still arriving at our network.

Update at 14:35:59(GMT): The earlier issue seem to have happened again we are working on resolving the issue as quickly as we can.

Update at 12:26:14(GMT): The cause of the network issues has been isolated and resolved. We will continue to closely monitor the sitaution

We are currently seeing large amounts of traffic across the 34SP networks, this is resulting in slow or no access to sites.

We are working on this issue at this time.

Start Time

3rd February 2011, at 12:07PM

Description

Update at 11:09:26(GMT): The Internet appears to be back to regular levels of health this morning -- status resolved.

---

We're aware that a large chunk of inter-ISP Internet routing traffic has disappeared into the ether this evening. This does not affect 34SP.com's network directly, but many customers will be connecting via ISPs that do peer directly via LINX.

Graphs illustrating the drop in 'normal' traffic levels can be found here.

Unfortunately there is little that we can do with regards to this issue, but hope that it was only momentary. If you experience consistent problems accessing portions of the Internet, hosted by 34SP.com or otherwise, please contact your ISP.

Start Time

2nd February 2011, at 8:57PM

Description

Update at 17:49:46(GMT): The issue has now been resolved with this server.We are presently rebooting the server3 server, services are presently offline.

Start Time

2nd February 2011, at 5:34PM

Description

Update at 21:12:30(GMT): The issue has now been resolved with this server.Services are all now back online and stabilising once more. We will continue to monitor the server through the night.

We are presently trying to restore services on server8. The system is under tremendous unexpected load, and several services are presently offline.

Start Time

1st February 2011, at 8:53PM

Description

Update at 17:48:00(GMT): Adam appears to be running well once again.

---

Update at 11:15:21(GMT): Adam is now back up and running and the original issue has now been resolved.

We will continue to monitor Adam for a short while to ensure the issue does not recur.

---

Adam has had to go down for an urgent reboot to solve an issue that have manifested this morning.

We hope to have the server back up and running shortly.

Start Time

28th January 2011, at 10:59AM

Description

Update at 20:53:29(GMT): This issue has since been resolved.

---

Update at 17:47:40(GMT): Server14 is back up and we are checking all databases as a precaution.

The high load has caused a DoS of all services which is proving difficult to track down.

We will continue to monitor this machine for a while this evening.

---

Update at 17:40:26(GMT): A similar issue has affected server14. We're again, trying to resolve this.

---

Update at 11:15:37(GMT): We've managed to regain control of xenserve14. We'll continue to monitor this machine for the next few hours to see if we can isolate any recurring fault.

---

Xenserve14 is currently experiencing an above-normal level of load that is effectively causing requests to fail. We're trying to regain control of the machine and hope to have service restored soon.

Start Time

24th January 2011, at 9:55AM

Description

Update at 16:55:18(GMT): This outage is now resolved.

---

Update at 16:26:13(GMT): The machine is now back up and running, we're checking the machine over.

---

Server6 (xenserve6) is currently down. We are investigating the situation and hope to have service restored shortly.

Start Time

21st January 2011, at 4:12PM

Description

Update at 16:55:06(GMT): This outage is now resolved.

---

Update at 13:00:21(GMT): Services on this server have now stabilised. Our engineers will continue to monitor this.

Update at 12:25:10(GMT): We are currently aware that server4 is having some issues relating to database and other services. Our engineers are currently working to resolve this issue. Apologies for any inconvenience caused.

Start Time

19th January 2011, at 12:24PM