34SP.com System Status History

Description

Update at 09:27:00(BST): Server8 has been running well for some time now and as a result, this outage is now marked as resolved.

---

Update at 22:08:32(BST): The new RAM has been fitted and xenserve8 is now back up and running! We'll be keeping a close eye on this machine for the next few hours.

---

Update at 21:51:49(BST): New RAM has been located and this is being fitted.

We apologise for the continued setbacks leading to this continued outage and appreciate your patience in this matter.

---

Update at 20:28:58(BST): Further setbacks have been experienced: this is due to the wrong type of RAM being sought from spares. This is due to a mistake in our records and we're working to find a solution (ideally the correct RAM).

More updates as they come.

---

Update at 19:54:05(BST): We apologies for extended outage to this server, the Engineer is still working on the machine and we hope to have service returned shortly.

---

Update at 19:17:36(BST): An Engineer is on-site now and will be replacing the memory in xenserve8. This means that the machine will be offline for 5-10 minutes whilst the memory is replaced with spares.

---

Update at 18:19:36(BST): A substantial portion of the memory in this server has failed. We have an engineer en route to replace the memory and will update as soon as this has been done.

This server is currently showing some hardware errors. We are attempting to restore functionality as quickly as possible.

Start Time

26th May 2011, at 5:44PM

Description

Update at 08:48:46(BST): The issue has now been resolved with this server.Update at 08:48:42(BST): The engineers have now resolved the issue affecting the MySQL service so all aspects should be back up and running.

We have received reports of MySQL issues on the server which will be affecting Emails and any sites with database driven content.

Our engineers are working on the issue now and normal service should resume shortly.

Start Time

26th May 2011, at 8:28AM

Description

Update at 17:42:55(BST): All now seems well. This outage is now marked as resolved.

---

Update at 17:19:43(BST): There is quite a high load on the MySQL datbase at present (which is in itself, rising and falling). We're investigating the cause of this, which is most-likely to be as a result of rogue/un-optimised queries.

---

Update at 16:47:48(BST): The machine rebooted cleanly and we are working on ascertaining the cause of the fault.

We will be checking the consistency of all databases and monitoring from this point onwards.

---

Business8 appears to have suffered a kernel panic, we've restarted the machine via the power controllers and will be monitoring the machine as it reboots.

Start Time

20th May 2011, at 4:45PM

Description

Update at 15:46:31(BST): This outage is now marked as resolved.

---

Update at 09:47:58(BST): The reboot has succeeded and the machine is back-up and running smoothly again.

We'll be checking all databases as a matter of course and shall continue to monitor the situation.

---

Update at 09:39:41(BST): Having no luck logging-in remotely, we have been forced to hard-reboot this machine. It is likely that the machine exhausted its swap memory during the surge.

The machine is coming back up now.

---

At present xenserve9 is unresponsive, causing issues with services. This is related to a surge in traffic that has caused a very high load.

We are working as quickly as possible to return the machine to a working state.

Start Time

18th May 2011, at 9:20AM

Description

Update at 17:29:23(BST): This outage is now marked as resolved.

---

Update at 17:10:52(BST): Both machines should now be functioning correctly once again.

---

At present xenserve9 is experiencing issues related to the MySQL process that is causing a number of services/websites to fail.

Coincidentally, xenserve17 is also experiencing a high level of traffic towards one or two websites, causing a high load that is making website loading particularly sluggish.

We hope to have both matters under control as soon as possible.

Start Time

17th May 2011, at 4:54PM

Description

Update at 09:28:27(BST): And as soon as we've posted, it appears that the developers have now resolved the issue.

---

We're aware of an issue affecting the speed of our main website, www.34sp.com and are working to resolve this.

Customers needing to manage their account are advised to visit https://account.34sp.com directly.

Thank you for your patience during this time.

Start Time

16th May 2011, at 9:26AM

Description

Update at 03:39:28(BST): Update at 18:09:43(BST): We continue to monitor the status of this machine but services are currently available.

Update at 17:41:55(BST): We are currently rebooting this server to clear the current issues..

We thank you for your patience and hope to have service restored as soon as possible.

---

We're currently investigating an issue that is affecting Business4. At present the machine is up and this looks more network related than anything.

More updates as they come.

Start Time

13th May 2011, at 5:19PM

Description

Update at 12:14:55(BST): Both these servers are back in service so this issue is now resolved

Update at 11:55:47(BST): Server14 and Server3 seem to have been the two servers the worst-affected and lingering load issues are unfortunately still continuing.

We're working to get this resolved ASAP.

---

We've been alerted to a number of servers with high loads. Upon further inspection this appears to have been an automated attempt to find insecure PHPMyAdmin installations on our servers.

Whilst the level of network traffic was not noticeably high, the requests to the web servers were driving-up the load placed on these servers, causing a loss of service for certain customers.

We have now blocked this IP at our network edge and will be contacting the appropriate abuse department.

Start Time

12th May 2011, at 11:34AM

Description

Update at 09:38:00(BST): This outage is now marked as resolved.

---

Update at 08:44:03(BST): Looking at the surge of incoming traffic, this was apparently a spate of DDoS traffic hitting a machine on the same switch as server12 & server14.

This has quickly subsided and we're working on identifying the target application.

---

We've noticed an issue that is affecting one of our switches and this currently appears to be resulting in no access to services on both xenserve12 and xenserve14.

We'll update as and when more information is available.

Start Time

6th May 2011, at 8:35AM

Description

Update at 10:46:52(BST): Update at 10:45:20(BST): The issue has now been resolved with this server.Fire alarm cleared and full service has now resumed. Apologies for any inconvenience caused. 10:03:15(BST): Support and Livechat are currently offline due to a firealarm and building evacuation.

Start Time

28th April 2011, at 10:20AM