34SP.com System Status History

Description

Update at 15:46:31(BST): This outage is now marked as resolved.

---

Update at 09:47:58(BST): The reboot has succeeded and the machine is back-up and running smoothly again.

We'll be checking all databases as a matter of course and shall continue to monitor the situation.

---

Update at 09:39:41(BST): Having no luck logging-in remotely, we have been forced to hard-reboot this machine. It is likely that the machine exhausted its swap memory during the surge.

The machine is coming back up now.

---

At present xenserve9 is unresponsive, causing issues with services. This is related to a surge in traffic that has caused a very high load.

We are working as quickly as possible to return the machine to a working state.

Start Time

18th May 2011, at 9:20AM

Description

Update at 17:29:23(BST): This outage is now marked as resolved.

---

Update at 17:10:52(BST): Both machines should now be functioning correctly once again.

---

At present xenserve9 is experiencing issues related to the MySQL process that is causing a number of services/websites to fail.

Coincidentally, xenserve17 is also experiencing a high level of traffic towards one or two websites, causing a high load that is making website loading particularly sluggish.

We hope to have both matters under control as soon as possible.

Start Time

17th May 2011, at 4:54PM

Description

Update at 09:28:27(BST): And as soon as we've posted, it appears that the developers have now resolved the issue.

---

We're aware of an issue affecting the speed of our main website, www.34sp.com and are working to resolve this.

Customers needing to manage their account are advised to visit https://account.34sp.com directly.

Thank you for your patience during this time.

Start Time

16th May 2011, at 9:26AM

Description

Update at 03:39:28(BST): Update at 18:09:43(BST): We continue to monitor the status of this machine but services are currently available.

Update at 17:41:55(BST): We are currently rebooting this server to clear the current issues..

We thank you for your patience and hope to have service restored as soon as possible.

---

We're currently investigating an issue that is affecting Business4. At present the machine is up and this looks more network related than anything.

More updates as they come.

Start Time

13th May 2011, at 5:19PM

Description

Update at 12:14:55(BST): Both these servers are back in service so this issue is now resolved

Update at 11:55:47(BST): Server14 and Server3 seem to have been the two servers the worst-affected and lingering load issues are unfortunately still continuing.

We're working to get this resolved ASAP.

---

We've been alerted to a number of servers with high loads. Upon further inspection this appears to have been an automated attempt to find insecure PHPMyAdmin installations on our servers.

Whilst the level of network traffic was not noticeably high, the requests to the web servers were driving-up the load placed on these servers, causing a loss of service for certain customers.

We have now blocked this IP at our network edge and will be contacting the appropriate abuse department.

Start Time

12th May 2011, at 11:34AM

Description

Update at 09:38:00(BST): This outage is now marked as resolved.

---

Update at 08:44:03(BST): Looking at the surge of incoming traffic, this was apparently a spate of DDoS traffic hitting a machine on the same switch as server12 & server14.

This has quickly subsided and we're working on identifying the target application.

---

We've noticed an issue that is affecting one of our switches and this currently appears to be resulting in no access to services on both xenserve12 and xenserve14.

We'll update as and when more information is available.

Start Time

6th May 2011, at 8:35AM

Description

Update at 10:46:52(BST): Update at 10:45:20(BST): The issue has now been resolved with this server.Fire alarm cleared and full service has now resumed. Apologies for any inconvenience caused. 10:03:15(BST): Support and Livechat are currently offline due to a firealarm and building evacuation.

Start Time

28th April 2011, at 10:20AM

Description

Update at 23:07:51(BST): The issue has now been resolved with this server.Update at 23:00:42(BST): Business8 has now been power cycled.

This server is presently offline. At this time we are investigating this fault.

Start Time

27th April 2011, at 10:31PM

Description

Update at 18:11:35(BST): Update at 21:36:31(BST): We are presently looking into a fault affecting this server.

Start Time

27th April 2011, at 7:41PM

Description

Update at 09:56:48(BST): This outage is now marked as resolved.

---

Customers with VPS' hosted by vps18.34sp.com may have to restart certain services after a number of issues resulted in the host itself running out of disk space.

This has since been resolved.

One of the VPS' present on the server had a malformed configuration file, which unfortunately invalidated the disk space quota -- allowing the VPS to thoroughly exceed its quota (more than the server can provide).

This mistake was not immediately apparent, given that under normal operation, the user's VPS did not exceed that of their provided disk space "limit". However, they today have been suffering DoS/injection attack-style traffic, which has pushed the size of their Apache error_log to well over 100GB.

Global fixes have been put in place for configuration files across ALL VPS hosts, to eliminate the same issue occurring elsewhere.

Similarly we will be re-evaluating the methods of monitoring disk space utilisation on these hosts (which is typically a very low percentage of available space).

34SP.com apologises to customers on this server, for the obvious inconvenience caused.

Any persons with lingering issues pertaining to this problem are advised to send an e-mail to support@34sp.com with the appropriate details.

Start Time

22nd April 2011, at 6:02PM