34SP.com System Status History

Description

Update at 12:14:55(BST): Both these servers are back in service so this issue is now resolved

Update at 11:55:47(BST): Server14 and Server3 seem to have been the two servers the worst-affected and lingering load issues are unfortunately still continuing.

We're working to get this resolved ASAP.

---

We've been alerted to a number of servers with high loads. Upon further inspection this appears to have been an automated attempt to find insecure PHPMyAdmin installations on our servers.

Whilst the level of network traffic was not noticeably high, the requests to the web servers were driving-up the load placed on these servers, causing a loss of service for certain customers.

We have now blocked this IP at our network edge and will be contacting the appropriate abuse department.

Start Time

12th May 2011, at 11:34AM

Description

Update at 09:38:00(BST): This outage is now marked as resolved.

---

Update at 08:44:03(BST): Looking at the surge of incoming traffic, this was apparently a spate of DDoS traffic hitting a machine on the same switch as server12 & server14.

This has quickly subsided and we're working on identifying the target application.

---

We've noticed an issue that is affecting one of our switches and this currently appears to be resulting in no access to services on both xenserve12 and xenserve14.

We'll update as and when more information is available.

Start Time

6th May 2011, at 8:35AM

Description

Update at 10:46:52(BST): Update at 10:45:20(BST): The issue has now been resolved with this server.Fire alarm cleared and full service has now resumed. Apologies for any inconvenience caused. 10:03:15(BST): Support and Livechat are currently offline due to a firealarm and building evacuation.

Start Time

28th April 2011, at 10:20AM

Description

Update at 23:07:51(BST): The issue has now been resolved with this server.Update at 23:00:42(BST): Business8 has now been power cycled.

This server is presently offline. At this time we are investigating this fault.

Start Time

27th April 2011, at 10:31PM

Description

Update at 18:11:35(BST): Update at 21:36:31(BST): We are presently looking into a fault affecting this server.

Start Time

27th April 2011, at 7:41PM

Description

Update at 09:56:48(BST): This outage is now marked as resolved.

---

Customers with VPS' hosted by vps18.34sp.com may have to restart certain services after a number of issues resulted in the host itself running out of disk space.

This has since been resolved.

One of the VPS' present on the server had a malformed configuration file, which unfortunately invalidated the disk space quota -- allowing the VPS to thoroughly exceed its quota (more than the server can provide).

This mistake was not immediately apparent, given that under normal operation, the user's VPS did not exceed that of their provided disk space "limit". However, they today have been suffering DoS/injection attack-style traffic, which has pushed the size of their Apache error_log to well over 100GB.

Global fixes have been put in place for configuration files across ALL VPS hosts, to eliminate the same issue occurring elsewhere.

Similarly we will be re-evaluating the methods of monitoring disk space utilisation on these hosts (which is typically a very low percentage of available space).

34SP.com apologises to customers on this server, for the obvious inconvenience caused.

Any persons with lingering issues pertaining to this problem are advised to send an e-mail to support@34sp.com with the appropriate details.

Start Time

22nd April 2011, at 6:02PM

Description

Update at 12:01:33(BST): The issue has now been resolved with this server.We would like to remind customers that phone and live chat cover is NOT available today, Good Friday, or on any of the upcoming Bank Holidays. Support is limited to email ONLY on these days. Full service is still available on weekends (Saturdays and Sundays), but is restricted to email on the following days:

22nd April (Good Friday)
25th April (Easter Monday)
29th April (Royal Wedding)
2nd May (May Day)
30th May (Spring Bank Hol)
29th August (Summer Bank Hol)

Start Time

22nd April 2011, at 1:15PM

Description

Update at 09:48:15(BST): This issue is now marked as resolved.

---

Update at 07:20:36(BST): Re-ordering previous update.

---

Update at 00:06:22(BST):
All services went back online around 15minutes ago, we are currently monitoring things incase of any issues from this update.

---

At present the Atmail ("Spam Filtering") mail system is stored across two machines. The storage (and performance) capacity of these machines is reaching its limits and we need to begin spreading the data across an extra server.

These actions also have the added side-effect of improving performance, by spreading the load placed upon the cluster, across more machines.

This work will commence in approximately 10 minutes and customers will experience no more than 20 minutes of downtime.

During this time any e-mail to be delivered will be queued-up and re-delivered after the maintenance is completed.

Sending e-mail via our SMTP service will not be affected.

Start Time

19th April 2011, at 11:22PM

Description

Update at 04:26:09(BST):
As per all other switch upgrades, switch3 was down and up very quickly.

This now means that all of our older switches are patched-up.

All further switches will require updates in the near future, but with the benefits brought about by forward-planning.

---

Update at 04:04:32(BST): Switch3 is also affected. We have unfortunately had to wait to upgrade this switch whilst certain backup routines were running.

There will be a momentary loss of all services served via switch3. This will amount to no more than around 20-30 seconds.

---

Update at 02:57:05(BST): The upgrade of switch6 is now completed.

---

Update at 02:53:58(BST): Another switch has exactly the same firmware version that was affected. We're reloading this one now, whilst traffic levels through it are relatively low (particularly given the time of day).

This will cause disruption to many services, but should only last a few seconds.

---

Update at 02:51:48(BST): Switch1 reloaded with new software.

We may need to reload other switches with upgrades (although this isn't quite as urgent; a number will be susceptible to the same issue) so this notice will remain and be updated if required.

---

Update at 02:45:59(BST): Software loaded and checksummed. Reloading the switch again.

---

Update at 02:39:47(BST): Switch rebooted. Now loading the new software.

---

We have identified an issue with one of our switches, which may affect a number of others running the same software release.

As a matter of urgency, we are having to restart one of our switches in London. Multiple services will be affected for a number of seconds, whilst the switch reboots.

Furthermore, we will be rebooting the switch once more, with a newer software version from Cisco. This will hopefully ensure that the same issue does not resurface in the future.

We thank you for your patience and co-operation in this matter.

Start Time

14th April 2011, at 2:34AM

Description

Update at 00:39:10(BST): The issue has now been resolved with this server.

---

Update at 23:55:56(BST): Machine rebooted, now coming back up.

---

As a reminder to Reseller customers on Xenserve9, there will be a momentary loss to all services on this machine as we need to shut it down in order to move to a new power controller.

This will be happening in the next few minutes and should take no longer than 20 minutes maximum.

Start Time

13th April 2011, at 11:47PM