34SP.com System Status History

Description

Update at 18:11:35(BST): Update at 21:36:31(BST): We are presently looking into a fault affecting this server.

Start Time

27th April 2011, at 7:41PM

Description

Update at 09:56:48(BST): This outage is now marked as resolved.

---

Customers with VPS' hosted by vps18.34sp.com may have to restart certain services after a number of issues resulted in the host itself running out of disk space.

This has since been resolved.

One of the VPS' present on the server had a malformed configuration file, which unfortunately invalidated the disk space quota -- allowing the VPS to thoroughly exceed its quota (more than the server can provide).

This mistake was not immediately apparent, given that under normal operation, the user's VPS did not exceed that of their provided disk space "limit". However, they today have been suffering DoS/injection attack-style traffic, which has pushed the size of their Apache error_log to well over 100GB.

Global fixes have been put in place for configuration files across ALL VPS hosts, to eliminate the same issue occurring elsewhere.

Similarly we will be re-evaluating the methods of monitoring disk space utilisation on these hosts (which is typically a very low percentage of available space).

34SP.com apologises to customers on this server, for the obvious inconvenience caused.

Any persons with lingering issues pertaining to this problem are advised to send an e-mail to support@34sp.com with the appropriate details.

Start Time

22nd April 2011, at 6:02PM

Description

Update at 12:01:33(BST): The issue has now been resolved with this server.We would like to remind customers that phone and live chat cover is NOT available today, Good Friday, or on any of the upcoming Bank Holidays. Support is limited to email ONLY on these days. Full service is still available on weekends (Saturdays and Sundays), but is restricted to email on the following days:

22nd April (Good Friday)
25th April (Easter Monday)
29th April (Royal Wedding)
2nd May (May Day)
30th May (Spring Bank Hol)
29th August (Summer Bank Hol)

Start Time

22nd April 2011, at 1:15PM

Description

Update at 09:48:15(BST): This issue is now marked as resolved.

---

Update at 07:20:36(BST): Re-ordering previous update.

---

Update at 00:06:22(BST):
All services went back online around 15minutes ago, we are currently monitoring things incase of any issues from this update.

---

At present the Atmail ("Spam Filtering") mail system is stored across two machines. The storage (and performance) capacity of these machines is reaching its limits and we need to begin spreading the data across an extra server.

These actions also have the added side-effect of improving performance, by spreading the load placed upon the cluster, across more machines.

This work will commence in approximately 10 minutes and customers will experience no more than 20 minutes of downtime.

During this time any e-mail to be delivered will be queued-up and re-delivered after the maintenance is completed.

Sending e-mail via our SMTP service will not be affected.

Start Time

19th April 2011, at 11:22PM

Description

Update at 04:26:09(BST):
As per all other switch upgrades, switch3 was down and up very quickly.

This now means that all of our older switches are patched-up.

All further switches will require updates in the near future, but with the benefits brought about by forward-planning.

---

Update at 04:04:32(BST): Switch3 is also affected. We have unfortunately had to wait to upgrade this switch whilst certain backup routines were running.

There will be a momentary loss of all services served via switch3. This will amount to no more than around 20-30 seconds.

---

Update at 02:57:05(BST): The upgrade of switch6 is now completed.

---

Update at 02:53:58(BST): Another switch has exactly the same firmware version that was affected. We're reloading this one now, whilst traffic levels through it are relatively low (particularly given the time of day).

This will cause disruption to many services, but should only last a few seconds.

---

Update at 02:51:48(BST): Switch1 reloaded with new software.

We may need to reload other switches with upgrades (although this isn't quite as urgent; a number will be susceptible to the same issue) so this notice will remain and be updated if required.

---

Update at 02:45:59(BST): Software loaded and checksummed. Reloading the switch again.

---

Update at 02:39:47(BST): Switch rebooted. Now loading the new software.

---

We have identified an issue with one of our switches, which may affect a number of others running the same software release.

As a matter of urgency, we are having to restart one of our switches in London. Multiple services will be affected for a number of seconds, whilst the switch reboots.

Furthermore, we will be rebooting the switch once more, with a newer software version from Cisco. This will hopefully ensure that the same issue does not resurface in the future.

We thank you for your patience and co-operation in this matter.

Start Time

14th April 2011, at 2:34AM

Description

Update at 00:39:10(BST): The issue has now been resolved with this server.

---

Update at 23:55:56(BST): Machine rebooted, now coming back up.

---

As a reminder to Reseller customers on Xenserve9, there will be a momentary loss to all services on this machine as we need to shut it down in order to move to a new power controller.

This will be happening in the next few minutes and should take no longer than 20 minutes maximum.

Start Time

13th April 2011, at 11:47PM

Description

Update at 15:43:48(BST): The issue has now been resolved with this server.The server reboot has fixed this issue - all sites should now be displaying OK.

Update at 15:42:59(BST): The apache web service is currently not responding. Engineers have restarted this with no effect. They are currently attempting to reboot the server to see if this fixes the issue - we anticipate that this will take at least 10-15 minutes after which we will be in a better position to advise further.

Start Time

13th April 2011, at 3:33PM

Description

Update at 22:38:54(BST): This outage is now set to resolved.

---

Update at 08:24:43(BST): 08:25 - Our engineers have now resolved the issue at hand so mail should now be working both on webmail.youdomain.com and using POP/IMAP connections.

We server is now being monitored to ensure the service is running as it should.

07:45 - We are currently experiencing issues with one our webmail servers, this will be affecting the receiving of mail both via webmail.yourdomain.com and also using POP/IMAP connections.

We have notified an engineer and they are looking into the issue now. As soon as we have further updates we will update this status. Please accept our apologies for any inconvenience caused.

Start Time

13th April 2011, at 7:57AM

Description

Update at 11:39:03(BST): Server3 appears to be running normally once again. We do still have every intention of replacing this hardware as soon as is feasible.

---

Update at 10:07:30(BST): The machine is now back up again. We're now checking that all is well.

---

Update at 09:58:50(BST): Server3 has gone down again and is currently rebooting. This machine is scheduled for replacement shortly and customers will be notified as soon as this can take place.

---

Update at 00:39:43(BST): The outage is now marked as resolved.

---

Update at 22:38:28(BST): Xenserve3 is now back online in its new location. We're just checking the databases over and will keep an eye on the machine over the next few hours.

---

Update at 22:11:13(BST): As we're on-site we've noticed that xenserve3 appears to be having a recurring issue with heat and this may well explain the increased instability.

We're going to move this machine to a rack with better cooling and less equipment in it. This should take no more than about 20 minutes.

We apologise sincerely for the obvious inconvenience; we'll be working to find a better solution in the near future.

---

Update at 22:36:22(BST): The issue has now been resolved with this server.The system is now up and services are being restored.

Update at 22:01:56(BST): The system has become unresponsive once more, we are presently power cycling the system

---

Update at 09:59:35(BST): This outage is now marked as resolved.

---

Update at 09:34:28(BST): Xenserve3 is back up -- we're currently checking all databases to ensure consistency.

---

Update at 09:19:00(BST): The machine has had to be rebooted. We're just watching it come back online.

---

We're currently aware that xenserve3 is not responding on a number of services. The machine is still up, but we're having trouble gaining access remotely.

We apologise for any inconvenience caused and hope to have this resolved shortly.

Start Time

11th April 2011, at 9:04AM

Description

Update at 12:13:50(BST): The issue has now been resolved with this server.

Update at 11:41:46(BST): An engineer is en route to this server to investigate the problem.

server9.xenserve.com is currently unavailable. We are investigating this as a matter of urgency and hope to restore functionality soon.

Start Time

8th April 2011, at 11:27AM