34SP.com System Status History

Description

Update at 09:48:15(BST): This issue is now marked as resolved.

---

Update at 07:20:36(BST): Re-ordering previous update.

---

Update at 00:06:22(BST):
All services went back online around 15minutes ago, we are currently monitoring things incase of any issues from this update.

---

At present the Atmail ("Spam Filtering") mail system is stored across two machines. The storage (and performance) capacity of these machines is reaching its limits and we need to begin spreading the data across an extra server.

These actions also have the added side-effect of improving performance, by spreading the load placed upon the cluster, across more machines.

This work will commence in approximately 10 minutes and customers will experience no more than 20 minutes of downtime.

During this time any e-mail to be delivered will be queued-up and re-delivered after the maintenance is completed.

Sending e-mail via our SMTP service will not be affected.

Start Time

19th April 2011, at 11:22PM

Description

Update at 04:26:09(BST):
As per all other switch upgrades, switch3 was down and up very quickly.

This now means that all of our older switches are patched-up.

All further switches will require updates in the near future, but with the benefits brought about by forward-planning.

---

Update at 04:04:32(BST): Switch3 is also affected. We have unfortunately had to wait to upgrade this switch whilst certain backup routines were running.

There will be a momentary loss of all services served via switch3. This will amount to no more than around 20-30 seconds.

---

Update at 02:57:05(BST): The upgrade of switch6 is now completed.

---

Update at 02:53:58(BST): Another switch has exactly the same firmware version that was affected. We're reloading this one now, whilst traffic levels through it are relatively low (particularly given the time of day).

This will cause disruption to many services, but should only last a few seconds.

---

Update at 02:51:48(BST): Switch1 reloaded with new software.

We may need to reload other switches with upgrades (although this isn't quite as urgent; a number will be susceptible to the same issue) so this notice will remain and be updated if required.

---

Update at 02:45:59(BST): Software loaded and checksummed. Reloading the switch again.

---

Update at 02:39:47(BST): Switch rebooted. Now loading the new software.

---

We have identified an issue with one of our switches, which may affect a number of others running the same software release.

As a matter of urgency, we are having to restart one of our switches in London. Multiple services will be affected for a number of seconds, whilst the switch reboots.

Furthermore, we will be rebooting the switch once more, with a newer software version from Cisco. This will hopefully ensure that the same issue does not resurface in the future.

We thank you for your patience and co-operation in this matter.

Start Time

14th April 2011, at 2:34AM

Description

Update at 00:39:10(BST): The issue has now been resolved with this server.

---

Update at 23:55:56(BST): Machine rebooted, now coming back up.

---

As a reminder to Reseller customers on Xenserve9, there will be a momentary loss to all services on this machine as we need to shut it down in order to move to a new power controller.

This will be happening in the next few minutes and should take no longer than 20 minutes maximum.

Start Time

13th April 2011, at 11:47PM

Description

Update at 15:43:48(BST): The issue has now been resolved with this server.The server reboot has fixed this issue - all sites should now be displaying OK.

Update at 15:42:59(BST): The apache web service is currently not responding. Engineers have restarted this with no effect. They are currently attempting to reboot the server to see if this fixes the issue - we anticipate that this will take at least 10-15 minutes after which we will be in a better position to advise further.

Start Time

13th April 2011, at 3:33PM

Description

Update at 22:38:54(BST): This outage is now set to resolved.

---

Update at 08:24:43(BST): 08:25 - Our engineers have now resolved the issue at hand so mail should now be working both on webmail.youdomain.com and using POP/IMAP connections.

We server is now being monitored to ensure the service is running as it should.

07:45 - We are currently experiencing issues with one our webmail servers, this will be affecting the receiving of mail both via webmail.yourdomain.com and also using POP/IMAP connections.

We have notified an engineer and they are looking into the issue now. As soon as we have further updates we will update this status. Please accept our apologies for any inconvenience caused.

Start Time

13th April 2011, at 7:57AM

Description

Update at 11:39:03(BST): Server3 appears to be running normally once again. We do still have every intention of replacing this hardware as soon as is feasible.

---

Update at 10:07:30(BST): The machine is now back up again. We're now checking that all is well.

---

Update at 09:58:50(BST): Server3 has gone down again and is currently rebooting. This machine is scheduled for replacement shortly and customers will be notified as soon as this can take place.

---

Update at 00:39:43(BST): The outage is now marked as resolved.

---

Update at 22:38:28(BST): Xenserve3 is now back online in its new location. We're just checking the databases over and will keep an eye on the machine over the next few hours.

---

Update at 22:11:13(BST): As we're on-site we've noticed that xenserve3 appears to be having a recurring issue with heat and this may well explain the increased instability.

We're going to move this machine to a rack with better cooling and less equipment in it. This should take no more than about 20 minutes.

We apologise sincerely for the obvious inconvenience; we'll be working to find a better solution in the near future.

---

Update at 22:36:22(BST): The issue has now been resolved with this server.The system is now up and services are being restored.

Update at 22:01:56(BST): The system has become unresponsive once more, we are presently power cycling the system

---

Update at 09:59:35(BST): This outage is now marked as resolved.

---

Update at 09:34:28(BST): Xenserve3 is back up -- we're currently checking all databases to ensure consistency.

---

Update at 09:19:00(BST): The machine has had to be rebooted. We're just watching it come back online.

---

We're currently aware that xenserve3 is not responding on a number of services. The machine is still up, but we're having trouble gaining access remotely.

We apologise for any inconvenience caused and hope to have this resolved shortly.

Start Time

11th April 2011, at 9:04AM

Description

Update at 12:13:50(BST): The issue has now been resolved with this server.

Update at 11:41:46(BST): An engineer is en route to this server to investigate the problem.

server9.xenserve.com is currently unavailable. We are investigating this as a matter of urgency and hope to restore functionality soon.

Start Time

8th April 2011, at 11:27AM

Description

Update at 15:55:30(BST):
All support operatives are now back in their seats. Thank you for your patience!

---

We have currently closed our support lines (telephone/live chats) to facilitate a full staff meeting, over the next 30-40 minutes.

Customers are advised to send an e-mail to support@34sp.com and we will get back to you as soon as possible.

Start Time

1st April 2011, at 3:32PM

Description

Update at 12:06:04(BST): We're satisfied that this was an isolated incident and will now resolve this outage.

---

Update at 10:53:26(BST): We have had confirmation from the facility in Manchester that 'some random devices tripped' during an un-planned power issue.

We will continue to investigate as to why this was possible, but it appears to be a very unfortunate problem.

---

Update at 10:23:12(BST): Servers are both back up now and services should be running.

This appears to be related to a power issue in our Manchester facility and we're still working to identify the exact cause.

---

Both servers went down and are currently being rebooted, we're looking into why this has happened.

Start Time

30th March 2011, at 10:12AM

Description

Update at 12:25:48(BST): The issue has now been resolved with this server.

---

We're having a few issues with Business7 this morning that are looking increasingly like only a reboot will clear such cobwebs out.

At the same time, we have a number of 'important security updates' to apply.

The reboot will take no more than around 10 minutes and e-mail services will not be affected.

We apologise for the inconvenience caused by having to do this during the day.

Start Time

29th March 2011, at 12:09PM