34SP.com System Status History

Description

Update at 15:26:23(BST): This maintenance was completed as planned last night.

---

Update at 00:09:01(BST): All email services have now been restored again.

---

Update at 23:58:29(BST): Email is currently offline - downtime expected to be less than 10minutes

---

Update at 23:45:45(BST): The mail is about to go offline, this will affect all users that are on the professional/business(new plan) and those using spamfiltering on the older personal accounts.

---

At 11PM tonight we will be taking the Atmail system briefly offline to perform some emergency maintenance tasks.

The maintenance work will allow us to deploy additional capacity in order to improve general services.

Maintenance is expected to take 10-15 minutes at most. During this time you will not be able to access email. Incoming email will not be bounced/lost during the procedure, senders will queue to resend later.

Start Time

27th March 2012, at 9:12PM

Description

Update at 09:59:11(BST): All appears to be working well since the reboot.


Update at 17:06:49(GMT): The server is back up and responding well once more. We'll continue to check this for any further issues.


Server16 has become unresponsive, so we're currently performing a reboot.

We expect the services to be back in a few minutes and we'll continue to monitor it once back.

Start Time

23rd March 2012, at 5:02PM

Description

Update at 12:08:13(GMT): The caching issues should now be resolved and we are seeing mail flow as normal through the servers.

---

Update at 09:58:16(GMT): We have resolved this issue but due to caching issues some customer may continue to see problems for the next couple of hours.

---

We are receiving reports from some customers that they are receiving a "451 temporary error" when trying to send mail both via SMTP and webmail. Engineers are checking into this at the moment - more details to follow.

Start Time

21st March 2012, at 9:44AM

Description

Update at 09:00:02(GMT): The issue has now been resolved with this server.The engineers have now resolved the issues that were causing the service failures. All services are back up and running again and under monitoring.

The server k9 appears be having trouble with the main Apache and MySQL services which will stop websites and email from working.

Our engineers have been notified and are investigating now. Please accept our apologies for any inconvenience caused during this time.

Start Time

16th March 2012, at 8:12AM

Description

Update at 14:46:48(GMT): The issue has now been resolved with this server.Reboot and update complete.



The server is undergoing a reboot and emergency maintenance . This was for security reasons. We apologise for any inconvenience caused.

Start Time

14th March 2012, at 1:38PM

Description

Update at 12:45:20(GMT): We are no longer experiencing any delays on the sending of SMTP mail.

---

Update at 15:50:51(GMT): Our tests are currently showing delays of no more than 10 minutes, but we continue to work on tracking this down.

---

We are currently looking into reports that mail sent via the SMTP (outgoing email) service in use for professional, business and reseller hosting is experiencing delays.

We will update further when we have made our investigations.

Start Time

7th March 2012, at 2:53PM

Description

Update at 10:19:43(GMT): The mail queue on adam has now been resolved and mail for users on this server will be processed normally.

---

Update at 14:55:22(GMT): The mail queue on adam has been significantly reduced and we are hopeful that normal service will be resumed today. We will post further updates as we have them.

---

We are presently clearing a large amount of spam email from the adam.34sp.com server mailqueue.

This is impacting the delivery of incoming emails to clients on this server.

Start Time

6th March 2012, at 10:10AM

Description

Update at 19:45:35(GMT): The issue has now been resolved with this server.Update at 19:29:21(GMT): All services should now be restored.

Update at 19:28:27(GMT): Our engineer has now restored most services. Any remaining issues will be addressed momentarily.

Update at 18:47:31(GMT): Our engineer is now on site to fix the last few remaining servers affected by this fault.

Update at 17:46:30(GMT): The following servers are still unavailable due to this network issue:

vps20
vps21

prohost10
prohost11
prohost12
prohost13

businesshost1

xenserve19
xenserve20

We have an engineer en route to the datacentre to fix this remaining set of servers now and will update this post again once this had been done.

---

Update at 16:43:03(GMT): Most services should be largely restored at this time. Several reseller servers are still unavailable, while we clean up the remainder of the attack. Again please note, that email sent to you during the outage should be queued by the sender for re-delivery and will not be lost.

---

Update at 15:37:21(GMT): The attack has been pinpointed to two of our switches. The engineers are working hard in cleaning any infected data.

This is affecting all services.

email services will be queued up awaiting delivery.

---

Update at 13:15:50(GMT): Unfortunately the issue has now escalated and appears to be affecting our entire network including our phone support systems. We apologise for any inconvenience caused and give our assurances we are doing everything possible to bring service back to normal.

---

Update at 12:48:15(GMT): Update at 11:27:25(GMT): Our engineers are currently resolving the issue and further information will be posted shortly.

---

Update at 10:53:44(GMT): The following servers are affected:

Business hosting
prohost10
prohost11
prohost13
rosie
vps20
vps21
vps22
xenserve16
xenserve18
xenserve19
xenserve20

We have an engineer on site at the data centre resolving this network issue. We apologise for the inconvenience caused and will update this post ASAP.

Start Time

5th March 2012, at 10:49AM

Description

Update at 09:50:41(GMT): The issue with this server is now resolved

---

Update at 15:33:23(GMT): A reboot needed a full disk check which took a while to fully sort. The server appears to be working fine now. We shall continue to monitor this.

---

We are currently investigating the unavailability of Business7 sever and will restore it to service as soon as possible.

Start Time

3rd March 2012, at 2:44PM

Description

Update at 14:04:44(GMT): The mail queue on max has now been cleared and we are seeing email delivered to mailboxes.

---

Update at 10:35:54(GMT): The queue reduced to approximately 25,000 mails, which then were held in the queue for a different reason. This has now been corrected and deliveries are happening. These should complete in the next two hours.

---

Update at 15:13:02(GMT): The queue is now under half a million emails and we anticipate mail returning to normal service tomorrow.

---

Update at 09:27:42(GMT): Work to clean the queue has continued overnight at the newly increased pace and we are seeing the count reduce very well. We expect to have mail flowing as normal again soon.

---

Update at 14:12:13(GMT): We have now brought the mail queue down to under 1 million e-mails and are looking at ways to reduce it further.

---

Update at 14:55:59(GMT): The mail queue on the max server is being reduced but is still very large. We will post further updates as we have them.

---

Update at 10:10:44(GMT): Work to reduce the mailqueue on max is continuing today.

---

Update at 09:51:26(GMT): The queues on the max server are reducing and we are looking at ways to accelerate this so that mail will be delivered normally again.

---

We are presently clearing a large amount of spam email from the max.34sp.com server mailqueue.

This is impacting the delivery of incoming emails to clients on this server.

Start Time

2nd March 2012, at 10:05PM