34SP.com System Status History

Description

Update at 09:59:11(BST): All appears to be working well since the reboot.


Update at 17:06:49(GMT): The server is back up and responding well once more. We'll continue to check this for any further issues.


Server16 has become unresponsive, so we're currently performing a reboot.

We expect the services to be back in a few minutes and we'll continue to monitor it once back.

Start Time

23rd March 2012, at 5:02PM

Description

Update at 12:08:13(GMT): The caching issues should now be resolved and we are seeing mail flow as normal through the servers.

---

Update at 09:58:16(GMT): We have resolved this issue but due to caching issues some customer may continue to see problems for the next couple of hours.

---

We are receiving reports from some customers that they are receiving a "451 temporary error" when trying to send mail both via SMTP and webmail. Engineers are checking into this at the moment - more details to follow.

Start Time

21st March 2012, at 9:44AM

Description

Update at 09:00:02(GMT): The issue has now been resolved with this server.The engineers have now resolved the issues that were causing the service failures. All services are back up and running again and under monitoring.

The server k9 appears be having trouble with the main Apache and MySQL services which will stop websites and email from working.

Our engineers have been notified and are investigating now. Please accept our apologies for any inconvenience caused during this time.

Start Time

16th March 2012, at 8:12AM

Description

Update at 14:46:48(GMT): The issue has now been resolved with this server.Reboot and update complete.



The server is undergoing a reboot and emergency maintenance . This was for security reasons. We apologise for any inconvenience caused.

Start Time

14th March 2012, at 1:38PM

Description

Update at 12:45:20(GMT): We are no longer experiencing any delays on the sending of SMTP mail.

---

Update at 15:50:51(GMT): Our tests are currently showing delays of no more than 10 minutes, but we continue to work on tracking this down.

---

We are currently looking into reports that mail sent via the SMTP (outgoing email) service in use for professional, business and reseller hosting is experiencing delays.

We will update further when we have made our investigations.

Start Time

7th March 2012, at 2:53PM

Description

Update at 10:19:43(GMT): The mail queue on adam has now been resolved and mail for users on this server will be processed normally.

---

Update at 14:55:22(GMT): The mail queue on adam has been significantly reduced and we are hopeful that normal service will be resumed today. We will post further updates as we have them.

---

We are presently clearing a large amount of spam email from the adam.34sp.com server mailqueue.

This is impacting the delivery of incoming emails to clients on this server.

Start Time

6th March 2012, at 10:10AM

Description

Update at 19:45:35(GMT): The issue has now been resolved with this server.Update at 19:29:21(GMT): All services should now be restored.

Update at 19:28:27(GMT): Our engineer has now restored most services. Any remaining issues will be addressed momentarily.

Update at 18:47:31(GMT): Our engineer is now on site to fix the last few remaining servers affected by this fault.

Update at 17:46:30(GMT): The following servers are still unavailable due to this network issue:

vps20
vps21

prohost10
prohost11
prohost12
prohost13

businesshost1

xenserve19
xenserve20

We have an engineer en route to the datacentre to fix this remaining set of servers now and will update this post again once this had been done.

---

Update at 16:43:03(GMT): Most services should be largely restored at this time. Several reseller servers are still unavailable, while we clean up the remainder of the attack. Again please note, that email sent to you during the outage should be queued by the sender for re-delivery and will not be lost.

---

Update at 15:37:21(GMT): The attack has been pinpointed to two of our switches. The engineers are working hard in cleaning any infected data.

This is affecting all services.

email services will be queued up awaiting delivery.

---

Update at 13:15:50(GMT): Unfortunately the issue has now escalated and appears to be affecting our entire network including our phone support systems. We apologise for any inconvenience caused and give our assurances we are doing everything possible to bring service back to normal.

---

Update at 12:48:15(GMT): Update at 11:27:25(GMT): Our engineers are currently resolving the issue and further information will be posted shortly.

---

Update at 10:53:44(GMT): The following servers are affected:

Business hosting
prohost10
prohost11
prohost13
rosie
vps20
vps21
vps22
xenserve16
xenserve18
xenserve19
xenserve20

We have an engineer on site at the data centre resolving this network issue. We apologise for the inconvenience caused and will update this post ASAP.

Start Time

5th March 2012, at 10:49AM

Description

Update at 09:50:41(GMT): The issue with this server is now resolved

---

Update at 15:33:23(GMT): A reboot needed a full disk check which took a while to fully sort. The server appears to be working fine now. We shall continue to monitor this.

---

We are currently investigating the unavailability of Business7 sever and will restore it to service as soon as possible.

Start Time

3rd March 2012, at 2:44PM

Description

Update at 14:04:44(GMT): The mail queue on max has now been cleared and we are seeing email delivered to mailboxes.

---

Update at 10:35:54(GMT): The queue reduced to approximately 25,000 mails, which then were held in the queue for a different reason. This has now been corrected and deliveries are happening. These should complete in the next two hours.

---

Update at 15:13:02(GMT): The queue is now under half a million emails and we anticipate mail returning to normal service tomorrow.

---

Update at 09:27:42(GMT): Work to clean the queue has continued overnight at the newly increased pace and we are seeing the count reduce very well. We expect to have mail flowing as normal again soon.

---

Update at 14:12:13(GMT): We have now brought the mail queue down to under 1 million e-mails and are looking at ways to reduce it further.

---

Update at 14:55:59(GMT): The mail queue on the max server is being reduced but is still very large. We will post further updates as we have them.

---

Update at 10:10:44(GMT): Work to reduce the mailqueue on max is continuing today.

---

Update at 09:51:26(GMT): The queues on the max server are reducing and we are looking at ways to accelerate this so that mail will be delivered normally again.

---

We are presently clearing a large amount of spam email from the max.34sp.com server mailqueue.

This is impacting the delivery of incoming emails to clients on this server.

Start Time

2nd March 2012, at 10:05PM

Description

Update at 13:34:51(GMT): Queues on the SMTP array have returned to nominal levels and there shouldn't be any delays remaining connected with the use of this service.

---

Update at 17:31:08(GMT): The queue on our SMTP array is reducing but still present. We expect this to resolve this evening and mail to be processed as quickly as normal.

---

Update at 09:55:13(GMT): This issue has now been resolved, and connections to our SMTP service are working correctly again. Mails that have been sent during the problems have been queued and will be delivered now, though there is a queue to be worked through so there will be a delay to emails in the queue.

---

We are aware that there is a queue of outbound mail on the SMTP array at the moment going back some 2 hours plus, resulting in some users being sent a notification email stating this.

Engineers are investigating the cause at present, and we will update status as soon as we have further details. There is no need to contact support or forward the notification email.

Start Time

29th February 2012, at 9:12AM