34SP.com System Status History

Description

Update at 09:00:02(GMT): The issue has now been resolved with this server.The engineers have now resolved the issues that were causing the service failures. All services are back up and running again and under monitoring.

The server k9 appears be having trouble with the main Apache and MySQL services which will stop websites and email from working.

Our engineers have been notified and are investigating now. Please accept our apologies for any inconvenience caused during this time.

Start Time

16th March 2012, at 8:12AM

Description

Update at 14:46:48(GMT): The issue has now been resolved with this server.Reboot and update complete.



The server is undergoing a reboot and emergency maintenance . This was for security reasons. We apologise for any inconvenience caused.

Start Time

14th March 2012, at 1:38PM

Description

Update at 12:45:20(GMT): We are no longer experiencing any delays on the sending of SMTP mail.

---

Update at 15:50:51(GMT): Our tests are currently showing delays of no more than 10 minutes, but we continue to work on tracking this down.

---

We are currently looking into reports that mail sent via the SMTP (outgoing email) service in use for professional, business and reseller hosting is experiencing delays.

We will update further when we have made our investigations.

Start Time

7th March 2012, at 2:53PM

Description

Update at 10:19:43(GMT): The mail queue on adam has now been resolved and mail for users on this server will be processed normally.

---

Update at 14:55:22(GMT): The mail queue on adam has been significantly reduced and we are hopeful that normal service will be resumed today. We will post further updates as we have them.

---

We are presently clearing a large amount of spam email from the adam.34sp.com server mailqueue.

This is impacting the delivery of incoming emails to clients on this server.

Start Time

6th March 2012, at 10:10AM

Description

Update at 19:45:35(GMT): The issue has now been resolved with this server.Update at 19:29:21(GMT): All services should now be restored.

Update at 19:28:27(GMT): Our engineer has now restored most services. Any remaining issues will be addressed momentarily.

Update at 18:47:31(GMT): Our engineer is now on site to fix the last few remaining servers affected by this fault.

Update at 17:46:30(GMT): The following servers are still unavailable due to this network issue:

vps20
vps21

prohost10
prohost11
prohost12
prohost13

businesshost1

xenserve19
xenserve20

We have an engineer en route to the datacentre to fix this remaining set of servers now and will update this post again once this had been done.

---

Update at 16:43:03(GMT): Most services should be largely restored at this time. Several reseller servers are still unavailable, while we clean up the remainder of the attack. Again please note, that email sent to you during the outage should be queued by the sender for re-delivery and will not be lost.

---

Update at 15:37:21(GMT): The attack has been pinpointed to two of our switches. The engineers are working hard in cleaning any infected data.

This is affecting all services.

email services will be queued up awaiting delivery.

---

Update at 13:15:50(GMT): Unfortunately the issue has now escalated and appears to be affecting our entire network including our phone support systems. We apologise for any inconvenience caused and give our assurances we are doing everything possible to bring service back to normal.

---

Update at 12:48:15(GMT): Update at 11:27:25(GMT): Our engineers are currently resolving the issue and further information will be posted shortly.

---

Update at 10:53:44(GMT): The following servers are affected:

Business hosting
prohost10
prohost11
prohost13
rosie
vps20
vps21
vps22
xenserve16
xenserve18
xenserve19
xenserve20

We have an engineer on site at the data centre resolving this network issue. We apologise for the inconvenience caused and will update this post ASAP.

Start Time

5th March 2012, at 10:49AM

Description

Update at 09:50:41(GMT): The issue with this server is now resolved

---

Update at 15:33:23(GMT): A reboot needed a full disk check which took a while to fully sort. The server appears to be working fine now. We shall continue to monitor this.

---

We are currently investigating the unavailability of Business7 sever and will restore it to service as soon as possible.

Start Time

3rd March 2012, at 2:44PM

Description

Update at 14:04:44(GMT): The mail queue on max has now been cleared and we are seeing email delivered to mailboxes.

---

Update at 10:35:54(GMT): The queue reduced to approximately 25,000 mails, which then were held in the queue for a different reason. This has now been corrected and deliveries are happening. These should complete in the next two hours.

---

Update at 15:13:02(GMT): The queue is now under half a million emails and we anticipate mail returning to normal service tomorrow.

---

Update at 09:27:42(GMT): Work to clean the queue has continued overnight at the newly increased pace and we are seeing the count reduce very well. We expect to have mail flowing as normal again soon.

---

Update at 14:12:13(GMT): We have now brought the mail queue down to under 1 million e-mails and are looking at ways to reduce it further.

---

Update at 14:55:59(GMT): The mail queue on the max server is being reduced but is still very large. We will post further updates as we have them.

---

Update at 10:10:44(GMT): Work to reduce the mailqueue on max is continuing today.

---

Update at 09:51:26(GMT): The queues on the max server are reducing and we are looking at ways to accelerate this so that mail will be delivered normally again.

---

We are presently clearing a large amount of spam email from the max.34sp.com server mailqueue.

This is impacting the delivery of incoming emails to clients on this server.

Start Time

2nd March 2012, at 10:05PM

Description

Update at 13:34:51(GMT): Queues on the SMTP array have returned to nominal levels and there shouldn't be any delays remaining connected with the use of this service.

---

Update at 17:31:08(GMT): The queue on our SMTP array is reducing but still present. We expect this to resolve this evening and mail to be processed as quickly as normal.

---

Update at 09:55:13(GMT): This issue has now been resolved, and connections to our SMTP service are working correctly again. Mails that have been sent during the problems have been queued and will be delivered now, though there is a queue to be worked through so there will be a delay to emails in the queue.

---

We are aware that there is a queue of outbound mail on the SMTP array at the moment going back some 2 hours plus, resulting in some users being sent a notification email stating this.

Engineers are investigating the cause at present, and we will update status as soon as we have further details. There is no need to contact support or forward the notification email.

Start Time

29th February 2012, at 9:12AM

Description

Update at 16:42:18(GMT): The issue has now been resolved with this server.Update at 10:16:57(GMT): The VPS18 issue will be resolved today and the final moves are taking place.


Update at 15:05:18(GMT): We only have a small amount of containers remaining on VPS18. We are contacting these clients as and when we migrate.


Update at 10:15:37(GMT): The restoration of containers is taking longer than initially expected due to manual work involved. We're informing clients as and when their container is being moved.


Update at 17:36:52(GMT): Almost half of the clients have been contacted and VPS servers moved.


Update at 15:55:28(GMT): We have moved some containers and we're going through them one by one. Unfortunately this can be a slow process. We will be ringing customers before moving them.


Update at 12:44:05(GMT): As a precaution engineers are moving all containers from VPS18 to a separate server. This will prevent any further issues. Customers will be contacted in due course.


Update at 10:23:55(GMT): We are currently going through each VPS container and contacting the owner. We will be updating them while we work on restoring their services.


Update at 09:32:27(GMT): The server has had firmware updates, however some data loss may have occurred during the reboot. Our engineers are looking into this. We will be updating this page as soon as we have any further information.


A fault was identified with the server this evening. It was rebooted a little after 11PM and is presently undergoing disk scans before coming back up.

We do realise this is a repeating fault. Please be advised replacement hardware is on order and set for installation and migration imminently.

Start Time

28th February 2012, at 11:44PM

Description

Update at 13:34:06(GMT): After 48 hours of close monitoring, we are resolving this status.

---

Update at 13:36:19(GMT): We have applied some firmware updates to this server which should help the stability, but will be monitoring it closely with a view to installing further updates is the server stability does not improve.

---

We are currently investigating the unavailability of server17.xenserve.com and will restore it to service as soon as possible.

Start Time

28th February 2012, at 12:35PM