34SP.com System Status History

Description

Update at 00:24:08(GMT): The issue has now been resolved with this server.Server13 is about to be rebooted in order to apply several critical firmware updates.

Expected downtime is approximately 10 minutes.

Start Time

27th February 2012, at 11:45PM

Description

Update at 16:42:09(GMT): The issue has now been resolved with this server.Update at 15:28:38(GMT): The delivery issue has now been resolved and mail has been verified as delivering successfully. This means that any new mail sent to users on server5.xenserve.com, or mail sent from the server should be delivered as usual.

The large queue of spam mails generated between Saturday and Tuesday afternoon is still in the process of being cleaned before delivery to mailboxes on server5.xenserve.com. It currently stands at approximately 1,000,000 e-mails and we would expect this to reduce a lot before delivery to customers is made.

---

Update at 14:29:30(GMT): We are currently sending this response to any enquiries regarding the situation on server5.xenserve.com:

Last weekend a compromise related to the Plesk software used on our servers was exploited to send a vast mount of spam e-mail, clogging up the e-mail queues on the server5.xenserve.com machine. Approximately 4.4 million spam e-mails were sent. Once identified, we stopped the sending of further mail and began work to clear the spam from the queues. Due to the quantity of mail involved, this took a lot longer than acceptable, so on Tuesday the decision was taken to move the existing queue aside, with a mix of spam and legitimate mail in it. A fresh queue was created so that mail would flow as normal, while work to clean the original queue continued.

The new current mail queue is in place and being used my the mail processes on server5.xenserve.com but we have experienced further issues with delivery of e- mail. At this time we are uncertain as to the cause of these are working hard to understand and correct this issue.

During this time, e-mail should have been queued for eventual delivery, but we understand from some users that some e-mail is being bounced and returned to sender. We are also attempting to understand why this is happening and prevent it such that legitimate e-mail will reach the intended recipient.

---

Update at 10:49:05(GMT): The mail queue on server5.xenserve.com has reduced but is still present. We are still working to resolve this issue.

---

Update at 15:12:44(GMT): A further queue of mail has developed on server5.xenserve.com and we are working to resolve this matter at the moment.

---

Update at 08:52:00(GMT): It appears there is still a delay in customers receiving emails on this server. Our engineers have been made aware of this are continuing to investigate.

---

Update at 17:28:10(GMT): Email delivery is now working normally on server5.xenserve.com again.

---

Update at 16:50:05(GMT): Most email is working normally on server5.xenserve.com, but a minority of addresses still aren't accepting e-mail. We are working on this at the moment.

---

Update at 13:52:18(GMT): Unfortunately the changes made last night has not resolved the issue as we hoped and is still causing email problems. The engineers are still investigating into this and hope to have this resolved as soon as possible.

---

Update at 17:28:22(GMT): We are now expecting the original queue to finish processing on Thursday morning, and will move all legitimate delayed e-mail into the queues at that point.

---

Update at 16:56:00(GMT): A new fresh, empty mail queue on server5 has been created. All new email received will be processed in real time once again. That means you should see newer email flowing at normal, fast speeds.

The original queue has not been deleted, it is still being processed in parallel with the new queue, meaning that email received during the incident will still be delayed, but wont be lost. Of course, we are also working on ratcheting up the processing speeds too, as an ongoing process.

---

Update at 12:04:36(GMT): We have worked to speed up the processing of the queue on server5.xenserve.com and have had some progress. The queue is still being processed, now at an increased rate. It is still very substantial, however and is likely to take some time to process.

---

Update at 09:44:28(GMT): The mail queue on server3.xenserve.com has now been cleared so mail will operate normally for accounts on this server. The queue on server5.xenserve.com is still being processed.

---

Update at 20:03:28(GMT): Please note we are still running automated tools to delete the spam email, while ensuring legitimate email is not affected. Due to the large numbers of email the entire process is expected to run into the evening. Mail delivery will be delayed as a result, but all users should receive emails in due course.

---

Due to a compromised account on each of these servers, we are currently seeing very large mail queues in excess of 1,000,000 mails. We are in the process of cleaning the spam from these queues, but this will cause a delay to legitimate mail until the process is complete.

Start Time

27th February 2012, at 2:07PM

Description

Update at 10:50:24(GMT): We have identified this issue and resolved it. We will report the problem to our firewall providers so that they can implement a permanent fix

---

An automated update of the firewall for our business hosting package has failed and prevented site being served. We are correcting this now and will have all sites back online shortly.

Start Time

25th February 2012, at 9:45AM

Description

Update at 22:15:42(GMT): The mail backlog has now been cleared and full service restored.

---

Update at 15:07:41(GMT): An issue on our outgoing mail servers has been identified and resolved. We are currently processing the backlog of mail and expect all mail to be delivered in the next few hours.

---

We are currently investigating into a possible issue causing delays with mail being sent through our SMTP server array. Please accept our apologies for any inconvenience caused and our assurances we are doing all we can to get this resolved quickly.

Start Time

24th February 2012, at 1:31PM

Description

Update at 14:20:29(GMT): We have cleaned out a large queue of e-mail that had built up and have seen the server behave reliably this morning, so we are marking this issue as resolved.

Our automated monitoring systems will continue to monitor adam.34sp.com and alert us to any issues.

---

We are presently investigating a repeating fault that is affecting the adam.34sp.com personal server.

Start Time

22nd February 2012, at 11:12PM

Description

Update at 16:04:59(GMT): All containers have now restarted.

---

Update at 15:43:40(GMT): The server is coming back up now and is starting the individual VPS containers. We expect full service to all affected clients to be restored in the next 10 minutes or so.

---

The server vps18.34sp.com has become unresponsive and we are investigating this matter. As soon as we have any further information we will post it here.

Start Time

22nd February 2012, at 3:35PM

Description

Update at 19:39:27(GMT): The issue has now been resolved with this server.The server and all containers are now back online.

This server is about to be rebooted for some brief emergency maintenance.

Downtime is anticipated to be approximately 10 minutes or so.

Start Time

20th February 2012, at 7:09PM

Description

Update at 11:43:21(GMT): The issue has now been resolved with this server.The engineers have now resolved the issue that was causing the outage and all is not running back as it should.

We are currently experiencing issues with our vps18 server. Our engineers have been notified to the issue and are investigating. We hope to have resolved shortly.

Please accept our apologies for any inconvenience caused.

Start Time

18th February 2012, at 10:17AM

Description

Update at 08:48:41(GMT): The issue has now been resolved with this server.Our phone support system is presently offline. It will be accessible again when this status message is resolved.

Start Time

13th February 2012, at 6:12PM

Description

Update at 20:02:06(GMT): Update at 19:09:54(GMT): Update at 23:18:38(GMT): Due to a recently discovered Plesk vulnerability, several servers have had their siteadmin login details reset.

Affected servers include:

business: business4
reseller: reseller6,7,8,12
professional: prohost12
personal: k9,robbie,ted

If you have been affected, we will have emailed your registered email address with full details, and how to proceed.

Start Time

12th February 2012, at 11:18PM