34SP.com System Status History

Description

Update at 13:34:51(GMT): Queues on the SMTP array have returned to nominal levels and there shouldn't be any delays remaining connected with the use of this service.

---

Update at 17:31:08(GMT): The queue on our SMTP array is reducing but still present. We expect this to resolve this evening and mail to be processed as quickly as normal.

---

Update at 09:55:13(GMT): This issue has now been resolved, and connections to our SMTP service are working correctly again. Mails that have been sent during the problems have been queued and will be delivered now, though there is a queue to be worked through so there will be a delay to emails in the queue.

---

We are aware that there is a queue of outbound mail on the SMTP array at the moment going back some 2 hours plus, resulting in some users being sent a notification email stating this.

Engineers are investigating the cause at present, and we will update status as soon as we have further details. There is no need to contact support or forward the notification email.

Start Time

29th February 2012, at 9:12AM

Description

Update at 16:42:18(GMT): The issue has now been resolved with this server.Update at 10:16:57(GMT): The VPS18 issue will be resolved today and the final moves are taking place.


Update at 15:05:18(GMT): We only have a small amount of containers remaining on VPS18. We are contacting these clients as and when we migrate.


Update at 10:15:37(GMT): The restoration of containers is taking longer than initially expected due to manual work involved. We're informing clients as and when their container is being moved.


Update at 17:36:52(GMT): Almost half of the clients have been contacted and VPS servers moved.


Update at 15:55:28(GMT): We have moved some containers and we're going through them one by one. Unfortunately this can be a slow process. We will be ringing customers before moving them.


Update at 12:44:05(GMT): As a precaution engineers are moving all containers from VPS18 to a separate server. This will prevent any further issues. Customers will be contacted in due course.


Update at 10:23:55(GMT): We are currently going through each VPS container and contacting the owner. We will be updating them while we work on restoring their services.


Update at 09:32:27(GMT): The server has had firmware updates, however some data loss may have occurred during the reboot. Our engineers are looking into this. We will be updating this page as soon as we have any further information.


A fault was identified with the server this evening. It was rebooted a little after 11PM and is presently undergoing disk scans before coming back up.

We do realise this is a repeating fault. Please be advised replacement hardware is on order and set for installation and migration imminently.

Start Time

28th February 2012, at 11:44PM

Description

Update at 13:34:06(GMT): After 48 hours of close monitoring, we are resolving this status.

---

Update at 13:36:19(GMT): We have applied some firmware updates to this server which should help the stability, but will be monitoring it closely with a view to installing further updates is the server stability does not improve.

---

We are currently investigating the unavailability of server17.xenserve.com and will restore it to service as soon as possible.

Start Time

28th February 2012, at 12:35PM

Description

Update at 00:24:08(GMT): The issue has now been resolved with this server.Server13 is about to be rebooted in order to apply several critical firmware updates.

Expected downtime is approximately 10 minutes.

Start Time

27th February 2012, at 11:45PM

Description

Update at 16:42:09(GMT): The issue has now been resolved with this server.Update at 15:28:38(GMT): The delivery issue has now been resolved and mail has been verified as delivering successfully. This means that any new mail sent to users on server5.xenserve.com, or mail sent from the server should be delivered as usual.

The large queue of spam mails generated between Saturday and Tuesday afternoon is still in the process of being cleaned before delivery to mailboxes on server5.xenserve.com. It currently stands at approximately 1,000,000 e-mails and we would expect this to reduce a lot before delivery to customers is made.

---

Update at 14:29:30(GMT): We are currently sending this response to any enquiries regarding the situation on server5.xenserve.com:

Last weekend a compromise related to the Plesk software used on our servers was exploited to send a vast mount of spam e-mail, clogging up the e-mail queues on the server5.xenserve.com machine. Approximately 4.4 million spam e-mails were sent. Once identified, we stopped the sending of further mail and began work to clear the spam from the queues. Due to the quantity of mail involved, this took a lot longer than acceptable, so on Tuesday the decision was taken to move the existing queue aside, with a mix of spam and legitimate mail in it. A fresh queue was created so that mail would flow as normal, while work to clean the original queue continued.

The new current mail queue is in place and being used my the mail processes on server5.xenserve.com but we have experienced further issues with delivery of e- mail. At this time we are uncertain as to the cause of these are working hard to understand and correct this issue.

During this time, e-mail should have been queued for eventual delivery, but we understand from some users that some e-mail is being bounced and returned to sender. We are also attempting to understand why this is happening and prevent it such that legitimate e-mail will reach the intended recipient.

---

Update at 10:49:05(GMT): The mail queue on server5.xenserve.com has reduced but is still present. We are still working to resolve this issue.

---

Update at 15:12:44(GMT): A further queue of mail has developed on server5.xenserve.com and we are working to resolve this matter at the moment.

---

Update at 08:52:00(GMT): It appears there is still a delay in customers receiving emails on this server. Our engineers have been made aware of this are continuing to investigate.

---

Update at 17:28:10(GMT): Email delivery is now working normally on server5.xenserve.com again.

---

Update at 16:50:05(GMT): Most email is working normally on server5.xenserve.com, but a minority of addresses still aren't accepting e-mail. We are working on this at the moment.

---

Update at 13:52:18(GMT): Unfortunately the changes made last night has not resolved the issue as we hoped and is still causing email problems. The engineers are still investigating into this and hope to have this resolved as soon as possible.

---

Update at 17:28:22(GMT): We are now expecting the original queue to finish processing on Thursday morning, and will move all legitimate delayed e-mail into the queues at that point.

---

Update at 16:56:00(GMT): A new fresh, empty mail queue on server5 has been created. All new email received will be processed in real time once again. That means you should see newer email flowing at normal, fast speeds.

The original queue has not been deleted, it is still being processed in parallel with the new queue, meaning that email received during the incident will still be delayed, but wont be lost. Of course, we are also working on ratcheting up the processing speeds too, as an ongoing process.

---

Update at 12:04:36(GMT): We have worked to speed up the processing of the queue on server5.xenserve.com and have had some progress. The queue is still being processed, now at an increased rate. It is still very substantial, however and is likely to take some time to process.

---

Update at 09:44:28(GMT): The mail queue on server3.xenserve.com has now been cleared so mail will operate normally for accounts on this server. The queue on server5.xenserve.com is still being processed.

---

Update at 20:03:28(GMT): Please note we are still running automated tools to delete the spam email, while ensuring legitimate email is not affected. Due to the large numbers of email the entire process is expected to run into the evening. Mail delivery will be delayed as a result, but all users should receive emails in due course.

---

Due to a compromised account on each of these servers, we are currently seeing very large mail queues in excess of 1,000,000 mails. We are in the process of cleaning the spam from these queues, but this will cause a delay to legitimate mail until the process is complete.

Start Time

27th February 2012, at 2:07PM

Description

Update at 10:50:24(GMT): We have identified this issue and resolved it. We will report the problem to our firewall providers so that they can implement a permanent fix

---

An automated update of the firewall for our business hosting package has failed and prevented site being served. We are correcting this now and will have all sites back online shortly.

Start Time

25th February 2012, at 9:45AM

Description

Update at 22:15:42(GMT): The mail backlog has now been cleared and full service restored.

---

Update at 15:07:41(GMT): An issue on our outgoing mail servers has been identified and resolved. We are currently processing the backlog of mail and expect all mail to be delivered in the next few hours.

---

We are currently investigating into a possible issue causing delays with mail being sent through our SMTP server array. Please accept our apologies for any inconvenience caused and our assurances we are doing all we can to get this resolved quickly.

Start Time

24th February 2012, at 1:31PM

Description

Update at 14:20:29(GMT): We have cleaned out a large queue of e-mail that had built up and have seen the server behave reliably this morning, so we are marking this issue as resolved.

Our automated monitoring systems will continue to monitor adam.34sp.com and alert us to any issues.

---

We are presently investigating a repeating fault that is affecting the adam.34sp.com personal server.

Start Time

22nd February 2012, at 11:12PM

Description

Update at 16:04:59(GMT): All containers have now restarted.

---

Update at 15:43:40(GMT): The server is coming back up now and is starting the individual VPS containers. We expect full service to all affected clients to be restored in the next 10 minutes or so.

---

The server vps18.34sp.com has become unresponsive and we are investigating this matter. As soon as we have any further information we will post it here.

Start Time

22nd February 2012, at 3:35PM

Description

Update at 19:39:27(GMT): The issue has now been resolved with this server.The server and all containers are now back online.

This server is about to be rebooted for some brief emergency maintenance.

Downtime is anticipated to be approximately 10 minutes or so.

Start Time

20th February 2012, at 7:09PM