34SP.com System Status History

Description

Update at 23:08:25(GMT): This outage is now marked as resolved.

---

Update at 17:05:05(GMT): Prohost6 is now back up (again) and we will continue to monitor this machine closely.

---

Update at 16:24:44(GMT): The machine has gone down again, we're currently performing kernel & environment updates in an attempt to improve stability.

---

Update at 15:30:44(GMT): The machine is back up and running now. We're investigating into the cause of the outage.

---

We're investigating a total outage of Prohost6, this does not affect e-mail services (these are kept separately) but all websites and databases hosted upon Prohost6 are currently offline whilst we try to rectify the situation.

Start Time

26th November 2010, at 3:12PM

Description

Update at 20:22:01(GMT): The issue has now been resolved with this server.This work has now been carried out.

---

Prohost7 is having some security updates applied to it, it will be being rebooted very shortly.

The downtime should only be a few minutes.

Start Time

25th November 2010, at 7:51PM

Description

Update at 15:01:12(GMT): For all e-mail that would be deliverable if not for the issues described below, this problem should now be resolved.

If your e-mail wasn't deliverable for some other reason, you should receive a bounce message as normal in due course, if not already.

Any customers that switched to a use a specific SMTP server to manually mitigate the issues are advised to ensure that they switch back to using smtp.34sp.com.

---

One of the many SMTP servers we operate had a large mail queue that was causing quite long delays in its outgoing mail queue.

This has now been resolved, it will take a short while for the queue to be processed, but all mail will eventually be delivered.

We apologise for the obvious inconvenience caused. Any customers sending large mailshots are asked to please wait a few hours before doing so, to avoid adding to the issue.

Start Time

23rd November 2010, at 2:05PM

Description

Update at 23:41:18(GMT): The issue has now been resolved with this server.Update at 23:09:50(GMT): Issues with the disks on server3 have surfaced. We are presently running a full disk/scan repair at this time.

We are presently looking into an issue affecting this server. All services are affected.

Start Time

18th November 2010, at 9:50PM

Description

Update at 21:50:08(GMT): The issue has now been resolved with this server.We are presently looking into an issue affecting this server. The server is presently unreachable, all services are affected.

Start Time

18th November 2010, at 7:47PM

Description

Update at 10:00:19(GMT): The issue has now been resolved with this server.A reminder that the joshua.34sp.com accounts are being migrated to a new server at this time. Please avoid making any changes to any sites or in the siteadmin area during this process.

Start Time

16th November 2010, at 8:47AM

Description

Update at 17:12:26(GMT): The issue has now been resolved with this server.

---

All affected domain 'index' files have now been restored from their respective backups. And all sites should now be working.

If there are any discrepancies between the backup restored (no older than a week) and the version previous to the attack, please re-upload your own local copy of the index file.

For any further assistance, please contact support by e-mailing support@34sp.com.

---

Update at 14:38:00(GMT): A global fix for the defacing done to domains on this server is currently under-way. This requires extracting files from compressed archives (backups) for each domain, therefore may take a short while to complete across all domains.

We appreciate your continued patience in this matter.

---

Update at 13:52:23(GMT): On further examination we have discovered an exploit in the FTP service that caused (indirectly) the earlier Apache outage.

We have now updated the service to prevent a recurrence and are dealing with the damage.

We are obviously receiving call after call to our support lines - if customers could please defer to sending an e-mail we will endeavour to inform you as and when this issue has been resolved.

---

Update at 13:15:19(GMT): The service is back up and running again. We shall continue to investigate the cause of this outage.

---

At present the Apache service on Prohost3 isn't responding. We're working to resolve this as quickly as we possibly can.

Start Time

11th November 2010, at 1:04PM

Description

Update at 13:31:49(GMT): The issue with this service is now resolved.

---

Update at 12:59:18(GMT): The Free Domain Redirects should now be working again. Any updates made to the Redirects during the downtime should take effect shortly.

---

The machine that handles our Free Domain Redirects service is a VPS hosted by VPS5.

We have added this as a separate status update to ensure that any customers experiencing issues regarding the Free Domain Redirects are aware that due to the current problems with VPS5 and that the service may well experience difficulties until we're able to resolve the issues with VPS5.

Start Time

10th November 2010, at 12:33PM

Description

Update at 13:31:14(GMT): The issue has now been resolved with this server.

---

All VPS' are now started-up again and the server looks to be functioning normally again. We will continue to keep a watchful eye on VPS5 for the next few days.

---

Update at 12:58:31(GMT): We've rebooted VPS5 and all VPS' should be starting-up again now - this takes a short while as all disk quotas need to be re-calculated.

Your continued patience is appreciated.

---

Update at 12:30:32(GMT): Our Engineer is on-site and we're diagnosing a fault with VPS5. At present it looks like there will be a full reboot of the machine, which will bring down all VPS' hosted upon it. We will do our best to keep this downtime to an absolute minimum from here on.

---

We're experiencing an issue with vps5.34sp.com and the VPS' within. The virtual machines, as well as the host, are experiencing difficulties with SSH access and other select services.

We have called an Engineer to attend and will update as soon as we have any further information.

Start Time

10th November 2010, at 9:39AM

Description

Update at 22:32:02(GMT): The issue has now been resolved with this server.We are presently looking into an issue affecting MySQL on server2. This will be affecting database, plesk and certain email services.

Start Time

6th November 2010, at 10:26PM