34SP.com System Status History

Description

Update at 21:50:08(GMT): The issue has now been resolved with this server.We are presently looking into an issue affecting this server. The server is presently unreachable, all services are affected.

Start Time

18th November 2010, at 7:47PM

Description

Update at 10:00:19(GMT): The issue has now been resolved with this server.A reminder that the joshua.34sp.com accounts are being migrated to a new server at this time. Please avoid making any changes to any sites or in the siteadmin area during this process.

Start Time

16th November 2010, at 8:47AM

Description

Update at 17:12:26(GMT): The issue has now been resolved with this server.

---

All affected domain 'index' files have now been restored from their respective backups. And all sites should now be working.

If there are any discrepancies between the backup restored (no older than a week) and the version previous to the attack, please re-upload your own local copy of the index file.

For any further assistance, please contact support by e-mailing support@34sp.com.

---

Update at 14:38:00(GMT): A global fix for the defacing done to domains on this server is currently under-way. This requires extracting files from compressed archives (backups) for each domain, therefore may take a short while to complete across all domains.

We appreciate your continued patience in this matter.

---

Update at 13:52:23(GMT): On further examination we have discovered an exploit in the FTP service that caused (indirectly) the earlier Apache outage.

We have now updated the service to prevent a recurrence and are dealing with the damage.

We are obviously receiving call after call to our support lines - if customers could please defer to sending an e-mail we will endeavour to inform you as and when this issue has been resolved.

---

Update at 13:15:19(GMT): The service is back up and running again. We shall continue to investigate the cause of this outage.

---

At present the Apache service on Prohost3 isn't responding. We're working to resolve this as quickly as we possibly can.

Start Time

11th November 2010, at 1:04PM

Description

Update at 13:31:49(GMT): The issue with this service is now resolved.

---

Update at 12:59:18(GMT): The Free Domain Redirects should now be working again. Any updates made to the Redirects during the downtime should take effect shortly.

---

The machine that handles our Free Domain Redirects service is a VPS hosted by VPS5.

We have added this as a separate status update to ensure that any customers experiencing issues regarding the Free Domain Redirects are aware that due to the current problems with VPS5 and that the service may well experience difficulties until we're able to resolve the issues with VPS5.

Start Time

10th November 2010, at 12:33PM

Description

Update at 13:31:14(GMT): The issue has now been resolved with this server.

---

All VPS' are now started-up again and the server looks to be functioning normally again. We will continue to keep a watchful eye on VPS5 for the next few days.

---

Update at 12:58:31(GMT): We've rebooted VPS5 and all VPS' should be starting-up again now - this takes a short while as all disk quotas need to be re-calculated.

Your continued patience is appreciated.

---

Update at 12:30:32(GMT): Our Engineer is on-site and we're diagnosing a fault with VPS5. At present it looks like there will be a full reboot of the machine, which will bring down all VPS' hosted upon it. We will do our best to keep this downtime to an absolute minimum from here on.

---

We're experiencing an issue with vps5.34sp.com and the VPS' within. The virtual machines, as well as the host, are experiencing difficulties with SSH access and other select services.

We have called an Engineer to attend and will update as soon as we have any further information.

Start Time

10th November 2010, at 9:39AM

Description

Update at 22:32:02(GMT): The issue has now been resolved with this server.We are presently looking into an issue affecting MySQL on server2. This will be affecting database, plesk and certain email services.

Start Time

6th November 2010, at 10:26PM

Description

Update at 22:30:49(GMT): The issue has now been resolved with this server.This migration has now been completed.

If you do not use us for dns services, please check your email and update your dns records as per the email we sent out a short while ago. (This should be 80.82.124.90 unless you are on a unique IP for SSL or anonymous ftp services)

If you have any issues with your site following the migration please do let us know

---

A reminder that the gort.34sp.com accounts are being migrated to a new server at this time. Please avoid making any changes to any sites or in the siteadmin area during this process.

Start Time

3rd November 2010, at 8:34AM

Description

Update at 22:29:50(GMT): We are presently tracking a recurring fault on this server that is causing issues the the Plesk and database systems.

Start Time

1st November 2010, at 7:26PM

Description

Update at 20:57:52(GMT): We have now resolved this issue by restoring certain tables from our routine backups. Customers affected by this action will shortly have an e-mail explaining further.

Furthermore, we apologise for any inconvenience that this issue has caused our customers today. If you have any further queries, please contact 34SP.com support via e-mail to support@34sp.com.

---

Update at 19:03:57(GMT): We're still working on resolving this issue, which appears to be related to a corrupt InnoDB table (or record) causing MySQL to restart frequently.

A resolution is currently being worked upon and we will provide more information as and when we have it.

---

The MySQL daemon on server4.xenserve.com is taking quite a punishing and some users are experiencing sporadic connection errors.

Whilst we debug the issue, please avoid any long-running administration/backup queries.

Start Time

31st October 2010, at 5:24PM

Description

Update at 08:33:25(GMT): Update at 10:27:20(GMT): This server is again experiencing some issues, this is due to a denial of service attack against the server - it is however on http so it not very easy to block this type, we are currently working on firewall rules to block items as we can to try to avoid this continuing.

---

Update at 14:20:17(BST): The server has been back online for a few hours now, with apache services going in and out of availability, we believe this is now under control but will be monitoring this for a while

---

Update at 10:54:52(BST): An engineer is on the way to the datacenter at this time to help further diagnose the issues on this server

---

We are looking into issues affecting this server at this time.

Start Time

30th October 2010, at 10:40AM