34SP.com System Status History

Description

Update at 01:38:49(GMT): The issue has now been resolved with this server.

---

The service-interrupting work was completed around an hour ago, but we've been running as many tests and checks across the new configuration to ensure all is sane. At present the network seems to be running well again.

Looking forward, this maintenance will provide future-proofing to the network performance in Manchester, as it grows.

If anyone is experiencing any lingering issues, please contact support@34sp.com as soon as is possible.

---

Update at 00:10:55(GMT): This work will be beginning now.

---

Customers residing on servers in our Manchester data centre are to be advised that we are going to have to perform some maintenance on our network switches, owing to a heavy increase in traffic that has recently brought to light several short-comings of this network.

We need to make some changes to the cabling topology and switch configurations to correct this. It will unfortunately mean that we will have some intermittent service outages as network paths are re-calculated, to account for the changes.

The work will be commencing in approximately 45-60 minutes, with patches of disruption during a window of around 30 minutes.

We will update this page as and when new information becomes available - thank you for your patience.

Start Time

2nd December 2010, at 11:15PM

Description

Update at 22:49:35(GMT): The issue has now been resolved with this server.This should now all be completed and all services back online. Please do let us know if you have any issues with anything

---

The services are currently offline on the server10.xenserve.com server, this is to do some emergency modifications to the database system on this server.

The expected downtime of all services should be less than 30minutes.

Start Time

2nd December 2010, at 10:24PM

Description

Update at 01:08:16(GMT): The issue has now been resolved with this server.We are presently looking into a fault affecting this VPS server.

If you have an affected VPS please do not restart your VPS through our control panel at this time.

Start Time

1st December 2010, at 12:19AM

Description

Update at 23:08:25(GMT): This outage is now marked as resolved.

---

Update at 17:05:05(GMT): Prohost6 is now back up (again) and we will continue to monitor this machine closely.

---

Update at 16:24:44(GMT): The machine has gone down again, we're currently performing kernel & environment updates in an attempt to improve stability.

---

Update at 15:30:44(GMT): The machine is back up and running now. We're investigating into the cause of the outage.

---

We're investigating a total outage of Prohost6, this does not affect e-mail services (these are kept separately) but all websites and databases hosted upon Prohost6 are currently offline whilst we try to rectify the situation.

Start Time

26th November 2010, at 3:12PM

Description

Update at 20:22:01(GMT): The issue has now been resolved with this server.This work has now been carried out.

---

Prohost7 is having some security updates applied to it, it will be being rebooted very shortly.

The downtime should only be a few minutes.

Start Time

25th November 2010, at 7:51PM

Description

Update at 15:01:12(GMT): For all e-mail that would be deliverable if not for the issues described below, this problem should now be resolved.

If your e-mail wasn't deliverable for some other reason, you should receive a bounce message as normal in due course, if not already.

Any customers that switched to a use a specific SMTP server to manually mitigate the issues are advised to ensure that they switch back to using smtp.34sp.com.

---

One of the many SMTP servers we operate had a large mail queue that was causing quite long delays in its outgoing mail queue.

This has now been resolved, it will take a short while for the queue to be processed, but all mail will eventually be delivered.

We apologise for the obvious inconvenience caused. Any customers sending large mailshots are asked to please wait a few hours before doing so, to avoid adding to the issue.

Start Time

23rd November 2010, at 2:05PM

Description

Update at 23:41:18(GMT): The issue has now been resolved with this server.Update at 23:09:50(GMT): Issues with the disks on server3 have surfaced. We are presently running a full disk/scan repair at this time.

We are presently looking into an issue affecting this server. All services are affected.

Start Time

18th November 2010, at 9:50PM

Description

Update at 21:50:08(GMT): The issue has now been resolved with this server.We are presently looking into an issue affecting this server. The server is presently unreachable, all services are affected.

Start Time

18th November 2010, at 7:47PM

Description

Update at 10:00:19(GMT): The issue has now been resolved with this server.A reminder that the joshua.34sp.com accounts are being migrated to a new server at this time. Please avoid making any changes to any sites or in the siteadmin area during this process.

Start Time

16th November 2010, at 8:47AM

Description

Update at 17:12:26(GMT): The issue has now been resolved with this server.

---

All affected domain 'index' files have now been restored from their respective backups. And all sites should now be working.

If there are any discrepancies between the backup restored (no older than a week) and the version previous to the attack, please re-upload your own local copy of the index file.

For any further assistance, please contact support by e-mailing support@34sp.com.

---

Update at 14:38:00(GMT): A global fix for the defacing done to domains on this server is currently under-way. This requires extracting files from compressed archives (backups) for each domain, therefore may take a short while to complete across all domains.

We appreciate your continued patience in this matter.

---

Update at 13:52:23(GMT): On further examination we have discovered an exploit in the FTP service that caused (indirectly) the earlier Apache outage.

We have now updated the service to prevent a recurrence and are dealing with the damage.

We are obviously receiving call after call to our support lines - if customers could please defer to sending an e-mail we will endeavour to inform you as and when this issue has been resolved.

---

Update at 13:15:19(GMT): The service is back up and running again. We shall continue to investigate the cause of this outage.

---

At present the Apache service on Prohost3 isn't responding. We're working to resolve this as quickly as we possibly can.

Start Time

11th November 2010, at 1:04PM