34SP.com System Status History

Description

Update at 21:32:38(BST): This status outage regarding xenserve9's DNS service is now marked as resolved following successful operation of the service for at least a few days.

---

Update at 17:53:55(BST): Apologies for the late update: xenserve9's DNS service has been running since approx. 17:22.

We will continue to monitor this machine for a short time.

---

At present, xenserve9 is experiencing a software issue with its local DNS service that may be causing an outage for customers that use the ns17/ns18.xenserve.com name server addresses.

We're working hard to get this resolved as soon as possible.

Start Time

19th September 2011, at 5:12PM

Description

Update at 21:29:47(BST): This outage is now resolved. Please see the current update regarding the planned maintenance to xenserve18.

---

Update at 18:01:12(BST): Upon further investigation, it appears that two faults were affecting xenserve18 at the time of the outage:

One is related to the RAID card. There is some updated firmware available for this and we shall be discussing downtime planning to have the RAID card updated, as soon as is possible.

The second is memory starvation. An upgrade to the system's memory has been planned recently, which we may have to bring forward.

Any 'final' details pertaining to this solution will be e-mailed to affected customers as soon as possible.

---

Update at 11:07:44(BST): Xenserve18 is now back up and running (and has been for a short while).

We have observed an error presented by the system's BIOS which required user interaction to boot correctly.

We will be checking all databases for consistency very shortly and shall continue to interrogate the hardware to see why this message was presented.

---

Update at 10:41:20(BST): Unfortunately xenserve18 has not booted cleanly. We have an Engineer on-site already to investigate.

As a result the machine will be down for another 15-20 minutes.

---

Update at 10:26:36(BST): We have been forced to reboot xenserve18. This will mean the machine should restart as normal. We'll be checking all databases as soon as the machine is up and running again.

---

We are experiencing very high loads with server18/xenserve18. This is causing an outage of all services.

We are working to recover the machine.

Start Time

19th September 2011, at 10:17AM

Description

Update at 16:12:06(BST): Update at 11:49:07(BST): All VPS containers are up and running again after the host restart.

At present it appears that this was caused by a live migration of a container from another VPS host (something that shouldn't cause such catastrophic behaviour in a host) but the logging after the migrated container started is notably sparse.

We will continue to investigate & monitor the server from this point.

---

VPS22 has rebooted itself, this will affect all current containers present on the machine.

We are investigating the cause of this and will provide updates ASAP.

Start Time

14th September 2011, at 11:34AM

Description

Update at 11:57:33(BST): Memory upgrades have been completed on these servers.

---

Update at 22:32:30(BST): Xenserve19 is now coming back online, and xenserve17 is being shutdown for its memory upgrade.

---

Update at 22:00:31(BST): Xenserve19 is now shutting down for the memory upgrade.

---

Update at 21:31:44(BST): Xenserve19 will be going down for its update in 30 minutes.

---

Reseller customers on servers xenserve19 & xenserve17 are reminded that we have a window of planned emergency maintenance to these servers this evening.

Both servers will be having their RAM (system memory) upgraded from 8GiB to 12GiB, in order to hopefully prevent further outages related to load (specifically, where memory starvation is the culprit.)

At 22:00, xenserve19 will be shutdown for its upgrade.
At 22:30, xenserve17 will be shutdown for its upgrade.

We expect the work to take no longer than 15 minutes, but have accounted for 30 minutes in the case of any unforeseen issues arising.

Relevant updates will be posted to this page, so please check back.

Start Time

13th September 2011, at 8:28PM

Description

Update at 17:54:37(BST): This outage is now set to resolved.

---

Update at 20:22:20(BST): Very few faults were observed from the migration but we will continue to monitor the new server closely.

Customers are advised to check their sites and raise a ticket to support@34sp.com if anything is still amiss.

---

Update at 13:50:50(BST): Server7 is now back online, work will continue as we test sites and work with customers to rectify any faults from the migration.

---

Update at 12:22:54(BST): IPs are about to be reconfigured between the servers, all going well services will be back online within an hour.

---

Update at 11:39:13(BST): WebServices are now going offline for a sync up of web files and mysql databases.

---

Update at 10:53:38(BST): All mailservices on server7 have now been stopped for a sync up of the email on the server.

---

The server7.xenserve.com is undergoing migration to new hardware today as per the email sent to all clients.

Try to avoid making any changes on any websites now to minimize loss of data when transferred to the new server.

We will update this post with progress throughout the day

Start Time

13th September 2011, at 8:11AM

Description

Update at 13:31:25(BST): The new core switches have been acting as gateways & routing quite flawlessly for several days now.

At this time we will mark this status post as resolved, but any customers with issues they suspect are related to this work are encouraged to contact support via e-mail.

---

Update at 01:28:16(BST): The routing issue between sites (Manchester and London) was resolved at around midnight. This was not planned and a full investigation into the cause of the outage to those routes will be carried-out in due course to improve resilience and prevent the same fault from occurring in the future.

Further apologies are required for customers sitting behind the access switch that failed to take the changes required for the new core switches.

This was eventually diagnosed as a faulty switch and was swapped for another model. Because of this event, multiple Reseller, Personal & VPS hosting customers will have seen some extended downtime.

All issues do now appear to be resolved. Our monitoring systems are no longer reporting any outages and randomly-tested websites are working.

Now that this upgrade is complete, it is intended that any future upgrades will be much less disruptive, due to the improved network design. More flexibility and better performance are also key benefits to the new hardware.

Any customer still experiencing issues this morning should contact support to raise a ticket regarding this.

---

Update at 23:50:21(BST): We've noticed some further errors with the routing changes that are now affecting certain routes in/out of our network. We're working on this as a priority as it is affecting some customers in other areas.

---

Update at 23:44:40(BST): Network upgrades were completed by 23:33, with a small routing oversight that caused further delay until 23:42.

We have one access switch that is resisting re-configuration. We should have this resolved shortly.

---

Update at 22:46:22(BST): Beginning in approx. 15 minutes.

---

As a reminder to customers, the planned maintenance window for upgrades to our core network will be beginning at 23:00 (BST) and finishing at most 60 minutes later, 00:00.

Any further updates will be posted to this status page. It is hosted externally and will be unaffected by any outage.

Start Time

8th September 2011, at 9:48PM

Description

Update at 12:00:32(BST): The issue has now been resolved with this server.GlobalSign certificates are now being ordered as usual. Thank you for your continues patience during this time.


As per the GlobalSign official statement below:

On Sep 5th 2011 the individual/group previously confirmed to have hacked several Comodo resellers, claimed responsibility for the recent DigiNotar hack. In his message posted on Pastebin, he also referred to having access to 4 further high profile Certificate Authorities, and named GlobalSign as one of the 4.

GlobalSign takes this claim very seriously and is currently investigating. As a responsible CA, we have decided to temporarily cease issuance of all Certificates until the investigation is complete. We will post updates as frequently as possible.

We apologise for any inconvenience.

Start Time

7th September 2011, at 9:20AM

Description

We would like to apologise for the lack of phone support during the evening of September 6th. Phone service will resume again at 8am on September 7th

Start Time

6th September 2011, at 7:21PM

Description

Update at 14:48:36(BST): Update at 14:48:10(BST): The issue has now been resolved with this server.

---

The personal hosting server max.34sp.com is currently unavailable. We are investigating this and will post an update as soon as we have any more information to give.

Start Time

6th September 2011, at 2:33PM

Description

Update at 10:19:06(BST): The issue has now been resolved with this server.The 4PSA SpamGuardian and CleanServer systems have now been re-added to this server, Please let us know if you find any issues with the migrated domains.

---

Update at 11:43:32(BST): IPs have now been moved over, we are still doing some minor post migration work, but most sites should be back up and running.

If you are experiencing any issues with your site since the move please do get in contact with us.

---

Update at 11:09:57(BST): Ip reconfiguration to move the ips to the new server is now in progress, we expect around 30minutes from this point.

---

Update at 10:38:25(BST): All webservices are now going offline for a sync up of files and databases.

---

Update at 10:12:56(BST): All mailservice on server6 have now been stopped for a sync up of the email on the server.

---

The server6.xenserve.com is undergoing migration to new hardware today as per the email sent to all clients.

Try to avoid making any changes on any websites now to minimize loss of data when transferred to the new server.

We will update this post with progress throughout the day

Start Time

6th September 2011, at 8:11AM