Update at 20:33:00(BST): Update at 19:01:04(BST): Phones are back online now, but may need to go off one more time this evening briefly.
Due to some quick office maintenance phone support is going offline. This should last no more than 60-90mins.
2nd May 2013, at 6:36PM
2nd May 2013, at 7:33PM
Update at 16:04:28(BST): Following several disk checks, the server is now available again.
29th April 2013, at 11:30AM
29th April 2013, at 3:04PM
Update at 22:36:46(BST): The issue has now been resolved with this server.The upgrade went as expected and all servers are checking out fine after the completed work. The basic systems now have double RAM (20>40GB) which should vastly improve performance.
We will continue to monitor system uptime and performance but expect things to dramatically improve from here on out.
Update at 22:16:24(BST): Upgrade is now complete and the servers are rebooting and being cross checked.
Update at 21:56:20(BST): Our engineer is now on site and starting the upgrade. PH20-22 will now go offline for 15-30 minutes.
Update at 21:24:13(BST): Several tweaks have improved stability through the evening and we also expect our engineer to be on site within the next 30 minute to commence the final upgrade.
Update at 20:27:31(BST): Due to the increasing instability of the platform, we have escalated the emergency work.
We are presently arranging for an engineer to head to the datacenter and begin the upgrade this evening (Apr 16th). More updates as this unfolds, but we expect to restore full availability this evening.
We have identified a load issue on this trio of servers.
We will be monitoring and maintaining these systems best we can through this evening (Apr 16th).
During the morning of April 17th we intend to complete an emergency upgrade on this system which will vastly increase system resources.
This should correct the issue long term.
16th April 2013, at 6:36PM
16th April 2013, at 9:36PM
Update at 02:29:14(BST): Update at 01:50:57(BST): All servers should now be back online that have been affected by this event.
Update at 01:20:21(BST): We have identified the area of our network generating the issue and are presently isolating the cause. Most systems will now be fine, but some may still be slowed.
We are presently examining a slow down of several systems on our network.
16th April 2013, at 12:21AM
16th April 2013, at 1:29AM
Update at 18:31:55(BST): Prohost 17-19 are now back online and serving. Each server has had its RAM doubled form 20GB to 40GB. This, in addition to some granular tweaks to the overall system should provide a vastly better uptime level for all users.
Update at 18:22:20(BST): PH17-19 have now been upgraded and are booting back up.
Update at 18:02:37(BST): PH17-19 are now going offline.
We will be shortly beginning an emergency maintenance window on the prohost 17 through 19 servers.
We will be doubling the servers RAM, which should take 15-30 minutes in total. This should help ease the repeating load issues the servers have experienced lately.
Once the maintenance begins and ends, we will update this post.
11th April 2013, at 5:03PM
11th April 2013, at 5:31PM
Update at 18:41:16(BST): Update at 18:41:02(BST): This issue has been resolved. We have identified a compromised system that has now been isolated.
We have identified an issue affecting the general speed of our network.
We are attending to this matter now.
7th April 2013, at 6:22PM
7th April 2013, at 5:41PM
Update at 10:53:47(BST): The issue has now been resolved with this server.
This server is now back online. Apologies for any inconvenience caused.
Engineers have had to perform an emergency reboot on this server.
Apologies for any inconvenience caused.
5th April 2013, at 10:48AM
5th April 2013, at 9:53AM
Update at 13:50:55(GMT): The issue has now been resolved. Our phone lines are now open. Apologies for any inconvenience caused.
Update at 09:58:41(GMT): Our phone lines are currently closed as a member of our weekend staff is unable to come to work, due to the adverse weather. Apologies for any inconvenience caused.
23rd March 2013, at 9:58AM
23rd March 2013, at 1:50PM
Update at 15:31:45(GMT): The issue has now been resolved with this server.
The server is now back online. Apologies for any inconvenience caused.
Update at 14:55:21(GMT): We are having to perform an emergency reboot.
Apologies for any inconvenience caused.
15th March 2013, at 2:54PM
15th March 2013, at 3:31PM
Update at 10:17:55(GMT): This issue is now marked as resolved.
Update at 14:24:09(GMT): The issue has now been corrected. We identified a rogue server creating excess traffic that has now been corrected.
We do maintain automated systems to prevent these floods and in this instance the system seems to have failed to prevent the attack. We are presently reviewing the piece of hardware used to protect against these floods, in order to prevent a repeat.
Update at 14:13:59(GMT): We have identified the cause of the network load and are working to quickly correct this. All affected services will be accessible again momentarily.
Engineers are currently investigating initial reports of network traffic that is failing to resolve to a server. We'll post further information as soon as it becomes clear where the issue actually lies.
11th March 2013, at 1:46PM
12th March 2013, at 10:17AM