34SP.com System Status History

Description

Update at 10:26:41(BST): Control panel access has now been restored

Update at 13:28:28(BST): After many emails back and forth with Plesk, they were unable to resolve this issue. At the point of us installing a new server to migrate everyone over to - our resident star system developer David, has fixed what plesk could not! We are currently checking everything over. It looks like the control panel is back working though.

Update at 16:11:46(BST): Parallels have been unable to assist us with this failure in their software. As a result we have only the final choice of deploying a new server, with new software, and migrating all affected sites over.

In the meantime please continue to contact support for any changes required, that are blocked by the Plesk fault.

We will be posting a migration date shortly.

Update at 15:56:58(BST): This system is still unavailable. We have requested that Parallels (Plesk softwre creators) assist with the matter as we are unable to proceed. Please note sites and email are up and running fine.

The control panel/plesk is offline due to a fault on this system. We are presently working on restoring this service.

Start Time

17th September 2014, at 5:10PM

Description

Update at 01:37:19(BST): The issue has now been resolved with this server.account.34sp.com, SMTP authentication and iRedmail based services (POP3, IMAP and webmail) will be unavailable for approximately 20 minutes from 00:30 for essential maintenance. No e-mail will be lost during this period. We apologise for any inconvenience.

Start Time

12th September 2014, at 12:30AM

Description

Update at 17:18:19(BST): Webmail is now authenticating correctly. If you see further problems, please contact support and we will do our best to help you.

---

Update at 16:44:20(BST): This problem is still affecting webmail users. Our engineers are working to resolve this as soon as possible.

---

Update at 16:29:53(BST): The problem with mail authentication has now been resolved.

---

We are currently undergoing an outage of our SMTP and email authentication systems. Our engineers are working on this problem now and will have it restored to service as soon as possible.

Start Time

11th September 2014, at 4:23PM

Description

Update at 10:47:21(BST): The issue has now been resolved with this server.Redundancy network now correctly active in place, restoring services to normal current levels.

-------------
Due to maintenance work being done by our primary network providers, a number of systems are currently inaccessible.

Apologies for any inconvenience this may be causing in the meantime, we are working to restore service as soon as possible.

Start Time

6th September 2014, at 10:31AM

Description

Update at 12:18:01(BST): All 3 servers now have over 99.5% uptime over the last few days. Please do contact support if you have any further questions or problems.

Update at 10:15:15(BST): All servers have been back up and running for a few minutes. All looks well, but we shall continue to closely monitor these for now.


Prohost26, Prohost27 and Prohost28 appear to have a software problem on the operating system. We're going to try a quick reboot and a change to these servers now, to hopefully improve things. Downtime may be around 10-15 minutes.

Start Time

5th September 2014, at 9:33AM

Description

Update at 16:20:06(BST): Control panel logins are available again and the correct domain names are now showing.

---

Update at 16:04:35(BST): Until we have this issue corrected, control panel logins are being disabled. We are working to correct the problem and allow logins again as soon as possible.

---

We are aware of the incorrect domain name showing for some customers in our control panel at https://account.34sp.com

We hope to have this issue resolved shortly.

Start Time

28th August 2014, at 3:45PM

Description

Update at 22:33:00(BST): Queues are now under 2,500, and we expect the remainder of mail to be delivered by 11PM.

Update at 21:19:43(BST): SMTP queues are down from 26,000 to 16,000. Based on this throughput all remaining email should go through in the next 90-120 minutes.

Update at 19:28:02(BST): Following on from today's email issues, our SMTP (outbound) queues are quite large at this time.

If you have used a 34SP SMTP server, and the email has been accepted by the server, it will be in the queue for delivery.

We're doing our best to speed through this backlog, which is around 26,000 emails at the time of writing. We expect the whole backlog to go through in the following hours.

Please note, you should not be seeing any authentication failures - logging into email at this time. If you do, please do try resetting your access details in the control panel.



Update at 17:22:34(BST): Email access has now been restored. If you reset your password earlier on, as mentioned then you may need to reset it again.

Thank you for your patience as we worked to resolve this major outage.

---

Update at 15:52:37(BST): This is a recap of email events of the day as of the current situation at 3.50PM:

In the early hours of the morning today we suffered a major fault on one of our back end load balanced authentication systems. The immediate knock on effect was the failure of outbound email (SMTP). Some users of our newer Roundcube platform may have also noticed a problem accessing webmail and IMAP mailboxes too.

This fault only affected logins, no stored email was lost and no inbound email was bounced. No websites were affected and all continued to function normally.

At 1:20pm we restored a backup of authentication data from July 21st onto a temporary replacement system and most users' services will have been accessible from this time onwards. Any SMTP accounts created after July 21st, or accounts that updated their password after this date would not be part of the backup and require a password reset.

SMTP passwords can be reset at:

https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com

Once logged in select Manage websites, followed by the domain in question and then select Email. From there you should see the SMTP tab where you can reset a new password - and begin sending email once more. Please note that if you do not see an authentication failure, email is flowing. As we are seeing a spike in load following service restoration, some outbound email delivery may be a queued for delivery.

Users of our newer Roundcube email system will not presently be able to reset their password. Our engineers are working presently to restore this functionality ASAP. To recap:

Atmail users:
IMAP/POP3/Webmail not affected.
SMTP down until 1:20 p.m., then online for accounts unchanged prior to July 21st

Roundcube:
IMAP/POP3/Webmail down until 1:20 p.m., then online for accounts unchanged prior to July 21st
SMTP down until 1:20 p.m., then online for accounts unchanged prior to July 21st
IMAP/POP3/Webmail/SMTP accounts updated post July 21st, pending immediate fix

Please note that phone support is presently offline while we catchup with the support volume we are experiencing.

---

Update at 15:36:40(BST): The fix below should work for most users. We are still working to restore full functionality.

---

Update at 14:02:31(BST): The password updating method described below will work for SMTP but currently not for POP or IMAP services.

We continue to work to restore full functionality.

---

Update at 13:49:59(BST): IMAP and POP3 authentication is now available on the same basis - using outdated details. The details are from the 21st July, so if you have updated your password or added a user since then, please log in at https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com and update the password for that user to restore functionality.

---

Update at 13:22:24(BST): SMTP (outgoing email) authentication should now be working for most users. We are using an older version of our user database so recent additions of user or changes of password may not be functional. You can add/update these through https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com as normal. Do be aware that such changes may not be permanent once the underlying issue is resolved.

Work continues on restoring IMAP/POP3 functionality. We will provide updates on this as soon as we are able.

---

Update at 12:43:31(BST): Our engineers are currently working on implementing a temporary solution to the problem which will allow most email to function as normal. We hope to be able to provide an update on this shortly.

---

Update at 11:03:55(BST): We are continuing to work on this problem as the absolute highest priority, but due to the nature of the problem are unable to provide an estimated time to repair. The servers affected normally operate in a redundant fashion and we are also looking to work out how the cluster as a whole was affected.

---

Update at 09:57:47(BST): Affected systems are outgoing (SMTP) email and some webmail/incoming (IMAP/POP3) email services.

We are unable to provide an estimated time to repair at the moment. Our engineers are working to repair the affected systems as quickly as possible. We will continue to post updates as we have them.

---

We're currently experiencing a problem with some of our mail servers. This is affecting quite a few customers currently who are trying to auth/log in or send emails. There are spare systems in place which for some reason are also having similar problems. This is all being looked at with the highest priority and we expect everything to be back working shortly. Apologies for the inconvenience caused.

Start Time

28th August 2014, at 9:28AM

Description

Update at 14:04:47(BST):

---

Update at 10:20:24(BST): This maintenance is now largely complete. We are working on some unresolved issues at the moment.

---

Update at 07:26:31(BST): 34sp.com and the customer control panel at account.34sp.com and currently unavailable due to essential maintenance.

---

The maintenance window is due to end at 9am.

We apologise for any inconvenience caused.

Start Time

27th August 2014, at 4:50AM

Description

Update at 11:19:36(BST): Service has now been restored.


Update at 10:50:53(BST): Engineers are still investigating a failure in this server. Please note that email is not affected by this outage.


We are currently investigating in to an unplanned outage on the professional hosting server, prohost27.

34SP apologies for any inconvenience caused by this service disruption.

Start Time

18th August 2014, at 6:27AM

Description

Update at 10:50:23(BST): Update at 11:59:30(BST): The majority of service issues are now resolved, however, there is potential for some intermittent interruption as one of our network providers is conducting line upgrades today.

------------------------------
Due to a localised issue on our nameserver, services may be slow or unresponsive. This includes access to your 34sp control panel.

Our engineers are working on resolving this issue. Apologies for any inconvenience this may be causing in the meantime.

Start Time

16th August 2014, at 11:04AM