34SP.com System Status History

Description

Update at 16:20:06(BST): Control panel logins are available again and the correct domain names are now showing.

---

Update at 16:04:35(BST): Until we have this issue corrected, control panel logins are being disabled. We are working to correct the problem and allow logins again as soon as possible.

---

We are aware of the incorrect domain name showing for some customers in our control panel at https://account.34sp.com

We hope to have this issue resolved shortly.

Start Time

28th August 2014, at 3:45PM

Description

Update at 22:33:00(BST): Queues are now under 2,500, and we expect the remainder of mail to be delivered by 11PM.

Update at 21:19:43(BST): SMTP queues are down from 26,000 to 16,000. Based on this throughput all remaining email should go through in the next 90-120 minutes.

Update at 19:28:02(BST): Following on from today's email issues, our SMTP (outbound) queues are quite large at this time.

If you have used a 34SP SMTP server, and the email has been accepted by the server, it will be in the queue for delivery.

We're doing our best to speed through this backlog, which is around 26,000 emails at the time of writing. We expect the whole backlog to go through in the following hours.

Please note, you should not be seeing any authentication failures - logging into email at this time. If you do, please do try resetting your access details in the control panel.



Update at 17:22:34(BST): Email access has now been restored. If you reset your password earlier on, as mentioned then you may need to reset it again.

Thank you for your patience as we worked to resolve this major outage.

---

Update at 15:52:37(BST): This is a recap of email events of the day as of the current situation at 3.50PM:

In the early hours of the morning today we suffered a major fault on one of our back end load balanced authentication systems. The immediate knock on effect was the failure of outbound email (SMTP). Some users of our newer Roundcube platform may have also noticed a problem accessing webmail and IMAP mailboxes too.

This fault only affected logins, no stored email was lost and no inbound email was bounced. No websites were affected and all continued to function normally.

At 1:20pm we restored a backup of authentication data from July 21st onto a temporary replacement system and most users' services will have been accessible from this time onwards. Any SMTP accounts created after July 21st, or accounts that updated their password after this date would not be part of the backup and require a password reset.

SMTP passwords can be reset at:

https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com

Once logged in select Manage websites, followed by the domain in question and then select Email. From there you should see the SMTP tab where you can reset a new password - and begin sending email once more. Please note that if you do not see an authentication failure, email is flowing. As we are seeing a spike in load following service restoration, some outbound email delivery may be a queued for delivery.

Users of our newer Roundcube email system will not presently be able to reset their password. Our engineers are working presently to restore this functionality ASAP. To recap:

Atmail users:
IMAP/POP3/Webmail not affected.
SMTP down until 1:20 p.m., then online for accounts unchanged prior to July 21st

Roundcube:
IMAP/POP3/Webmail down until 1:20 p.m., then online for accounts unchanged prior to July 21st
SMTP down until 1:20 p.m., then online for accounts unchanged prior to July 21st
IMAP/POP3/Webmail/SMTP accounts updated post July 21st, pending immediate fix

Please note that phone support is presently offline while we catchup with the support volume we are experiencing.

---

Update at 15:36:40(BST): The fix below should work for most users. We are still working to restore full functionality.

---

Update at 14:02:31(BST): The password updating method described below will work for SMTP but currently not for POP or IMAP services.

We continue to work to restore full functionality.

---

Update at 13:49:59(BST): IMAP and POP3 authentication is now available on the same basis - using outdated details. The details are from the 21st July, so if you have updated your password or added a user since then, please log in at https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com and update the password for that user to restore functionality.

---

Update at 13:22:24(BST): SMTP (outgoing email) authentication should now be working for most users. We are using an older version of our user database so recent additions of user or changes of password may not be functional. You can add/update these through https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com' target='_blank'>https://account.34sp.com as normal. Do be aware that such changes may not be permanent once the underlying issue is resolved.

Work continues on restoring IMAP/POP3 functionality. We will provide updates on this as soon as we are able.

---

Update at 12:43:31(BST): Our engineers are currently working on implementing a temporary solution to the problem which will allow most email to function as normal. We hope to be able to provide an update on this shortly.

---

Update at 11:03:55(BST): We are continuing to work on this problem as the absolute highest priority, but due to the nature of the problem are unable to provide an estimated time to repair. The servers affected normally operate in a redundant fashion and we are also looking to work out how the cluster as a whole was affected.

---

Update at 09:57:47(BST): Affected systems are outgoing (SMTP) email and some webmail/incoming (IMAP/POP3) email services.

We are unable to provide an estimated time to repair at the moment. Our engineers are working to repair the affected systems as quickly as possible. We will continue to post updates as we have them.

---

We're currently experiencing a problem with some of our mail servers. This is affecting quite a few customers currently who are trying to auth/log in or send emails. There are spare systems in place which for some reason are also having similar problems. This is all being looked at with the highest priority and we expect everything to be back working shortly. Apologies for the inconvenience caused.

Start Time

28th August 2014, at 9:28AM

Description

Update at 14:04:47(BST):

---

Update at 10:20:24(BST): This maintenance is now largely complete. We are working on some unresolved issues at the moment.

---

Update at 07:26:31(BST): 34sp.com and the customer control panel at account.34sp.com and currently unavailable due to essential maintenance.

---

The maintenance window is due to end at 9am.

We apologise for any inconvenience caused.

Start Time

27th August 2014, at 4:50AM

Description

Update at 11:19:36(BST): Service has now been restored.


Update at 10:50:53(BST): Engineers are still investigating a failure in this server. Please note that email is not affected by this outage.


We are currently investigating in to an unplanned outage on the professional hosting server, prohost27.

34SP apologies for any inconvenience caused by this service disruption.

Start Time

18th August 2014, at 6:27AM

Description

Update at 10:50:23(BST): Update at 11:59:30(BST): The majority of service issues are now resolved, however, there is potential for some intermittent interruption as one of our network providers is conducting line upgrades today.

------------------------------
Due to a localised issue on our nameserver, services may be slow or unresponsive. This includes access to your 34sp control panel.

Our engineers are working on resolving this issue. Apologies for any inconvenience this may be causing in the meantime.

Start Time

16th August 2014, at 11:04AM

Description

Update at 09:45:57(BST): Affected customers were contacted and stability has returned. Please contact support if you are still having any further problems.

We're currently seeing high loads on this server which is causing sites to load slowly. There looks to be a problem with a couple of customer websites which our engineers are currently looking into.

Start Time

12th August 2014, at 10:00AM

Description

Update at 00:22:34(BST): This work has been completed ahead of time and full network functionality has been restored.

---

We are currently replacing a core network router. Services may be intermittent for the duration of this replacement, which should be complete before 01:00 BST.

Start Time

3rd August 2014, at 11:59PM

Description

Update at 10:43:18(BST): No further issues have been seen.

Update at 21:16:57(BST): Services are running well again. We're continuing to investigate and monitor the server for now.

Engineers are currently working on the current high loads on this server. Apologies for any inconvenience caused.

Start Time

1st August 2014, at 7:04PM

Description

Update at 10:42:27(BST): No further maintenance is now planned.

Update at 21:20:11(BST): We're continuing to monitor the situation with this server. Since the reboot, we're seeing near 100% uptime again but we're still working on and monitoring this server closely. A further reboot may be needed which we will try to sort late Sunday night, if this is required.

We've seen an increased amount of attacks to this server recently which has also caused some poor performance whilst this has been happening. We've been working closely with our security team to quickly resolve this and improve things. Unfortunately a quick server reboot is needed to apply some new updates. We will do this now and the downtime should be no longer than a couple of minutes. Apologies for any inconvenience this may cause.

Start Time

31st July 2014, at 3:28PM

Description

Update at 08:14:01(BST): The issue has now been resolved with this server.We are currently investigating into a server side issue with the server, prohost27. Once we receive an update from our engineers we will advise further.

We apologies for any inconvenience caused.

The server unfortunately required a restart and system check, this has now been completed and normal service has resumed.

Start Time

21st July 2014, at 8:02AM