34SP.com System Status History

Network and SMTP Issues (Multiple)

resolved

Description

Update at 00:36:47(BST): Update at 19:09:11(BST): Update at 16:06:03(BST): Some services are being brought back online while we continue to resolve the situation.

Update at 15:44:46(BST): The network issue has been restricted to a certain section of our network. This will still be affecting some clients and SMTP services. We're doing everything we can to resolve these issues as soon as possible.

The network issues are also affecting our SMTP services. We're hoping to have these resolved shortly

Update at 15:03:31(BST): Our engineers are on site at the datacentre and are investigating.


Update at 14:20:01(BST): We are aware of a problem affecting multiple servers at this time. Our engineers are investigating this. We are finding the route of the issue and will restore normal service as soon as possible. We apologise for the inconvenience caused.

Start Time

13th September 2012, at 3:07PM

Resolution Time

15th September 2012, at 11:36PM

Atmail MailStorage (ATMAIL CLUSTER)

resolved

Description

Update at 11:12:34(BST): No further issues reported


Update at 23:15:14(BST): All services are now back online, we will monitor this for a while just incase of any problems, please do let us know if you see any issues by contacting support.


Update at 23:11:14(BST):

Mail services are now offline whilst email is synced to new storage.





Over the next 30-40 minutes we will be bringing additional storage online for the mail servers, this will involve approximatly 20-25minutes of downtime where email will not be accessible.

This is necessary work and we hope to have the service back up quicker than this.

If you do have any issues after this has taken place please do get in contact with us.

Thanks

Start Time

6th September 2012, at 11:06PM

Resolution Time

10th September 2012, at 10:12AM

Delays to outgoing mails

resolved

Description

Update at 19:26:31(BST): The issue has now been resolved with this server.

We are currently experiencing delays to outgoing email. This was caused by an abused customer SMTP account. Spam emails have been removed and the back log is now being processed.

We would like to take this opportunity to remind customers to have up to date anti virus on their devices and to run scans regularly.
Please also do not use passwords that contain dictionary words as these can be hacked easily.

Apologies for any inconvenience caused.

Start Time

5th September 2012, at 10:27AM

Resolution Time

5th September 2012, at 6:26PM

Server migrations (Various)

resolved

Description

Update at 11:20:34(BST): The issue has now been resolved with this server.Engineers have now resolved this issue and all sites known to have been affected are now resolving correctly. If your own site appears to still have issues, please contact support@34sp.com and we can look into this directly for you.

Update at 09:58:47(BST): It seems that a minority of sites on personal hosting are now not responding following this migration. This will affect some sites on the following servers:

holly.34sp.com
robbie.34sp.com
erasmus.34sp.com
k9.34sp.com
ted.34sp.com
ash.34sp.com

We are working to resolve this as soon as possible.


Update at 07:11:59(BST): This migration has been completed successfully. If you have any further issues please contact support as usual.


The following servers are now down for their migration, as notified to users on the servers. We will restore them to service as soon as possible, and expect this to be in line with the notified outage.

business4.34sp.com
business6.43sp.com

holly.34sp.com
robbie.34sp.com
erasmus.34sp.com
k9.34sp.com
ted.34sp.com
ash.34sp.com

server15.xenserve.com

Start Time

2nd September 2012, at 12:00AM

Resolution Time

2nd September 2012, at 10:20AM

Slow mail delivery (server12.xenserver.com)

resolved

Description

Update at 11:47:08(BST):
All spam mail has now been deleted and all services are running normally. Again, thank you for your patience and apologies for any inconvenience caused.

Update at 09:22:10(BST):
We are continuing to delete the spam emails. As there was over 100,000 this is casing some slow down to website load times and email retrieval / delivery. We thank you for your patience. We would like to take this opportunity to remind customers that scripts on your sites are your responsibility and we urge you to keep all scripts up to date. Failure to do so, leaves your sites open to these types of spam attack and other vulnerabilities.

Update at 08:47:31(BST): The engineers are presently working on this and it appears the mail is not genuine so is in the process of being removed. Normal mail delivery will resume and no mail should be lost.

There is currently a large mail queue on the server which has formed overnight which will result in slow mail delivery to and from accounts on reseller server12.

Engineers are currently investigating into whether this is due to a genuine mail shot or a result of an exploited website being used to send spam email.

We will provide an update as soon as we receive confirmation.

34SP apologises for any inconvenience caused and gives assurances it will be resolved as soon as possible.

Start Time

30th August 2012, at 8:04AM

Resolution Time

30th August 2012, at 10:47AM

Network issues (Various)

resolved

Description

Update at 16:41:08(BST): This has now been resolved and connectivity restored for these servers.


We are aware of connectivity issues to the following servers:

business2
vps24
vps11
xenserve13
prohost6
vps16
business7
xenserve12
business1
xenserve14

Sites on these servers will currently be unavailable.

We are working to restore this as soon as possible.

Start Time

24th August 2012, at 4:11PM

Resolution Time

24th August 2012, at 3:41PM

prohost7 (Prohost7)

resolved

Description

Update at 19:16:15(BST): The issue has now been resolved with this server.This issue is now resolved

We are aware that the server prohost7.34sp.com is currently unavailable and are investigating this as a matter of urgency.

Start Time

23rd August 2012, at 7:06PM

Resolution Time

23rd August 2012, at 6:16PM

Mail upgrade beginning (Atmail services)

resolved

Description

Update at 23:04:57(BST): The issue has now been resolved with this server.Upgrades completed successfully at approximately 22.57. All services should now be working.

Update at 22:46:25(BST): The upgrade is now beginning, we expect it to take no longer than 20 minutes.

Atmail is going offline in the next few minutes. This is in order to implement some emergency upgrades to deploy additional capacity.

Email services hosted on Atmail will be inaccessible for 10-20 minutes at most.

Start Time

16th August 2012, at 10:25PM

Resolution Time

16th August 2012, at 10:04PM

DNS fault (Various)

resolved

Description

Update at 22:24:34(BST): Update at 00:30:52(BST): At around 8.30PM (August 14th) we were forced to restart bind on our primary DNS server (ns.34sp.com). Bind is the system that converts domain name addresses to IP addresses. Previously undetected configuration errors on the system meant that bind did not restart automatically. Work began immediately on investigating and correcting several configuration records that were corrupted. During this time our secondary DNS server (ns2.34sp.com) was unaffected and continued to serve DNS.

The primary DNS system was restored at approximately 11.40PM. Any affected users would have seen services restored at this time. Please note that each end user would have been affected differently; depending on ISP caching and access of nameservers, many users would experience no loss of service. For affected users, email sent during the event, should be queued by the sending mail server and redelivered without fault.

The nature of this fault was quite rare, and the causes now rectified. We do not anticipate a repeat of this event.

Start Time

14th August 2012, at 11:47PM

Resolution Time

16th August 2012, at 9:24PM

Rebooting Prohost4 (prohost4.34sp.com)

resolved

Description

Update at 16:31:26(BST): The issue has now been resolved with this server.We are currently rebooting the server as an emergency measure. Apologies for any inconvenience caused.

Start Time

14th August 2012, at 12:44PM

Resolution Time

14th August 2012, at 3:31PM