Mercury email

resolved

Description

We are experiencing a large influx of email causing intermittent slowness and availability issues on the Mercury email platform. Engineers are working to resolve this issue as a matter of urgency.


Updates

23rd April 2019, at 11:25AM

[resolved] It has now been over 2 weeks since any reported incidents and this issue is now marked as resolved.

3rd April 2019, at 5:45PM

[monitoring] Today has not seen a repeat of the email access issues but we continue to keep a close eye on the system and work to improve capacity.

2nd April 2019, at 11:34AM

[unresolved] With increasing utility during the day, we're seeing Mercury email access slow down again. We're attempting to mitigate that and some customers may see their mail access affected. Today we are also deploying additional hardware to provide extra capacity where a need has been identified.

1st April 2019, at 5:36PM

[monitoring] All access is currently working. We are closely monitoring the situation and in the coming days will be bringing new hardware online with the aim of increasing the capacity of the system.

1st April 2019, at 2:19PM

[unresolved] Almost all access has now been re-enabled so the majority of users will find their email working. We continue to work on fixing access for those that are still having difficulty.

1st April 2019, at 11:06AM

[unresolved] We have lowered the load on our email servers sufficiently and are gradually enabling access for all users. We hope that full access will be restored again if all goes according to plan in the next hour. We will then continue to closely monitor the servers and take action as necessary.

1st April 2019, at 10:12AM

[unresolved] We're seeing a recurrence of slow email access this morning and are continuing work to resolve this matter.

30th March 2019, at 5:23PM

[monitoring] Maintenance work continued through Friday 29th well into the early hours helping to alleviate some load. Access should have been restored for all users late Friday evening. Access remains fine during Saturday 30th.

The underlying issues remain a work in progress, and we continue to work on the longer term solution.

29th March 2019, at 4:49PM

[unresolved] The issue seems to be traced back from our hardware supplier (Nexenta) to file locking. This file locking means that a smaller issue triggers a load increase; as load increases, services slow down, causing in turn more load as user demand is increased.

We plan to start moving users off their server provision into our own in house development in due course.

We expect the issues being seen today will calm down as the evening progresses.

29th March 2019, at 1:29PM

[unresolved] User log in indexes are now being served from a separate server in an attempt to reduce load spikes. This has automatically synced into the system which in itself has caused a peak in load. We will continue to work on the issue and update status as we know more.

28th March 2019, at 9:02PM

[monitoring] In the last half hour, service looks to have returned to normal. We are continuing to work on the underlying issues and will keep a close eye on the Mercury email system.

28th March 2019, at 4:44PM

[unresolved] The reboots (including some minor configuration changes) have failed to make more than a temporary impact on the load we're seeing. Investigation continues.

28th March 2019, at 2:57PM

[unresolved] Engineers are going to attempt a sequential reboot of servers in the affected array. We believe that this may lower the load as they rejoin the array in turn with empty caches. We would like to assure customers that we have multiple engineers working on this issue and it is receiving our highest priority.

28th March 2019, at 1:06PM

[unresolved] We are seeing load intermittently reduce on the Mercury platform but this issue is ongoing and our engineers are working on a resolution.