Cortex services are Unavailable

Summary:

On 15th October, internal monitoring indicated that a subset of features in the US PROD and UAT environments were unavailable. Upon investigation, our teams identified a configuration issue that resulted in requests being incorrectly directed. This misconfiguration was primarily attributed to limitations imposed by a vendor, which caused traffic to be unable to successfully redirect. Our teams acted swiftly to address the issue and restart the relevant service, thereby successfully restoring access to the application.

Customer Impact:

Customers reported issues accessing the Cortex feature of the Aera application

Root Cause:

A configuration issue was identified, resulting in unsuccessful request direction. This misconfiguration was primarily due to limitations imposed by a vendor.

Remediations:

To resolve the issue, the teams adjusted the configuration and restarted the relevant service, which successfully restored access to the application.

Future Mitigating Actions:

A medium term fix has been identified and implemented to prevent this issue from reoccurring. A long term fix has been added to the roadmap.
Internal change management processes are under review and will be updated accordingly
Monitoring and alerting for the impacted service have been reviewed, and a gap has been identified. Steps are in progress to address this gap.

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at Aera Support Portal

We are continuing to investigate the Cortex issues. Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the Cortex services on the platform. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.