UATIRL - Cortex services are unavailable

Minor incident UAT IRL (Retired) Cortex
2024-07-29 07:19 UTC · 9 hours, 9 minutes

Updates

Post-mortem

Summary:
Following recent infrastructure maintenance, an issue was identified on July 28th that caused an interruption of Cortex services. The issue was escalated to the Infrastructure Engineering team, where it was identified that the unintentional loss of previous configuration changes during the maintenance led to issues. To rectify the issue, changes were reverted to their previous state, which restored service and enabled Cortex services to be reinstated. Subsequent monitoring of the restored services detected that some services remained intermittent with their availability. An additional configuration change was identified as being needed and was swiftly implemented by the Infrastructure Engineering team.

Customer Impact:
Users experienced Cortex services being intermittently unavailable in UATIRL01

Root Cause:
As identified by Infrastructure Engineering, an omission of previous configuration changes during the infrastructure maintenance, and subsequent restoration, resulted in intermittent disruption of Cortex services.

Remediations:
The required configuration settings were updated allowing full functionality to be restored.

Future Mitigating Actions:
Internal change management processes have undergone review. A gap within the process was identified, and necessary updates have been made and communicated to all stakeholder teams.

August 8, 2024 · 15:07 UTC
Resolved

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at support@aeratechnology.com

July 29, 2024 · 16:27 UTC
Investigating

Our engineers are continuing to investigate the root cause of the Cortex issues. We understand the business impact this issue may have and are working to restore service as quickly as possible. Again, we thank you for your continued patience and understanding.

July 29, 2024 · 11:26 UTC
Investigating

We are continuing to work towards restoring service for the Cortex issues. Our engineers are diligently working to narrow down the root cause. We will continue to keep you informed as the investigation progresses. We appreciate your continued patience whilst we work towards resolution.

July 29, 2024 · 09:13 UTC
Investigating

We are continuing to investigate the Cortex issues. Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

July 29, 2024 · 07:55 UTC
Issue

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with Cortex Services. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.

July 29, 2024 · 07:19 UTC

← Back