PROD IRL is unavailable

Major incident Production IRL Platform Access Analytics Cognitive WorkBench Data WorkBench Discovery Integrations Other Features Skills
2023-09-18 12:59 UTC · 1 hour, 21 minutes

Updates

Post-mortem

Summary:

On 18th Sep 2023, users experienced disruption when trying to access the Aera Application in the PRODIRL environment. Upon investigation, it was discovered that the unintended simultaneous occurrence of multiple transactions was the cause of the issue, leading to disruption in access. To restore service, the impacting transactions were terminated and service subsequently resumed. To permanently resolve the issue the Engineering team has identified and implemented a fix within all environments

Customer Impact:

The issue affected users as they couldn’t use the PROD IRL platform, causing disruptions in their workflow

Root Cause:

Simultaneous transactions occurred blocking further incoming requests causing the application to become unavailable

Remediations:
The team terminated the relevant transactions allowing service to resume.
A permanent fix has been identified and implemented to prevent a similar occurrence in the future

Future Mitigating Actions:

The Engineering team will be implementing further enhancements focused within this area of the application to improve the stability and robustness of the application

November 8, 2023 · 08:45 UTC
Resolved

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at support@aeratechnology.com

September 18, 2023 · 14:18 UTC
Monitoring

We have identified the cause of the reported issues. PROD IRL environment has been stable for the last 15 Minutes. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm the resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience and understanding whilst our engineers restored service.

September 18, 2023 · 13:33 UTC
Investigating

We are continuing to investigate the platform issues. Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

September 18, 2023 · 13:19 UTC
Issue

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the platform. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.

September 18, 2023 · 12:59 UTC

← Back