PRODUS is unavailable

Summary:

On 30th Apr 2024, internal monitoring alerted the Production Services Engineering (PSE) team that the Production US (ProdUS) environment was unavailable, leading to users experiencing disruption while attempting to access the Aera Application. Upon investigation, it was discovered that a small subset of complex transaction types were not being managed as expected, resulting in the access disruption. To restore service, the impacting transactions were terminated and service subsequently resumed. To permanently resolve the issue, the Engineering team identified and coded a fix, which will be deployed shortly.

Customer Impact:

The ProdUS platform was unavailable, causing disruptions to access and business workflows.

Root Cause:

The root cause was the occurrence of a small subset of complex transaction types not being managed as expected, which blocked further incoming requests, thereby causing the application to become unavailable.

Remediations:

The team terminated the relevant transactions, allowing service to resume. A permanent fix has been identified and implemented to prevent a similar occurrence in the future. The fix will be deployed to customer environments shortly.

Future Mitigating Actions:

Enhanced and expanded use cases, focused within this area of the application, will be added to Engineering’s test suite.

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at support@aeratechnology.com

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the platform. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.