UATIRL is unavailable

Summary:

On 19th Mar 2024 (UTC), Aera monitoring tools began alerting for intermittent service availability issues across various components within the UAT IRL environment, affecting user’s ability to access and utilise the platform.
Our Production Service (PSE) and Site Reliability Engineering (SRE) teams conducted investigations and identified the issue as stemming from an automated configuration update that was unable to complete successfully.
This resulted in cascading effects throughout the environment, necessitating significant time to restore the platform to full functionality. During the restoration process, the platform underwent updates to prevent a recurrence of this issue within the same environment. Additionally, the PSE team documented all mitigation and resolution steps to expedite recovery in the event of a similar issue arising in a different environment.

Customer Impact:

Users were unable to access or utilise the UAT IRL environment successfully.

Root Cause:

An automated configuration update was unable to complete successfully, leading to a cascading effect throughout the environment that prevented services from establishing successful connections with each other, resulting in the platform becoming unavailable.

Remediations:

Services were reconstructed with the correct configuration.

Future Mitigating Actions:

All recovery steps have been documented to expedite resolution in future occurrences.
The service initially responsible for the auto-update failure has been retired within our next generation infrastructure; migration from the current to next generation infrastructure is currently in progress for all projects

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at support@aeratechnology.com

We have identified the cause of the reported issues. Our engineers have restored the services and application is up now. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience and understanding whilst our engineers restored service.

We have identified the cause of the reported issues. Our engineers have Restored the Issue UATIRL is unavailable and service has been restored. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience and understanding whilst our engineers restored service.

We’ve identified the cause of the UATIRL01 platform issue and are working on a fix. We don’t have an estimated time for completion yet, but we’ll keep you updated once there is a change in the current status. We understand the business impact this issue may have and are working to restore service as quickly as possible. We thank you for your continued patience and understanding.

Our engineers are continuing to investigate the root cause of the UATIRL is unavailable issues. We understand the business impact this issue may have and are working to restore service as quickly as possible. Again, we thank you for your continued patience and understanding.

We are continuing to work towards restoring service for the UATIRL Platform. Our engineers are diligently working to narrow down the root cause. We will continue to keep you informed as the investigation progresses. We appreciate your continued patience while we work towards a resolution.

We are continuing to investigate the platform issues. Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the platform. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.