PROD US - Experiencing issues with Analytics service

Summary:

On Sep 26th, users reported multiple issues with accessing and using Analytics reports. These issues manifested themselves in various ways including UI errors and duplicate data displays. Our team identified the root cause as a bug within our internal migration tool, causing some reports across the environment to be inaccessible or miscalculated. To restore functionality as quickly as possible the impacted reports were reverted to their last good state. To address this permanently, a code change has been implemented and deployed to fix the bug.

Customer Impact:

Users experienced various issues accessing and using Analytics reports.

Root Cause:

This originated from a bug within our internal migration tool that impacted reports in the Analytics service across all projects within the same environment.

Remediations:

To accelerate resolution the impacted reports were reverted to their previous last good state
In response to the identified root cause, our engineering team deployed a code change to fix and prevent the recurrence of the issue.

Future Mitigating Actions:

This bug scenario has been added to the regression test suite.
Regression testing for this service is under review and will be enhanced and updated accordingly.
Internal change management processes are under review and will be updated accordingly.

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at support@aeratechnology.com

We are continuing to investigate the Analytics service issues. Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the Analytics service in PROD US. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.