UAT US - Integrations are Unavailable

Major incident UAT US Integrations
2024-02-01 13:00 UTC · 14 hours, 25 minutes

Updates

Post-mortem

Summary:

On 1st Feb 2024, our Operations team reported that crawlers were not executing in UATUS01. This caused disruption for users attempting to extract data. The issue stemmed from a defect in the application where, under certain circumstances, it caused the scheduler engine to become unresponsive. Due to this no schedules were executed and crawlers were stalled.
To address the issue our team restarted the impacted services restoring integrations to their normal state

Customer Impact:

The issue of crawlers not moving in UATUS01 disrupted the workflow of users resulting in on time data unavailability

Root Cause:

The underlying cause of the issue was a defect in the application

Remediations:

To address the issue, our team restarted impacted services, restoring integrations to their usual state

Future Mitigating Actions:

Our engineers have identified a fix for the defect, which is currently being tested in a lower environment

February 9, 2024 · 10:32 UTC
Resolved

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at support@aeratechnology.com

February 2, 2024 · 03:24 UTC
Monitoring

We have identified the cause of the reported issues. Our engineers recommend restarting the agents manually from the respective project to get agents back online and retrigger integrations. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm the resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience.

February 1, 2024 · 15:42 UTC
Investigating

We are continuing to work towards restoring service for the integration issues. Our engineers are diligently working to narrow down the root cause. We will continue to keep you informed as the investigation progresses. We appreciate your continued patience whilst we work towards resolution.

February 1, 2024 · 15:11 UTC
Investigating

We are continuing to investigate the integration issues. Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

February 1, 2024 · 14:18 UTC
Issue

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the platform. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.

February 1, 2024 · 14:00 UTC

← Back