PROD US - Integration Delays

Summary:

On March 12th, 2024, our Production Service Engineering team observed that a small subset of integration jobs within the PRODUS environment were taking longer than usual, resulting in reports of delays. Investigations revealed that a cascading impact from the previous incident on March 11th was affecting a small number of large file transfers. The Production Service and Product Engineering teams adjusted additional file transfer configuration settings and service resumed. Subsequently, a product fix was identified to permanently alleviate the issue and was deployed to all environments. Following the fix, all integrations were validated and confirmed to be functioning smoothly without further issues.

Customer Impact:

A small subset of customer integrations were impacted by performance degradation.

Root Cause:

The root cause lies in the system’s handling of large file transfers, resulting in a delay in data processing.

Remediations:

• To address the issue, adjustments were made to the file transfer configuration settings to allow the crawlers and integrations to start working as expected again. In parallel, Engineering worked on a permanent mitigation for the issue that was released to all environments the following day.

Future Mitigating Actions:

• The engineering team has implemented the fix across all environments.
• Additional test cases have been added to the full suite of regression tests.

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at support@aeratechnology.com

Our engineers have restored the service for all integrations. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm the resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience and understanding whilst our engineers restored service.

Our engineers have restored the service for all integrations. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm the resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience and understanding whilst our engineers restored service.

Our engineers have restored the service for all integrations. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm the resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience and understanding whilst our engineers restored service.

Our engineers have restored the service for all integrations. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm the resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience and understanding whilst our engineers restored service.

We are continuing to work towards restoring service for the delayed integrations. Our engineers are diligently working to narrow down the root cause. We will continue to keep you informed as the investigation progresses. We appreciate your continued patience whilst we work towards resolution.

We are continuing to work towards restoring service for the delayed integrations. Our engineers are diligently working to narrow down the root cause. We will continue to keep you informed as the investigation progresses. We appreciate your continued patience whilst we work towards resolution.

We are continuing to investigate the delayed integration issues. Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the delayed integrations. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.