PRODUS- Integrations are Unavailable

Major incident Production US Integrations
2023-05-26 08:46 UTC · 3 hours, 26 minutes

Updates

Post-mortem

Summary:

Our planned maintenance on May 25th involved restarting our services in order to apply important configuration changes. During the restart process, the execution service encountered an issue and did not come back online as expected. Upon investigation, it was determined that the service was unable to establish connections with the database due to the existing connections not being properly cleared during the restart process. Our technical team promptly identified and cleared all the connections in the database that were causing the service disruption. Subsequently, we initiated a fresh restart of the service, which successfully brought it back online and restored its full functionality.

Customer Impact:

Timeout errors encountered while running the EPs.

Root Cause:

The execution service was unable to establish connections with the database due to the existing connections not being properly cleared during the restart process. This led to service unavailability.

Remediations:

After implementing the necessary measures, we successfully restarted the service, and it returned to being fully operational.

Future Mitigating Actions:

We will be reviewing the service resiliency, specifically with respect to connection closures during restarts, to ensure that all services are able to resume their normal operations without any interruptions.
Internal change management processes are under review and will be updated accordingly.

June 14, 2023 · 11:54 UTC
Resolved

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at support@aeratechnology.com

May 26, 2023 · 12:12 UTC
Monitoring

We have identified the cause of the reported issues and service has been restored. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm the resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team support@aeratechnology.com

Thank you for your patience and understanding whilst our engineers restored service.

May 26, 2023 · 10:25 UTC
Investigating

We are continuing to work towards restoring service for the Execution Processing in PRODUS environment. Our engineers are diligently working to narrow down the root cause. We will continue to keep you informed as the investigation progresses. We appreciate your continued patience whilst we work towards a resolution.

May 26, 2023 · 09:38 UTC
Investigating

We continue to investigate the integration issues impacting Execution Processings(EPs). Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

May 26, 2023 · 09:01 UTC
Issue

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the platform. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.

May 26, 2023 · 08:46 UTC

← Back