PRODUS is unavailable

Summary:

On 5th February 2025, a sector of the PRODUS environment experienced a temporary outage, affecting access for several customers. The issue was caused by an unexpected spike in utilisation on the application nodes, preventing normal operation. To restore service, the affected services were restarted, stabilising the environment. Engineering is conducting further analysis and implementing measures to prevent recurrence.

Customer Impact:

A subset of customers were unable to access the PRODUS environment during the outage.

Root Cause:

A request exceeding the API usage limit caused a spike in CPU utilisation on the application nodes, resulting in a service disruption as it surpassed the system’s processing capacity.

Remediations:

The Production Services Engineering team immediately restarted the affected application nodes to restore service. This action successfully stabilised the environment and enabled users and integrations to resume normal operation.

Future Mitigating Actions:

The Engineering team is investigating stricter and more dynamic enforcement of API usage limits.
Implement enhanced granular monitoring and alerting for API usage.
Provide clearer documentation and communication regarding API usage limits and best practices.

We have confirmed internally and with our customers that the Aera platform is now fully restored.

We appreciate your patience during this incident and apologise for any inconvenience that this issue may have caused. Our teams are now working on documenting a comprehensive root cause analysis which we will share with you shortly.

If you have any questions or experience any further problems please don’t hesitate to reach out to our Support team at Aera Support Portal

We have identified the cause of the reported issues. Our engineers have confirmed that services have been restored. We will continue to monitor to ensure no additional issues arise and will send a further update to confirm the resolution.

You should now be able to resume normal activities however if you continue to experience any problems please contact our support team through the Aera Support Portal

Thank you for your patience and understanding whilst our engineers restored service.

We are continuing to investigate the PRODUS unavailability issues. Our engineers are actively working to restore service as quickly as possible. Thank you for bearing with us whilst we work through these issues.

This notice is to advise you that we are receiving reports of our customers experiencing difficulties with the platform. We are actively investigating and will provide regular updates until the issues are resolved.

Our apologies for the inconvenience this may be causing and we appreciate your patience as we investigate further.