IVAO - Notice history

All affected services have remained stable following the recovery actions taken earlier, and we are not currently observing any service impact.

We will continue to closely monitor the situation, as our service provider's maintenance activities are still ongoing. Whilst we do not anticipate further disruption, there remains a possibility of service degradation until their work has been fully completed.

Thank you for your patience and understanding. Happy flying/controlling!

Most of our services have now stabilised and are operating normally. We are currently verifying the status of all remaining affected components.

One of our service providers inadvertently degraded their service whilst performing maintenance, resulting in reduced capacity. This caused several nodes in our Kubernetes cluster, which hosts the majority of our websites and APIs, to become unresponsive.

Our team is actively working to restore service and is manually scaling the cluster to recover affected systems as quickly as possible. We will continue to provide updates as more information becomes available.

During a manual cluster resource rebalance, a load surge caused a temporary unavailability of our Web Cluster control plane.
This prevented the normal scaling and operations from completing successfully.

For future cluster maintenance operations, a slower and more gradual rollout of our deployments will be made to avoid the surge we faced today.

We are not seeing any further impact at the moment.
We will continue monitoring for the time being.

We are seeing a recovery of the services.
We will continue monitoring this issue.