IVAO - Tracker API – Incident details

All systems operational

Tracker API

Resolved
Major outage
Started about 2 months agoLasted about 2 hours

Affected

REST APIs

Partial outage from 2:51 AM to 2:57 AM, Degraded performance from 2:57 AM to 3:01 AM, Operational from 3:01 AM to 4:47 AM

Tracker

Partial outage from 2:51 AM to 2:57 AM, Degraded performance from 2:57 AM to 3:01 AM, Operational from 3:01 AM to 4:47 AM

Updates
  • Postmortem
    UTC
    Postmortem

    During a manual cluster resource rebalance, a load surge caused a temporary unavailability of our Web Cluster control plane.
    This prevented the normal scaling and operations from completing successfully.

    For future cluster maintenance operations, a slower and more gradual rollout of our deployments will be made to avoid the surge we faced today.

  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    We are not seeing any further impact at the moment.
    We will continue monitoring for the time being.

  • Identified
    UTC
    Identified

    We are seeing a recovery of the services.
    We will continue monitoring this issue.

  • Investigating
    UTC
    Investigating

    We are experiencing a partial outage on the Tracker API. A cluster upgrade is currently in progress.
    We are currently investigating this incident.