UI and parts of the API were temporarily unavailable

  • Now
  • Jul 6, 2023, 7:08:08 AM GMT+2
    Resolved
    We have resolved the incident.
  • Jul 6, 2023, 6:49:50 AM GMT+2
    Monitoring
    UI and affected API parts are fully operational again since 06:26 am - we are closely monitoring the recovery of all affected systems.
  • Jul 6, 2023, 6:01:00 AM GMT+2
    Identified
    A minor Kubernetes update broke internal DNS services, causing parts of the API and UI to become unavailable. Event queue processing and notification delivery continued to work. We identfied the issue during the upgrade process and were able to roll back after 2 minutes, however the rollback process of all node pools took a moment.
Affected services
API
Partial outage
Web Application
Major outage