Themo - Unexpected downtime due to DST adjustment – Incident details

Unexpected downtime due to DST adjustment

Resolved
Major outage
Started 25 days agoLasted about 9 hours

Affected

Themo API

Major outage from 5:00 AM to 2:00 PM

IoT Hub

Major outage from 5:00 AM to 2:00 PM

Optimisation Service

Major outage from 5:00 AM to 2:00 PM

Data Services

Major outage from 5:00 AM to 2:00 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Identified
    Identified

    This disruption resulted from a chain of technical complications beyond our initial controls. A daylight-saving time (DST) adjustment introduced duplicate price hours, causing errors in optimization. This, in turn, led to an increased volume of requests, resulting in substantial system overload and rate-limiting. Eventually, our systems crashed, which escalated to IP blocking and domain service issues. Furthermore, on October 25th, Microsoft Azure released a network layer update affecting a subset of services essential to our microservices infrastructure, complicating recovery efforts.