Duplicate alert responses sent via SMS and duplicate alerts for inbound SMS alert sources

Resolved
Duration: 10hMay 17, 2024

Affected services

SMS

Degraded

Timeline

Resolved
20:07May 17, 20241y ago

We are pleased to inform you that the incident has been resolved. While we are continuing to observe SMS delivery delays in a few instances when sending messages to the Telefonica O2 Network, most SMS messages are now arriving at the usual speed. We appreciate your patience and understanding during this time.

SMS
Monitoring
16:08May 17, 20241y ago

Our provider has implemented a change that resolved the duplicates received for inbound SMS from the Telefonica O2 Network in Germany. We recommend customers of inbound SMS alert sources to turn on their alert sources again. However, we are continuing to observe SMS delivery delays when sending messages to the Telefonica O2 Network in Germany. We apologize for any inconvenience this may cause and appreciate your patience as we continue to monitor the situation.

SMS
Monitoring
13:08May 17, 20241y ago

Our provider has posted an update: They are continuing to observe SMS delivery delays when sending messages to the Telefonica O2 Network in Germany, and duplicates are being received for inbound SMS from this network. They are working with their carrier partner and will provide another update in 2 hours or as soon as more information becomes available.

SMS
Monitoring
12:22May 17, 20241y ago

We have deployed a workaround to address the issue of customers receiving multiple unnecessary SMS responses.

Our provider has informed us of ongoing SMS delivery delays and duplicates for inbound SMS from the Telefonica O2 Network in Germany.

We recommend customers with inbound SMS alert sources to pause or put their alert source into maintenance mode. We will post another update in 30 minutes. We apologize for any inconvenience this may cause and appreciate your patience.

SMS
Identified
10:46May 17, 20241y ago

We have localized the problem causing inbound SMS duplicates from our provider every 1-2 hours since May 17th. A P1 ticket has been opened with our provider, and we have implemented a fix to remove the standard response when an alert has already been accepted or resolved.

Please note that this incident is also affecting inbound SMS alert sources, and we currently have no workaround for this.

We apologize for the inconvenience and thank you for your patience.

SMS
Identified
10:13May 17, 20241y ago

We have identified the root cause of the issue where customers receive multiple unnecessary SMS alerts after acknowledging or resolving an alert. Our team is actively working on a fix. We apologize for any inconvenience this may cause and appreciate your patience.

SMS
Investigating
09:26May 17, 20241y ago

We are currently investigating an issue where customers are receiving multiple unnecessary alert updates via SMS. Our team is working to identify the root cause and resolve it as soon as possible. We apologize for any inconvenience this may cause.

SMS