Störung: Microsoft Teams rumpelt (16./17.3.2020)

Bei der Cloud-Lösung Microsoft Teams hat es zum 16. März 2020 wohl ziemliche Hakeleien gegeben. Die Cloud-gestützte Collaborationslösung aus dem Haus Microsoft war plötzlich nicht mehr zu gebrauchen. Ergänzung: Auch am 17. März 2020 klemmt es.

Das es mit Microsofts Cloud-Lösungen wöchentlich mehrfach irgendwo hakt, lässt sich an inzwischen zahlreichen Blog-Beiträgen nachlesen. In Zeiten von Coronakrise und Home Office setzen möglicherweise viele Nutzer genau auf diese Collaborationslösungen. Die Macher von Slack haben gerade einen Bug gefixt, der die Übernahme eines Benutzerkontos ermöglicht hätte (siehe diesen Artikel auf Bleeping Computer).

Bei Microsoft war das zumindest gestern kein Problem, denn dort konnte man sich als Benutzer (wenn man zu den Pechvögeln gehörte) unter Teams schlicht nicht mal mehr anmelden. Blog-Leser Daniel S. schickte mir zum gestrigen Abend eine Mail:

FYI

Die MS Cloud dreht wieder komplett am Rad heute. Vermutlich steigen heute alle ins Home Office ein – ausgerechnet jetzt funktioniert kein Teams.

Aktuell sind 7 Rot Meldungen im Admin-Portal

Gruß

Daniel

Auf Nachfrage hat er mir noch den nachfolgenden Screenshot des Admin-Status geschickt, die eine Störung zeigt.

Die Auflistung zeigt zwar viele grüne Häkchen – aber das sind Problemchen, die das Support-Team im Laufe des Tages behoben hat. Ergänzung: Gerade habe ich noch einen Status bekommen, dass es vor allem Europa betrag.

Microsoft Teams went down for two hours as Europe logged in https://t.co/4FHC09qnUc

— Aryeh Goretsky (@goretsky) March 17, 2020

Martin Geuß hatte es bereits zum Vormittag auf Dr. Windows als Digitalisierung am Limit: Microsoft Teams kämpft mit Aussetzern thematisiert. Inzwischen sollte das Problem weitgehend behoben sein. Blog-Leser Andreas P. hat mir die Statusauszüge aus dem Administrator-Dashboard geschickt:

-16.03.2020 18:57:54
Title: Teams messaging functionality issues
User Impact: Some users may be unable to send and receive messages or schedule meetings.
More info: Additionally, some users may have been unable to create teams and update team memberships within the Microsoft Teams service.
Current status: After rerouting user connections, our telemetry indicates that service has been restored as of Monday, March 16, 2020, at 3:37 PM UTC. We'll continue to actively monitor the service for the next 24 hours to ensure additional impact does not occur.
Scope of impact: This issue could have impacted any user who was served through the affected infrastructure. Our telemetry indicated that this issue only affected users hosted within Europe.
Start time: Monday, March 16, 2020, at 2:20 PM UTC
Root cause: A downstream service experienced latency processing requests, resulting in errors within the service that processes chat functionality. We're actively determining what caused the downstream latency and putting in any corrective measures required.
Next update by: Tuesday, March 17, 2020, at 11:00 AM UTC
-16.03.2020 17:32:56
Title: Teams messaging functionality issues
User Impact: Some users may be unable to send and receive messages or schedule meetings.
More info: Additionally, some users may be unable to create a team and update team memberships within the Microsoft Teams service.
Current status: We've identified a potential issue with the service responsible for chat functionality and we're reviewing diagnostic data to validate whether this is the source of the functionality degradation. We're continuing to reroute user connections to alternate components and have observed that the majority of users are seeing recovery.
Scope of impact: This issue could impact any user who is served through the affected infrastructure.
Next update by: Monday, March 16, 2020, at 6:00 PM UTC
-16.03.2020 16:29:46
Title: Teams messaging functionality issues
User Impact: Some users may be unable to send and receive messages or schedule meetings.
More info: Additionally, some users may be unable to create a team and update team memberships within the Microsoft Teams service.
Current status: We're restarting affected components and rerouting user requests to alternate systems to alleviate impact.
Scope of impact: This issue could impact any user who is served through the affected infrastructure.
Next update by: Monday, March 16, 2020, at 5:00 PM UTC
-16.03.2020 15:42:37
Title: Teams messaging functionality issues
User Impact: Some users may be unable to send and receive messages or schedule meetings.
More info: Additionally, some users may be unable to create a team and update team memberships within the Microsoft Teams service.
Current status: We've received reports of impact to multiple Microsoft Teams functionalities. We're reviewing system telemetry to isolate the source of the issue.
Scope of impact: This issue could impact any user who is served through the affected infrastructure.
Next update by: Monday, March 16, 2020, at 3:30 PM UTC

Kommt natürlich zur Unzeit, wenn mehr Leute diesen Dienst nutzen möchten. War jemand betroffen?

Neue Störung zum 17. März

Ergänzung: Auch am 17. März 2020 klemmt es. Zumindest entnehme ich dies dem aktuellen Statusbeitrag des Service-Teams auf Twitter.

We investigated and resolved an issue with the Teams chat service that affected some of our users in the Europe region. We determined this to be a caching issue within a component of our infrastructure. More details will updated under the SI# TM206556.

— Microsoft 365 Status (@MSFT365Status) March 17, 2020

Hier noch die Statusmeldungen vom 17. März, die mit Blog-Leser Andreas P. zur Verfügung gestellt hat.

-17.03.2020 14:51:53

Title: Teams messaging functionality issues

User Impact: Some users may have been unable to send and receive messages or schedule meetings and live events.

More info: Additionally, some users may have been unable to create teams and update team memberships within the Microsoft Teams service.

Current status: We connected the service to alternate, healthy infrastructure to mitigate impact for the affected users. Additionally, we applied a mitigation to address the caching issue that resulted in impact. Our investigation determined that the impact that occurred on Monday, March 16, was symptomatically the same as the impact from Tuesday, March 17, but with a different root cause. We'll publish a Post-Incident Report within five business days to provide additional context into the root cause of these problems.

This communication will remain open to provide additional updates in the event this issue reoccurs.

Scope of impact: Our telemetry indicated that this issue affected some users hosted within the Europe region.

Start time: Monday, March 16, 2020, at 2:20 PM UTC

Root cause: On Tuesday, March 17, a caching issue occurred within a component of the service that supports the messaging feature, resulting in impact.

Root cause: On Monday, March 16, a downstream service was experiencing latency when processing requests, resulting in errors within the service that processes chat functionality. We're actively determining what caused the downstream latency and putting in any corrective measures required.

Next update by: Wednesday, March 18, 2020, at 10:00 AM UTC

-17.03.2020 13:12:29

Title: Teams messaging functionality issues

User Impact: Some users may be unable to send and receive messages or schedule meetings and live events.

More info: Additionally, some users may be unable to create teams and update team memberships within the Microsoft Teams service.

Current status: We've identified that a caching issue has caused the service which handles the messaging feature to become degraded. We've completed connecting the user connections to additional, healthy infrastructure and are monitoring the service while it recovers.

Scope of impact: This issue could impact any user who is served through the affected infrastructure. Our telemetry indicates that this issue only affects users hosted within Europe.

Start time: Monday, March 16, 2020, at 2:20 PM UTC

Next update by: Tuesday, March 17, 2020, at 2:00 PM UTC

-17.03.2020 10:34:23

Title: Teams messaging functionality issues

User Impact: Some users may be unable to send and receive messages or schedule meetings and live events.

More info: Additionally, some users may be unable to create teams and update team memberships within the Microsoft Teams service.

Current status: We've identified an issue with service that handles the messaging feature, which is causing impact. While we determine the underlying cause of this issue, we're connecting the user connections to additional, healthy infrastructure to mitigate impact.

Scope of impact: This issue could impact any user who is served through the affected infrastructure. Our telemetry indicates that this issue only affects users hosted within Europe.

Start time: Monday, March 16, 2020, at 2:20 PM UTC

Next update by: Tuesday, March 17, 2020, at 12:00 PM UTC

-17.03.2020 09:33:55

Title: Teams messaging functionality issues

User Impact: Some users may be unable to send and receive messages or schedule meetings.

More info: Additionally, some users may be unable to create teams and update team memberships within the Microsoft Teams service.

Current status: Our telemetry indicates that the impact related to this issue has returned. We're reviewing logs and additional service data to determine next steps.

Scope of impact: This issue could impact any user who is served through the affected infrastructure. Our telemetry indicates that this issue only affects users hosted within Europe.

Start time: Monday, March 16, 2020, at 2:20 PM UTC

Next update by: Tuesday, March 17, 2020, at 11:00 AM UTC

-17.03.2020 03:00:52

Title: Teams messaging functionality issues

User Impact: Some users may be unable to send and receive messages or schedule meetings.

More info: Additionally, some users may be unable to create teams and update team memberships within the Microsoft Teams service.

Current status: Our service monitoring and data analysis indicates the issue is resolved for the majority of users; though, we've identified that a small number of users hosted within Europe remain affected and we're working to remediate impact for these users. Furthermore, we're continuing to monitor service availability for the groups of users that seem to have been disproportionately affected by the issue to curb the potential for further impact.

Scope of impact: This issue could impact any user who is served through the affected infrastructure. Our telemetry indicates that this issue only affects users hosted within Europe.

Start time: Monday, March 16, 2020, at 2:20 PM UTC

Root cause: A downstream service is experiencing latency processing requests, resulting in errors within the service that processes chat functionality. We're actively determining what caused the downstream latency and putting in any corrective measures required.

Next update by: Tuesday, March 17, 2020, at 11:00 AM UTC