Skip to main content

Microsoft links global 365 outage to dormant defect in July update

The company said increased traffic and reduced server capacity triggered the bug, while its automated recovery systems could not handle the scale of the failure.

By Mei Ling Tan

Rows of server racks in a data center
Photo: TNW (Financial Times)

Microsoft said a defect in a July update to its network gateway servers caused the global Microsoft 365 outage after remaining dormant for more than three months. Its preliminary findings link the failure to increased traffic combined with reduced capacity.

The company said Teams traffic switched to a gateway also serving Exchange Online, increasing demand while an unrelated problem had left nearly 14% of that gateway’s server capacity unavailable. As individual servers became unresponsive, their traffic shifted to working machines and triggered further failures. Within two hours, more than 80% were unresponsive at the worst point.

According to Microsoft, the defect caused lock contention in an internal Windows process, exhausting server CPU and memory resources. Testing had concentrated on less powerful hardware with fewer cores, missing a problem that was worse on newer, higher-core machines.

Recovery was slowed because queued traffic overwhelmed servers as they restarted, and automated recovery systems were not built for a failure of that scale, Microsoft said. Engineers developed an update to restore the previous Windows code path and deployed it across successive groups of servers.

Teams, OneDrive, SharePoint and Office applications recovered sooner because they could bypass the affected gateways. Exchange Online could not do so because of its scale, leaving Outlook unavailable longest. The timeline reported by The Register puts deployment of the fix at 07:20 UTC on 13 November and Microsoft’s final customer update at 04:15 UTC on 14 November.

Microsoft apologized on 13 November and ruled out a cyberattack or AI agents as the cause. The company plans to issue its final incident report on 20 November, according to The Register.

Keep reading: Tech News

All Tech News stories

Latest in Technology

All latest
  1. Tech News

    LangChain valued at US$3 billion in funding round for AI agent tools

  2. Tech News

    Anthropic sets 30-day disclosure deadline for serious software flaws

  3. Tech News

    AI tool turns software patches into exploits without human involvement

  4. Tech News

    Noeri announces US$110 million funding round

  5. Tech News

    Apollo to embed six evaluators at OpenAI for safety tests

  6. Tech News

    FAA router investigation follows outage that halted U.S. flights