Thursday, November 12: Microsoft email, Teams and Copilot fall over, with Downdetector reports from more than 70 countries.
On that same Thursday, Nvidia CEO Jensen Huang resigned after reports of affairs, one with a suspected Chinese spy.
Friday the 13th: the FAA grounds every departing flight in the United States for more than four hours.
If you were looking for a villain – a hacker, a rogue AI, a Friday-the-13th hex – the week offered plenty of candidates and some valuable IT outage lessons. In addition, Nvidia gave us a useful reminder along the way: even a very large company can have a single point of failure, and sometimes it is a person.
Now let’s talk about the two outages that grounded people.
What Happened Last Week…
Microsoft, Thursday. Microsoft said it was a technical issue, not a cyberattack and not AI. Its preliminary explanation: a problem left behind by an earlier network capacity upgrade surfaced after a later network change, thus overwhelming the process on affected servers that sets up secure connections. In plain English, something old and something new met and the servers stopped answering.
FAA, Friday. The ground stop ran from mid-afternoon Eastern until 8:15 p.m. About 4,000 flights were airborne when it started and all were accounted for. Transportation Secretary Sean Duffy said switches and routers “got overloaded with data,” and reporting tied it to equipment work during a copper-to-fiber upgrade at one facility. By Sunday, Duffy said the root cause was still unknown but there was no sign of a hack. Officials also said the FAA and Microsoft events were unrelated.
The cost according to sources: referencing FlightAware, NPR reported more than 9,000 cancellations on Friday alone and over 4,000 on Saturday; The Points Guy counted more than 18,000 cancelled flights across three days.
Notice what both root causes have in common. Neither is a hostile act, as far as anyone has shown. Both involve infrastructure change on a complex, aging or layered network. That is not a curse. That is “Legacy” Tuesday – or in this case, Thursday and Friday.
“Shit Happens” Is a Pattern, Not a Shrug
The honest version of the headline is that big systems fail in boring ways and they will keep doing so. Look at the recent record:
- FAA NOTAM outage (January 2023). The FAA traced the outage to a damaged database file in the system that delivers pilot safety alerts. As daytime traffic picked up, it overwhelmed the backup system, which forced the grounding of flights. Domestic departures were paused until 9 a.m. ET. This was the first nationwide ground stop since 9/11. It was not an attack: it was a damaged file and a backup that couldn’t carry the load.
- CrowdStrike (July 19, 2024). A content update crashed roughly 8.5 million Windows machines. It was not an attack by a rogue actor. The post-incident analysis pointed to gaps in content validation and testing. The incident was exacerbated by the fact the file reached every online host at once instead of a small canary group first.
- Iberian blackout (April 28, 2025). More than 50 million people lost power; some regions waited up to 16 hours. The expert panel’s final report found no single cause. Voltage control, protection settings and poor visibility of rooftop solar interacted, and the grid collapsed in under 90 seconds. The defense schemes worked as designed for frequency problems, not for a fast overvoltage cascade. The protections did what they were supposed to do. The problem was they were built for a different kind of failure.
- AWS us-east-1 (October 2025). A latent race condition in DynamoDB’s DNS automation produced an empty record, and one region’s hiccup disrupted services across the internet for roughly 15 hours.
- Azure Front Door (October 2025) was an inadvertent configuration change. Cloudflare (November 2025) was a database permission change that doubled a config file past a hard limit. Microsoft (July 2026) was a bug in an automated network maintenance system that removed more IP routes than intended.
Count them: validation gap, interacting defaults, race condition, config change, permission change, maintenance bug. Not one needed an adversary. The common thread is change, automation, and hidden dependencies. And our appetite for all three keeps growing.
CrowdStrike vs. Cloudflare: Same Cause, Different Damage
CrowdStrike and Cloudflare look like the same story on paper. Neither was an attack and in both a file changed and was not caught before it reached production. The difference is where each failure landed and who could fix it. That decides what you need to prepare for.
In the CrowdStrike incident the failure lived on your machines. A content update to a security agent crashed roughly 8.5 million Windows devices. CrowdStrike could pull the bad update, but every affected machine still needed hands-on repair. Delta said its IT teams had to manually repair and reboot each of the affected systems and its recovery ran over five days. Recovery speed depended on how prepared each company was and that is why airlines recovered so differently.
In the case with Cloudflare the failure lived in the vendor’s network. A database permissions change caused a Bot Management feature file to double in size, exceeding hard-coded memory limits in Cloudflare’s proxy system. Cloudflare customers’ own systems were fine, but their traffic could not get through. Cloudflare fixed it centrally and everyone came back at about the same time. Customers could do little but wait.
What this means for you:
- For a CrowdStrike-style event, your own recovery plan decides how bad the day gets. Know which critical systems depend on the affected software and how you would restore them at scale and in what order.
- For a Cloudflare-style event, the question is dependency and concentration. Ask what happens to your business if one provider is down and whether you have a fallback path that doesn’t run through it.
What the New Cloud Reality Changes
Cloud is not less reliable than what came before. It is differently shaped, and IT leaders who run it like a data center are the ones who get hurt.
- You inherit your vendor’s change management. When Microsoft pushes a network change, your uptime rides on it. You will never see that change calendar. So you have to ask for what you can see: notification, staged releases, and a root-cause report within a committed window.
- Blast radius is concentrated. One identity service, one edge network, one region can sit under thousands of businesses. A failure that once hit one company’s mail server now hits a continent’s.
- It is dependencies all the way down. Picture a crew-scheduling tool that runs on a cloud, which leans on a vendor’s DNS, on machines that need a security agent. CrowdStrike showed how long that chain can be. Delta said 60 percent of its mission-critical applications, including backups, relied on Windows and that it had to physically reset 40,000 servers… although CrowdStrike and Microsoft disputed Delta’s account of its own recovery.
- Your comms channel is the thing that broke. If your incident plan says “email the team” and Outlook is down, you have no plan. The same goes for the admin portal you would use to check status.
- The fog of war is now part of the outage. Within hours, social media had attack theories and a claimed link between Microsoft and the FAA that officials denied. Security teams burn real hours chasing a rumor. Decide in advance who answers “is it us, is it them, is it an attack?” and by when.
Where the Airlines Got It Wrong
A fair caveat first: the ground stop was not the airlines’ fault, and the evidence on their internal recovery is still thin. But the mess after 8:15 p.m. Friday was theirs to manage and some patterns are visible.
- Slow resets. As of Sunday night, American had cancelled about 23 percent of its mainline schedule and JetBlue 28 percent, against roughly 5 percent at Delta and Southwest and about 3 percent at United. Aircraft and crews were out of position; planes that did not fly were not where the next flight needed them. We saw this movie in 2024, when Delta’s crew-tracking tools could not absorb the volume of changes and the airline cancelled more than 7,000 flights over five days and claimed $500 million in losses. Rare is not – and should not be – the same as unrehearsed-for.
- Service channels that buckle under load. NPR reported a JetBlue passenger pair spent more than nine hours overnight on hold for reimbursement. Air Canada said it cancelled about 250 flights affecting 27,000 passengers; travelers told CBC they were promised a rebooking update within 30 minutes that never came. A United passenger told CNN she was given no food. These are individual accounts, not audits, but the pattern is the point: the apps and call centers were not sized for everyone needing help at once.
- Waivers that differ airline to airline and arrive in pieces. Rebooking windows ran from November 18 to November 21 depending on the carrier with different booking cut-offs. Airline customers do not read fare rules during a crisis, but they do remember who made it simple.
- Hiding behind “not our fault.” The airlines were right that the ground stop was outside their control. Legally speaking, they owed passengers only a free rebooking or a refund. And many carriers stopped there, leaving stranded travelers to cover their own hotels and meals. But being legally in the clear is not the same as serving your customers well. The airlines that offered even modest help beyond the minimum are the ones that will be remembered for it.
- No visible playbook for a known risk. The 2023 stop proved a nationwide ground stop can happen. After last Friday Transportation Secretary, Sean Duffy, reportedly told reporters to expect more of these. A risk that is both proven and likely to recur deserves a written playbook that isn’t too hard to script. Decide in advance who sends the first message to passengers, what the app shows, and which crews and aircraft get repositioned first. Then rehearse it, so the first time anyone runs the plan isn’t during a real crisis. We don’t know whether any airline had such a plan, but the passenger accounts suggest the gaps were not covered.
The IT Outage Lessons-Learned Framework: Know, Limit, Degrade, Rehearse, Tell
When clients ask us what to do after an outage like last week’s, I start by taking the blame out of the conversation. You can’t prevent every vendor failure nor can you stop a network change you never see. What you can control is how much damage a failure does to your business and how quickly you recover. That comes down to decisions made well before the bad day. It also helps a lot to have those decisions written down rather than left to instinct.
As a rule of thumb, we give clients a five-step framework: Know, Limit, Degrade, Rehearse, Tell. Each step maps to a policy that a named person can own, report on, and be asked about at the next leadership meeting. Treat it as an agenda for your next IT leadership review. And for each step ask whether you have a policy, an owner, and evidence it has been tested.
1. Know: Map your dependencies
- Policy: maintain a living map of your top 10 business-critical services, including the vendor’s vendor, with a named owner for each.
- Policy: put concentration risk (one cloud, one identity provider, one security agent, one region) on the risk register and review it with the board.
2. Limit: Shrink the blast radius
- Policy: staged rollouts and rollback plans for your own changes; no big changes the week before a peak (holiday travel, quarter-end).
- Policy: put the same expectations in vendor contracts – advance notice of major changes, staged release practices, and a root-cause report within a fixed number of days.
3. Degrade: Plan to run badly, not to run perfectly
- Policy: every critical process has a documented manual fallback tested by someone who did not write it.
- Policy: out-of-band communications. Assume email, chat and the admin portal are down; keep an alternate channel and a phone tree outside your primary tenant.
- Policy: cloud-independent break-glass admin accounts and an offline, protected copy of your identity system.
4. Rehearse: Practice the day it all stops
- Policy: two tabletops a year with the business in the room, one of them an “everything stops, then everything restarts at once” drill. Time the restart: that is where airlines lose days.
The Iberian report is the grown-up version: it recommends black-start tests every three years and at least 24 hours of blackout-proof communications. Recovery plans that have never been run are hopes.
5. Tell: Communicate fast and plainly
- Policy: one incident commander, one channel, updates on a fixed cadence even when the update is “no change.”
- Policy: blameless postmortem within a week with an owner and a due date for every fix.
Who Handled It Well: Three Examples Worth Stealing from
Maersk after NotPetya (2017)
When NotPetya wiped Maersk’s environment, the company reinstalled about 4,000 servers, 45,000 PCs and 2,500 applications in ten days, work that would normally take around six months, according to chairman Jim Hagemann Snabe.
Staff improvised: WhatsApp groups became a lifeline and the company bought roughly 2,000 laptops from retail stores. The part people love to retell is that a single domain controller in Ghana survived because a local power outage had knocked it offline. Maersk’s own CISO, Andy Powell, has been candid that this was luck.
What to copy: the decision to rebuild rather than patch; getting identity back first; and radical transparency. Powell has said clients valued being told what was happening from day one.
What not to copy: the luck. Keep an offline, immutable copy of identity on purpose. (And note the irony: a power outage saved Maersk, the Iberian one tried to destroy a continent’s grid.)
Cloudflare after its November 2025 outage
Cloudflare’s outage came from a permission change that doubled a configuration file beyond a hard limit. Within a day, CEO Matthew Prince put his own name on a detailed postmortem, said there was no attack, apologized, and committed to stricter validation and better fault isolation. What to copy: the speed, the specificity, and the ownership. A leader who explains the failure plainly buys more trust than one who explains it carefully.
Iberian grid operators after the 2025 blackout
Iberian grid operators restored power faster than many expected. Portugal’s grid was fully restored by 12:22 a.m. on April 29 and Spain’s transmission system by about 4:00 a.m. That was roughly 12 and 16 hours after the outage began. Getting there was not smooth. Several bottom-up black-start attempts failed and had to be re-energized from neighboring zones. Operators adapted and kept going.
The final report also found real gaps. Voice communications failed at some operators and generators, and distributed energy resources were hard to see. That is why the panel now recommends black-start tests every three years and at least 24 hours of blackout-proof communications.
The Common-Sense IT Outages Checklist
Print it. Tape it up. If you can’t tick a box, make it your next project.
Before
☐ Do we have a map of our top 10 critical services and every vendor underneath them?
☐ Has the business, not IT, agreed how long each can be down and how much data it can lose?
☐ Does each critical process have a manual fallback someone has actually tried?
☐ If email and Teams are down, how do we reach the team in the next ten minutes?
☐ Are our break-glass admin accounts independent of the cloud they protect and have they been tested this quarter?
☐ Do we hold an offline, protected copy of our identity system?
☐ Do vendor contracts require change notice and a root-cause report on a fixed clock?
☐ Is concentration risk (one cloud, one agent, one region) on the board’s radar?
During
☐ Does someone own the “us, them, or attack?” question with a deadline?
☐ Will one person run the incident, in one channel, with updates on a set schedule?
☐ Have customers and staff heard from us in plain language within the first hour?
After
☐ Do we complete a blameless postmortem within a week of an incident with an owner and a due date for every fix?
☐ Have we run and timed a full restart drill where everything comes back at once in the last six months?
The Bottom Line
Friday the 13th did not do this. A change window, an old network and a lot of hidden dependency did. Moreover, similar things will do it again, probably not on a Friday. The leaders who look calm in the next one will not be the ones who prevented every failure. They will be the ones who assumed it was coming, knew what depended on what, and had already practiced the bad day.



