Millions of people have now experienced the joy of an empty inbox, thanks to the extended outage on one of Microsoft’s most important business software offerings. Many people are saying that Microsoft, having gotten rid of a lot of its veteran engineering staff over the last few years, may not be capable of fixing the problem. On X, John Ringo offers his thoughts:
CAN Microsoft recover?
(This was a reply to something. I decided it was better as a post.)
Oh, I agree. They’ve been getting rid of the nerds that built the system and have this little undocumented bit of knowledge in their heads for years. That is a huge issue and definitely contributing to the downtime.
But they (and AWS and others) have also built a system that is so complex and interwoven, the real issue is:
If it crashes this hard can they bring it back up as it was at all?
We get into discussions of telos and kosmos. It’s distinctly possible if you brought back EVERYONE who built it that they still couldn’t fix Humpty-Dumpty.
We’ll see by Monday.
Telos and kosmos.
Telos is a simple simple system. Every point in the system is understood and is comprehensible by a single person. (A single server. A large but not enormous set of servers.)
Then there are simple complex systems. A system where every individual part of the system can be comprehended but it’s impossible to comprehend how all the parts interact.
That is what Microsoft and AWS and other cloud providers have created. A system where you can maybe understand why this bit isn’t working but not why/how its interacting to cause all these other bits to not work.
It’s too complex.
It’s one of the reasons that the people saying “Nobody is going to migrate off the cloud. Nobody wants to run their own servers. No IT guy wants to run email. These guys are literally the experts.” are all wrong.
When a single company, from a two man shop to fortune 500, uses its own server system, the entire system can be understood. It is telos. Every bit can be comprehended. It may be badly documented. There may be this little “thing” that only Joe Neckbeard knows and if you fire him you may have an outage.
But it’s NOT so big nobody can really rebuild it in a serious meltdown.
Rebuilding this ecosystem is so complex that it’s like nuking Manhattan and expecting every single piece to go back just like magic.
It won’t. And those missing pieces are Gladys’ payroll spreadsheet that she inherited from Gloria who inherited it from Sandra who inherited it from Carol who got some nerd who left in 1996 to do it for her.
The question becomes, what to do if it’s not back by Monday?
