The world is full of important people performing vital tasks either without fail or wherein failures can be caught by other individuals and processes that serve as guard rails.
The solution isn't merely to be perfect it is to have a process that is tolerant of the level of imperfection that one would expect. For example it is perfectly reasonable that a someone could have foreseen it not clobbering a license file without prompting the user this could have brought the matter to immediate attention, its reasonable to suppose that if the first person didn't foresee it someone reviewing their choices might have thought of it, its reasonable that someone might have viewed the commit logs and noticed what was going on.
None of this is an acceptable level of failure from people that drawing a 6 figure salary for a position of expertise or leadership.
I am not a MS employee but I do work as a software engineer at a small company and make six figures a year. Mistakes like this happen. In fact they happen all the time. There are a lot of people out there writing code on any given day. Some of them may have been staring at the screen for 12 straights hours trying to make a release date. Sure its a lot more visible when someone like MS makes one, but think about the actual issue here. They did not create a feature with the intent to overwrite existing licenses files, they made a feature to ensure the correct licenses were added to new repos. Now they forgot one edge case (out of how many thousands or millions?), took ownership of the mistake, and are taking actions to correct it. Overall to me that seems like a very small bug that they took/are taking steps to fix on Christmas no less.
>>None of this is an acceptable level of failure from people that drawing a 6 figure salary for a position of expertise or leadership.
If the rest of the world thought like you we would not have any people left in high paying jobs. Everyone is bound to make mistakes. Owning up and setting things right is what is important in my eyes.
> The world is full of important people performing vital tasks either without fail or wherein failures can be caught by other individuals and processes that serve as guard rails.
Sometimes planes literally fall out of the sky. NASA caused a space shuttle to explode killing the entire crew. It is fundamentally untrue that anywhere in the world we have perfect people or processes.
> The solution isn't merely to be perfect it is to have a process that is tolerant of the level of imperfection that one would expect.
I expect that occasionally sometimes people will ship bugs and neither they nor the processes in place will catch them. This is a very minor bug. This is not the sort of bug that keeps engineering leaders up at night.
> For example it is perfectly reasonable that a someone could have foreseen it not clobbering a license file without prompting the user this could have brought the matter to immediate attention, its reasonable to suppose that if the first person didn't foresee it someone reviewing their choices might have thought of it, its reasonable that someone might have viewed the commit logs and noticed what was going on.
It’s really easy to point at others’ errors with the clarity of hindsight. But everything you said here boils down to just predicting the specific broken case, which is useless advice. Even your suggestion to check commit logs is useless unless you know the bug you’re looking for. It’s not as if Git spews errors when a license file is changed and it shouldn’t be. And if they knew what to look for in the logs, they wouldn’t have checked in the bug in the first place.
This team has doubtless shipped thousands of features. It’s unsurprising that they’ve shipped some bugs.
> None of this is an acceptable level of failure from people that drawing a 6 figure salary for a position of expertise or leadership.
You’re talking about a bug that caused a license file to be incorrectly changed and the fix is a revert. There is an audit trail of what happened and what repos were affected. It literally had no meaningful negative effect except to Microsoft’s image with the severest of critics, and to Jeff Wilcox’s holiday break. This is not a Therac-25 level incident.
I find it very interesting that none of the hyper-critical people in the comments here have discussed catching any broader class of issues. There are many comments criticizing the author for failing to check for an existing license file. (e.g.” How can someone be so bad at their job that they realize that a bot that writes to the license file needs to avoid overwriting an existing file?”) None of those comments that I’ve seen addresses the fact that adding a license file during a fork at all is probably an error. No, just criticism about this one case because someone told them about the exact observed bug so now it’s obvious.
The ability to criticize an error someone else already found is not valuable or constructive. And if this is the best you can bring, your judgement is certainly no better than the engineer who created the bug in the first place.
Disclosure: I’m a Microsoft employee.
Further disclosure: If someone showed up to a postmortem for my team with this level of condescension and no useful advice, I’d tell them to be quiet or leave.
> Further disclosure: If someone showed up to a postmortem for my team with this level of condescension and no useful advice, I’d tell them to be quiet or leave.
If I worked with you it would be my responsibility to support my coworker and help them improve. As it is neither of us has much of an obligation to the other we are uninvolved third parties giving our unsolicited opinions to one another. It is a privilege as an uninvolved observer to another to analyze it frankly without worrying about hurting someone's ego. It is OK I think to look at some mess or other and the inevitable conciliatory excuses and or hostile push back against criticism and call bullshit.
> None of those comments that I’ve seen addresses the fact that adding a license file during a fork at all is probably an error. No, just criticism about this one case because someone told them about the exact observed bug so now it’s obvious.
This is an excellent point insofar as explaining why the bug might have happened. I disagree however that it is sufficient explanation. Such code should still check for the presence of an existing license file for another license and raise an error that requires human intervention to override. That is constructive and actionable.
> This is an excellent point insofar as explaining why the bug might have happened. I disagree however that it is sufficient explanation. Such code should still check for the presence of an existing license file for another license and raise an error that requires human intervention to override. That is constructive and actionable.
Sure. In retrospect, that seems like a really good idea. But I'm sure you've done an awful lot of stuff that is dumb in retrospect, too.
Legal asks for tooling to ensure that the correct license file ends up on all developer projects. It's known that developers will often Google license agreements, copy them from places that don't belong, end up with the wrong LICENSE from a templating mistake. Someone builds tooling to just smash the correct LICENSE file in as a first commit on any repo that shows up in the Microsoft org: if it's an active developer and this is the rare case where this logic is wrong, surely they'll notice that this has happened, and this will prevent code from being up with an incorrect license.
Then, the use case evolves: people start to fork things into the Microsoft organization, and this code does the wrong thing an appreciable fraction of the time. But no one is scrutinizing commits on forked repos to the same extent and doesn't notice.
Maybe you do much better and have never made a mistake like this. But real people make real mistakes. Yes, when you encounter a mistake, you try to get better for next time! But that doesn't mean that we should expect a 0 mistake rate out of ______ org or not assume good faith with an ordinary rate of bad outcomes until proven otherwise.
Unlike the GP poster, I am not a MSFT employee-- I am often skeptical of MSFT actions but they get a pass here. Obliterating the license file has no advantage for MSFT, and has the risk of legal exposure and this kind of public black eye. Obviously someone screwed up, but screw ups happen. Now that Jeff Wilcox has had to work on Christmas, you can be sure he will be even more attuned to this type of issue in the future.
The solution isn't merely to be perfect it is to have a process that is tolerant of the level of imperfection that one would expect. For example it is perfectly reasonable that a someone could have foreseen it not clobbering a license file without prompting the user this could have brought the matter to immediate attention, its reasonable to suppose that if the first person didn't foresee it someone reviewing their choices might have thought of it, its reasonable that someone might have viewed the commit logs and noticed what was going on.
None of this is an acceptable level of failure from people that drawing a 6 figure salary for a position of expertise or leadership.