I've always considered Goodhart's Law to be "of the people," but the article offers only top down solutions. What is the solution when the decision-makers have subjected you to a Goodhartable system of incentives and don't have enough awareness, vision, or humility themselves to change the system as described in the article? Well, you play the game or get beat by those who do. Right up until the point when you all lose.
I agree that if you're the one in control, complaining about Goodhart's Law is a sort of learned helplessness. However for those not in control -- and this goes as well for scenarios with diffuse control such as large crowds and democracy -- Goodhart's Law is useful, in the way that naming any systemic ailment is useful.
Perhaps we can talk about the 'hard problem' of Goodhart's Law to separate it from the problem solved in the article. Does anyone have good advice for those living inside the hard problem?
Even as someone who is in charge, it can be incredibly difficult to effect change once this type of culture is established. The only way it can happen is if enough people know it's wrong up and down the chain and are able to consciously recognize the systemic problems and rally individuals to start pulling the cart out of the mud. This is easier said than done, because most people are willing to put up with a tremendous amount of bullshit as long as it is predictable bullshit and they don't feel responsible for it.
I also want to challenge your framing about learned helplessness by "the one in control". It's true that in many situations you won't have power and you need to pick your battles. But on the other hand, everyone has some degree of agency, and programming especially requires folks to understand what they are doing and how it fits into larger systems (both technological and human). As a manager, I have seen a tremendous amount of learned helplessness based on assumed constraints that simply were not true. Yes, in some cases it's justified, but the most successful people tend to have fewer assumed constraints, regularly take action to do what they can to improve things, and they aren't easily discouraged when (inevitably) outcomes fall short of their ideal vision.
If people are serious about rooting out people who lie or cheat to appear on-target (beneath them in the management chain) they need employ anonymous whistleblowing straight up to them.
But I think many people in charge see the corruptness of the people below them as "headaches to deal with" and "if it's not squeaking it must be working" which I think is exactly backwards (it's the "if it compiles it's working" approach to management).
develop a network of senior people and build consensus around change. if you try to go this alone you are perceived (and may very well be) a shortsighted iconoclast acting out some narcissistic fantasy.
modulo that, once we have established some legitimacy and buy-in from the people that matter..just lose your fear and do the right thing. sure, they can fire you.
One of the subtler points Eli Goldratt makes indirectly is essentially that, "this type of culture is established" as you put it, in situations of abundance.
So Goldratt's description of "crunch time" is a company which starts out every quarter doing things in the (as they see it) "right way," then they close out every quarter doing everything in the "wrong way" or taking shortcuts or spending considerable costs to finish projects to make their revenue come in.
The spin I'm offering is: when you have a season of abundance, you do not have to face crunch time, and you get, as you put it, "predictable bullshit and they don't feel responsible for it." But effecting change during crunch time, by contrast, seems much easier. As long as you can keep your head straight. Goldratt's starting point begins, "maybe these 'shorctuts' we take under pressure are not the 'wrong way' per se."
I personally hate learning abstractions without examples, so let's get concrete. Your software project gets overdue and you and your peers start approving each others' merge-requests after only skimming them for flagrantly bad code, rather than "is this method in the right class?" and "does this have adequate test coverage?" you are now instead looking for "okay that's a raw SQL query, could that accidentally drop the whole database? no? approved! anything else we can add tests and refactor in a month." That attitude, right?
Goldratt says, "no, pay attention to that!" ... you say "but that's not how it should be done, we must review code as we go, tests are good to have" and my-imaginary-version-of-him replies, "sure I understand that you have a need for code safety, I am not saying that you don't. But the fact remains, you accepted the bypassing of this process when crunch-time came, which means that it constrained the team in unacceptable ways overall. So if we believe in working smarter-not-harder, we need to get imaginative now, because the 'logical place' for code safety measures to get injected, is too slow. And there are lots of imaginative solutions. We could hire someone whose only job will be code review, they could start reviewing your code before you're even done writing it. Or we could mandate that everyone must commit their code to the main branch every single day with only pro-forma code review, this will force us to adopt out-of-band practices to make things safe. Other things like that."
So what does effecting change look like in a system that does not have the luxury of abundance? It looks like a "crunch time" that never "ends", but does "get better." Goldratt's point is essentially that you don't want to meet this punch with a counterforce, as that will hurt: you instead want to yield with it and let its own momentum carry the both of you to a better solution.
I have a theory about how this works on software projects but I didn't quite get the chance to run the experiment at my last job, so if this intrigues you and you have the headcount to hire me... :)
But notice in a public school environment the hard incentives ($$$) are all on the administration. So, it's attendance, tests and grades, maybe followed by class size. Yes, there's incentives for students to cheat but cheating is much more widespread and likely on the part of the teachers and administration. The blogger didn't talk about this, but you need to look to see for any metric or group of metrics who is incentivized to cheat and how it might manifest.
You say this like there is an obvious and right way to handle the situation. But the reason such a situation might end up Goodhartable is if you had three tests every day with no time to study. Unless the requirements change, it would be impossible to "succeed" without adopting #2 or #3. This is why the GP is saying this concept is great for a top-down approach, but if you are in a system that doesn't give a shit, then you are forced to play the game or lose to those who do.
> Does anyone have good advice for those living inside the hard problem?
Keep speaking up. Make the problem evident, on an ongoing basis. Mention possibly solutions or alternatives when you think of them. People may not like complainers, but they'd probably have a more difficult time dismissing a sincere person who wants to make things better.
-----
With respect to education one of the main points of credentials is prestige (https://www.science20.com/quantum_gravity/blog/phd_octopus_w... ), from the point of view of the hirer. Particular demonstrable skills are also another point, but from the other end, from the education institute. Even in the case when the hirer is also an educational institute, the people in control of one set of reasons for choosing the measuring stick are not the ones in control of the other set of reasons.
Prestige requires a measuring stick, and the point of that measuring stick is exclusivity and marketing power. Neither of which particularly require a skill.
Basically fighting back against Goodhart's law is so difficult because no particular person or group has total control over all of the incentives for using a particular measure.
If people were more honest with themselves about why they are using a particular measuring stick then maybe we could tackle Goodhart's law.
Are you hiring Ph.D.s because on average they are more capable to the demands of the job (Extremal Goodhart)? Because you also have a Ph.D. and so identify with Ph.D. holders (affinity pseudo-Goodhart [my own creation])? Because you also have a Ph.D. and know a whole ton of Ph.D. holders who have difficulty getting good paying jobs (a form of Causal Goodhart, probably)? Because you have too many decent resumes to sort through (Regressive Goodhart)?
But they either aren't honest, or have too many differing goals for the use of a measure to justify to themselves minimizing the measure.
I think trying to identify a separate "Hard Goodhart's Law" or whatever is a bit silly. Very little you've said here is really specific to Goodhart's Law. If you don't have the power to solve the problem, then, well, you can't solve the problem. You're left with the standard options for working with any bad system: go with the flow, leave, try to gain more power, etc.
Which is not too say that asking for advice is a bad idea, quite the opposite. Just that said advice doesn't need a special name.
That's like- the whole point. If you as a manager set a goal, those three choices are all the choices the people under you have to meet the goal, and you often set it up so that the preferred choice (actually make things better) is impossible. Its saying that decision makers need to be aware of this and adjust your behavior.
Like you said, if they don't, the people dealing with it have no choice- so, it'd be impossible to offer advice for those people, since they have no way of actioning it (unless its convincing the decision maker of that point)
I agree that if you're the one in control, complaining about Goodhart's Law is a sort of learned helplessness. However for those not in control -- and this goes as well for scenarios with diffuse control such as large crowds and democracy -- Goodhart's Law is useful, in the way that naming any systemic ailment is useful.
Perhaps we can talk about the 'hard problem' of Goodhart's Law to separate it from the problem solved in the article. Does anyone have good advice for those living inside the hard problem?