This is kind of an ironic claim on a community that believes so strongly in "black boxes" like machine learning. Golly, if their heuristics point in a particular direction isn't that a ~signal~ with predictive power and market value?
Take heed of this the next time you extoll the virtues of blind inference - whether human or machine. Maybe the next time it'll be _your_ soap dispenser that doesn't recognize black people. Or redlines black people and keeps them from getting a credit card, or mortgage, etc.
Because the reality is - there are all kinds of signals like these that are real. People in certain areas are more likely to be poor, to default on their loans, etc - that's a real signal, just like these high-value financial transactions. People doing big transactions in cash are way more likely to be involved in crime - just like people in certain areas are way more likely to default or whatever.
But relative risk says nothing about the false positive rate. People do cash transactions all the time, legitimately. Poor people manage to pay their mortgage without defaulting all the time.
Also - the reason people are trapped in that area in the first place is social, and economic. The decision to value a particular signal - or trust a model that blindly ignores confounding factors - ultimately lies with you, the human creating the model. Never hide behind it.
The machine is just a machine, it just does math for you. It only calculates the model you tell it to. If you let your model be racist, that's on you. You designed it that way. You designed it to be racist.
The creation of the "black box" model is one of the scariest things we've done in the last couple decades. Because now, nobody can tell you why it happened, it just was a bunch of neurons that attributed various weights to factors in your profile. If you're lucky they'll even be able to tell you what some of the weighting factors might be (but usually not). And now it's nobody's fault, it's just ~the model~. It's a blatant abdication of any responsibility for the faults or consequences of your work.
It's problematic in any situation where a programmer sets up model X and lets a mathematical model run on the input, and then accepts the result without social consideration, handing it over to a register jockey who must blindly accept the output without any ability to countermand the model.
It used to be that if you tried to get a loan there was a loan officer in your town who had the final say. Someone who knew the locals and could override the "model" based on human knowledge. The Kennedys are good for it, their business is still solid enough, the Johnsons are really unreliable and I wouldn't do that.
I'm much more concerned with the "false positives" here than the "false negatives". The loan officer can give a few bad loans based on his gut feeling... and then he's going to lose his job. There's a direct feedback loop on them. Someone who doesn't get the loan to expand their business is going to suffer much more immediately and much more deeply.
That's what I'm fundamentally against - the abolition of the "loan officer" in this situation, a human who can countermand the models when they're obviously wrong. At the end of the day these are truly just classifier models and there's no guarantee that any given output is valid for a given input - someone has to maintain the feedback loop and keep training the model back.
And not only that but these aren't meaningless "training runs", each one can potentially screw up someone's life. So again, big consequences for error here.
And indeed the socially-just answer may not even be the mathematically correct one. Is there actually a check that your training model doesn't discriminate against black people? If you weight that to zero, are you sure it isn't going to start picking up the addresses where they live instead? Or names?
The problem with black-box models is they are designed to identify arbitrary or hidden features. Even if you forbid one feature they often will just find another proxy. That's what the models are supposed to do, actually. That's super problematic when there's nobody around to tell the model "no", and it can ruin someone's life.
I'm picking on black people as an example here because redlining is a blatantly obvious case of a rational individual decision with massive social consequences. But you can substitute in "high risk financial transactions" like handling lots of cash if you like. Those are pretty obviously prone to false positives just like redlining.
Frankly there's a lot of things about yourself that have recently become "public knowledge" that you probably don't want a government agent to analyze with a blunt instrument. For example, the USPS saves "mail covers" i.e. addressing information for all letters/parcels in the US. Or, based on commercial information that can be gathered without a warrant (reddit/HN datasets are on BigQuery let alone actual ISP- or forum-level data which can be subpoena'd - similarly courier services like UPS/Fedex can be subpoena'd without a warrant) they could analyze users to see what topics they post about on the internet. Post a lot about drug legalization? Our model says that's suspicious. Don't mind the dogs, they're just sniffing.
This argument really doesn't hold water to me. It seems dramatically more likely to me that the human agent overriding the model is going to be prejudiced than the mathematical model itself.
Of course it's possible (in fact, almost certain) that a math model trained on a large set of data is going to pick up on some problematic features. However, is it really more likely that these statistical inferences are more biased than a human being?
I'm sorry, but in my experience the number of racist human beings outweighs the number of racist computers.
Your examples seem so fraught. The Johnsons are unreliable, from a human, seems as likely to mean that John Johnson and Mr. Overriding Agent's sister had a nasty breakup as it does to mean they're likely to bounce checks. The Kennedys are good for it just sounds like code for, The Kennedys are of the racial group Agent prefers.
I agree with you that we can't blindly follow computer models, but I don't think I follow you to your conclusion that the loan officer was a valuable safety net.
But that loan officer brings their own bias to the scenario. They could just as easily say the Johnson's are unreliable because they are black. It wasn't that long ago that saying that was institutionalized and I still suspect it occurs. An algorithm is colorblind. This isn't to say an algorithm is necessarily good, but that humans aren't either. One of the reasons bureaucratic red tape exists is as an effort to overcome individual judgement in favor of consistent and fair judgement. So though there's human intervention or policy considerations that need to occur in this situation due to massive abuses and obvious unfairness, I don't think it damns the mathematical model as a concept.
That's really what I'm arguing overall, mathematical models are great as a second opinion, or even a first opinion, but at some point you do really need a human in the loop to say "nah that doesn't make sense".
The nice thing about this, versus - say - a self-driving car - is that you don't have to make a decision within 0.4 seconds before the car crashes. There is plenty of time to get a human into the loop here, and to show the human why the model thinks what it does. Models that can't explain themselves, well, that's going to be of limited social value unless the need is absolutely dire like the car scenario. If the car gets me out of a crash, great, but I would want to know why my loan was denied.
If you can't put it into words or show a plot of the classifier... well... condorcet voting hasn't taken off either, everyone seems to prefer IRV so far even if it's "less optimal".
We have plenty of experience with "if you screw up as loan officer too much you lose your job". That's a pretty well-proven model. Especially now that you have an additional guidance signal on when the person is going "off the reservation" so to speak. Be a loose cannon too much and you're going to be the first guy carrying their stuff out the door in a box.
Frankly I'm much more enthused about the reverse here - I want to see the model tell me who's a bad loan officer given the hand they're dealt, not what a bad loan is. We can even incorporate social outcomes into that scoring metric.
It's a lot like teachers. Would you try to black-box yourself to the perfect textbook? Or would you be better off trying to figure out who's the struggling teacher in a shitty school and who's the lazy teacher half-assing it in the rich 'burbs?
This is kind of an ironic claim on a community that believes so strongly in "black boxes" like machine learning. Golly, if their heuristics point in a particular direction isn't that a ~signal~ with predictive power and market value?
Take heed of this the next time you extoll the virtues of blind inference - whether human or machine. Maybe the next time it'll be _your_ soap dispenser that doesn't recognize black people. Or redlines black people and keeps them from getting a credit card, or mortgage, etc.
Because the reality is - there are all kinds of signals like these that are real. People in certain areas are more likely to be poor, to default on their loans, etc - that's a real signal, just like these high-value financial transactions. People doing big transactions in cash are way more likely to be involved in crime - just like people in certain areas are way more likely to default or whatever.
But relative risk says nothing about the false positive rate. People do cash transactions all the time, legitimately. Poor people manage to pay their mortgage without defaulting all the time.
Also - the reason people are trapped in that area in the first place is social, and economic. The decision to value a particular signal - or trust a model that blindly ignores confounding factors - ultimately lies with you, the human creating the model. Never hide behind it.
The machine is just a machine, it just does math for you. It only calculates the model you tell it to. If you let your model be racist, that's on you. You designed it that way. You designed it to be racist.
The creation of the "black box" model is one of the scariest things we've done in the last couple decades. Because now, nobody can tell you why it happened, it just was a bunch of neurons that attributed various weights to factors in your profile. If you're lucky they'll even be able to tell you what some of the weighting factors might be (but usually not). And now it's nobody's fault, it's just ~the model~. It's a blatant abdication of any responsibility for the faults or consequences of your work.