Back in the days of Old Man Potter, the bank solved this problem by being local and the banker actually was your friend, so he already knew your social graph.
Yes, but that banker used heuristics like "Tom is a good white Christian who goes to church every Sunday, of course he is good for the money" and this algorithm can be proven not to, even if it comes to very similar credit decisions as Bill the Biased Banker.
This is part of the reason why credit scores took over the world in the first place: they're demonstrably blind to protected classifications. (They're also more accurate than Bill the Biased Banker and orders of magnitude cheaper to run and execute virtually instantly at 3 AM in the morning, but if their only feature was "As effective as traditional underwriting but immune to anti-redlining legislation" they'd still have taken over the world.)
Disparate Impact is not a real explanation, because it's equivalently useful at any stage of the argument. For example, "Disparate Impact" might explain why banks don't lend so much to group X. So if we find some good underlying variable that explains why banks would be disproportionately unlikely to lend to group X, "Disparate Impact" now explains that factor. And when that factor gets explained, it explains the next factor.
Disparate Impact is just "God of the Gaps" for discrimination.
The point was that in a small town the banker knows who you are and can evaluate your risk just by being acquainted with you. If he trusts you, he might just be your friend, if he doesn't then he is probably not your banker.
(Yes the word "friend" is a bit hyperbolic in this case)
