I thought this was an excellent treatment of the garbage-in-garbage-out aspect of this debate. Too often it seems people get wrapped up in the "Is it ethical to use a computer to judge someone?" implications, which are non-trivial, but also seem to be borne out of a general ignorance about the many ways we do allow computers (or rather, computation) to make judgements about people.
Also, it's an excellent example of the power of research and tabulation, though it seems the link to the list of departments they researched is broken.
I think the use of crime modeling doesn't have to be controversial, but the first step is demanding transparency. It's scary enough when the police are the primary collectors and judgers of data, but even worse when they leave it to a vendor and assume it's hunky-dory out of the circular reasoning that proprietary algorithms are good because they are proprietary.
> When the Fresno police briefed the city council about the Beware system, the council president asked a simple question about those color-coded threat levels: “How does a person get to red?” The officer didn’t know, because the vendor wasn’t saying. As the officer delivering the briefing explained: “We don’t know what their algorithm is exactly… We don’t have any kind of a list that will tell us, this is their criteria, this is what would make a person red, this is what would make a person yellow. It’s their own system
Aren't there numerous ML techniques potentially be applicable to evaluating potential future crimes where even the people responsible for writing the algorithm and feeding the massive dataset in don't really understand how a particular person might "get to red"? Transparency seems to be only part of the problem.
The corollary to this: if an algorithm uses AI or machine learning to the extent that nobody precisely understands why it makes the decisions it does, it will be very difficult to change its behavior in specific cases, e.g. "make it stop doing that," especially when the inputs cannot be changed.
This is going to come up at some point when a self-driving car does something that appears totally irrational and ends up causing an accident. Engineers will need to come up with some kind of explanation, and I suspect the general public will not be satisfied when they learn that the explanation may be unknowable, or may reduce only to probability instead of certainty.
I deal with some of this at work in far more trivial use cases, and non-engineers just can't seem to accept that sometimes you cannot fix the imperfections without introducing worse imperfections in other areas of the system, and that ML generally leads to output that is "good enough" instead of perfect.
Maybe. But at least we can demand to know the system's input data and what measure(s) the training process is designed to optimize. For example, in the case of Beware, which public records are being used, and what is the benchmark for red/yellow/green? Also, are there any systems in place to try to reduce things like racial bias or the self-reinforcement effect described in the report? Why or why not?
Maybe, but I'm not aware of any that have produced predictions on a broad in vivo population that are better than existing methods. It's one thing to do a heat map of a region at a certain time of day, and express that in terms of overall statistical risk. It's quite another to try and read the future intentions of a human being based on what amounts to a wealth of noise and a paucity of signal.
Also, it's an excellent example of the power of research and tabulation, though it seems the link to the list of departments they researched is broken.
I think the use of crime modeling doesn't have to be controversial, but the first step is demanding transparency. It's scary enough when the police are the primary collectors and judgers of data, but even worse when they leave it to a vendor and assume it's hunky-dory out of the circular reasoning that proprietary algorithms are good because they are proprietary.
> When the Fresno police briefed the city council about the Beware system, the council president asked a simple question about those color-coded threat levels: “How does a person get to red?” The officer didn’t know, because the vendor wasn’t saying. As the officer delivering the briefing explained: “We don’t know what their algorithm is exactly… We don’t have any kind of a list that will tell us, this is their criteria, this is what would make a person red, this is what would make a person yellow. It’s their own system