Trust is not just about understanding the model's behaviour - although that is clearly helpful.
1) The model may be applied to millions or billions of instances, humans can't check them all, or even a small percentage. Explanations appear extensive and general (it's using the right features) but it's very hard to prove or convincingly demonstrate that this is so - it could be using the right features or good features in a good way in this case - and then not in other cases.
2) Experts / owners are generally doing the checking; users are not victims, they should not have to simply accept what they are told. They need a means of protection and representation in the process to have real trust. If this doesn't happen adoption will be impeded, and I believe that once lost trust can be very hard to regain especially in systems with stabilisation mechanisms (memory)
3) Some users are less privileged than others, they don't have the freedom to make decisions about trust. Models are imposed on them, there needs to be a mechanism of wider accountability that prevents them from being abused.
I think that in addition to explanation three things are required :
- mechanisms for managing model development accountably and transparently; why were the decisions to use or discard data or techniques taken? What were the tests that were applied and when?
- mechanisms to manage models in production - what models are being used, where? Are they demonstrated to be being used appropriately? Can any harm that they do be discovered and repaired?
- Market, legislative and social structures that impose these requirements and penalise actors who fail to use them.
If not then expect AI to go the way of nuclear fission, airships and GMO.
1) The model may be applied to millions or billions of instances, humans can't check them all, or even a small percentage. Explanations appear extensive and general (it's using the right features) but it's very hard to prove or convincingly demonstrate that this is so - it could be using the right features or good features in a good way in this case - and then not in other cases.
2) Experts / owners are generally doing the checking; users are not victims, they should not have to simply accept what they are told. They need a means of protection and representation in the process to have real trust. If this doesn't happen adoption will be impeded, and I believe that once lost trust can be very hard to regain especially in systems with stabilisation mechanisms (memory)
3) Some users are less privileged than others, they don't have the freedom to make decisions about trust. Models are imposed on them, there needs to be a mechanism of wider accountability that prevents them from being abused.
I think that in addition to explanation three things are required :
- mechanisms for managing model development accountably and transparently; why were the decisions to use or discard data or techniques taken? What were the tests that were applied and when?
- mechanisms to manage models in production - what models are being used, where? Are they demonstrated to be being used appropriately? Can any harm that they do be discovered and repaired?
- Market, legislative and social structures that impose these requirements and penalise actors who fail to use them.
If not then expect AI to go the way of nuclear fission, airships and GMO.