Nvidia's main advantage over AMD for deep learning is software, not the hardware itself. Nvidia's software advantage has lead to network effects where almost all GPU supporting deep learning libraries build on top of Nvidia. There's no Tensorflow/Theano/Torch/Keras/etc for AMD/OpenCL for the most part. Nvidia also releases high performance kernels for common neural net operations (such as matrix multiply and convolution).
On the hardware end, Nvidia has slightly superior floating point performance (which is the only thing that matters for neural nets). Pascal also contains 16 bit floating point instructions, which will also be a big boost to neural net training performance.
I would be interested to hear about the difference in floating point performance. I would have guessed that, at this point, pretty much every chip designer in the world knows equally well how to make a floating-point multiplier. So it must be that nvidia vs amd have made different trade-offs in when it comes to caching memory near the FP registers or some such thing?
I'm unsure about the floating point performance differences, but 2 other reasons for potential differences are (1) number of floating point units and (2) different clock speeds.
On the hardware end, Nvidia has slightly superior floating point performance (which is the only thing that matters for neural nets). Pascal also contains 16 bit floating point instructions, which will also be a big boost to neural net training performance.