Yeah, that's where they apply the activation function and that happens per neuron so you can't rotate everything and expect the same result.