The slowdown can be enormous if you use SIMD, I believe MSVC emits a write to memory after every single SIMD op in debug, thus the code is incredibly slow(more than 10x).
If you have any SIMD kernels you will suffer, and likely switch to release, with manual markings used to disable optimizations per function/file.