Just speaking from experience, GAN upscalers pick up artifacts in the training dataset like a bloodhound.

You can use this to your advantage by purposely introducing them into the lowres inputs so they will be removed.