So it could be picking up the sound of the fan in the new psu, or on the new graphics card.
Is there a headphone port in the front? What happens if you plug in speakers or different headphones? Maybe plug in speakers and then plug the headphones into those?