I suspect its a DELL computer. am i right?
Dell likes to do shit like that with their cases. 7 years ago i also wanted to upgrade my dell computer with a new power supply.
Problem was that the case had a bump that was in the way of the power supply. Dell want's you to buy Dell power supplys. They also once had their own RAM. They where also the a**holes that invented the BTX standard.
Your computers CPU is actually still quite good. So i understand if you don't feel like buying a hole new system.
You can cut off the plug in your case, then get a better power supply.
I'll not recommend that GPU for gaming btw.
It's not really meant to be a gaming gpu.
Something like a 560 is WAY better.
Maybe also upgrade the storage to an SSD. That will further speed up the system.
And a 2x4 gig RAM kit wont hurt ether. 4 gigabyte RAM is beginning to become outdated.