Shouldn't the video card be using its own memory only? Why would it use system memory, too?
Anyway, freezing can be caused by lots of things. The two biggest culprits are hardware problems and overheating. Check to make sure all your connections are tight and everything is installed properly. Check your temps, too.