9600GT
Chipset nVidia Geforce 9600GT
Video Memory Installed 512MB DDR3
Core Clock Speed 650MHz
Memory Clock Speed 1800MHz
Shader Clock 1625MHz
Memory Bus 256 bit
Stream Processors (not stated from source
www.novatech.co.uk - but I believe they are 112 or 128 (much like the 8800GT(S)) although could be more)
3870
Chipset ATI Radeon HD 3870
Core Clock 775MHz
Video Memory Installed 512MB DDR4
Memory Interface 256bit
Memory Clock 2250MHz
Stream Processors 320
Based on the numbers there is no real way a 9600GT can compete with the Radeon. Bus width being equal and the fact the radeon uses GDDR4 instead of 3 should also help (only if overclocked as DDR4 dissipates less heat as it runs at a lower voltage).
The only real situation in which the 9600GT could (statistically) beat the Radeon is at low resolution settings (800x600 or less really). For many years Nvidia cards have always been better at lower resolutions than ATI cards, for no apparent reason. But when resolutions of 1280x1024 upwards are used the memory clock advantage and gpu core advantage should easily be enough for the 3870 to beat the 9600GT.
As I'm sure you already know, the amount of memory on a graphics card is pretty irrelevant unless the desired resolution requires it. A 1024x768 display with 32 bits per pixel depth buffer only needs approximately 3MB of memory to represent it. This can be doubled for storing the following sequential frame in the frame buffer alongside (to prevent flickering - technique known as double buffering). Most of todays GPU memory is used to store high detail texture maps / light maps / height maps / transparency maps / shadow maps and any other detail that game developers want to throw in.
If you're getting consistent results in favour of 9600GT at all resolutions then chances are something is up with the 8750 or the application doesn't like it.
EDIT: having looked at some benchmark results (
http://www.techspot.com/article/88-geforce-9600gt-versus-radeon-hd3870/page4.html), they appear to fluctuate a decent amount. A trend can be seen which shows when AA and AF are off the Radeon easily beats the 9600GT but when 4xAA and 16xAF it reverses slightly in favour of the 9600GT. I think nvidia accidentally made a 'budget' card that is in fact a lot better than its price tag implies.
The results are a lot closer than I expected which leads me to believe it is probably a ATI driver issue that is prohibiting the maximum potential of the card. (That being said, the 9600GT is a fantastic card as practically every source will tell you). Either of those would be an excellent purchase for that money.
Hope that answers your question.
