My old system ran Vista Ultimate just fine, and heres what was in it:
AMD Sempron 3300+ @ 2.0 GHz
2 GB Kingston RAM, PC3200
ATi Radeon X1300Pro 256MB GPU, in AGP 4x/8x
It actually ran pretty good, but I did have 2 gigs of RAM and a dedicated GPU. Of course, this PC met impending doom 2 months after I installed Vista when I blew the mobo by putting in a new GPU that needed a 400w+ PSU when I only had a generic 300w. I miss that computer, but the one I have now is SOOO much better.
And yes, you are probably right, if you want even better performance, go back to XP. That beats Vista in a lot of ways.