Re: Graphics card or new modem?
You'd need either a GeForce 570 or better, or a Radeon HD 6970 or better.
I have two Radeon HD 5870's in CrossFire and have played Rift, and it's a pretty graphically demanding game. You'd also need a CPU with a lot of horsepower, at least an Intel i5, if not an i7. An AMD Phenom quad core could also do it, but it would need to be an upper end model, at least 3.2 GHz.
Sorry, forgot to reply to the modem part. You don't need a new modem if the current one is doing its job. They're pretty simple devices anymore.