I agree with David on the dependancy issues with visual libraries. VC++ isn't really pure C++ and would need modifying to be used on a non windows platform. Of course, if you're just a casual coder that probably isn't an issue to you. I use Eclipse IDE which, although is harder to get something working quickly, does allow flexibility and greater debugging facilities (which will ultimately save you time in the long run - trust me on that)
At the end of the day its personal preference when it comes to compilers. Once you learn one well (and its limitations i.e. libraries) you should have no trouble switching to whatever you need to use.