On mine if I plug/unplug the speakers xp forgets all about the sound card. It even unloads the drivers for it. Pain in the butt for such an otherwise nice sounding setup.
What make and model card is it and what are you doing when the drivers drop out?