Yes a speaker in your case. The beep would not come from the desktop speakers plugged into an audio jack. There should be pins on your motherboard to connect a speaker plug
Speaker plug example
This would connect to the speaker in a case like this
And on the motherboard to something like this
Sorry the pictures are terrible, can't find my camera.