external modems sometimes have volume switches.
internal modem sometimes have jumpers to turn off the speaker.
You'll find a volume setting in the modem properties in the control pannel
failing all of the above
you can:
disconnect the speaker, there is no practical use for it except to tell you it's there...
or seal it in a box with some foam around it to damped the noise.