I believe what you are referring to is frames.
Here is an explanation: http://www.w3schools.com/htmL/html_frames.asp
I would STRONGLY advise against using frames, since many browsers have compatibility issues with displaying them.
The other option is using an iframe (essentially the same thing, still not supported well, but more so than traditional frames).
Here is a link: http://www.w3schools.com/TAGS/tag_iframe.asp
The problem you run into here is the stateless nature of the http protocol. If you were to use ajax, it would have to be part of a much larger solution than copy-pasting some code.
Take this website for example:
The music remains (without re-loading) on every page, BUT...
The entire page is flash. While their appears to be a re-load of the whole page, closer inspection of the url on each page reveals that you are still looking at the same page. As a rule of thumb for me: Unless the audio is loaded into an swf and pre-loaded properly, don't use audio on your website. Partially because you run into codec problems and bandwidth problems. The ajax requests for returning an audio file would most likely kill any bandwidth limit you have setup with your ISP.
Hope this helps. Let me know if you have any other questions/concerns.