I'm guessing that only happens with large files? It's because it originally downloads the file to a temporary folder, and that pause is just from it copying the file to your download folder. Which will take a while with big files.
One of the main things that made me switch back to XP was the enormous amount of time it took to unzip a zipped folder within Vista. It would take me 40 minutes to unzip an 11MB zip file. Hopefully that is fixed in Vista.