Something had to be downloading in the background on the non IE browser. Just look at your ping time there. 520ms vs the 20ms time on IE. A ping spike like that suggests something was being downloaded while you ran that test.
Oops yeah I meant Chrome fail.
Took a quick look at the browsers then went to the numbers.
Then I mixed which was which in the post.
As my friend would say "Dee-taa-Dee".
Yeah the ping is really odd.. Are you able to get the IP of that test server?
If can I would then open up command prompt, try a ping, and see what you get.
Of course it should be no where near 500ms.
I don't know what you've been reading, but with the latest security flaws unveiled I've been reading the opposite!
Regarding the Chrome issue, run a "manual" speed test by checking how long it takes to load a webpage and download a file, roughly speaking. Unless you notice huge differences between the two, that test is wrong - they're not always brilliantly reliable.