Well, it could be the server you are pinging on speedtest.
Open command prompt and ping
www.google.com
I'm getting about 100ms response time, let me know what you get.
without router:
Pinging
www.l.google.com [74.125.95.104] with 32 bytes of data:
Reply from 74.125.95.104: bytes=32 time=106ms TTL=50
Reply from 74.125.95.104: bytes=32 time=61ms TTL=50
Reply from 74.125.95.104: bytes=32 time=71ms TTL=50
Reply from 74.125.95.104: bytes=32 time=80ms TTL=50
Ping statistics for 74.125.95.104:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 61ms, Maximum = 106ms, Average = 79ms
with router:
Pinging
www.l.google.com [74.125.95.99] with 32 bytes of data:
Reply from 74.125.95.99: bytes=32 time=73ms TTL=50
Reply from 74.125.95.99: bytes=32 time=85ms TTL=50
Reply from 74.125.95.99: bytes=32 time=60ms TTL=50
Reply from 74.125.95.99: bytes=32 time=66ms TTL=50
Ping statistics for 74.125.95.99:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 60ms, Maximum = 85ms, Average = 71ms