But it will take a while to guess a correct MAC address right?
Nope, just a matter of seconds. For a lab in school, we used a network scanner (similar to wireshark) to monitor wireless traffic. The app could find a network and pull in any data attempting to reach that access point even when we were not connected to it. The source and destination MAC addresses on the packet are not encrypted. How easy it is to spoof your MAC to match one that is allow, varies based on the OS you are using. It was pretty easy on back-track linux.
That said, I doubt the OP's neighbors are smart enough to work this out but it's still surprisingly simple.