Access to the specific folder will be slow, (I used to have the same on my music folder when I just dumped thousands of MP3's in a single folder).
access will be especially slow if you have thumbnail view turned on in that folder.
this should not affect the computer speed in general, though it is good housekeeping...
you've nothing to loose, (other than hours of your life) by sorting the pictures into subfolders...