Empty Folders & Slows

Does having a lot of empty folders slow down servers? I.E. Will it slow down access times, the time it takes to defrag, etc. I manage a network with approximately 4000 users and our policy in the past is to give personal network drives only as requested. But as we are migrating file servers to a Windows 2003 Server, I would be inclined to just give everyone a network drive, but is it going to slow things down significantly? I have a coworker who seems to think it will, but I tend to disagree. Any thoughts?