Understand that I'm not there in person, and no amount of clarification in this forum is going to give me the same kind of eyes-on understanding of the situation as what you have. That being said, however:
1. You need to research what tools on a MacOS X server might help with what you're doing.
2. You don't need a bigger, better server so much as you do some separation of tasks.
3. You need some kind of backup capability for your server(s) in case something goes drastically wrong.
While your server is not running at capacity all the time (or even most of the time), it's nice to have some cushion between current loads and what it can handle. That being the case, it's a good idea to try to stay ahead of what the network will demand. Since you have run at peak capacity a couple times, it might be a good idea to start looking into getting more capability in your server setup.
If a MacOS X server will actually provide you with better tools for dealing with the tasks you describe, you should definitely look into getting one. You must balance that against other needs, though.
Whether or not you get a MacOS X server, you need at least two servers in operation that are not Macs: either one main server and one backup server to store important data, or two "main" servers that share network load and each of which can entirely take over the job of the other (including both having full copies of important data) if a server fails. If you go with a MacOS X server, you should probably make that second server a backup rather than a second "main" server, because the Mac server should be backed up as well.
Considering your circumstances, it sounds like the backup server option is the way to go for now, and you can always add redundant/parallel server capability later. With the already present server, you'd only need to buy two other servers to diversify your server base by tasks. Thus, you could have a MacOS X server and a Windows server, with the current machine turned into a backup machine to store data backups from which you can restore either of the other servers in case one of them goes down and needs to be recovered.
Since you're not in an environment where lives depend on 24/7 uptime, I'd recommend the above over going with an attempted redundant server setup first: for your purposes, splitting tasks among servers is probably the better way to go.
Since you're in a production environment, if you're not familiar with unix systems, I'd recommend going with a Windows machine, a MacOS X machine, and your backup server being set up on Linux. For a backup server and basic file services, Linux is fairly trivial to learn as a server environment, and the added stability and ease of maintainability of the OS will serve you well in that role. While Linux is fully capable of providing the services you need for both Windows and Mac clients, unless you are already quite familiar with unix systems, I would not recommend trying to replace all of your servers with Linux systems at this time.
The downside of this arrangement is that you'd then have three different OSes on three servers. The upside, of course, is that you've only got three servers to manage, and properly configured you'll almost never even notice the Linux server: it'll just quietly do its job and, barring hardware failure, will not require any management beyond the usual administrative log-checking tasks and the like.
One more time: Take all this with a grain of salt. I'm not there, so I can't give you perfect advice. This is all just based on what I know of your situation from here.
If you DO go with the "buy one huge, new server" option, don't throw away your old server. You definitely need to set up some kind of data redundancy, particularly when handling that many client systems. In case of primary server failure, you'll be glad to have the data saved somewhere for quick recovery. Again, Linux would serve admirably in that role, and with basic backup server functionality it's relatively trivial to familiarize yourself with Linux.