I guess if I really wanted to fine-tune a machine for some high-demand task, I might compile the kernel so that the performance-critical elements are statically linked. For example, if I were setting up a file server that, for some strange reason, was going to get pounded, I'd compile the kernel with the network card driver statically linked. It wouldn't make a huge difference, but if I could squeeze a little more out of it, I'd do it. It's not as if it's a lot of trouble, after all.

Also, if I were going to use a Linux firewall, I'd probably clean out anything other than exactly what was needed to do the job.