Shared Media on 1 server

I am responsible for setting up and maintaining what right now is 2 websites, and will potentially become 5 in the future. I already notice that between the 2 sites existing (each on 2 different servers) I use duplicate images and scripts. I can foresee a maintenance nightmare when I have 5 sites that use the same images and scripts. 1 of these upcoming sites will be an Intranet, the others will be available to the outside.

What if I set up 1 site to host only shared media (images, movies, scripts, etc.). Do you see any cautionary measures to take, any performance hits to expect? Of course there’s always the chance the server will go down, and all sites lose access to the files. Anyone else dealt with this issue before?