Terminal Server RAM

Hi

My office runs Windows terminal server 2003.
We have about 15 users per application servers (We have 3) User always complains the system is slow. I must agree it is indeed. When I look at the task manager I can see the processor rarly go up to 12% uage but the page file is always over 2.2 G. The servers only have 2G of RAM and 4G of page file. Could this be slowing everybody down? My it subcontractant says 2G is enought but I feel putting 1 or 2 G more could not hurt? How can I judge the right amount of RAM to put on my servers? Thanks in advance for the tips.