No definitive answer here... just some thoughts.
Is this an Excel error... or Windows error?? If Excel error, take what Jim said to heart. (I wasn't aware of these limitations.)
Do you get same error if you reverse opening order of the files? Does either file have autorun macros... that perhaps do a lot of calculations?? [Disable macros and try opening files.] Must you open both at same time? Can you combine the files and open as one?? Will it open this 'unified' file???
I also question use of Excel as application of choice when file size hits 65Mb... and another one at 30 Mb???
I noted that you already changed 'virtual memory' size. I don't think this will have any effect, as I don't think this is a swapfile.sys issue.
I think you can ignore 'normal' apps or services running in background. Use Process Explorer (free download on web) to view running processes. Task Manager will also disply some memory usage figures. Obviously, there could be other issues... run virus scan lately??? Any services running that you don't recognize???
And finally... bear in mind what I always taught my students: WINDOWS LIES!!! If the OS encounters an error, it will present its 'best guess' what the problem is. BUT... sometimes this is NOT correct and leads you down the WRONG troubleshooting path!
Try these steps and repost results/your thoughts.
GL