Hi,
You don't say how you tried to change the resolution. I assume that you used some GUI tool.
You need to be root to do the following. If you've tried your existing method as some other user, you might want to try that as root first.
I assume that RH9.0 uses Xfree version 4, which will by default poll the monitor and set everything up for the highest resolution that the monitor can manage.
If you can locate the file /var/log/XFree86.0.log which should be created every time you run X, look in there to find where the XF86Config file is. You can use:
grep XF86 /var/log/XFree86.0.log
then edit that file (make a backup copy first), and look for Virtual. There is a big chance that you won't find it, and will need to add it!
Look for 'Section "Screen"'. If there is a line with DefaultDepth 24 (or some other number) go down to the section:
SubSection "Display"
Depth 24
Then put in a line
Virtual 1024 768
after the Depth.
Then try starting X again.
Best of luck.