Mac systems are pretty trouble free, and OS X is a pretty solid system. However, as with all things, an ounce of prevention is better than a pound of cure, and there are some things that you can do with OS X to keep it running in tip-top shape.
#1 Back it up
The first and most crucial thing to do is to keep a backup of your important files, or even the entire system. To clone the entire system (great for backups and quick restoration in the case of hardware failure), an external hard drive and SuperDuper! are the best solution as far as I am concerned. SuperDuper! makes a bootable clone of your hard drive; a great thing to do before performing large system updates, or something to schedule routinely. Other options include buying an Apple Time Machine and scheduling backups that way.
#2 Keep software up to date
Keeping software current is another important thing. Software updates not only provide new functionality and correct bugs, they often correct security flaws as well. Keeping your system up to date via the OS X Software Update is important, but so is keeping the third-party applications installed current. Many applications "phone home" to check for updates and will alert you if updates are available. Otherwise, you can point your RSS feed reader to MacUpdate's web site to keep you alert of new software releases. You can also elect to pay MacUpdate's yearly subscription fee and be alerted via email or use their desktop application (which is quite useful as well, as it will inform you of what installed applications are out of date).
#3 Perform a Permissions Repair
This can be done by launching Disk Utility, selecting the boot drive, and clicking Repair Disk Permissions. It is a good habit to repair permissions before and after large system upgrades (i.e., when upgrading from OS X 10.6.3 to 10.6.4). As well, in Disk Utility you can use the "Verify Disk" button to scan the integrity of the filesystem on the hard drive. If it identifies problems, you will need to boot from the install disc or a cloned drive (another reason SuperDuper! is useful) and use Disk Utility to repair them.
#4 Get the right tools
Tools exist to keep an eye on your system, and free ones are even better. To be informed of the impending decline of hard drives, use the free SMARTReporter menu bar tool. This tool checks the S.M.A.R.T. output of your hard drive(s) and reports if there are any failures or inconsistencies; however, keep in mind that this only works with internal SATA/eSATA/IDE drives, and not external drives.
Another great monitoring tool is iStat Menus. This is another menu bar app that displays useful information such as a CPU usage meter, memory usage, disk usage, disk activity, temperature sensors, and more. If it feels like the system is slowing down, clicking on the menu bar icons will show you what applications are taking the most CPU or consuming the most memory.
#5 Beware of memory leaks
With the stability of OS X, some people may leave their systems running constantly, or put them to sleep without restarting the system. If any running software has memory leaks, even small ones, leaving them open constantly will start to consume more and more system memory. Some applications that use a lot of memory, such as Safari and other browsers, may not fully release that used memory when tabs are closed, and it isn't uncommon to see Safari using far more memory than a browser warrants.
I've seen Safari consume over 1GB of memory all on its own (granted I tend to have a whole lot of tabs open at any given time). Quitting and restarting these memory hogs will reduce memory consumption, at least for a time. Restarting the entire computer will reset OS X's virtual memory system and clean things up; after restarting, the system should typically feel snappier.
#6 Watch your hard drive
Another common sense tips include keeping an eye on hard drive space. Generally speaking, you should always ensure you have about 10% of your drive available as free space; if you find yourself filling the drive quickly or often, consider replacing it with a larger drive, delete some unused applications or files, off-loading some archive data to an external drive, or simply emptying the trash every once in a while will help.
#7 Check Login Items
Finally, a lot of programs like to run little helpers in the background, or think that if you run them once, you want to run them all the time. Firing up System Preferences and looking in the Accounts pane, then selecting the Login Items tab for your account can be a real eye opener. These programs start when you log in, so if there are applications here that you do not want running, here is where you can remove them. Simply highlight the application you no longer want starting at login and click the "-" button to remove it.
These are all fairly straightforward tips, but continuing to keep them in mind will keep your system running at an optimal level of performance.
Vincent Danen works on the Red Hat Security Response Team and lives in Canada. He has been writing about and developing on Linux for over 10 years and is a veteran Mac user.