It all depends on what you want to do. First let's define our terms...
A Personal Information Manager (PIM) keeps up with things like phone numbers, addresses, email addresses, personal calendar and notes. Basically your Franklin(tm) Planner kind of things.
Contact Management Software does that and more. Contact management software allows you to keep all the information on a given contact (let's assume business for now) grouped and easy to search. You can keep notes of each phone call, sale, email, etc. all under one listing. Bill calls, you pull up his record and it shows you the last call you had with him, what emails you have been sending back and forth as well as all his contact information. Or, on Tuesday you look at your calendar and see you have a follow-up call with Bill that you made last week. Click on it and you get the history leading up to this call.
Outlook is not really very good as a contact manager as there is not a good way to set automatic follow-ups, emails, etc. It is, however a pretty good Personal Information Manager as it does track your phone book, appointment list and handle email and news groups. Even with the business content add-on it still falls way short of being a true contact manager.
ACT, Maximizer and Goldmine come to mind as better alternatives. All handle basic PIM work as well as the contact management tasks.
All three have trial versions and are worth looking at to see which one fits your way of working.
I would dismiss Excell and Yahoo Address Bool immediately if you really want Contact Management. Access is only good if you are a good programer and are basically going to write your own version of ACT or Goldmine.
Good luck