Critical Thinking

It's a shame Tech Republic does not follow your advice.  I read your column in minuscule type with two very large grey columns on either side.  Had to do a lot of scrolling and squinting to read.

Imagine that you could see the following improvements on your Web site, with a total of only four weeks of work:
* 244% increase in conversion rate
* 286% increase in overall sales
Those numbers are from http://www.creativegood.com/casestudies/shure.html

Here is another set of numbers to consider:
* Sales/Conversion rate improvement: 100%
* Traffic/visitor count improvement: 150%
* User performance/productivity improvement: 161%
* Use of specific (target) features improvement: 202%
That set of numbers came from http://www.useit.com/alertbox/20030107.html

Wow! If you are in charge of (or derive income from) a Web site or software development, how many fingers would you cut off for these levels of improvement? Well, there is good news for you: you do not have to lose any fingers. You just need to follow usability theory.

The first set of numbers comes from Creative Good, a design consulting firm run by Mark Hurst after performing four weeks of site redesign. The second set of numbers comes from Jakob Nielsen's Alertbox, as a result of a 135% increase in site usability. Both of these people have proven, decades long track records of significantly improving the usability of applications and Web sites. They are also the two main influences on my running water Web design philosophy.

Mr. Hurst pays special attention to the user's experience. Mr. Nielsen uses scientific methodologies (control groups, eye tracking, in depth statistical analysis, and so on) to choose amongst proposed designs. I would have to say that the numbers above (just a brief sampling of the numbers available on each of their Web sites) speaks for itself.

As I have already written, usability is the killer feature. There is not a single site out there on the Web which is unique. Someone else is always offering the same products, content, or service for the same price you are, sometimes cheaper. Your Web site, product, content, and service is a commodity and users have choices. If you are first to market today, tomorrow you will be one of a dozen, and by next week you will be in a sea of hundreds. Google makes their living by stealing everyone else's lunch by improving usability.

Remember, you are not the end user. Your high priced design consultant is not the end user. The Internet elite pushing AJAX and Web 2.0 are the not majority of end users. My grandmother, your six year old daughter, the person who thinks that the CD-ROM drive is a coffee holder is the typical end user. Unless you are making a tool for use by programmers, asking your programming friends if something is usable is worse than worthless, it is harmful. Think about that. Re-read those numbers.

I have never, ever spoken to an end user who said, "gee, I had a really hard time figuring out how to make a purchase from that site and I just gave up, but wow, it sure was pretty, I will be sure to attempt to shop there again." I have never seen an AJAX or Web 2.0 proponent show hard evidence using testing methods that follow sound methodology prove that their techniques improve sales, conversion rates, traffic levels, hit counts, new user registrations, or any other bottom line number. I have never seen a consultant or graphics designer who specializes in confusing yet beautiful site design to offer a single number that shows that by reducing usability they improved the metrics that truly matter.

Another interesting fact is that the rules of usability are fairly static. For example, initial efforts at 3D interfaces showed that they were just as useless in 2006 as they were in 1986. Improvements and changes in technology do not equate to a change in the users' needs; they are merely different ways of addressing the users' needs. TiVo did not suddenly create the desire for people to view movies whenever they wanted. It simply addressed the desire, and did it better than a traditional VCR did, which did it better than HBO did, which did it better than the movie theater did. The Internet did not create a demand for adult content. It simply did it better than? you can imagine the list. These are examples of how improvements in usability generate big bucks. If you do not believe me, compare the sales figures for iPods versus portable CD players.

So yes, if you want to improve your Web site's performance, by all means consider a redesign. But do not settle for the most attractive design or the design with the most features. Choose a design that will demonstratively improve you bottom line. That means listening labs, observations of users at work, log analysis comparing the new design being given to random users versus the current design, and so on. That means using scientific rigor. Even if you do not have a big budget, just grabbing ten people who have never used your site or application and represent typical users (family, friends, spouses, etc. are a great source of free testers) and watching them trying to perform typical tasks while timing them can help you. But to sit in an ivory tower or a conference room where groupthink kicks in and nodding your heads saying, "Yes, I like this much better" will not help you.

J.Ja

I agree with David Mercer - if it wasn't for the fact that TechRepublic occassionally delivers pearls of wisdom like this article, I would have abandoned it precisely because it breaks so many of the rules advocated.

What is it with only using 30% of my screen width to display information that I care about!

PS. The pearls are becoming more rare.

One of my earliest experiences with IT work was volunteering for a local charity. It was a home for developmentally disabled adults. I assisted with their computer systems, creating accounting and budgeting spreadsheets, building a Web site, and more. It was also one of the most personally satisfying IT experiences I have ever had. In high school, I was extremely active with charity work through the Air Force Junior ROTC program that I was in. Ringing bells for the Salvation Army, visiting local nursing homes, and more were part of that experience. None of these had any kind of payoff for me other than a sense of satisfaction with myself.

Over the years, I fell out of this practice. Recently however, I have begun doing work like this again (not in an IT capacity, however). I still find it immensely rewarding. I have decided to offer my services as an IT professional to some non-profit organizations as well. IT services are typically quite expensive. If I can help where I can, I want to.

The Open Source Software folks talk about how they are changing the world by giving away free software. Yes, it is great that businesses can save money and new technologies are being developed. Right now, there are children who have been severely abused in shelters, addicts trying to recover from their disease, prisoners trying to learn a new way of life... the list is endless. And there are non-profit organizations trying to do something about of these problems. It is easy to look at some of the recent scandals and problems with the large charities out there and say, "well that does not look like something I want to be involved in, and they have enough money to hire IT professionals anyways." That may be true. However, for each huge charity that may put only 10% of its budget to actually helping people, there are a hundred local charities operating on a shoestring budget. These charities are where you can deliver the most help, and they are the ones with the least ability to get IT help.

One of the great things about volunteer work is that your resume is not nearly as important as your willingness to help. If you are looking to learn a new skill, volunteering is a great way to put it into practice. It can also help build your resume, because you will have the opportunity to try things which you are not qualified to do "on paper." For example, a local group that I am involved with is beginning a VoIP project. I have no experience with VoIP, but I have a lot of experience in general networking, systems administration, telecommunications, power protection, and other things peripheral to a VoIP project. The person who was asked to do the VoIP project has zero knowledge of any of these things. This organization was delighted when I offered to help out, and now I will have the chance to learn VoIP when I might not otherwise have the chance to do so.

We all know what kinds of productivity gains IT can deliver to businesses. But non-profit organizations, particularly those that operate on the local level, often cannot afford the capital investment to leverage IT properly. I am not familiar with any services or Web sites that help worthy organizations find volunteer IT help, or help IT pros looking to do some good deeds assist a good cause. If you know of such an organization, please let me know. I am also considering starting a Web site to assist with this. If you are interested in helping, please let me know via TechRepublic private message. I think it is high time we put the spirit of the OSS community to work in helping others with problems more important than the TPS reports.

J.Ja

Wow.  Could you please forward this blog entry to who ever designed the TR site?  After I copied and pasted the entry onto Word so I could get the font large enough to read, I decided that I needed to have the web designer who maintains my site read the entry, maybe he will understand the way you wrote it, because he sure doesn't seen to understand me!  Oops, there is that session ID you talked about in the URL I copied and pasted, If this site had a built in "forward this article" feature I could use that, I guess I could forward the word document I pasted to to get the font large enough to read, but I really don't like doing that.  I guess I'll send him the URL, and if he isn't smart enough to figure it out its time for a new web designer.

Oops, I just realized that the URL I was copying was the one from the comments editor page.  The actual blog page does not have a session ID.  My bad.

As I type this I notice that the icons at the top ot the edit window are reloading every time I hit a key.  What's up with that?!  I'm sure glad I have a really fast connection, that could get real annoying at 28.2.

EDITED - Added Information

Jason -

If you honestly beleive that Web usability has a low ROI, I suggest you check out my post "You do NOT Need to Cut Off Limbs to See a 286% Sales Boost". Yes, many of these are "common sense". However, "common sense" is pretty rare, especially when the IT industry is in the grip of massive groupthink. It seems to me that the vast majority of designers are stuck on a few concepts which seem to "look cool" but in reality lower usability, and therefore kill profitability. If you are using a quality CMS system (they are also quite rare) or even just making an HTML template, it is not hard to follow these suggestions. In fact, my experience has been that it is easier to let a Web site flow and adjust to the user's environment, than to attempt to force a particular layout in all cases. If you cannot have a Web design which does not require horizontal scrolling at 800x600, I suggest you re-evaluate your design. Web designers have been creating great looking designs which work quite well at 800x600 for well over a decade now.

To other comments, thanks much for your feedback. Wesley, I know that the TR site has some issues. These are outside of my control, but I know that the TR staff is extremely responsive to user feedback. If you make your thoughts heard to "those that be", and enough people do, I am sure they will at least take it into consideration. I too experience the "buzillion image downloads with each keystroke" problem, so I am forced to write blogs in Word and copy/paste to the blog, then strip Word's bad HTML. That is not a good solution at all.

J.Ja

I am certainly no hardware expert, but for whatever reason, I think we lost something in the world of computers when we went to a purely binary world of computing. Maybe I have just read Destination: Void by Frank Herbert one too many times. But something inside of me is just nagging that a system of many parallel circuits performing identical functions, but that allows some randomness to occur, could be incredibly useful, particularly in the field of Artificial Intelligence. Any kind of randomness would be fine, whether it be slight delays in processing by some circuits, slightly inaccurate results due to minor differences in physical state, or even just random interference from stray EMF radiation.

Why do I think this? It is because I believe that making mistakes is a crucial function of the human learning process. Until one makes a mistake, it is impossible between understanding and the repetition of memorized information. Currently, with binary computing, all "intelligence" is simply a form of memorization. The computer is handed a set of initial patterns, and some rules to build new patterns, and it is supposed to "understand" linkages between abstract concepts. But since the programmer is giving the computer the rules to making new patterns (even if those rules are patterns to develop new rules for pattern generation), the computer is ultimately tied to the programmer's inherent biases and concepts.

What I envision instead is a tabula rasa of hardware, with some sort of input mechanism giving the computer access to data feeds (camera data, audio, keyboard, or whatever), a simple output mechanism, and a mechanism to display approval or disapproval of the results. It is up to the computer to reprogram itself in a method that avoids disapproval at the very least, and seek approval. In a nutshell, we create "pain" for making a mistake, and "pleasure" for success.

Why the tabula rasa? Because that is how human minds start. Humans may have an instinct for language like any other animal, but the language itself is undefined. There are language portions of the brain, but no particular language is needed, thus the proliferation of languages all over the world. It is quite interesting to note that certain cultures reflect themselves in language, but even more interesting that language itself affects the way in which people think. I believe that this is because language imprints patterns upon our thought, and our thought tends to follow those patterns.

Underneath it all should essentially be a massive pattern recognition system that the computer can rewrite as it sees fit, possibly in a method of evolution with some random mutation thrown in for good measure. Hopefully, many of these systems can be networked together in a less than perfect manner, so that they may swap patterns, but the communication system would occasionally alter or even completely corrupt the pattern. It would seem to me that the transmission of mutated ideas could help spur additional evolution of the pattern system.

I think that a good beginning of this system would lie in evolutionary game theory. Current techniques involve establishing set strategies in advance; this system would develop its own strategies. The evolutionary game itself would provide the reward and punishment system. I think many techniques from functional programming may be used for such a system. Overall, I think it would be an absolutely fascinating project. In my mind, I feel that this type of system could be used for a wide variety of tasks. Often in data, it is the exceptions to the norm (outliers) that are the truly interesting piece of information. If you are a police detective, the lives of the average citizen are not of much interest to you, but the behavior of a small segment is much more important. This kind of system would be excellent at finding the exceptionally interesting data.

Another great use of this kind of system would be to replace the folksonomy systems that are sprouting up everywhere. Folksonomy systems are a direct result of the fact that quality contextually relevant searching mechanisms are incredibly difficult to program. I have a major problem with the "tagging" trend, which is the same problem I have with Wikipedia. Only the truly enthusiastic bother to do it consistently, and the truly enthusiastic (or worse, fanboys) are not very reliable because of their enthusiasm. Just like it is very difficult to get good information about any OS thanks to the fanboys and zealots, I do not like the idea of using tagging or folksonomy to generate metadata. A system that is capable of generating its own contextually relevant patterns would be excellent at this.

If you have any ideas on this subject, I would love to hear them. Pattern matching is one of my personal pleasures (I drive my friends nuts with high-speed association games), and AI completely fascinates me.

J.Ja

In April, I touched on the topic of "Geeks and Communications Skills." Reading through some of David Berlind's (over at ZDNet) recent blogs about poor customer service via call centers, I am following up my original post with this series. This series refers specifically to communicating with call centers, a potentially unpleasant task that most IT professionals need to deal with on a regular basis. This post discusses how to get maximum results from a call into the support hotline, and the next post in the series will show how to provide quality phone support.

Getting quality support over a phone for a technical matter is not the easiest thing in the world to do. As someone who has been on both sides of the phone call, I know that it can be frustrating both to get help and to be helped. As IT professionals, making contact with phone-based support is part and parcel of our jobs, and it is inevitable. Chances are, when you are calling the support hotline, you need help now, all too frequently with a mission critical problem. While we should not need special skills just to deal with phone support, it is the unfortunate case that without being prepared and knowing how to effectively communicate with call centers, it is hard to get the best support possible.

Be prepared before calling

If at all possible, have any relevant logs or error messages handy. Telling the technical support person "it told me something about an interrupted communications with something or other" is not going to help them resolve your problem. Have the exact error message (particularly any error code or status code numbers) will cut the time needed to get help significantly. Also, make sure that you have a paper and pen on hand, to take any pertinent notes. If the problem is with a piece of equipment in another room, have a way of "walking and talking," and make the call from a cell or cordless phone if possible, in case "hands on" troubleshooting is needed. For pieces of equipment like networking gear or headless servers, make sure that you have a laptop and the correct console cables nearby. If you have to put the support technician on hold for ten minutes to dig for your tools, you are not helping the call go well at all. And of course, always have the serial number and support contract number, or previous ticket number in your hands before dialing the phone; many support technicians must reject your call, even a "quick question" without that information.

The first few moments

Make sure to get and record the technician's name (as well as the spelling) or ID number as soon as they answer the phone. Also try your best to pronounce the technician's name correctly. Do not be dismayed if the technician requires full details on you and your unit even if it is a brief question, this is the process that the support person is required to follow. If the technician sounds harried or insincere with their greeting, let it slide. Chances are, the technician is harried, and it is difficult to make "good afternoon" sound genuine when it is the 78th time you have said it that day. Keep in mind that you are calling to get technical support, not exchange pleasantries with a complete stranger. When the technician asks "how may I help you?" do not tell a long story. Describe in as few words as possible what the immediate problem is. The technician will determine if they need to hear about the preceding events. Here is an example:

WRONG

Technician: How may I help you?

Customer: We got this unit about seven months ago, I think we bought it from a place we found on the Internet, but it may have been through our local reseller. It has worked great up until a few days ago, but lately it has been a bit flaky. It was making some odd noises like the noise my brakes make when the pad is worn out, and I think I may have smelt something strange in the server room near the unit, but the other technician who works in the server room did eat chili that day, so it might not have been the unit making the smell. Well, today it was down, and when I powered it back on, it sent me an email about having a bad hard drive in it.

RIGHT

Technician: How may I help you?

Customer: The unit was not powered on this morning, and when we powered it up, it sent me an email about having a bad hard drive.

Technician: Did you hear it making any strange noises?

Customer: Yes, it make some grinding and clicking noises yesterday.

Technician: Yes sir, that is definitely a sign of a bad hard drive. I see that you are still under warrantly for parts and I will be delighted to get a replacement drive sent out immediately. Are you able to access the unit's Web-based administration system so we can find out which drive is bad?

See the difference? The first conversation fed the technician a lot of information that seemed relevant to the customer, but was not relevant to resolving the problem. The second conversation was short and to the point. By the time the first customer got done telling his, the second customer was nearly finished with getting a replacement part.

Respect the technician

When calling into technical support, always be respectful the technician, no matter how frustrated or upset you may be. Remember, the support technician did not cause the problem, create the design flaw, write the manual, write the code, assemble the hardware, perform your configuration, or have any other part in the problem when you call in. They are there to resolve your problem. If you vent your frustrations on the technician, they are going to quickly put you in the "jerk customer" bucket and the quality of your support will likely reflect that. The technician who may have been willing and ready to go the extra mile or provide support above and beyond contractual obligations will barely meet the bare minimum if you are abusive to them. Under no circumstances should you use foul language or raise your voice.

WRONG

Technician: Thank you for calling Servers Express, my name is John. May I please have the serial number of the unit?

Customer: Sure, the serial number of this piece of junk is I-A-M-B-R-O-K-E-N, I mean, 55-66-1212. Listen, this thing is the worst purchasing decision I ever made. Just get me my money back and I won't have to drive over there and knock some skulls.

Technician (rolls his eyes, puts the phone on "mute" and curses the day his was born, then take the phone off of "mute"): Sir, I understand that you are frustrated and upset, but I am here to help resolve your problem. However, I will request that you not be abusive towards me. Let's try to resolve the problem first, it may be something simple that can be immediately resolved. I am here to help you sir, and if a refund is needed, I will be happy to help you with that.

RIGHT

Technician: Thank you for calling Servers Express, my name is John. May I please have the serial number of the unit?

Customer: Sure, the serial number of this unit is 55-66-1212. We have been having a lot of problems with this unit, and I would like to work towards getting a refund.

Technician: Sir, let's try to resolve the problem first, it may be something simple that can be immediately resolved. I am here to help you sir, and if a refund is needed, I will be happy to help you with that. Could you please describe exactly what the problem is occurring, so that I may determine if I am authorized to get you an immediate refund?

If you feel that the technician is not treating you with due amounts of respect, do not yell or curse, request to speak to their supervisor. If the call heads south, do not hang up and call back in hoping to get to a different technician, get the supervisor on the phone. Most call center employees need to work very closely together, and are in close contact with each other to find out what happened on a call or to ask each other for advice. This also means that when you call back in, the next technician you speak to is quite likely to be already aware of the argument that you have had with the previous technician, and is now expecting confrontation. Also remember that the notes in the ticket often contain more than just details of the technical problem. A good support technician immediately documents any kind of problem with the customer and alerts their supervisor (whether or not they are truthful is a separate matter) when a call goes wrong. This means that the ticket notes may have a warning such as "customer was extremely abuse and dropped the call" in them. Finally, keep in mind that you do not have to enter every fight that you are invited to. Again, if the technician is indeed being rude or abusive, you simply need to request to speak to a supervisor. After all, you are a customer.

It is also important to remember that the technician is doing a repetitive, thankless, and probably underpaid job. That technician spends eight hours a day taking phone calls from people who vent their anger on the technician, who do not have the technical knowledge to be trying to resolve the problem, and has a boss breathing down their neck about "hold time" and "average talk time" and "average number of escalations." In addition to handling your phone call, they may be simultaneously trying to help their fellow technicians via email or instant message, as well as supporting some customers through email or online chat. Although that is no excuse for the technician to deliver less than excellent service, it is something to keep in mind when talking to the technician.

Also keep in mind that the technician frequently has a script or process that they must not deviate from. If the technician gets away from that script or process, and there is a problem, they are the ones who get in trouble. Support technicians quickly learn that they keep their jobs by following the script. If the technicians will not budge from the script or process, do not blame them. That does not help you at all. If it is obvious that the technician's script or process prevents them from helping you the way you need to be helped, request an escalation to a higher level of support or to speak to a manager, depending upon the problem.

Stay focused

For a variety of reasons, try to stay focused on the problem at hand. Many outsourced call centers get paid by the ticket. This means that when you roll four different problems into one ticket, they are losing money. In fact, some call centers will request that if you have any separate issues, you must call back in. If this happens, do not get angry. Remember the technician is just following the process and procedures laid out for him by his management. Instead, write a letter or email to customer service complaining about this policy. Also remember, when you pack many issues into one call or ticket, the notes get very confusing very quickly, and follow up calls will take longer than needed since the technician who answers the later call will need to wade through pages of notes about related issues. The support technician's performance metrics suffer when you put them on hold to answer a call or chat with your boss or otherwise stray from resolving the problem. When the technician is worried about their boss coming over to find out why they have been on the phone for so long, they are not thinking about helping you anymore, they are thinking about ways to get you off the phone.

Personally, I have been in situations where I was supporting a NAS device over the phone, troubleshooting a frame relay connection on the computer, clearing and investigating SNMP alerts, helping other technicians via IM, emailing customer service to have an RMA filled, and writing documentation regarding tape drives simultaneously, while being the only technician on staff and not having had a bathroom break in nearly three hours. The technician is doing more than talking to you on the phone, so try to be respectful of their time.

Know the lingo

Call centers have a unique language all to themselves. Knowing this jargon helps significantly in resolving problems and smoothing the call. It is also extremely helpful to know the "military alphabet" for spelling out words.

Escalation: Referring the call higher up the chain. A call can be escalated on a technical level (involving a more experienced technician for a hard to solve problem) or the business level (involving a manager or supervisor in order to request something outside of the process or contract).

T&M/Time and Material: When service outside of the contractual obligations is requested, it is a "T&M Case," meaning that the customer is responsible for paying any labor or parts charges involved in the resolution. If you are told that a service request is under a T&M basis, find out what the T&M charges will be before authorizing service. T&M charges can be surprisingly high, and there is often a minimum charge as to the number of labor hours, regardless of how long the repair actually takes.

RMA: Return of Material Authorization, the process (as well as the ID number) of getting a replacement part sent.

Advance RMA: Most RMA processes will not send a replacement part until they receive the defective part. An advanced RMA allows the company to ship the replacement part before receiving the defective part. This often involves putting a hold on a credit card for the cost of the part for a period of time which will be charged to the card if the replacement part is not received within a certain amount of time. Make sure that you clearly understand the exact terms of an advance RMA before agreeing to it. If you require a replacement part immediately, ask if an advance RMA is possible, even if the technician does not tell you about the option for it. Many companies have an advance RMA option that they do not widely announce.

Supervisor: Be very careful when requesting to speak to a "manager." A technician may be truthful in stating that a "manager" is not available, meanwhile a "supervisor" is sitting right next to them. When escalating a call for business reasons or to lodge a complaint, request a "supervisor," not a manager.

Technical Lead: A support technician within that level or support with more knowledge and experience than the average technician, but not at the next level of support. If you have requested an escalation for technical reasons and are told that they will call you back or they are unavailable, try requesting to speak to a "technical lead." They may not be quite as knowledgeable as the next full level of support, but they may know the answer to your problem.

Level X: Most call centers have three tiers of technicians: Level 1, Level 2, and Level 3. Level 1 technicians typically must stick 100% to a script, and often (but not always) do not know anything outside of their support database of common problems. The Level 2 technicians are much more knowledgeable, and they know things that are not in the Level 1 database. Sometimes Level 2 support is needed for particularly difficult or potentially damaging procedures. A call is escalated to Level 2 either because Level 1 is unable to resolve the problem at all, or because the procedure takes some time to perform and Level 1 needs to be available to answer immediate calls. Level 2 escalations often have a longer hold time than the initial call to Level 1, and frequently Level 2 will return your call instead of having you wait on hold for them. The Level 3 technicians are usually the highest level of support. They nearly always have direct contact with the engineering team or actually are members of the engineering team. If Level 3 cannot resolve your problem, chances are you will be getting a refund or full unit replacement. Because Level 3 problems are often "one off" issues or reveal an entirely new problem that has never been encountered before, it can sometimes take days or weeks to resolve a ticket that has been escalated to Level 3. My experience has been that Level 1 tries to cap a call at 10 ? 15 minutes, Level 2 tries not to hold onto a ticket for more than a few days, and Level 3 can take days or weeks to resolve a problem. Remember, the higher you go, the more difficult the problem is to solve. Some upscale contracts allow you to request immediate escalation to Level 2, or immediately direct your call to Level 2. Depending upon your needs and ability to solve minor problems yourself, this contract option may be perfect for you.

Technician Dispatch/Technician Roll: Physically sending a technician to resolve a problem.

Vendor Meet: Some problems (particularly networking issues) require a technician dispatch from more than one company. When this is needed, a "technician dispatch" becomes a "vendor meet" and the different companies arrange the arrival time of their technicians so that they all arrive at the location of the problem simultaneously. A vendor meet typically eliminates any response requirements, because each company has a different SLA with the customer. For example, if Vendor A's SLA is a 4 hour response time, and Vendor B's SLA is an 8 hour response time, Vendor A cannot be held to their 4 hour response time if Vendor B cannot arrange the vendor meet within 4 hours. A vendor meet though often carries its own SLA, where a vendor that is late for the vendor meet is penalized.

Response time: The amount of time that a technician has to arrive on site. Some contracts measure response time from the moment the initial call is placed, other contracts measure response time from the time at which it is determined that onsite service is required.

X by Y by Z service: This is a shorthand way of describing SLA obligations. "X" is the number of hours that service is provided (24 meaning all day, 8 or 9 meaning "9 to 5" or business hours only), "Y" is the number of days (5 for weekdays only, 7 for all week long), and "Z" is the response time. A "9 by 5 by 4" contract means that the response time is four hours from the time of call in during business hours in the week. A 7 day contract typically requires service on holidays, while a 5 day contract typically does not count holidays as days of possible service.

Military alphabet/phonetic alphabet: Alpha, Bravo, Charlie, Delta, Echo, Foxtrot, Golf, Hotel, India, Juliet, Kilo, Lima, Mike, November, Oscar, Papa, Quebec, Romeo, Sierra, Tango, Uniform, Victor, Whiskey, X-Ray, Yankee, Zulu

Being transferred

All too often, your call needs to be transferred to someone else. A common problem is that a transferred call gets dropped. Especially during a call that has been less than perfect, it is tempting to think that the technician dropped you entirely. This does indeed happen on occasion, but it could be any number of problems. Sometimes the person on the receiving end drops the call when they pick up. A frequent issue in a call center is that a call is in the queue, and a technician goes to hang up a call that they are on. Just as they push the button to drop the call that they are on, the customer drops off and their phone automatically picks up the next call. As a result, instead of dropping the finished call, the drop the new call. Although this can be very frustrating, particularly if there is a long hold time, this can be a less frustrating situation if you get the direct contact information for where you are being transferred to before the transfer attempt is made. Never assume that you were dropped on purpose, no matter how bad the call may have been. For the years that I worked at call centers, I only met a small number of technicians that would deliberately drop calls. Since the phone system would record how the call was ended and by whom, after more than one or two customer complaints, they were terminated. If your call is dropped more than once (especially if it is by the same technician), request to speak with a supervisor immediately; you may have hit upon the bad apple in the bunch.

Ending the call

At the end of the call, do not forget to thank your support technician. If you were particularly impressed or disappointed with the level of service you received, request to speak to a supervisor and make your thoughts known, or send an email to customer service. Kudos to technicians is particularly well received, as it is rare that a customer lets the company know of a job well done. Appropriate compliments on service to a supervisor have a funny way of percolating through a call center, helping to ensure top-flight service in the future. If you are given a ticket number, do not forget to write it down, even if the problem is resolved. It is always easier to refer back to a previous ticket for a recurrent problem than to try to describe something that happened weeks or months ago, if the technician needs a full history of the problem.

Following up

Some tickets, particularly chronic problems, may require follow up. If the technician said you will get a call back, and you have not received one, do not get mad or panic. They may be under unusually high levels of calls that day. Simply call back and let them know that you were expecting a callback and did not receive it yet. If you repeatedly do not get a call back by the promised time, escalate the call to a supervisor. Repeated failure to call back is unacceptable, regardless of contractual obligations or reasons for the failure. If the technician provided you with an email address to send log files or other troubleshooting information, do not treat this as your direct pipeline to that technician! That technician may no longer be on duty, or may be unable to accept support requests via email. Try to make follow up calls through the same path (online chat, email, phone, etc.) as the initial contact; some companies separate different groups of technicians that may not be as familiar with your problem, or may not even have access to each other's systems. If a ticket has been referred or escalated to another group, find out how to contact that group directly if possible. Following up with Level 1 for a call that has been escalated to Level 3 or talking to Level 2 to get the status of an RMA just wastes time.

I hope these tips help you to get the best possible support from a cell center! My next post will provide advice for call center technicians looking to provide the best possible support for customers.

J.Ja

In April, I touched on the topic of "Geeks and Communications Skills." Reading through some of David Berlind's (over at ZDNet) recent blogs about poor customer service via call centers, I am following up my original post with this series. This series refers specifically to communicating with call centers, a potentially unpleasant task that most IT professionals need to deal with on a regular basis. The previous post discussed how to get maximum results from a call into the support hotline, and this post in the series will show how to provide quality phone support.

Call centers are an entry point into the IT industry for many people. Although my time at a call center occurred well after I had entered IT, it was still a great experience for me. Some of the most rewarding jobs I have had over the years involved direct customer contact for a full eight hours a day. However, delivering quality customer service, particularly trying to assist a user with technical problems over a phone, is a difficult art to master. These tips should help the call center employer (or anyone trying to assist a customer over the phone) to deliver the best help possible.

Answer the phone properly

A proper phone greeting should include a greeting, the name of the company, the precise department that the customer has reached, and the support person's name. It should be as sincere as possible. Treat every call as the first call of the day, and remember that no matter how difficult the last call was, it has nothing to do with the current call. The standard greeting I have always used was, "Hello, this is XYZ Technical Support. My name is Justin, how may I help you?" Although the script for that company suggested that I ask for the serial number first, I discovered (after much experimentation) that my greeting let the user know that I was there to help them, not to tie them up with process. After the user gave me a brief explanation of their problem, I would then request the serial number. If your company allows you a bit of freedom with your greeting, try this out. Your users will tend to be more pleasant to deal with if you greet them like this.

Empathize with the user

When a user calls technical support, they frequent are overcome by a feeling of powerlessness. They may be embarrassed that they cannot solve the problem on their own, or that their job rests upon a successful resolution of the problem. I have literally heard grown men break out into sobbing tears on a support call. To compensate, some users will attempt to direct the call, or refuse to follow directions. Other users try to pump up their egos or impress you with their wisdom. I have had callers repeatedly brag about their Cisco or Microsoft (or whoever) certification; meanwhile their problem is extremely basic. While it may be tempting to ask them "if you are so smart, why are you calling me?" do not do it. You have the information needed and the process established to help the user, and it is up to you to direct the call. You need to quickly establish this with the user by asking pertinent questions up front. Once the user sees that the questions you are asking will solve the problem, they are happy to follow along. It is very rare to have a caller refuse help once you have demonstrated that you can actually help them. But always keep in mind that the user feels helpless. Do not embarrass them or make them feel little, but show in a kind and gentle way that you are there to help, but only if they let you help.

Respect the customer

Keep in mind that by the time the user calls you, they have probably been trying to solve the problem themselves. They made be under the gun with a tight deadline or a mission critical problem. As a result, users often sound harried, tense, or may be terse or short with you. They may even take out their frustrations with your company, the product, or your co-workers out on you, and act as if you are the problem. Let it slide. The last thing you need is to be in an argument or abusive to a customer because they are having a bad day. Your job is to put a smile back on their face and resolve their problem, not to dictate to them a lesson in manners.

Of course, if a customer is extremely rude or abusive, such as using foul language directed at you personally, you have the right to politely request that they calm down. I have found it helpful to remind them that I am not the unit, the company, or the other support technician who was not helpful. One example of an effective phrase to do this is, "Sir, I understand that you are frustrated and angry right now. But I am here to help you, and it is difficult for me to do that if you are abusive to me. If you would like, I can transfer you to my supervisor who may be able to help you." By doing this, you have put the ball in the customer's lap. They can either calm down, or they can speak to your supervisor while you help the next customer. But you cannot assist the user if they insist on insulting you.

Another aspect of respect is to learn the customer's preferred name, to use it frequently, and to pronounce it correctly. When the customer gives you their name, always refer to them as "Mr. [Last Name]" or "Ms. [Last Name]" unless they request that you use their first name. Never refer to a woman as "Miss" or "Missus" unless you distinctly hear them say that; "Ms." Is the appropriate prefix for woman of unknown marital status. "Sir" and "Ma'am" also go a long way towards being polite and respectful. If you have accidentally confused a "Mister" with a "Ms." or vice versa (it happens quite frequently, especially for men with high voice or women with low voices, and a gender-ambiguous first name) and they correct you, simply apologize and make sure that you do not repeat the mistake. Using the customer's name is important on many levels. It shows that they you are treating them as an individual, not as a nameless, faceless voice emanating from a telephone. It also lets them know that you are paying close attention to them. Be wary though, if you mispronounce the customer's name, you are letting them know that you are not paying attention. If they have a difficult name, repeat the name and ask them if you have the pronunciation correction ("Mr. Wick-cow-sky, is that correct?"). My experience has been that customers really like this level of personalized treatment.

No matter what happens, never, ever raise your voice to a customer or use anything less than polite, respectful language. If the customer is in a loud environment such as a server room, there is a way to your raise voice to be heard and still sound pleasant, and a way to sound like a grouch when raising your voice. Learn the former. If the call is getting "too hot for comfort," it may be time to involve your supervisor, or possibly place the user on hold for a few (very brief) moments to take a deep breath before getting back onto the call. For particularly angry customers, I found it helpful to have my supervisor jack into my phone to monitor the call on the spot, or be by my side, so that they were available if needed. This also allowed my supervisor to directly hear what was going on, so that if a complaint or a problem arose, it would not be my word against the customer's.

No matter what the caller sounds like, never, ever mention their accent. I cannot stress this enough. While I do not advocate lying to a customer, "Sir, could you please speak a little more slowly? It is very loud in here at the moment," will be taken much better than, "Sir, I have a hard time understanding your accent. Could you please speak slower?" Many people may not speak your language natively, or may be from an area with a different dialect of your language. If one is available, ask the user if they would prefer for you to get a translator on the line. Many people who do not speak a language natively are very self conscious about their accent. To bring it up is impolite at best, and quite offensive at worst. Just about the worst mistake  ever made regarding accents was offering to get a translator on the phone for someone from Scotland. I simply could not understand the Scottish accent at all, and I thought that maybe Scots had a regional language (like Wales). Boy was I wrong! The only thing that saved me on that call was that it was an internal call, and it was still rather difficult to smooth it over after that.

Learn the customer's lingo and use it

Each call center has its own subsection of the English language, a combination of product-specific words and phrases as well as regional sayings. However, your customers may be from all over the country or even from all over the world. They are not steeped in the culture of your company either. So if the user calls the device in question a "doodad" and your literature calls it a "widget," just call it a "doodad." Similarly, if your user says something along the lines of "power cycle," say "power cycle" not "reboot." Adapting to the user's language allows you to communicate more effectively. If you work at a call center long enough, you pick up bits and pieces of different regional phrases. Feel free to use these, as long as you do not sound like a pretender or like you are mocking the user. I worked in one call center where we dealt with a large number of people in Australia. One of my co-workers would do his best version of "G'Day mate!" in an exaggerated Queensland accent (think Paul Hogan). Not only did he sound ridiculous to begin with, but the callers were typically in New South Wales and had an entirely different accent. Most of them did not find it particularly funny; they thought he was making fun of them.

Pick up on the user's mood and match it

If your customer is laughing and joking, it is OK to be a bit lighthearted. Do not take it too far of course, of make any type of even potentially offensive jokes, of course. On the other hand, if the user is being a bit down, or is upset, or is all business, then you should be 100% professional. Someone with a lot of pressure on them to solve a problem will treat your lightheartedness as you not taking the problem seriously. Even if the call is in a moment of "down time" like during a unit reboot, filling the silence with small talk is not appreciated under those circumstances. The happy shiny customer, on the other hand, will most likely be happy to exchange pleasantries like "how is the weather?" or "I have never been there, what are some good places to visit if I ever get out there?" But remember to always be professional.

Be sincere, honest, and upbeat

Whenever you open your mouth, make sure that your heart agrees with the words you are saying. "I am having a great day!" is an obvious lie when you grumble it, and the customer does not appreciate that. If you are having a lousy day, there are ways to be honest yet professional. "Today has had a lot of challenges, but I learned some great things," is a lot better than, "I am looking forwards to the end of the shift." On that note, never mention how long you have been on duty or if you are near the end of your shift. The last thing you ever want is to give the caller the impression that you are looking to get them off of the phone so that you can go home or take a break. The user is calling in to get help with their problems, not to hear about yours. They do not want to be talking to someone with a bad attitude either.

Always tell the truth to your customers. They have an uncanny ability to spot a lie, even over a phone. If you make a mistake, immediately correct it as soon as you know better, or admit to it if the customer catches it first. Along those lines, if a co-worker has given incorrect information to a customer (for whatever reason), be careful with how you correct the problem. "Well, I am sorry sir, but John is new and really is clueless," is not a very good way to handle this. "John may have made a mistake or miscommunicated what he really meant sir, I will let him know and make sure that he gets the correct information," is a much better response.

"This call may be monitored or recorded for quality and training purposes."

Unless you are working for a very small company, chances are that there is always the possibility that any given call may be monitored in real time or recorded. Treat every call as if your supervisor is standing next to you. That means always be polite, do not lie to your customer, and delivery the best support possible, on every single call.

Make sure that your ticket notes are accurate and appropriately worded

If the customer calls back, you want to make sure that your co-workers are able to quickly read the notes in the ticket and understand exactly what happened. If a questionable situation arose, make sure that you document why you made the decision that you did. "As per paragraph 1.X.a of the system manual" or "as per team lead Joe" are wonderful ways of covering yourself if a problem comes up later. It also lets the next technician know where you got the information from. On the other hand, the ticket notes should not be too verbose. When someone has to read four paragraphs of "he said, she said" just to find out that the network connection is down and a technician is being dispatched, you have been too wordy. When writing your ticket notes, ask yourself, "if this user called back, what information would I need to know to pick up where we left off?" The user who calls back will appreciate that you co-workers will be able to quickly and accurately understand the exact situation, instead of being put on hold forever.

Always give your users a wide range of options, and explain them clearly and simply

All too often, a major source of customer complaints is that if they had known about an option, they would have taken it. A common example is the advance RMA. Most users do not know about it, and accept the basic RMA policy. If they later find out that there was an advance RMA option, they may get upset because they would have preferred to use it. Always give your users the wide spectrum of options, but be sure to explain what they are as well. For example, "we also offer an advance RMA" is not nearly as helpful as, "if you require the replacement part sooner, we have an option for that as well." By giving your users options, they feel as if they are in control. When the user feels as if they were a crucial part of the success of the call, they are happy with the call.

Ending the call

Always try to end the call on a positive note, even if their problem cannot be solved. Remind the customer of your name, and thank them for calling. Like the greeting, it can be difficult to sound sincere when thanking you for calling in, but also remember that callers can detect insincerity quite easily. Even if the call was "challenging" (a euphemism for, "the caller insulted my religion and my mother, told me I was born out of wedlock, and ended up taking a blowtorch to the unit"), be polite until the very end. Especially when the user is upset or the problem could not be solved, any sounds or hints of displeasure on your end (even a loud sigh) will be interpreted as displeasure with them and they will react as if you are angry with them. Your job is to solve problems, not create them. Any negativity on your part will ruin the call. At the end of the call, always ask the user if they have any additional question or concerns, or if there is anything else that you may help them with. Never give the user the impression that you are trying to push them off the phone.

I hope that these guidelines help you deliver great customer service to your customers!

J.Ja

I just finished reading a very interesting article ("The Lumi?re Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users") from Microsoft Research. Lumi?re was the name of the project that developed mathematical models of assessing the needs and goals of software users in real time in order to present users with accurate, relevant assistance as they worked. Does this sound like something that you think users would be interested in? I sure do!

However, after reading the Lumi?re paper, I may be changing my mind a bit. To understand why, we need to take a brief trip through time.

In 1993, the Lumi?re project began, and was initially shown to another team at Microsoft in 1994 (to tell you what team ruins the surprise!). That other team liked what they saw, and built a special version of one of their products that allowed the Lumi?re group to hook into it and begin showing how the Lumi?re techniques could improve the product.

During this period of time, teams of experts (primarily psychologists, from what I can tell) were set up to observe users in a sort of "Chinese Room" type experiment. Users were told that they were using an experimental system to help them work. The experts observed the users, but were not told what the users were trying to do. The experts tried to identify the tasks that the users were accomplishing, and then provide the users with relevant assistance. The way the experiment was set up, users were not sure if it was a computer or humans presenting the help, because they were isolated from the experts. This was done in order to understand what types of tasks were easily identified and for which of those tasks assistance would be helpful to the user.

Lumi?re users Bayesian networks of probabilities (commonly used in spam filters) to identify what the user is trying to do and what they need help with based upon their past tasks, takes into account the users' demonstrated abilities, levels of expertise, and historical "pain points." After determining the users' needs and goals (using a set of probabilities that age and become less relevant over time) through a variety of events (and non-events, such as idling the mouse over a menu option for a period of time), Lumi?re makes suggestions based upon what the user is most likely stuck on, or helps give them a shortcut to accomplish their goals faster, even if they already know what they are doing.

Does this sound familiar yet? It should. Let us finish the history of Lumi?re though?

The application development team was so impressed with the results of the Lumi?re/application integration, that Lumi?re was adapted to that particular application (and related applications) in a less advanced form. I am not sure if the full Lumi?re project was stripped down to meet shipping deadlines, reduce the system resource needs, or meet the needs of a broad user base, but much of its advanced functionality (including tracking and profiling users to determine individual skill levels) did not make it to the final product. Lumi?re powered what was probably the most hated feature in software history.

The application? Microsoft Office 97. Lumi?re's contribution? Clippy.

Maybe if Lumi?re was shipped in its entirety, and users knew that it could be "trained" they would have been willing to put up with it long enough to have it actually be helpful. I do not know, and trying to re-predict the past based on "what-if scenarios" and "just-so stories" is not very helpful. What is known is that Clippy was so universally hated that it is probably the only feature (definitely the only high-profile feature) that Microsoft has ever removed from a product that I can recall. In fact, when Clippy was left out of Office XP (aka Office 2002), it was actually a selling point in advertisements!

Before I read the Lumi?re paper, I had assumed that Clippy was some second-rate hack, or maybe not much work had gone into it. Lumi?re had been worked on for four years by a wide variety of psychologists, usability experts, statisticians, and a ton of other highly qualified and experienced experts. It was developed by the best of the best. From what I can tell, users in the Lumi?re testing group must have been very impressed by it for it to end up in the Microsoft Office Suite. But real-world users despised it. It was annoying. It impeded work flow. It was grossly inaccurate.

Personally, I wanted to love Clippy. I tried a number of times to leave him on. I always felt sad, after the first use of an Office application after installation, to see him blink his eyes and commit feature suicide when I turned him off. Then again, I am also a touch attached by the little puppy in Windows file search. I tried letting Clippy help me do my job. Clippy always made me miserable, and I think he made 99% of users miserable.

And this is why I am reconsidering my stance towards applications that anticipate users' needs. If all of the effort that Microsoft put into Lumi?re could not produce a system that was usable and workable in the real world, it is pretty hard to imagine being able to do it well at all. Granted, Microsoft Office is an incredibly complex suite of applications that get used for a wide variety of tasks, many of which are outside their intended purposes (Excel as a database system, Access for client/server applications, etc.). But I can imagine the difficulty and frustration of using a system which attempts to adjust itself to the users' needs on the fly. If it is less than 100% accurate, users will get mad and frustrated, and their lives will become painful, instead of the anticipation system making them more productive. That is the real lesson to be learned from Lumi?re and Clippy. User anticipation is a great idea, but only if it is perfect in its execution.

J.Ja

Longtime readers of mine will know that I have a lot of love and respect for some of the less popular languages and programming models like Perl and functional programming (in the order of "love" and "respect"). With that in mind, I am strongly encouraging Microsoft to put as many resources as possible into getting the .Net platform ported to *Nix immediately. Why? Because .Net is quickly emerging as a powerhouse for less mainstream languages! With the impending release of IronPython and the recent release of F#, .Net is establishing major credibility in areas traditionally dominated by *Nix.

Many of the lesser known languages suffered from a lack of proper tools such as IDEs and debuggers. Debugging Perl, for example, is very similar to the way I debugged BASIC in 1991: with statements to dump variable and program state to the screen or a log file. All too many languages are in this boat. Visual Studio, on the other hand, is a fantastic programming environment.

In addition to a lack of tools, many of these less used languages suffer from a shortage of quality libraries. For too long, Visual Basic, C#, and Java were easier and quicker to write code in, not based on any particular strength of the language itself, but on the huge base of quality libraries available. Indeed, the .Net Framework is an incredibly comprehensive library, encompassing most of the Windows API in a rational, consistent fashion. Meanwhile, other languages lag behind with the libraries.

As more and more languages move to the .Net Framework, it is increasingly possible and important to be able to write code in more than one language. A function that may be 100 lines of code in VB.Net may only be 10 lines in Perl. The 4,000 line monstrosity in C# can become 100 lines in a functional language such as O'Caml. With IronPython and F# (an O'Caml derivative) now in a usable state on the .Net Framework, the game is changing. Now, you can use VB.Net to handle the interface, catch events, and handle database operations which it is great for, pass data back and forth to C# code to deal with the multithreading aspects of the code, and have F# perform some intense analysis of the data. All at nearly the same speed of natively compiled code. Indeed, I have read of F# spanking C# in terms of speed on some projects!

And the only conclusion I can come to is that Microsoft needs to get their act together and get the .Net Framework ported to *Nix as soon as possible. If possible, they should put a GUI abstraction layer in it, to interface with X with no changes to code needed. At the very least, as long as one is not using concepts specific to Windows and its associated technologies (Active Directory, NTFS file permissions, some aspects of networking, and so on), software should run on a *Nix platform with no changes needed. For example, if a piece of code receives a request from a Web server, connects to a database, works some magic, and outputs HTML, that should work flawlessly.

I recognize that the Mono project exists, but it is not perfect, from a number of standpoints. The most obvious one is that Microsoft has never been great about sharing the nitty-gritty details of their technology with others, particularly when it gives them an edge. Another problem is that Mono is steered by Novell, who is on less than friendly terms with Microsoft. At the very least, the Mono community needs to be working closely with Microsoft to make sure that they get things right. From what I have read and heard, porting applications from the Microsoft .Net Framework to Mono can involve a good deal of work. I believe that unless someone is using Windows-specific code, the transition should be minimal.

At this point in the game, the only nail .Net is currently unable to put into Java's coffin (as well as become the de facto platform for many languages) is cross-platform support. If Microsoft can get .Net running on Linux, it should be fairly easy to then get it to Solaris and the BSDs, and then finally to MacOSX. Every article I read about a new language being ported to the .Net Framework, particularly dynamic languages, say the same thing: the .Net CLR is an excellent system to be basing languages on. People set out to show the shortcomings of the .Net CLR, and end up becoming its biggest supporters. Ruby is quickly moving to .Net. IronPython is nearly out. F# is already released. ActiveState's Perl is now able to work with the .Net Framework, although not as seamlessly as I would like. Within a year or so, every major language, most of the mid-level languages (Perl, Ruby, Python, etc.), and many of the minor languages (O'Caml, COBOL, and so on) will be fully ported to the .Net Framework. That means that languages that used to be relegated to special-purpose uses are suddenly viable options for mainstream developers. It also means that it is now possible to perform much special-purpose programming on Windows, more easily than ever, with those special-purpose languages. This brings Windows well into the realm of science, statistics, GIS, and other niche markets.

And the only thing missing is top-shelf *Nix support. Microsoft, are you listening?

J.Ja

Justin Rattner over at ZDNet recently posted a blog about the end of applications. Mr. Rattner is right, yet incredibly wrong. He is absolutely right in that "all of the interesting applications have been written." In fact, I am on the record as stating that it is virtually impossible to create a single new application that could not be done (however inefficiently) with other means and tools.

That being said, Mr. Rattner's analysis of where the future lies is misguided at best. He believes that the future lies in changes to reduced operating costs and increasing manageability. I am all for reducing operating costs and increased manageability. But usability is the most important missing piece. Computers are completely unusable. Even cell phones are unusable. Most cell phone consumers pay big bucks for phone where the only features they use are the same features they were using 10 years ago, making phone calls. Why? Because the interface and OS for cell phones is atrocious.

Mr. Rattner's viewpoint is that of a hardware person. This is understandable; he is the CTO of Intel (amongst other technical titles). Hardware people have never understood end users terribly well. Not that application developers do such a great job of it either, but at least they make the product that the customer is actually using. Even worse, Intel specifically understands neither application development nor end users. They make general purpose CPUs designed to disappear in the user's life. Intel has never once done anything to help me as a developer. When Intel started pushing HyperThreading and dual core CPUs, did they put anything out to help tech the average programmer how to write programs to perform well on single core and dual core CPUs? No. Did they put any effort into helping Open Source Software projects optimize their performance to take advantage of the new hardware? No. Has Intel ever put any effort at all into writing code that end users see? Nothing except for a few drivers and system tray applications for their NICs and video chipsets. So why would I think that anyone at Intel has any understanding of the developers, the development process, or what end users want?

If you are an IT manager, would you prefer to go through yet another round of hardware purchasing to consolidate a few servers and reduce the power bill... or would you prefer to initiate a usability improvement session on your existing software? Most IT managers will choose to buy the hardware. I do not care how much hardware or how powerful it is. If the users cannot use it, it is wasted money. It's like owning a Corvette but driving the Long Island Expressway to work, it is wasted power. On the other hand, increased usability, by making interface (and some underlying logic changes) yields rich results with the hardware and applications we have today. For example, a website that performs a round of usability improvements can expect to see sales increase by a dramatic amount .

We have been playing Intel's game for the last 20 years, and it has not helped the end users. IT does not exist for the sake of enriching the wallets of hardware vendors, application developers, or consultants. Let me rephrase that. IT should not exist for those reasons. But it does. As a result, users hate their computers. Companies spend a fortune on IT, buying hardware to compensate for sloppy programmers and software that is "the version you have been waiting for your whole life" that still does not address the underlying flaws in all of the previous versions, but has a few new featured buried in a menu someone, and it now lets you skin the interface.

I have been a computer user for the last 20 years. I have owned a personal computer for at least 15 of those years. Yet, at the end of the day, my usage habits have been frozen since about 1994. Here is what I use a computer for (in no particular order):
* Checking email
* Chat/instant messaging
* Consuming content, primarily text, but sometimes audio/visual
* Creating content (95% text, 5% visual)
* Programming
* Playing video games
* SSH'ing to *Nix servers

I was able to do all of this, in one form or another, on a 486. I was able to do most of it on a 286. Indeed, my 286 with that 2400 baud modem was actually able to feed me remote text (my primary target of consumption) about as fast as my current PC does over a broadband connection. Why? Because it was not dealing with a Web server or having to parse HTML, and then fire up a JavaScript interpreter just to make a menu pop up!

And sad to say, the actual usability of the computer has gotten worse, not better.

I postulate that while achieving a minimum level of proficiency with a computer is now much easier, the difficulty and time needed to become a power user has increased substantially. For example, learning the basics of WordPerfect 5.1 was a bear. But once you had that little template laid over your F keys, you could do anything you wanted to a document with a few short keystrokes, without ever taking your hand off of the keyboard. Compare this to Word 2003. Word does not have many word processing features that WP5.1 did not have. Its extra features are on the periphery, in the form of things like Smart Tags, the "Research" system, and so on. Yet, doing the same tasks that I did in WP5.1 in fractions of a second now take many seconds! Take my hands off the keyboard, reach for the mouse, highlight the text, go to the menu (because there are more items than keyboard combinations), scan through the endless cascading menus looking for the tasks I want to perform (instead of glancing at the template laid over the F keys), sometimes taking a minute, then walking through some dialog or wizard. All to create the exact same table which took under 10 keystrokes and five seconds in WP5.1.

If anyone considers this "progress" they are mistaken.

Mr. Rattner is correct in that the vast majority of "improvements" really fall under the eye candy category. RSS is eye candy. HTML is eye candy. AJAX and "Web 2.0" are irrelevant eye candy. Granted, my definition of "eye candy" is rather broad; much of what these technologies deliver is added convenience and even the occasional efficiency to the end user. But the value of the content or the application itself is not increased by much at all. None of these technologies truly improve the human/computer interaction. There is nothing that these technologies deliver that could not be done 20 years ago. But the hardware vendors love these technologies. Want to sell faster CPUs? Get programmers to ditch flat files for tabular data in favor of XML. Want to sell memory? Get the users onto AJAX'ed systems ("JavaScript is the COBOL of the Web" to quote Roger Ramjet from ZDNet TalkBacks) and off of natively compiled code. Pushing horribly cluttered interfaces is a great way to get people to buy gigantic monitors and the video cards that go along with those higher resolutions. Convincing people that it somehow makes more sense to watch a movie on a 19" monitor instead of their 35" TV is another smart move.

So while I think that Mr. Rattner's starting premise is right, I think that his analysis of the situation in general is not correct.

J.Ja

Oracle is the Terrell Owens of software: you absolutely despise playing with it, but its performance is almost good enough to make up for its unforgivable manners. Almost. Today I had the misfortune of having to perform some development work with an Oracle database. I had not touched the Oracle client installation on my desktop machine in months, except to update the Oracle .Net tools a few weeks ago. I updated them, but had not used them. When I tried to make a connection to my development database, the drivers were just giving me a big "New York Greeting." I never did resolve the problem; I kludged together a work around. I ended up just using a different system on my client's VPN and working with their development database. I put two hours into resolving the problem, and simply did not have anymore time to waste.

Over the last year or so, I have spent well over 40 hours wrestling with Oracle. That is a full 2% of my yearly productivity devoted to nothing but trying to get their "tools" to stop giving me grief. I can easily chalk up at least another 20 hours over the last year trying to find error in SQL statements that Oracle does not help me find. "Missing right parenthesis" is not very helpful when one is doing a JOIN on 106 subqueries. MySQL is nice enough to show me exactly where the problem is in the error message.

And then there is the sewer known as "sqlldr.exe." Another complete waste of time to deal with. Again, comparing to MySQL, I have to write a script file to do what MySQL can do in one "LOAD DATA INFILE?" statement. Even the command line for sqlldr is complicated. For whatever reason, my current working directory needs to be the data directory, or else the files do not seem to be found.

Oracle Enterprise Manager is pretty foul too. It has menu and toolbar items that tell me to go use "Oracle Enterprise Manager Console," which, despite its name, is not the same program, but a Web based app. The software itself barely works. It leaves a Java console window in the background, which is constantly spitting out various uncaught exceptions. The software has a positively miserable interface. Its worst habit, though, is dropping the first character typed into any text box. I have never seen that kind of behavior in any other application.

Even the Oracle client is a shrew. Each time you install an Oracle product, it tramples all over the file permissions on the client, rendering it unusable. Due to a quirk in Windows' file permission check system, after correcting the problem you need to reboot your PC. Oracle is the only database that requires a monstrosity of a client; it is like Novell Netware circa Windows 3.1. Do not even get me started on the idiotic tnsnames.ora file contraption. Talk about a manageability nightmare! Every other database out there just needs to install a small ODBC driver (or JDBC driver in a convenient JAR file, for Java developers), make an ODBC connection, a poof, you are connected. Not Oracle. It needs to turn your computer into a mess, and you still need to create the ODBC connections anyways.

Dealing with the Oracle corporation itself is a David Lynch film at best. Their Web site makes it clear to me that they are not interested in helping me. The search system on their Web site is horribly broken which reflects quite poorly on their database products. Trying to find updated tools and drivers is a hassle. Not that they update their tools very often, Oracle has a poor track record at bug/security fixes. And again, when you do update the tools (using their awful Java installer, which takes two minutes just to start, and always gives me grief about where to install stuff), it destroys your existing setup, requiring a three hour trip down search engine lane to fix the problems.

I am not even sure if Oracle's performance is that great. Microsoft SQL Server, when properly configured, can be pretty fast, close to and occasionally exceeded Oracle's. MySQL is catching up very quickly, and PostgreSQL is no slouch either. And all of them are infinitely easier to deal with than Oracle. Oracle's dominance of the database market is proof positive that the people who make IT purchasing decisions are not the people who have to live with those decisions. Why is it that there are no MySQL, PostgreSQL, or SQL Server consultants billing out at $250/hour, but there are Oracle consultants doing so? What does that tell you about Oracle?

Oracle just does not get it. When your developers tell you that using Oracle over another database makes development take 10% longer, or your DBAs cost you 25% more, or management on users' desktops is 250% more difficult, or integration costs are $250,000 versus $10,000 for anyone else, eventually you are going to start to wonder if a 5% performance edge is worth it. I do not think it is. In its current state, I would not recommend Oracle to anyone. I simply fail to see its advantage, and I have yet to hear from a developer who likes dealing with it. The only people I have talked to who like Oracle are billing out at divorce lawyer rates to integrate and support it.

I would love to hear your Oracle horror stories, or any defense of them.

J.Ja

The company that I work for had a pressing need for some battery backup. We were adding new servers with more power draw, and our existing UPS just was not cutting it. One of the biggest challenges that we faced was a lack of 20 amp (or bigger) outlets. Our office is only equipped with 15 amp outlets, no one ever expected our needs to grow as quickly as they have been. As a result, we were limited to about 1.5 kVa worth of power.

Originally, due to cost considerations, we were going to stick with some low end, line interactive units (the Powerware 5120, believe). One of the reasons we suddenly needed so much power though was our new server, a dual Xeon 3 gHz system with 4 SATA II drives in it, rated at 700 watts. Although it was fairly obvious that the unit would not be drawing anywhere near 700 watts, 300 ? 350 seemed to be a reasonable estimate of the power draw of that server. Add to that the load of three other servers, and a single 1.5 kVa system would be quite near its maximum capacity, and well over the recommended capacity.

We decided to not go with the Powerware unit, and instead opt for a pair of GE GT rackmount units, each with an external battery pack for additional runtime. Here is why we opted for this particular unit:
* Its maximum load is 900 watts, while many competing units were rated at 700 watts
* 2U rackmount form factor for each unit, as well as each battery pack
* Dual conversion, as opposed to line interactive
* Able to daisy chain up to three external battery packs per UPS unit
* Plenty of outlets on the unit
* Outstanding price point, especially in comparison to other 1.5 kVa dual conversion units
The last point was the most important. Although the cost was significantly higher than any line interactive unit, it was less expensive (by at least 20%) than any competing product. For a number of reasons, I will not name the vendor or the price paid for the units.

The units arrived via freight. Initially, I was rather scared by them; the freight company listed four boxes at a total of 290 lbs. The prospect of manhandling 75 lb. units did not thrill me. When the boxes arrived, I was delighted to discover that they were not nearly as heavy as I had originally expected. Initial inspection of the units showed them to be extremely well constructed. The power cords were, by far, the most heavy duty 15 amp power cords that I have ever seen. Indeed, they were of a higher, heavier duty quality than on the APC SmartUPS 2200 that I have at home. Likewise, the connectors for the external battery packs were extremely heavy duty.

Installation and set up of the units was a breeze. The only sticking point was the cords for the external battery packs. They were pre-bent into a curve, and extremely resistant to bending. If I wanted to swap the position of the UPS and the battery (putting the battery above the unit, as opposed to under), I would have had an extremely difficult time. I also had some difficulty inserting one of the ends of the battery's power cord into one plug. A little bit of wiggling cured that, however. The plugs were secured with bolts, but the holes in the power cord's "ears" was slightly small. I was very happy to discover that both the UPS's and the batteries were not nearly as heavy as I expected. The handles on the front of the units made it easy to move the units, and picking them up without a helper was not a problem.

After installation, the units continued to delight. They are whisper quiet, and generate extremely little heat. While this is not a prime concern in a server rack buried in a server room, I believe that a tower version of this unit would be extremely pleasant to have next to a desk. Indeed, the units are quieter and generate less heat than my desktop PC. It was nice to deal with rackmount equipment that did not forgo the creature comforts. Visually, the units are quite attractive. Although the average systems administrator probably could not care less, some people do like a good looking unit. The units are a pleasant silver color, with a sporty red stripe across the front. The control panel is extremely bright and easy to read. The LEDs are spaced far apart, so light from one LED does not appear to light the other LEDs. The control panel has a general "step meter" that displays load while receiving external power, and shows battery charge while on battery. I would have preferred to see dual meters. In my experience, getting an idea of how much load can be reduced by shutting down non-critical systems first to extend run time for more important systems can be important.

I must admit though to being a bit disappointed in GE's "JUMPStart" software for the units. [Correction 8/23/2006: The software is named "JUMP Manager", not "JUMPStart"; I got the name confused with a Sun software package I was recently dealing with.] My first impression of it was less than favorable: it requested an installation path without spaces. It is a Java application that seems to run on both Windows and *Nix. I tested it on a Windows 2003 Enterprise Edition server. After installation, it was not courteous enough to make a group for itself in the Programs menu. It just dumped itself into the root of the menu. It also did not add itself to "Add/Remove Programs." It also did not add itself as a Windows service. When I did run it manually, it left a nasty Java console window open. The Java console was spitting out errors, but the software itself seemed to be running fine. This software, as far as I am concerned, is not ready for primetime for general UPS monitoring purposes. I would definitely recommend purchasing the add-on SNMP card, and monitoring the unit that way. I will be looking into that myself, and add it to my MOM configuration, along with scripts to shut down servers in the case of a low battery.

Overall, I think the GE GT unit is a fantastic item. It is available in 1 kVa ? 3 kVa capacities. The only problem that was beyond "one time nuisance" was the lack of good software for the serial connection, pretty much so mandating the purchase of the add-on SNMP card. You should factor this into your purchase price if you decide to buy this unit. The lightweight of the units and the battery packs lets me feel more comfortable that I will not develop a hernia when it comes time to replace the batteries.

J.Ja

This is the first of a multi-part series demonstrating multithreading techniques and performance characteristics in VB.Net.

Introduction

Multithreading is treated as a black art by many programmers, and for good reason. Even for an experienced programmer, knowing when to multithread an application, where to multithread an application, and how to multithread an application can be difficult to determine. Multithreading is definitely not a science, there are no hard and fast rules whatsoever to guide the programmer. In preparation for writing this series, I contacted Rick Brewster at Microsoft (who worked on the excellent Paint.net software) for advice. He quoted Rico Mariani at Microsoft: "measure, measure, measure." This is sound advice. This series of articles is not going to provide any pat advice, but will offer some suggestions, sample code, and demonstrations of various multithreading techniques as well as showing some performance characteristics, so that you can get an idea of your multithreading options.

Where We Were, Where We Are

The idea behind multithreading is not new, it goes back decades. However, up until very recently, with the advent of HyperThreading processors and then multi-core processors did multithreading become as critical to understand when writing applications as it suddenly is. Before these advances in hardware, unless you were writing applications for use of "big iron," chances were that the system you were coding for would probably not have more than one physical or logical CPU, unless you were writing applications for expensive SMP or parallel processing servers. The Windows world has been almost entirely single processor until recently, while only the high end Macintosh machines had multiple processors. The game has completely changed, particularly with Intel introducing the Core 2 CPUs (codenamed "Conroe"). With this line of CPUs and the guaranteed to follow AMD counterpunch, along with the previous generations of multi-core CPUs, even low end desktop PCs are coming out with multiple logical CPUs.

On a single core, single CPU computer, all multithreading is performed by the operating system with a trick called "time slicing." As a basic idea, time slicing essentially multiplexes requests to the CPU, interleaving them and giving a bigger portion of the pipeline to higher priority processes. While the system is effective and works, it is quite inefficient. The operating system needs to understand which threads go with which requests, and there is quite a bit of overhead associated with the technique. With the new systems entering (and already on) the market, the multithreading game has changed completely, allowing the operating system to dynamically allocate different threads to different logical CPUs based upon the threads' needs. For example, one long running thread can devour all of one core's resources, while the other core handles short lived, simple to process threads such as user input and mouse clicks. As a result, applications that make maximum use of the new architecture will be more responsive, run more smoothly, and provide a better experience to the user.

As an interesting sideline to the new architechtural changes, while the number of cores per physical CPU is increasing (Intel and AMD are currently producing dual core systems, and quad core systems are on their way!), the clock speed of the individual cores has remained stagnant and even gone done a bit in some cases. In other words, while the new CPUs can juggle multiple tasks better than before, single tasks pay a price. This makes it more crucial than ever before that applications be written to use multiple threads in the right places in the right way. Most desktop applications tend to not use multithreading, or if they do use multithreading, the main processing occurs on one thread, leaving the main application's thread free to provide status updates and process user input such as handling a "Cancel" button. We have all seen what a single threaded application looks like: the window goes grey, Task Manager reports that the process is not responding until it suddenly finishes and updates the screen, we see the spinning hourglass, and have no way of stopping it from cranking away without killing the whole process. Even worse, these applications frequently dominate the CPU, causing the entire system to run slow. A properly multithreaded application, particularly one on a multi-core or multi-CPU system does not inflict itself on the user like this.

To sum it all up, an application that is written to be properly multithreaded, all else being equal, will be preferred by a user over one that runs in single thread. And programmers who understand multithreading and have demonstrable experience in the techniques of multithreading will be in increasingly hot demand as the desktop market shifts to the new CPUs.

A Quick Primer to Multithreading Concepts

Multithreading seems to have a language all to itself, and it does! Many of the concepts involved in multithreading simply do not get discussed in traditional procedural or object oriented code, because the problems do not exist. If they are encountered, it is with working with a database, in terms of maintaining data integrity. Here is a brief primer to the ideas in the world of multithreading.

Physical CPU: An actual CPU or processor that fits into a socket on the motherboard. A computer with only one core per CPU will have the same number of physical and logical CPUs.

Logical CPU: A separate CPU pipeline. A HyperThreaded processor will appear to have two logical CPUs per core, and a multi-core processor will have a logical CPU for each core per processor. For example, a computer with two dual-core, HyperThreaded CPUs will appear to have eight logical CPUs (2 physical CPUs x 2 cores per CPU x 2 pipelines per core).

Atomic Operation: An operation in code which must only be executed by one thread at a time, typically to maintain data integrity. For example, the following code block will not work correctly if other threads try updated the variable intSharedVariable, necessitating making this block of code an atomic operation:

intThreadIterations = 0
while (intSharedVariable < intMaximumIterations && intThreadIterations <= 5) {
	# Do some work
	intSharedVariable++;
	intThreadIterations++;
}

Block: A thread (or process) is "blocking" when it is in such a state that all other threads must wait until it is finished to continue their work. Using the example for "Atomic Operation," any thread trying to access intSharedVariable will block until the atomic operation is completed.

Lock: A system for marking a restricting access to a particular variable or object to other threads. The other threads will block until the lock is released. Some locks pertain to both reading and writing (any thread accessing the object will block no matter what), and others are write locks, allowing other threads to read the data but will block when trying to change its value. It is extremely rare to find a read lock, allowing data to be written but blocking on read operations.

Mutex: A locking primitive used to ensure that only one thread at a time has access to a resource.

Semaphore: A counter that counts how many resources are being used at any given time. When the maximum is reached, calls to decrement the semaphore block until another thread releases the semaphore. Semaphores are used to limit the number of threads performing work or accessing a resource, such as in a thread pool.

Monitor: A method of controlling and synchronizing access to objects.

Thread Pool: The .Net thread pool dynamically manages the number of threads and their priority for processes that use it. Any number of threads may be requested to run from the thread pool, but the .Net Framework controls how many are active at any given moment.

A Brief Aside Regarding Virtual Machines

Because of the increasing prevalence of virtual machine (both the host/client type and the hypervisor variety), it is unreliable to rely upon a count of the number of physical processors in a machine. Any low level functions are unreliable at best. Instead, you should count on the OS and/or the .Net Framework to determine the number of processors that are apparent to the OS. The best way to do this within .Net is a call to System.Environment.ProcessorCount which returns the number of logical processors visible to the operating system. That is an important distinction, as a hypervisor or virtual machine system may make only one logical or physical processor accessible by the operating system running your application, while more logical or physical CPUs actually exist. As an example, if you have two dual core CPUs installed and the hypervisor only makes one core available to the OS running your application, but you base your thread usage on the number of logical cores at a low level (which would be four, not the one usable by your application and OS), you will be using four times as many threads as you meant to be using.

The Challenges

Writing applications that use multithreading are not very difficult, particularly if it is of the type where a single thread for processing runs separately from the main thread, to allow for progress bars and cancel buttons. In that case, the programmer has not strayed very far from having a single threaded application running on a multithreaded operating system. Other systems have processing of data that is in an N-dimensional format that does not require any atomic operations, so dividing the workload by logical CPU and processing them simultaneously is not too difficult; many multimedia applications are written in this manner. For example, a graphics program might divide an image into quarters on a system with four logical CPUs, and use four threads to simultaneously apply a filter to each quarter. Other applications like database or application servers will created and queue a thread for each operation, and perform them as possible in the order they were entered, carefully locking key data. Yet other applications will perform as much processing as possible, while network or file I/O processes on a separate thread, splitting the two processes as early as possible and joining them as late as possible to give the I/O the most possible time to finish on its own. And yet other applications need to perform a good number of atomic operations, and weigh the cost of each context switch and blocking against the benefit of each additional thread.

By far, the biggest difficulty with writing multithreaded applications is maintaining data integrity. It is tempting to lock as much code as possible within atomic operations or tie objects up with mutex's and just hope for the best. While that may have a lot of data integrity, the cost of the atomic operations and locking is so high that your application will probably run significantly slower than a single threaded application!

After the data integrity issue has been solved, performance is the next concern. A profiler such as the one included with Visual Studio 2005 is invaluable for this. By using the profiler on a single threaded piece of code, you can see where the bottlenecks are, and judge whether or not they are big enough to justify the cost of multithreading them. Once the application is multithreaded, the profiler helps you measure how many threads is best for your application.

Testing is another challenge. Due to the nature of a multithreaded application, stepping through the application is frequently unrealistic. The best way to test these types of applications is to prepare a battery of input/expected output scenarios and test against those. Code review, particular by peers, can pick up critical details that paper planning and even testing might not catch. Repeated tests are quite important, because depending on your threading technique, the threads may not always be processed in the same order, providing different results if there is a problem. Finally, having a wide range of CPU architectures to test on is extremely important if you are relying upon the number of processors to manage a thread pool or number of concurrent threads.

Building a Multithreaded Application

The next blog post in this series will work us through building the outline of an application designed to demonstrate various multithreading techniques and their performance characteristics. Stay tuned!

J.Ja

This is the second of a multi-part series demonstrating multithreading techniques and performance characteristics in VB.Net.

In the last post, I outlined some of the basics of multithreading. In today?s installment, we will be constructing the outline of an application that will show off some different multithreading tricks, techniques, and their performance characteristics. We will be using a modular design, so that we can replace the test functionality with code from other projects to see what multithreading works best for that code.

Without going into any details (it is a very simple Windows application), we put together a basic interface that looks like this:

The only thing in this basic application that is of interest is the status bar. In the status bar, we use a call to System.Environment.ProcessorCount to report the number of logical CPUs (the screenshot was taken on an AMD Sempron system).

Internally, we are putting the processing logic into an entirely separate class as a Shared function, so that we may easily create a command line version of the application, if desired. This class, ThreadTestUtilities also contains two Enumerations for the test types:

  Public Enum MultiThreadTestType
    SingleThread = 1
    DotNetThreadPool = 2
    OnePerLogicalCPU = 3
    OnePerPhysicalCPU = 4
    Arbitrary = 5
  End Enum

  Public Enum AtomicType
    NonAtomic = 1
    Mutex = 2
    Monitor = 3
    SyncLocking = 4
  End Enum

Although MultiThreadTestType contains the member OnePerPhysicalCPU, you will note that the screenshot does not show this option. The reason is that relying upon the physical CPU count is quite a bad idea, as explained in the previous post. But we put it into our Enumeration anyways, in case we decide later on to allow testing on that basis.

The Shared function for actually running the tests looks like this for the moment:

  Public Shared Function RunTest(ByVal TestType As MultiThreadTestType, ByVal AtomicOperations As Boolean, ByVal Iterations As Integer, Optional ByVal ThreadCount As Integer = 0) AsDouble
    Dim DateTimeStartTime As DateTime
    Dim DoubleReturnValue As Double

    DateTimeStartTime = DateTime.Now

    DoubleReturnValue = (DateTime.Now - DateTimeStartTime).TotalMilliseconds

    DateTimeStartTime = Nothing
    Return DoubleReturnValue
  End Function

The function?s return results are the total number of milliseconds that it takes for the test to run. You will note that it looks like no multithreading is occurring yet. This is on purpose. We want as little multithreading in the application outside of the test itself. We will later constructing different functions for actually running the tests and calling them in a standard procedural way from within RunTest().

The final step before we can start constructing the tests themselves is to write a baseline test function. Each of the tests will use the exact same function to perform some computations in order to keep a level playing field amongst all tests. Here is our test function:

  Private Function Compute(ByVal InputValue As Double) As Double
    Dim DoubleOutputValue As Double
    Dim DateTimeNow As DateTime
    Dim rndNumberGenerator As System.Random

    DateTimeNow = New DateTime(DateTime.Now.Ticks)
    rndNumberGenerator = New System.Random(DateTimeNow.Hour + DateTimeNow.Minute + DateTimeNow.Millisecond)

    If DateTimeNow.Millisecond > 500 Then
      DoubleOutputValue = System.Math.IEEERemainder(System.Math.Exp(rndNumberGenerator.Next * (InputValue + 5000) * System.Math.E), rndNumberGenerator.Next)
    Else
      DoubleOutputValue = rndNumberGenerator.Next(InputValue, System.Math.Max(Integer.MaxValue, System.Math.Log(System.Math.Pow(System.Math.PI, InputValue))))
    End If

    DateTimeNow = Nothing
    rndNumberGenerator = Nothing

    Return DoubleOutputValue
  End Function

As you can see, our test function is designed to put the processor through its paces (at least floating point operations) without using much memory at all or to require any disk access. You may want to modify this code to use a lot of memory or to perform disk access or network operations, to meet the conditions that the applications that you are working on will be facing.

In the next post in the series, we will write our first battery of tests, to show single threaded performance, with no locking mechanism required or needed. While the development machine that I am currently working on has only one logical CPU (AMD Sempron 3200, x64 architecture, single core CPU), we will get to see this code in action on an AMD Athlon 3200+ (x64 architecture, single core CPU), an Intel Pentium 4 2.8 gHz (x386 architecture, single core CPU, one logical CPU), and a dual Xeon 3.0 gHz system (two dual-core, HyperThreaded Xeon CPUs with x64 architecture for a total of eight logical CPUs). These will be our test systems for the duration of this blog series, and should let you see how identical multithreading models perform on very different systems.

J.Ja

This is the third installment of a multi-part series demonstrating multithreading techniques and performance characteristics in VB.Net.

In my previous post, I created the skeleton of a project that demonstrates multithreaded performance. In this post, we will be filling in the skeleton to dispatch the work to the correct function, and creating a performance baseline using a single thread.

During the testing for this post, it was determined that the Compute() function outlined in the previous post did not work as expected, so it has been revised slightly. Its concept is the same, but the computation has been tweaked a bit to eliminate overflow errors in the math. So now we have the following Compute() function:

Private Function Compute(ByVal InputValue As Double) As Double
  Dim DoubleOutputValue As Double
  Dim DateTimeNow As DateTime
  Dim rndNumberGenerator As System.Random

  DateTimeNow = New DateTime(DateTime.Now.Ticks)
  rndNumberGenerator = New System.Random(DateTimeNow.Hour + DateTimeNow.Minute + DateTimeNow.Millisecond)

  If DateTimeNow.Millisecond > 500 Then
    DoubleOutputValue = System.Math.IEEERemainder(System.Math.Exp(rndNumberGenerator.Next * (InputValue + 5000) * System.Math.E), rndNumberGenerator.Next)
  Else
    DoubleOutputValue = rndNumberGenerator.Next(InputValue) / System.Math.Max(Double.MaxValue - 1, System.Math.Log(System.Math.Pow(System.Math.PI, InputValue)))
  End If

  DateTimeNow = Nothing
  rndNumberGenerator = Nothing

  Return DoubleOutputValue
End Function

Our single thread run looks like:

Public Sub SingleThreadComputation(ByVal Iterations As Integer)
  Dim IntegerIterationCounter As Integer

  For IntegerIterationCounter = 1 To Iterations
    Compute(Double.Parse(IntegerIterationCounter))
  Next
End Sub

Finally, here are our performance characteristics for 1,000,000 iterations, in milliseconds per test:

System A: AMD Sempron 3200 (1 logical x64 CPU), 1 GB RAM
System B: AMD Athlon 3200+ (1 logical x64 CPU), 1 GB RAM
System C: Intel Pentium 4 2.8 gHz (1 logical x86 CPU), 1 GB RAM
System D: Two Intel Xeon 3.0 gHz (2 dual core, HyperThreaded CPUs providing 8 logical x64 CPUs), 2 GB RAM

It is extremely important to understand the following information and disclaimers regarding these benchmark figures:

They are not to be taken as absolute numbers. They are taken on real-world systems with real-world OS installations, not clean benchmark systems. They are not to be used as any concrete measure of relative CPU performance; they simply illustrate the different relative performance characteristics of different multithreading techniques on different numbers of logical CPUs, in order to show how different processors can perform differently with different techniques.

The performance numbers on the single thread test show some truly fascinating results. System D (the dual Xeon machine) was actually our worst performer on the single threaded test. Although the Xeons seemed to suffer a bit on the single threaded performance, it is expected that they will maintain nearly identical performance when running 8 simultaneous threads non-atomically, while the single core CPUs should suffer penalty for running multithreaded.

Another mildly interesting item to note is the difference in the clock reports between the AMD systems (A and B) and the Intel systems (C and D). The AMD CPUs rounded milliseconds up to units of 0.025 milliseconds (I showed 3 significant digits in the table, but the program returned 4 significant digits). Since all of the test machines are running the same version of the .Net Framework, this is obviously a difference between the two chipmakers. It is not relevant to the results of this test, but it is an interesting data point to remember for future use, in case it ever comes up.

Stay tuned for the next post, which will show our first multithreaded test.

J.Ja

[Update 9/11/2006: corrected Joel Spolsky's name, and added relevant links]

There are a few people in the tech world that I would really like to meet. Some of them just seem like they would be cool people to hang out with, some of them would make awesome people to have as a boss or co-worker, and others are just blindingly smart. Here is my list of tech folks that I would like to know, and why.

Joel Splosky Spolsky [Edited 9/11/2006: Corrected Joel's Name]

http://www.joelonsoftware.com/
This guy is really, really smart. Even better, he understands developers and development better than anyone else, and owns his own company. Every blog post of his, I want to forward to my boss (but usually do not, in case it is thought of as criticism). I really like how he is able to put into words the feelings that I have felt about a lot of things so succinctly.

Steve Ballmer

Say what you want about the guy and his business practices, he has an enthusiasm for his job that is rarely seen outside of professional athletics. He treats running Microsoft like he in on the Olympic team. Rarely do you see employees, let alone managers or executives get so excited about stuff (remember the "Monkey Boy" video?). It would be great to have that level of enthusiasm in a manager.

Jakob Neilsen

http://www.alertbox.com
I have been reading his work for well over five years, and I have yet to find something to disagree with him on. Regular readers know how important I think usability is, and he does a better job than anyone else at turning it into a science that is able to work within business realities. "Usability Engineer" is a job title I would love to have, and I would like to get together with him and figure out how to become one.

Mark Hurst

http://www.goodexperience.com/
Mark Hurst is up there with Jakob Neilsen, but he is a lot more focused on the user experience itself, and not the objective study of usability. He is also much like Joel Splosky Spolsky Edited 9/19/2006: corrected name, in that he really "gets" how tech people work, and how to help them work within the confines of business realities. His newsletters are smart and funny, and packed with unique information that you will not find anywhere else.

John Dvorak

http://www.pcmag.com/category2/0,1874,3574,00.asp
I have been reading John Dvorak since I was about 11 years old, or something like that. He is extremely witty, and when he is talking about basic principles, few analysts are smarter. He is rather contrarian, like myself, which I also like. Just because everyone else thinks that something is true does not mean that he will jump on board. Lately I think he has been slipping, but much of that is because he has not been in the trenches for a long time. That still does not diminish his abilities to analyze, and he is still top notch when he has the right information. I also respect that he does not buy into the vendor buzz and media hype, unlike far too many tech writers.

Daniel C. Dennett

http://ase.tufts.edu/cogstud/~ddennett.htm
While Daniel C. Dennett is not a tech industry person per se, his writing and research has a lot to do with technology, and is extremely applicable in technology. He was the first person to write about cognitive science who really made sense to me. Even when he is wrong, he is a good sport about it.

Bill Gates

The man is absolutely ruthless, to the point where it takes the entire government to put the brakes on him. Like Ballmer, Gates has an uncontrolled exuberance for his work and his company. While he may get dinged for copying other people?s ideas, he is excellent and combining the ideas of others in ways which they never thought of. I also really like his philanthropy work; I get the impression that he wants to win for the sake of winning, not for the rewards. While that type of predatory attitude is not entirely admirable, it is certainly respected.

Larry Wall

http://www.wall.org/~larry/
The father of Perl. 'Nuff said.

Mark Cuban

http://www.blogmaverick.com/
Mark Cuban rocks, because he understands what the consumer wants. More specifically, he ignores what the consumer thinks or claims they want, and talks about what they will actually pay money for. That is a huge difference. For example, I know a lot of people who want fancy cars, but prefer to drive economy cars and not pay for a fancy car. Mark Cuban is also a very funny person to read. His whole to-do with Overtsock.com was probably the funniest thing in the tech business world for quite some time.

James Gosling

http://blogs.sun.com/jag/
A geek?s geek, if you know what I mean. He is totally, brutally open and honest to the point where you wonder that his employer (Sun) allows him to appear in public without a muzzle. I remember reading an interview with him a while back, where he actually answered the question of "what are your biggest mistakes" with legitimate answers.

Who is on your list of people in the Tech World that you would like to meet?

J.Ja

OK, so a temporary Ubuntu bug disabled X. And this is a problem because? In the years that I have run *Nix servers, I have never once felt the need to install X. The only time I installed X on a server, was because someone else in the organization needed an easy way to follow the progress of some code they were running, and letting them login to X to view a graphical version of top was what they wanted.

Folks, there is no good reason to run (let alone install!) X on a server, unless it is being used as a server for thin clients. If you have a ton of people using X applications, it is easier to get a bunch of X clients and hook them into a big server.

Outside of that fairly rare scenario, why is X on your server? There is nothing that needs to be done on a *Nix server than cannot be done via CLI, without the hassle and headache of dealing with X. X is one of the most miserable pieces of software out there to deal with, right behind BIND. So unless you are running an X farm, rip X off of your server.

J.Ja

This is the fourth installment of a multi-part series demonstrating multithreading techniques and performance characteristics in VB.Net.

In today's post, we will take a look at using the SyncLock system to maintain data integrity during operations. We will also be showing off the .Net ThreadPool object as well. Initially, I planned to write my own version of the ThreadPool. However, there is really no reason to do so within the scope of this series. The .Net ThreadPool works great for our needs and is well designed to manage the number of active threads. The .Net ThreadPool is a very smart way of managing multiple threads, and it allows us to set the maximum number of executing threads, which allows us to achieve our goal of seeing what the effects of limiting or expanding the number of running threads has on performance. Writing a thread pool on our own is a rather difficult task, and is not needed unless your project has very specialized needs.

Here is the code for the SyncLockThreadWorker object that performs the computations and atomic operations:

Public Class SyncLockThreadWorker
  Public Shared objStorageLock As New Object
  Public Shared objCompletedComputationsLock As New Object
  Public Shared IntegerCompletedComputations As Integer = 0
  Private Shared DoubleStorage As Double

  Public Property Storage() As Double
    Get
      SyncLock objStorageLock
        Return DoubleStorage
      End SyncLock
    End Get
    Set(ByVal value As Double)
      SyncLock objStorageLock
        DoubleStorage = value
      End SyncLock
    End Set
  End Property

  Public Property CompletedComputations() As Integer
    Get
      Return IntegerCompletedComputations
    End Get
    Set(ByVal value As Integer)
      IntegerCompletedComputations = value
    End Set
  End Property

  Public Sub ThreadProc(ByVal StateObject As Object)
    Dim ttuComputation As ThreadTestUtilities

    ttuComputation = New ThreadTestUtilities

    Storage = ttuComputation.Compute(CDbl(StateObject))

    SyncLock objCompletedComputationsLock
      CompletedComputations += 1
    End SyncLock

    ttuComputation = Nothing
  End Sub

  Public Sub New()

  End Sub
End Class

Inspection of the class shows that we use SyncLock in two places. The first place is in updating the Storage property. The other place is in updating the number of completed threads. Because we do not ever read the Storage property while simultaneously writing to it, placing the lock within the property accessor is acceptable. For the CompletedComputations property, however, we read and write to it in the same line, making it necessary to lock it as we work with it, as opposed to within the accessor.

This is the code that creates and runs the threads, as well as waits for them all to finish before completing. If we do not wait for all of the threads to be finished, the Sub will exit as soon as the threads are queued.

Public Sub SyncLockMultiThreadComputation(ByVal Iterations As Integer, Optional ByVal ThreadCount As Integer = 0)
  Dim twSyncLock As SyncLockThreadWorker
  Dim IntegerIterationCounter As Integer
  Dim iOriginalMaxThreads As Integer
  Dim iOriginalMinThreads As Integer
  Dim iOriginalMaxIOThreads As Integer
  Dim iOriginalMinIOThreads As Integer

  twSyncLock = New SyncLockThreadWorker

  Threading.ThreadPool.GetMaxThreads(iOriginalMaxThreads, iOriginalMaxIOThreads)
  Threading.ThreadPool.GetMinThreads(iOriginalMinThreads, iOriginalMinIOThreads)

  If ThreadCount > 0 Then
    Threading.ThreadPool.SetMaxThreads(ThreadCount, ThreadCount)
    Threading.ThreadPool.SetMinThreads(ThreadCount, ThreadCount)
  End If

  For IntegerIterationCounter = 1 To Iterations
    Threading.ThreadPool.QueueUserWorkItem(AddressOf twSyncLock.ThreadProc, Double.Parse(IntegerIterationCounter))
  Next

  While SyncLockThreadWorker.IntegerCompletedComputations < Iterations

  End While

  Threading.ThreadPool.SetMaxThreads(iOriginalMaxThreads, iOriginalMaxIOThreads)
  Threading.ThreadPool.SetMinThreads(iOriginalMinThreads, iOriginalMinIOThreads)

  twSyncLock = Nothing
  IntegerIterationCounter = Nothing
End Sub

Here are the results of our tests. All tests are for 1,000,000 iterations, and the results are in milliseconds per test run.

TEST 1

This test allows the ThreadPool to manage the total number of minimum and maximum threads on its own:

TEST 2

In this test, we limit the maximum number of threads to one per logical processor:

TEST 3

This test uses only one thread:

TEST 4

This test uses two concurrent threads:

TEST 5

Here we show four concurrent threads:

TEST 6

This test uses eight concurrent threads:

TEST 7

Finally, this test runs 16 simultaneous threads:

System A: AMD Sempron 3200 (1 logical x64 CPU), 1 GB RAM
System B: AMD Athlon 3200+ (1 logical x64 CPU), 1 GB RAM
System C: Intel Pentium 4 2.8 gHz (1 logical x86 CPU), 1 GB RAM
System D: Two Intel Xeon 3.0 gHz (2 dual core, HyperThreaded CPUs providing 8 logical x64 CPUs), 2 GB RAM

It is extremely important to understand the following information and disclaimers regarding these benchmark figures:

They are not to be taken as absolute numbers. They are taken on real-world systems with real-world OS installations, not clean benchmark systems. They are not to be used as any concrete measure of relative CPU performance; they simply illustrate the different relative performance characteristics of different multithreading techniques on different numbers of logical CPUs, in order to show how different processors can perform differently with different techniques.

Compared to our previous test results, it is very easy to see that atomic operations have a very high cost. Additionally, each active thread above and beyond the number of logical processors degrades performance. This is because the OS (I say "the OS" instead of "Windows" because this will apply to any multithreading OS that uses time slices to manage multithreading) has to perform quite a bit of work to manage each thread that runs per processor. It is also easy to see that the ThreadPool object does not make the best possible decisions regarding the number of active threads. While this is not evidence that the ThreadPool object is poorly written, it is evidence that it is wise to override its settings in order to tune performance to the application's needs, based on the type of operations occurring in the running threads. An application running threads waiting on asynchronous I/O, for example, will be able to sustain many more active threads than one such as this one that is performing raw computations. It would also be interesting to force a number of concurrent threads, instead of allowing ThreadPool to manage it. It is difficult to tell if the relatively static results beyond a few threads are because ThreadPool is deliberately throttling the number of running threads or if the SyncLock is creating a traffic jam that no amount of pooling can resolve. The only way to test this would be to rewrite the test to use a homegrown thread pool, and to possibly have each thread perform a much higher work-to-locking ratio. It is also interesting to see that the dual Xeon system still plods behind the other systems. It may be that with a much higher number of forced concurrent threads that the Xeon truly shines.

My next post will have extremely similar code, but use the Mutex object instead of SyncLock to manage the atomic operations.

J.Ja

It tends to be the case that when there are three or more competitors for something, the race gets narrowed down to two real contenders fairly quickly. In the case of ASP.Net, J2EE, and PHP, I think the odd man out is going to be PHP. "But," you cry out, "PHP is the language of choice for so many open source Web projects!" So was Perl at one point. It did not save Perl, and it will not save PHP. Why not? PHP does not allow the programmer to multithread their application.

I have been doing a series of posts about multithreading techniques with VB.Net (next week I will be posting Part V, using Mutex's for atomic operations). The code is almost directly translatable into C#, and the principles are identical in Java. They will not work in PHP though. PHP, as far as I can tell by searching and browsing its documentation, does not support multithreading. This is going to be a major vulnerability for PHP in the very near future.

One of the major drivers of the Internet explosion has been the plummeting cost of running a server, thanks in large part to the ability of commodity hardware to handle the load of serving pages over the last few years. Between the LAMP stack, J2EE, and the Microsoft stack, there are plenty of options, and they all work very well on a server that can be had for well under $5,000. Up until recently, however, these commodity servers were using one physical and logical CPU. AMD and Intel have changed the game with their dual core (and soon, quad core) CPUs. SMP motherboards have come down significantly in price as well. The $5,000 number can now get you a server that has two dual core Xeon, Core 2 Duo, or Opteron CPUs. In other words, a modern server is at minimum a two logical CPU machine and is headed towards the four-to-sixteen logical core CPU quite soon (two quad core CPUs with HyperThreading is sixteen logical cores). The Sun T1 CPU has put a 32 thread pipeline into the under-$10,000 price range.

Furthermore, more and more Internet applications retrieve ? but do not require ? data to be retrieved from elsewhere, possibly across a WAN link. If you are a Web developer, would you prefer your application to continue to process its work (for example, retrieving data from the local database, dynamically creating images, etc.) while waiting on the third party chunk of code (such as a Web service), and only have part of the page show a "Sorry!" message? Or do you think it is better for the entire page to take forever if the third party data source has a problem? Personally, I would prefer to process those WAN requests as a separate thread. But PHP cannot do that, because it does not support multithreading.

Sure, PHP supports UNIX style process forking. But the documentation has this to say about forking:
Process Control support in PHP implements the Unix style of process creation, program execution, signal handling and process termination. Process Control should not be enabled within a webserver environment and unexpected results may happen if any Process Control functions are used within a webserver environment.
In other words, PHP, a language that is not terribly useful outside of Web development language, should not use process forking in a Web server. Huh?

PHP is weak in many other areas as well. Its documentation is not very good, and frequently the comment threads provide the information that the documentation should have included but did not. It lacks high quality tools such as IDEs and debuggers in a "ready-to-use" state. I will give PHP this: it is easier to install on top of Apache or IIS than any J2EE server that I have encountered. But outside of that, its only strength was being a Perl-like language (allowing Perl developers to transition to it) while being open source (which neither Java nor .Net are) and being adapted to the Web (which Perl was not). With Ruby and Python in the game, PHP is no longer unique. From what I have heard, read, and seen, Ruby on Rails kicks PHP six ways to Sunday.

PHP is getting hit from all sides, from where I sit. It took RHEL about two years to support PHP5. PHP lacks multithreading support. The .Net Framework and J2EE ecologies are exploding, while PHP5 still feels like a limited knock off of Perl designed to work exclusively with Web pages with a hacky-feeling object model. If you learn Java or a .Net language, you can transition very easily to desktop applications or non-Web server applications; you just need to learn new methods of user interaction. Not so with PHP, it is pretty much so useless outside of a Web server. Meanwhile, Ruby (which does support multithreading) is an excellent general purpose language and the Ruby on Rails framework makes PHP look downright Mickey Mouse in comparison, and it natively supports AJAX (as much as I dislike AJAX, I recognize that it does have its uses, and it is currently quite hot). And Python has been threatening to be "the next big thing" for long enough to believe it may have a chance.

To put it another way, PHP does not do anything that every other language does not do, and it lacks multithreading support, which is quickly becoming a showstopper. If PHP does not catch up soon, I believe that it is doomed. Ruby and Python are poised to take its crown for the open source Web development platform of choice. Remember, Perl and CGI used to own the Web development space entirely; now Perl barely exists in Web development outside of legacy applications. Things change, and PHP needs to change quickly, or it will follow Perl.

J.Ja