Critical Thinking

I am very grateful that I am no longer at my last, super corporate job, and that at my current job, I run the IT systems. Why? Because Skype probably would have gotten me fired from my last job. Skype, like every other major instant messaging client, has a conversation history feature. Unlike the others, Skype has a synchronization system. For example, if I am at work at talk to someone via Skype, and then get home and they are logged on from that computer, it will dump the previous chat session's contents to my home PC.

This is all well and good. Rather obnoxious (when the other person signs on, you suddenly get dumped an entire conversation all at once, and it looks like a new message), but harmless. Or so I thought.

Just a few minutes ago, someone I talk to frequently via Skype signed on, using the same PC he was using last night when I talked to him. The contents of last night's conversation (when we were both at home) got dumped to my screen. Some of the language was not "work safe" and some of the contents were not "work safe." Certainly, much of the conversation was something that, even if it did not get me fired, would certainly have me blush if my boss ever confronted me with it.

We do not do any kind of Web/Internet monitoring at my current company. If we did, I would be the one doing it any ways. My previous employer was rather different. All IMs went through a proxy which performed logging, and each manager had full access to the logs of their direct reports. In other words, had this happened at my previous employer, my manager would have been able to read the full contents of that conversation. Even worse, it would have appeared to occur during working hours as the proxy would have timestamped it.

So, if you work for an employer that might be (or definitely is) performing any type of Internet monitoring, I highly recommend that you either avoid Skype entirely as an IM client and use it strictly for its VoIP features, or disable to conversation logging functionality to prevent your "non work safe" conversations from outside the office from leaking into the office. This is yet another example of a new feature that its creators did not consider the implementation of or consequences of before they implemented it.

J.Ja

The process of short-release cycles is known in our company as making 'banana-ware' - in other words, the product is green when we ship it and it ripens at the customers.

For us this works - because the software is not the primary focus - the database information we ship and the reports that you can retrieve are what people want - if the software has the occasional glitch it will be patched asap, or very rarely when it lasts that long, fixed on the next release, which will be out every 3 months ( this ensures we cover our legal obligations to have up to date safety data). The software expires every 7 months. The continual getting ready for release cycle is supported by 5 testers and they are starting to automate this.

Best bit about the short release cycle is that you only have to support bugs for at most 7 months.

I think Justin is correct in many respects.  Buggy software is
regulary shipped, and we are in a constant patch cycle.  The
ramifications are huge.  But, I do not think that the development
methodology is the cause.  You can create bad software using a
short release cycle as in Agile Programming as well as with a longer
release cycle as in the Water Fall development cycle.

The problem is real, but the solution is not going to come from the frontline developers, who rarely have the authority to stall the demands of their masters even when they have the skills. The problem is that expectations of businesses (and managers) exceed the real possibilities of clean development on a compressed timeline. Any time a business is managed by the quarterly balance sheet, the problem will be that anything more than quarter-long project is impossible for the managers to wrap their minds around. Regardless of all good intentions, no developer can expect to change that management style, and most software development is not going to respond effectively to the demand of ROI by the quarter.
 
As for methodology being the culprit? Any methodology can be abused, and blaming RAD methodologies for the problems of buggy software (or blaming short cycle releases) is nonsensical. I have seen long-form methods that generate thousands of pages of inscrutable documentation fail as certainly, and they usually fail for reasons that are harder to qualify because the documentation is generally bad. Good developers exceed the limitations of their methodology; bad ones degrade the value of their methodology -- no specific methodology is better or worse. Documentation only has value when it is focused, correct for the project, and communicative -- most I have seen is not.
 
Software development used to be a creative enterprise, where those doing it had real passion for the work. Having been kicking around 20+ years, and actually making development my primary source of income that entire time, I have watched the field reduced by an influx of people who look at it as a money-making opportunity first and foremost. When they find out how much actual work it is to do it well, many of them simply enter the "get-by" mode of operation. This attitude is more the cause of the degradation of software quality than anything else, because without passion the process of creativity behind the development of software becomes mechanical. Problem-solving is rarely mechanical, and never effectively so when timelines are as compressed as real developers face daily. But blaming this lack of passion on the methods of choice on is arsy-versy: the problem is that the weakest developers are choosing methodologies that demand the least of them, but they would do as poorly in tougher frameworks.
 
The development community doesn't need more methodologies, should stop pursuing the pipe-dream that it is an "engineering" undertaking, and should focus on its actual value-proposition. With good, passionate developers, software provides quality tools, regardless of cycle, method, or timelines. Craftsmanship -- almost unknown to the modern world -- counts in the field.

All code has bugs. The dilemna is how do we know whether we've found the crucial bugs. In non-trivial code you have too many code paths to exhaustively test each path, so you use beta releases to verify whether you have fixed all non-crucial bugs.

All development teams, even Microsoft's, have limited resources. It is not a "code with known bugs for the sake of getting new feature in the face of users", it's "which bugs are serious enough to not put new features in the face of users".

As to "rapid application development, agile programming
techniques, "test in production," perpetual betas, the continuous patch
cycle, and other current programming trends", in which methodology is the mantra "ship really bad code quickly".

Regardless of how you develop code, all code contains bugs. The idea behind many of the new techniques is to get a working version of the product out in front of users quickly so you can discover whether or not you are building a solution that works for them. That and building the right features into the product, rather than features they never use. That and find bugs and fix them as early as possible. What's wrong with that?

Which development model did Hyatt use? How do you know whether it was agile or waterfall or "throw it over the wall to an outside consulting company and pray"? What the h*ll does http/https have to do with the development model? I see no connection. How did this simple mistake get by code reviews? Oh wait, I'll bet there were no code reviews. How is that the fault of developers? It is the fault of development management. Most of the mistakes you blame on developers are the result of pressure from above to "just do it". Management is who won't let us attend seminars on creating secure Web sites. Management is who wants us to ship buggy code. Management is who insists on getting out crappy beta after crappy beta.

All I see is guilt by association--current code must have been written in a current development model.

Here's my take: upper management did not allow the product to "bake". As problems were discovered, developers were instructed to "code around them" rather than fix the underlying cause. I'll bet they were instructed to base their Web software on Windows/ASP from the get-go, not from any cost/benefit analysis.

You are barking up the wrong tree. Developers are not hot to ship buggy code. That decision is not theirs.

doug in Seattle

There are two aspects intertwined here. One is the quality of the code itself. As one post indicated the discussion seems to be toward sloppy programming. The other aspect of rapid releases is "quick to market" products. By releasing rapidly, one can incrementally add functionality to a product, make it available to the customer and start seeking very valudable customer feedback, to futher enhance the product. This is very important for customer facing applications. Some organizations have a tendency to create a huge release and jam all the functionality that they can think of. The fact remains that most of that functionality is hardly used by their customers.

Some time ago, Joel Spolsky ("Joel on Software") wrote a good little piece entitles "The Development Abstraction Layer." As always, Joel was right on the money. The last few weeks, I have been trying to arrange training to go to training for Cognos ReportNet, at the request of one of our clients. Arranging the training has been an amazing waste of time. I spent hours on the phone with Cognos, hours searching and filtering and refining course searches, hours trying to make travel arrangements; in short, the kind of things that Joel says should be abstracted out of the development process.

It is not that I feel that this level of work was demeaning. If anything, it was refreshing to not have my head buried in a compiler or an Excel spreadsheet all day long. But all things considered, an administrative assistant could have handled the training arrangements just as competently as I did (maybe even better) at a significantly reduced cost. This is not the first time that this has happened at the company I work for. My current employer is a five person company, and we all have our separate responsibilities and areas of specialization. Whenever some non-core competency item comes up (accounts payable, accounts receivable, travel arrangements, going to the Post Office, etc.) we have to take time away from a project and cease incurring billable hours to do work that an administrative assistant could do just as well if not better.

Even at large companies, this is standard. At every job I have ever had, administrative assistants and other support staff simply did not exist for the IT staff. It seems like every other department had support, except for IT. Ironically, IT professionals (even after the dot-com crash and the offshoring craze) make more than Marketing, Public Relations, Accounting, and all of the other departments. Yet it is still standard to have IT professionals waste their time doing things that have nothing to do with their jobs. I am simply baffled at the attitude that says that spending the time of someone who earns $70,000 per year on trying to order stationary instead of developing software. The modern enterprise is filled with an extraordinary amount of layers. Simply ordering pens can take an hour. Travel arrangements take days to set up through the corporate travel department. Purchasing something from an unapproved vendor or software not on the corporate installation list is nightmarish at best (and IT professionals typically need at lot of non-standard software and hardware). Yet, IT professionals are asked to spend their time doing this. At some large companies I have worked for, the kinds of tasks have frequently consumed 5% of my work time on average over the span of my employment. A 5% reduction in productivity, multiplied by the cost of an IT department is a huge hole in the budget. But this is considered acceptable.

I do not consider this acceptable. I consider it bad business. If you could buy a product from one store for $50 and at a different store for $20, you would get it from the cheaper store, right? So why are companies forcing high priced IT professionals to do the same work that a high school graduate could be doing at $10 per hour? Is it simply prejudice against IT? The mistaken belief that IT professionals do not do anything that is not their core mission? Or something else?

I would love to hear your feedback about this problem.

J.Ja

Excellent posting, Justin.

I haven't seriously drilled into the Agile Programming methodology yet, so I'm not in a position to make a serious comment about it.  In general though, if a methodology, programming technique, whatever, raises even the suggestion of "short cuts" to coding, questions must be asked. If it can be used as a crutch by lazy coders to legitimise their lack of good practice under the banner of "It's so hot right now", questions must be asked.  If un-tested code is going to find it's way out into userland and be viewed as even remotely "production - ready", questions must be asked.

For my money, coding has always been about one key concept: Quality.  I think Justin describes it as "craftsmanship".  The professional coder is always aware that, no matter who s/he happens to be working for at the time, their code is inextricably linked to their name as a programmer.  If, over the space of say 2 years, the code constantly has to be patched for serious flaws that could have been avoided with some proper review and testing, you can bet your life those with the job of maintaining that code will be asking "well who wrote this in the first place?" On the other hand, if over that same 2 year period their code has performed admirably and withstood even unexpected challenges, those same people are going to be wanting him/her back again to help with the next major release, and will no doubt be willing to pay accordingly.

Amongst other things, a good methodology enforces documentation, rigour and order on a project, and these are obviously good things.  Coders, however, should never overlook their personal responsibility to develop code that, to the best of their knowledge, fits the bill.  This principle will never change, no matter which methodology is "so hot right now".

Stephen

Agree, agree, agree. Let me just add the following to indicate my feelings on most web site design:

AAAAAAAAAAAAARGH!

Good web site design is a mix of technical skill, experience, and understanding and empathising with the user. The good designer will try to understand what their target users want, what is important to them, and their state of mind when they arrive at the web site.

Let me use Google as an example. Please don't flame me for mentioning Google - I know strong feelings exist around them at the moment. But Google understands one thing very well - the average user arriving at a search engine wants one thing: to find other websites quickly, and leave! They don't want to muck around in useless "portals" and have news, adverts, and celeb gossip pop up in front of them, with the actual search bar hidden in some unobtrusive location, with space for about 3 characters using a 3 point font. If you go to Google, the home page loads in seconds.(milliseconds if the graphics load from a cache) Compare this with the MSN home page, which is definitely in portal mode.

If, on the other hand, your web site is in the "celeb gossip" category, you might get away with more flashy graphics, photos, ads, etc, because your users are more likely to be in "browse" mode. If each page takes 5 minutes to load, however, you will probably lose the user.

A final suggestion that might help keep the user in mind, is to make up a fictitious user who represents the average person you want on your site. Give them a name and a personality! Give them a realistic level of tech literacy. The more realistic the better. This helps take away the vague idea of "the user" who is some legendary entity out there, to a real person. It is easier to imagine what "Bob Sportsnut" the 19 year old college student thinks of your site than "the college demographic". Marketers are particularly prone to this habit of dehumanising their target audience, and they wonder why nobody is flocking to their web site which is 90% ads that "appeal to our demographic" and 10% actual information of interest to Terence and Susan out there.

* the characters mentioned in this entry are fictitious. Any resemblance to real persons living or dead is unintentional. All trademarks are the property of their respective owners. Batteries not included. Barbie and Ken not included. *

Nice set of articles. My pet peeve: "This site is best [or worse, only]
viewed with the _____ browser." Any boob who uses a page building tool
that restricts users to only one browser deserves to take it in the
shorts. Another: "This site requires the latest version of Shockwave/Flash/unameit."

A question: Is it worth going to CSS for sites? Can we depend on 99%+
of customers using a browser that either correctly handles CSS, or at
least degrades gracefully? There are a few nice things that can be done
with CSS that can't be done with plain HTML, but I don't want my sites
looking crappy because the IE5 or NS4.5 browser messes up all my CSS code. Any
guidelines here? I'm happy to write pure HTML pages, and if CSS will
make them look and act even nicer, that's great, but I don't want to
end up with broken pages.

Phil

Phil -

My recommendation with CSS is to use static HTML with an external style sheet, and inline style tags where needed for "one off" styling needs. I have had great success with this in the past, and I stick with CSS 1, using it mostly for basic things (no layers, for example). The beauty of this is that even ancient browsers can view the site, and since what you are using CSS for is non-critical, even a site without CSS will work. I will follow up on this in the near future with a blog exsplaining my "running water" philosophy of HTML, which has provided great sucess with zero need to change it (outside of changing the HTML to be standards compliant as the standards change) in my 10 years of working with HTML.

J.Ja

I have been re-reading some of my blog posts, and I noticed just how often I mention usability. I consider usability to be the most important feature a piece of software (or hardware, for that matter) can have, right next to reliability and security. Anything else is less important. The case for reliability and security is well established. Usability has been proven to be a vital feature as well, but for whatever reason, programmers do not seem to care for it too much. And when they do, they seem to frequently fail at getting it right.

Why is usability so important? Cell phones are a great example. A huge portion (40% is a number I read recently) of cell phones users do not use any feature of their phone except basic voice calls. Each user that only makes basic calls is a lost revenue opportunity for the cell phone provider. These users, by and large, are not using the advanced features, because they cannot figure out how to use them. Indeed, cell phones are so hard to use, I recently read that around 25% of cell phones users do not even use the address book, a feature that has been available for over a decade on cell phones. That is ten years to get a feature right, and it is still not usable by a whopping quarter of users.

If I opened a store, and 25% of my customers could not figure out how to open the door, I would be out of business. If I opened a hamburger stand, and 40% of my customers found ordering fries or a drink with their basic hamburger to be more work than it was worth, I would be out of business.

Software is just as bad. Microsoft Office is the extreme example of this. Most of its innovative or high powered features are buried in menus, non-standard toolbars, or otherwise hidden from the user. A user could be doing something manually for years before discovering that Office had a feature to do the exact same thing already built it.

A significant portion of computer users will not experiment above and beyond a single click to accomplish their goals. These users will not drag/drop, will not right click, will not double click, and certainly will never learn a keyboard shortcut. These are the users we see helplessly scanning the menus to perform even the most basic tasks. As a result, some programmers choose to dumb down their software to the point where it is nearly impossible to do anything faster than the speed of the menu system.

At the other end of the spectrum, there is software that is simply impossible to use efficiently, no matter how long you have been using it. Sadly, Microsoft Office also falls under this category. Although it is feature packed, there are few keyboard shortcuts to assist in the work. Even worse, using its advanced features is cumbersome at best. The auto formatting and other things it does to help the newcomer hold the advanced user back, forcing the document into undesired formats.

Usability is indeed a fine line; expose too much advanced functionality upfront, or force users to work in uncomfortable ways, and they develop a fear of the product. Making the software too easy to use holds people back from using your product to the fullest. But usability needs to be a top concern for anyone making any kind of interface, whether it be hardware or software, or even something as mundane as a common phone. Without usability, people will not use your product to the fullest, leading them to get little value from it and to not buy it. Poor usability nearly always results in poor market results (there are a few exceptions to this, like MySpace). Great usability almost always results in better market performance (the Macintosh is the exception to this rule). Google's stunning rise to the top of the Web search market probably would have happened even if they were delivering the exact same results that MSN or Yahoo! was delivering, just because Google's interface is superior to their competitors' interfaces in every measurable way (note that Google employs zero JavaScript for their core search engine, let alone Flash, AJAX, spinning banners and dancing bears). You can ignore usability, but do so at your own risk.

J.Ja

Recently, there was a very interesting and heated discussion on TechRepublic concerning what the best programming language for cross platform development is. I weighed in, of course (my vote went to Perl and standard C/C++). Afterwards, I began to ask myself, "does cross platform development even matter?" The answer to this is "maybe," and "yes."

Where cross platform development might not matter much is in the desktop application space. Most large enterprises have standardized on the Windows platform. If you are a Windows application developer, you can sell your product over 90% of the desktop users out there. As frequent readers know, I am not fond of the idea of locking yourself out of a market. Obviously, if you are writing a Windows system utility, UNIX and Macintosh users are not important to your strategy. But if you are looking to write a general purpose application, you may need to ask yourself if losing 5% - 10% of potential sales outweighs the efforts of writing your program in a cross platform manner. Even in the cross platform space, there are pure desktop applications, where the data is stored in files, and client/server applications that require some sort communications between the desktop application and a backend server of some sort.

Cross platform development definitely matters in the server application space, though. Windows does not command the huge market share in the data room that it does on the desktop. It is very difficult indeed to justify being able to sell your product to 40% - 60% of the potential market. This is one reason why I am not a fan of using either .Net or Java on the backend. .Net, unless you are using the portion of the .Net Framework that Mono supports, is hardly cross platform. And most of .Net's advantages stem from its tight integration with Windows. I still do not trust Java's cross platform portability. I have simply had too many problems with it in the past. Many of the portability problems I have had with Java stem not just from the JVM, but from the application server itself. Targeting a particular application server cuts you off from potential sales even worse than targeting a particular operating system. And using only standard Java reduces its usefulness on the backend significantly.

Web applications require still more cross platform considerations. Internet Explorer on Windows no longer holds the 90% of the market like it used to. A few years ago, you could target a Web site towards IE on Windows, and allow for it to gracefully degrade for UNIX, Apple, Netscape/Mozilla, Safari, Opera, and so on. This is no longer the case. The front end needs to work just as well over various browsers running on a multitude of operating systems. And the back end is subject to the same OS considerations as other server applications, with even more problems: you cannot rely upon a particular Web server being installed (let alone a particular application server). Even worse is depending upon specific backend technologies such as Windows Indexing Service or a particular vendor's RDBMS.

Taking all of this into consideration, we have the following grid of application varieties:

Looking at the grid, we can see that there are some tough choices to be made. Even when writing a desktop application, if that application requires communications with a server, the server backend will probably need to be written in a cross platform manner, even if the client portion is strictly Windows only. With Web applications, the choice becomes even more tricky: you have to account for a wide variety of Web browsers on the front end, while simultaneously writing the backend components to be used in many different environments.

It is also very difficult to choose between the headaches and hassles of writing cross platform code versus non-portable code. With Web applications, if you follow my "running water Web design" theory (to be discussed in an upcoming blog), your only cross platform concerns occur on the backend, reducing the number of platform combinations that you need to support. If you are writing a desktop application, your choices are really limited to the following development strategies:

* Java

* A cross platform language such as C/C++ to be compiled into libraries and wrapped in a platform-specific front end such as the Windows API, X, .Net, etc.
* Something such as Tk which allows cross platform GUI design

On the backend, you need to decide how much advantage leveraging server specific functions can give you as opposed to writing generic code in Java, C# (using Mono), C/C++, Perl, or some other highly portable language.

Unless you are willing to sacrifice potential sales, you are really forced to be writing your code in a cross platform manner. As the market shows, most desktop application developers are willing to give up about 10% of their sales in order to target Windows systems; only the truly large projects (or popular OSS projects) get aimed at multiple platforms on the desktop. The backend story is much murkier. If you have a high profit/low volume sales strategy, targeting a particular OS or combination of OS, Web server, and application server is a viable strategy. You cannot ignore cross platform code if you are shooting for a low profit/high volume sales strategy, particularly in a commodity market.

J.Ja

As a few others have mentioned, the problem is not the methodology used for building the code, it is the objectives of the managers directing the development effort.  If the people at the top perceive that they will be at serious risk if a bad version gets out the door, then the emphasis will be on bug-free products.  The current evidence is the turning of Microsoft.  It remains to be seen whether anyone else in the industry will feel similarly threatened by a market perception of bad quality product.  When the marketing department or the internal user decides that new features are more important than quality, the pressure on the development team will be to produce to timelines, not to quality measures.  Organizational controls are all based on a simple idea.  Do what we want and we will reward you, do what we don't want and we will reward your successor.  The technical term for a developer, tester, manager, architect or business analyst who bucks the system is 'unemployed'.

Waterfall, RAD, or Whatchamajigit Development Paradigm can all produce good code or bad code, and usually a combination of the two.  A development team which schedules for design and code reviews, walkthroughs, test plan development and execution, configuration management, release management and all of the other useful quality processes, and then is overridden by project management to discard the 'unnecessary' parts in order to meet deadlines, reduce the development budget, fit into the cost model, and make up for other project delays, cannot help but get poor results.  It doesn't do developers any good to have a development methodology which is routinely overridden or ignored outright, but far too many companies have a wonderfully robust methodology for display purposes only.

The idea behind too many methodologies, certifications, CMM guidelines and other gimmicks is that if you have some sort of process written down, then you cannot blame anyone for the quality of the end product.  Processes and procedures are worthless if they are ignored in the real world of deadlines and budgets, or if they are blindly applied without thought.  Which methodology you choose to abuse is inconsequential.

In my most recent blog, I took a look at whether or not you may need to consider writing your software in a cross platform portable fashion. In this blog, I will examine the various options for writing cross platform portable code. There are two ends to writing cross platform code: the interface, and the processing portion of the code. In a pure desktop application, they are contained within the same set of code. The front end is almost always some sort of GUI, or occasionally a command line. A server application tends to take input from a network socket, and produces output to another network socket, a database, a file, or a combination of the three. In a client/server situation (including dynamic Web sites), the front end communicates with a backend server application (ultimately via a network socket, but sometimes not directly) which in turn produces output which arrives over a network socket (again, not always directly).

No matter what type of application you are writing, you only have a few options in terms of how to go about writing cross platform code. Your first choice is to separate your presentation code from your logic, write the presentation in platform-specific code, and reference a library that you wrote to be portable. Your other choice is to use a language and interface toolkit that is portable.

Whichever route you take, part of your program will need to be written in a portable language. This is where the truly tough decisions come into play. Look back on what languages I consider to be highly portable: Perl and standard C/C++. These languages have their roots in the UNIX world, where portability is a concern. Perl has the advantage of being a high level, interpreted language; the language does not allow you to deal with anything that may be system-specific, and the interpreter handles any incompatibilities for you. Cross platform compatible C/C++, unfortunately, is rather limited in what you can and cannot do without delving into area where you will break its portability. Perl is a very good language to be working with, in and of itself, but it does not have the support of application servers like Java or the .Net languages have. C/C++ is just downright not nice to work with. Both Perl and C/C++ completely blow Java and .Net out of the water in terms of speed. I have not worked with Ruby, but knowing its properties, I think that it is fair to assume that it is also significantly faster to execute than Java, and has no cross platform compatibility problems either. There are also less popular (but still viable) alternatives such as Python, Lisp, O'Caml, and more.

What is a programmer to do?

I think that for a smaller project, it is possible to manage the incompatibilities amongst various JVM's and avoid application server specific code and use Java. For a larger project, I need to question this approach. For a large project, the current trend seems to be to just throw hardware at the speed issue. The common logic is, "hardware is cheaper than developers." I do not believe this to be true. Is it truly less expensive to maintain 500 servers than 400? The power bill alone is a small mint. Additionally, this logic assumes that development in Java or .Net is so significantly faster and easier than development in C/C++, Perl, Ruby, or whatever. When you are looking specifically at Web applications, Ruby on Rails and PHP also come into play. Personally, I am not a fan of PHP. I think development in it is downright miserable. But PHP does have better performance than Java, from what I have read. I cannot judge RoR at all, since I have not personally worked with it.

If you are working on a large application, it is impossible to ignore the performance factor. Hardware costs are ongoing; a well written application gets written once, and after the initial development, support and maintenance and new feature costs are a fraction of what the initial development costs were. It is my contention that for a large Web based application it is less expensive to spend a little bit more time, effort, and money working with Perl, Ruby, C/C++, or some other highly portable language in a CGI environment than to write slower, less efficient code in Java or .Net.

In addition, the advantages of Java or .Net quickly disappear when you need to start writing routines or functions outside of the core libraries. Perl, C/C++, and Lisp all have excellent support for a wide variety of scientific functions, AI, text parsing, and many other less mainstream applications. For a variety of factors, Java and .Net do not have these kinds of libraries, and the languages themselves do not lend themselves well to this type of development. If your application goes beyond basic C/R/U/D functionality, then your secret sauce is in the backend logic. Backend logic in Perl tends to take about 75% less lines of code than the equivalent logic in Java, VB, or C#, because Perl is a much more powerful language. This holds true for any functional programming language, in my experience. As soon as your program goes beyond gluing various libraries together with small dabs of code, the deficiencies of Java and .Net quickly become obvious. If your differentiating factor is going to come from advanced logic or high performance, Java or .Net just will not cut the mustard.

All said and done, writing cross platform coding is always a juggling act. There are no hard and fast rules to follow. How you decide to go about it is entirely dependent upon your needs. Java is a good choice for projects that do not need to scale much or require much business logic, but as the operational demands placed upon the software increase, and/or the amount of business logic increases, it is my belief that the additional effort of coding in Perl, C/C++, or a similar language more than pays for itself in terms of reduced hardware needs, faster execution, and less effort needed to write complex logic.

J.Ja

I read thru your post. I agree for becoming a good programmer, one should have undergone the rigors you have mentioned. This gives you the understanding of how, why and what is happening.

But, I believe the market has moved to a point where anybody who can drag and drop controls over a form is calling himself/herself a programmer. Is the use of IDE's made life easier ? Yes, to a point they have taken out of the pain out of the job. When you meet these people and they are not able to comprehend or appreciate the code or the design, people tend to get upset. 

The analogy to the same is the car making. Robots dish out cars in the assembly line. All looking same and doing the same function. But that does not mean, the market for custom cars or niche cars is over. The car makers making custom cars/niche cars command a premium for thier work. So, over the time, the tools(software) will become more intelligent to the point that they can generate the code themselves. All kind of applications that offer no more intelligence then what can be programmer will write2, will be rolled out by the tools. The market for the drag and drop programmers will vanish. But the market for custom, hand written software will still exist. The real programmers will command a premium and they will be able to carve out a niche for themselves.

How long the intelligent tools will take to appear? My guess, once the cost of the programmers goes beyond a certain level, the cost of creating such tools will become justified. Just like the oil vs the bio fuels.

I was in a bit of a funk today, wondering when I will get to start working on some truly challenging projects any time soon. For the last year or so, most of the programming projects I have been involved with boiled down to a lot of Excel macros or C/R/U/D operations in VB.Net. Mind you, there is no shame in this; it pays the bills. Some programmers are delighted with these kinds of projects, and there is nothing wrong with that. But I like a little more meat with my potatoes. So I started reflecting on some of my favorite programming projects from the past, to find out what they all had in common.

One great project I worked on (in terms of my enjoyment factor; personally I think the product was a dog and the marketplace did not want it) was writing filters for screen scraping software. The software was selective, it was made to extract information from various online databases. The job was a true challenge. In a nutshell, I would spend hours or even days analyzing the HTML code from search results, then crafting a king sized regex to parse the page. The debugging process involved changing maybe a few dozen characters in the regex per day and constant testing. What I really liked about this project was that it was highly creative; there was typically no one "right" regex to write, and the work was mostly searching for patterns. It was like being a treasure hunter. To this day, I still know my regex's inside and out, thanks to that job, and I have a tendency to use regex's for just about everything I can.

Another wonderful project I was on was a personal project for myself. I was writing my own shopping cart system. Like any good hacker (in the purest sense of the word), I was not satisfied to create an online shopping cart using existing components. I devised the most devilish project specs possible: no libraries at all (even standard ones), no database server, upload-and-go installation, and so on. At the end of the day, I ended up writing my own session handlers, logging methods, a flat file database system, a template engine, and emulating a significant portion of .htaccess's functionality (including emulating the status code 403 system), all in under 2,000 lines of code. This project taught me a lot (especially about RFC 2616) about good coding techniques, and making the most out of what I had on hand.

In college, I had the opportunity to use StarLogo to put a scientist's theories on ant behavior to the test. I closely modeled her theory in code, and then set it running. I also prepared a full post-mortem of the project. This project was a ton of fun for me on many levels. It was a great experience to model theoretical behavior in code. Even better, I was using code in an abstract level to create visual patterns and data sets to be analyzed. At the end of the day, I determine that the theory I was testing had a serious flaw, and by adjusting the program to work around it, I was able to make a much needed correction to the theory.

What all of these projects have in common is my love for writing code that goes beyond the C/R/U/D process, and involves deep logic. They also all required a substantial amount of planning and thought before the first line of code was written. None of them could have been created by the "drag-n-drop" or "shake and bake" techniques that are filling the days of more and more programmers. Don't get me wrong, I am grateful that I no longer need to keep writing the same database access code into every project. But my passion for programming goes far beyond seeing a project reach an end user and getting a paycheck. I truly love problem solving and attention to details. I have written more "get data from the user, validate it, stick it in the database, and display it to another user" projects than I can count, but I barely remember any of them. Projects that involve less mainstream code techniques like heavy use of recursion, functional programming, pattern matching, those are the projects that I remember. If I did not love programming, I would not be doing it. And those are the projects that I love the best.

What are you favorite programming projects from the past?

J.Ja

George Ou just posted another blog regarding Net neutrality, and I felt that the time was right for me to weigh in on this issue as well. His point is valid; why is it reasonable for a content provider to block or grant access, based upon who carries the last mile, but the last mile is not allowed to block or grant access? It isn't reasonable at all. Even worse, let's say that ESPN is able to extort a dollar per thousand users from my ISP. I know that my ISP is not going to start offering "premium broadband," they are going to pass that cost onto me, regardless of whether or not I personally view ESPN's premium content only.

I am also quite disturbed at what this implies about Internet users. Web sites have been trying for years to derive revenue stream from memberships, micro payments on a per-view basis, and other techniques. Thanks to the availability of free alternatives, no Internet user is will to pay for content, regardless of how much better it may be than the free alternatives (example: Encyclopedia Britannica vs. Wikipedia). "Adult content," some specialist niche content sites, and content where the access comes with a paid membership in a non-Web entity (such as access to The Economist's Web site, provided with a subscription to the magazine) seems to be the only way a company can get a consumer to pony up cash. The fact that they need to extort the ISPs in order to generate revenues from content that cost them millions of dollars to generate is sad.

You can thank sites like YouTube for this situation, as far as I am concerned. YouTube is quickly becoming a giant, public Tivo. I own a television, but do not have cable TV. However, I am sure that any TV show I may want to watch is on YouTube right now for free, copyrights be damned. If I cannot get it through YouTube, I can find it on a Gnutella network or BitTorrent. So the content providers are getting ripped off. First, they are losing viewers on the TV channel to the Internet, which is OK with me. Then, when they try to compete by putting their content on the Internet, often ad supported, they get ripped off and that content (or their TV content) gets put up on YouTube or P2P'ed, so the company cannot even generate ad dollars on their Web site.

The most dangerous man in the world is a man with nothing left to lose, memories of what he used to have, and his back up against the wall. This is the position that the traditional content providers are finding themselves in. And they are getting more and more desperate. ESPN's attempted power play is not the move of a clear thinking, rational company. It smacks of desperation. It is like having a heated discussion with someone and they cannot budge you from your position, so they hit you in the head out of frustration.

I have a problem with "Net Neutrality" legislation on a number of levels, but the biggest one is that definitions are completely impossible. How do you define what is considered an ISP or even "the Internet"? Is my cell phone part of "the Internet" if I turn on my WAP browser? Is my home PC considered a content provider, simply because I have a static, public IP, a domain name pointing to it, and a BSD server on it doling out Web pages? Is an Internet anonymizing service considered a backbone provider because they act as a pass through for packets? If I follow the RFC for TCP/IP using carrier pigeons on the physical layer, are the "flying rats" New York now part of "the cloud" and subject to regulation? What if they refuse to fly to certain locations to deliver the packets?

Any good law needs to have clear, understandable definitions. Period. When grey area, loopholes, wiggle room, and fudge factor exist in legislature, the people who suffer are those who do not have the resources to hire lawyers to interpret and develop justifications. That means you and me. Look at tax laws. I have the pleasure of paying full income tax for my bracket with only minor deductions. But a major company pays no taxes, because they hired a lawyer to base them out of the Bahamas. When I try to do that, it is called a "tax shelter" and I enjoy the benefits of three free square meals a day courtesy of the prison system. When someone with a clever lawyer does it, they get a bonus and stock options. And someone with billions pays less of a percentage of their income in taxes than I do, thanks to the magic of deductions (unless the Alternate Minimum Tax kicks in).

"Net neutrality" legislation, thanks to the impossibility of ever writing it right, does nothing to protect you or me, or the small entrepreneur or anyone else, and will simply be sidestepped by any major, "evil" corporation. Furthermore, paying a premium for upgraded service is a time honored tradition. FedEx charges me more for next day delivery than for "best effort" delivery. Toll roads have no traffic lights. Airlines charge more for non-stop flight than for flights with layovers. And so on and so on.

Thanks, but no thanks. I would rather see a total lack of regulations here and let the market place sort this out. If ESPN (or any other content provider) wants to try to force ISPs to pay for their content, I suspect it will fail. And if ISPs try blocking or limiting access to certain content providers will not probably succeed either. Customer hate cable TV packages with a passion. If ISPs try to move to the same formula, they will fail too. The marketplace has already made it clear that as much as they want premium content online, they are unwilling to pay for it. The marketplace simply does not care where the originator of the charge is, they are too accustomed and happy with paying one, simple, ISP bill and having full, unlimited access to accept anything else.

J.Ja

I was playing Warcraft 40K: Dawn of War (correction 6/24/06: I mean, Warhammer 40K: Dawn of War) a few days ago, and noticed something intriguing about the victory screen's statistics: while numerically accurate, they presented an incredibly misleading picture of how the game played out. The victory screen would bold the number in each category that it deemed to be "best," but in reality, what they called the "best number" was not necessarily so. Numbers and graphs are a very powerful way of presenting information, and sometimes the only way of presenting information, but without the proper thought going into how the information is presented it is useless or even misleading.

What Warcraft 40K (correction 6/24/06: again, Warhammer 40K) did wrong, in this case, was to make assumptions about whether a high number was better or worse. For example, the player who lost the least number of units was deemed the "best" in the category. However, that player was knocked out of the game very quickly! In other words, they lost 10 units and I lost 80, but the 10 units they lost were every unit they had, and the 80 I lost was a small percentage of my overall unit count. To put it simply, the units lost count lack an inadequate baseline to be compared against.

Here are just a few examples of how the data you present to your users can be quite inaccurate in terms of what the consumers wants to know while being numerically accurate at the same time.

To use some more tangible, real world examples, imagine that you are viewing a sales report. You see that a particular sales person sold $500,000 worth of product. That sounds great, right? But what if the previous sales person for that area used to sell $2,000,000 worth of product in the same span of time? What if the competitor's product is selling $5,000,000 in the same amount of time? What if the other territories have twice the sales, but five times as many accounts? Again, without a proper comparison, the simple sales number of $500,000 is useless or misleading.

Another example of this is stock market indexes. If the DJIA drops 400 points in one day, is that nothing to worry about, or should you start hoarding food? Well, it all depends on what the percentage of the drop is from its starting value. If the starting value was 1,115 then it is time to head for the hills. But if the starting value was 10,516 then 400 points is a much smaller problem.

Even with a baseline presented, how do you know that the baseline is appropriate? Take market share, for example. What is the relevant way to measure market share? Is it by units sold, or by dollar amount sold? Let us compare two products. Product A has a price of $10 per unit, and Product B has a price of $5 per unit. If Product A sells 500 units, and Product B sells 750 units, then a unit-based market share shows Product A's market share as 40%. But a dollar amount market share shows Product A's market share as 57%. That can be some pretty useful math, being able to increase market share by 17%, simple by changing your definition of market share. But wait, let us add in one more piece of information: one unit of Product A has the efficacy of five units of Product B. In other words, you need to buy five times as much of Product B to do the job of one unit of Product A. Let us re-write our market share calculations in terms of "efficacy sold." Product A sold 500 units of efficacy, but Product B sold 250 units of efficacy. Now, Product A's market share based on simple efficacy sold is 66%. So, what really is the best way of calculating your market share in this situation? In this case, I would argue in favor of an efficacy-based market share.

Percentages are another dangerous area. "Percentages can be tricky?" you ask. Yes, they can be. The value of a percentage all depends upon the relevancy of the total that is being used for comparison. For example, if I say "I am almost 20," most people will assume that I am only a month or two away from my 20th birthday; after all, most people measure their distance from a certain age based on a 12 month time span. But what if I just turned 19 last month? By the standard measure of "almost" a certain age, I am only 8% of the way to the age of 20. But compared to how long I have lived, I am 95% of the way to the age of 20. Even a number as simple as a percentage can be misleading.

In the same vein, averages can paint a different picture of reality too. Averages have a way of reducing outliers to the status of irrelevant when the data set gets large enough. Imagine that you are managing a call center, and you see an average hold time of thirty seconds. That sounds good. Now, what if you later discovered (deep diving after numerous customer complaints) that what was really happening was that for most of the day, the hold time was zero seconds, but during peak hours, the hold time was actually five minutes. Now your average no longer looks so great; it disguised your need for additional workers doing peak hours.

Another dangerous data trap are time span comparisons. Take a toy store as an example here. If you compare Quarter 1 2006 sales to Quarter 4 2005 sales (current quarter vs. previous quarter) then you will be panicking every year because you will see a huge drop in sales. But if you compare Quarter 1 2006 to Quarter 1 2005, you are able to see a proper "apples-to-apples" comparison. On the other hand, for the sales for a non-cyclical product such as medication for chronic diseases, a current quarter vs. previous quarter comparison gives a more accurate snapshot of sales trends.

What can you, as a developer do about this?

Your goal is to serve your end users' need for accurate, reliable, and understandable information. The first thing you should be doing is to always explain the methodology used in language that the vast majority of your users can understand. You can use technical mathematics language if statisticians are going to be using your data, but you cannot do this if your data is headed for a major news Web site. Another thing you should do is to put yourself in the shoes of your end user, or ask them what they need to know. Learn about the numbers. Find out what the numbers actually represent, and the typical data trends for that data.

Typically, first order calculations such as a raw sales number are not very useful. Second order and third order (velocity and acceleration) calculations tend to provide much more data. It is one thing to say that a car is 800 feet down a quarter mile race track (first order number, distance). It is more useful to state that its current speed is 80 MPH. It may be even more useful to say that its acceleration is 9 ft/second2. And if you are the race car mechanic, you probably want a plot of the instantaneous velocity and acceleration throughout the race, to find out where the vehicle's performance needs to be turned for maximum race speed. Meanwhile, the fans just want to know the total time and trap speed.

This brings up the issue of graphical representations of data. We all like charts and graphs. My boss likes to joke with me and ask, "Can I get that as a pie chart?" In reality, many graphs are not as useful as they may appear. Like raw numbers, a graph without an appropriate baseline is not very useful. A chart of current sales figures for a year is good. The same chart showing the total market size and competitor's performance lets the viewer see if the trend is specific to their product, or the market as a whole. A line chart showing a second or third order derivative over time is excellent (for example: market share over time, change in market share over time, etc.).

You also need to decide how you are going to handle outliers. Are exceptional cases what your users are looking for? Or are exceptional cases to be discarded? Or maybe your users need you to remove the exceptions from the overall data set (so as to not mess up averages), but to point them out elsewhere. Again, you really need to work closely with your users.

There are all sorts of data traps for the unwary, and these are just a few of them. Be on the lookout, and do not be afraid to ask your users questions if you think something in the project spec does not make sense. Most users would rather have you say to them, "I know this is what you asked for, but I am looking at the raw data and I think we can devise a more useful metric" than to be using a product that gives them a misleading idea of the data due to a poor metric selection.

J.Ja

I have been spending a bit of time working on a small utility-type application. This application is designed to solve a problem that I have to deal with constantly, a problem that I am sure many other people have to deal with. It analyzes a delimited file, and determines the field types and maximum width of each field. All too frequently, a customer will dump a delimited file on me without providing any metadata (I consider myself lucky if I get field names, I often have to guess which field is which). To ensure accurate use of the data, it is crucial that I create a database table that is not too narrow, or contains any incorrect field types. On the other hand, making fields too wide or using more generic field types makes the database slow and uses storage inefficiently. After all, which is faster to perform a JOIN on, an integer column, or a varchar? How quickly will I fill up my storage space if I used 200 character width columns when I really only need 50 characters?

So I wrote the application in VB.Net, a quick and dirty language for a quick and dirty job. Sadly, the performance just outright stunk. Once again, my low expectations of .Net were met. My first thought was that Perl would have run through the test file (about 30 MB large) as fast as my hard drive could dish it out. So I was about to set out to rewrite the program in Perl, when I decided to see if maybe re-working the VB.Net code could yield any speed benefits.

With about five minutes worth of code editing, I reduced execution speed to approximately 1% of its former speed. In other words, execution speed is 100 times faster now. In fact, it now executes about as fast as I would expect equivalent Perl code to execute. I did not bother to rewrite it in Perl, because it is more than fast enough for my needs. Sadly, any performance numbers I would have gotten from it could not be published anyways, thanks to MSDN's license agreement. (Correction 6/29/06: after re-reviewing the MSDN EULA, this is incorrect. I may publish benchmarks on the .Net Framework, it is one of the few exceptions to the general "no benchmarks allowed" rules in the EULA)

What changes did I make to the VB.Net code to achieve this performance miracle?

I dumped as much of the object oriented code as I could.

Here is an example (clarification 6/26/06: this is not the real code, just some similar code for demonstration purposes):

"Proper" Object Oriented Code

For iRowCounter = 0 to cStringList.Length ? 1
      aSplitString = cStringList(iRowCounter).Split(sDelimiter.ToCharArray)
 
      For iColumnCounter = 0 to aFieldNames.Length ? 1
            If a.SplitString(iColumnCounter).Empty Then
                  sOutput = sOutput & "Empty"
            Else
                  sOutput = sOutput & aSplitString(iColumnCounter).Length.ToString
            End If
 
            sOutput = sOutput & " "
      Next
 
            sOutput = sOutput & vbNewLine
Next
 

Procedural Style Code

iNumberOfRows = cStringList.Length ? 1
iNumberOfColumns = aFieldNames.Length ? 1
cDelimiters = sDelimiter.ToCharArray
 
For iRowCounter = 0 to iNumberOfRows
      aSplitString = cStringList(iRowCounter).Split(cDelimiters)
 
      For iColumnCounter = 0 to iNumberOfColumns
            If a.SplitString(iColumnCounter).Empty Then
                  sOutput = sOutput & "Empty"
            Else
                  sOutput = sOutput & aSplitString(iColumnCounter).Length.ToString
            End If
 
            sOutput = sOutput & " "
      Next
 
            sOutput = sOutput & vbNewLine
Next

These examples are quite short, but you can see the difference. By making these type of small changes, I took execution speed from about 7.5 minutes to under 10 seconds.

Unfortunately, as an example of clean OO code, the program now looks a little too much like procedural code for the comfort of some people. After all, I am using an OO language, right? So why use all of that procedural code? Well, I think the results speak for themselves.

This example highlights an interesting thing: the less elegant code wins out. Is it easier to maintain code like this? Absolutely not. I have always said that each line of code is a potential point of failure; reduce your lines of code, reduce your errors and debugging time. Furthermore, what is the point of using an OO language, if you are going to strip away much of the OO aspects of it as soon as you can? And why take up memory duplicating all of that information when it is already available in the objects?

I have read through a lot of source code in my life, and many programmers do indeed choose to use the OO style as much as possible. The code does look better, and it retains an element of elegance. But as my experience, time and time again has shown, the OO style executes extremely slowly. The reason lies in the very nature of OO code. For each of those row length lookups, the system is not caching the length, even if it has not changed. Even if the length is stored as a property (as opposed to being), that information still has to be dug up. This means burrowing through the object tree to get at the underlying data, and then bubbling it back up until it hits that For condition. In the procedural style, that information is already on hand in the final type needed.

As a rule of thumb, if a property (or method return) will be used more than once before it changes, it is nearly always faster to store that information to a temporary variable. Of course, there are exceptions to this. If you are going to be working with an insane amount of temporary variables at once, the memory used to store them may go up so quickly that you are burying the needle on RAM usage. An example of that is if the items in the property are absolutely huge (say, 100 MB of data in each one) or if the application is highly multithreaded. In a situation like that, you need to decide between memory usage and CPU utilization. But for your average utility loops, it is a sure bet that storing those values in temporary space (especially things used in a condition) is going to reap huge performance benefits.

So choose your poison: ugly, harder to maintain, less elegant procedural-esque code, or slower to execute, pure OO code. Regular readers know my emphasis on end user satisfaction; slow code makes no one happy. I prefer to break the OO paradigm in order to reap speed benefits any day of the week.

J.Ja

I must admit, the Callisto release of Eclipse has intrigued me, enough to sit on their Web site waiting to download it. For days (probably much longer), the site has had a clock counting down to the moment of release: 6/30/2006 10:00 EST (45 minutes ago). The site has replaced the clock with a "Callisto is coming soon" message. Maybe I am just being silly, but how hard is it to simply post a link, when the thing has supposedly been ready for some time? (update 6/30/06 13:00 EST: the "Calliston is coming soon" message now has added onto it: "no really, we promise." Meanwhile there is already an article linked to on their Web site about how Callisto has shipped on time; I recognize that a few hour slip is nothing, but I just find this thing funny).

What is even funnier, the front page of their site has a link to an article about how Eclipse/Callisto is an example of how large, OSS projects are able to hit deadlines. Whoops!

J.Ja

My most recent post ("How I Improved Execution Speed By 100 Times In 5 Minutes") generated some tremendous feedback. One of the things that was touched upon a number of times was the possibility of the compiler or managed code environment caching values used in loops, or automagically transferring the value to an invariant, as I did in my sample code. One poster, Kevmeister, made some great points, which I would like to discuss in more detail.

Here is what Kevmeister says regarding the compiler performing these kinds of optimizations:

The chief problem is that "properties" might be public member variables or they might be accessor functions. How in fact does a compiler actually determine that the expression it is dealing with is in fact constant? If it is an accessor function how does it know what the behaviour of the accessor function is in computing the value of that property? Even if it is a public member variable, how does it know that the some other part of the loop will not cause a side-effect resulting in that public member variable being modified. I think an optimiser needs to be thoroughly clever to determine these kinds of things, if it is possible at all.

Well said indeed. The compiler really has little way of knowing for sure that the value has not changed. Even if the compiler tore apart the source code to the object used to generate the property, it has little way of deciding what could be cached and what could not be cached, unless it was able to track the value back to a hardcoded number or something declared as a constant.

Another issue that Kevmeister did not mention is multithreading. What happens if the looping code is in one thread, and another thread does something that would change that value? It is bad enough in a For loop, where the idea of the check value suddenly jumping around could be quite dangerous, but that may indeed be the desired behavior in a While or Until loop. Some people will put a loop in one thread, checking a value that gets changed by a separate thread. So in a multithreaded situation, caching the value is not very helpful.

Another unmentioned idea for caching these variable would be for the compiler to attach a dirty bit to some values. If the value itself (or any underlying data) were to every change, the dirty bit would be set. It would barely impact performance. Any variable can subscribe to the dirty bit of another variable, allowing a bubble up effect. This idea has some merit, but although the overhead on an individual level is rather small in most cases, on a mass scale, it could be devastating to performance. And it still does not address the issue of what happens when the value to be checked is derived rather than stated. Consider the following loop statement (I just made up a timer class here, please ignore any similarity to an actual object):

while dtTimeToStop >= timerProcessTimer.CurrentRunTime()

If the compiler attempts to cache CurrentRunTime(), there will be a severe performance ding. Even if the compiler is updating its cached value periodically, the cure is worse than the disease.

Kevmeister also pointed out that something within the loop may change that value. What he does not say, and blow the whole idea out of the water, is what if accessing the value itself changes it? Here is a fake class:

Public Class Class1
    Private intWhenToStop As Integer
    ReadOnly Property WhenToStop() As Integer        Get            If intWhenToStop > 0 Then                intWhenToStop -= 1            End If
            Return intWhenToStop        End Get
    End Property
    Sub KeepGoingUntilDone()
        Dim intCounter As Integer
        For intCounter = 0 To WhenToStop            'Do Stuff        Next    End Sub
    Public Sub New()        intWhenToStop = 500
    End Sub
End Class

See the problems with caching or otherwise touching WhenToStop() by anything other than the programmer's code? Yes, it is a rather bizarre example. But I have seen situations where this kind of code does indeed make the most sense (especially before the first few pots of coffee).

So the idea of the compiler or managed code environment automatically handling these situations is really not such a great one.

But that still does not let the compiler off the hook completely. It is my belief that just as the compiler throws out warnings and errors, it should also throw out suggestions and hints as well. Visual Studio is very intelligent about many things. I could see the IntelliSense system being beefed up to give some performance tips. Not many programmers get taught any particular language, and far too few programmers read about programming outside of reference books for syntax lookup. With that in mind, I think that while the compiler automatically handling these types of issues is not a good idea, the idea of the compiler making some helpful suggestions is not a bad one at all.

J.Ja

I recently attended training for Cognos Metadata Modeling. This experience highlighted for me one of the key difficulties in becoming a top-notch IT professional. Maybe it is a result of the IT profession still being a relatively new industry, or the demands of students, or something else, but training and documentation tend to almost exclusively address the how of performing a task, without explaining the why.

For example, the Cognos training manuals showed us how, step-by-step, mouse click-by-keystroke, to go through the motions of transforming the source data into a usable data model, suitable for report creators and other data consumers. What the manual did not spend much time on at all was why we were making the decisions that we were making in the first place. Luckily, the instructor (Brian) did a top flight job explaining this as we went along, as well as answering all questions about that aspect of the material. The fact remains, if it were not for Brian, I would not have learned anything that F1 could not have taught me. In short, the curriculum as formally laid out was a five day course in how to click, copy/paste, and drag/drop.

This training course was not an isolated incident. During my brief stint as a Computer Science major (regular readers may remember that I double majored in "can't-get-a-job-ology"), the courses were extremely concentrated upon theory, the why of what programmers do. The coursework at my college was extremely why oriented. The vast majority of the students were constantly complaining about this. They wanted to be taught how to program in a particular language. Indeed, it was assumed that you knew the language used for the class, and it was up to the student to learn it on their own. For example, CS 111: Introduction to Programming used Pascal to teach the basics such as variables, loops, control structures, and so on. The next course (CS 112: Data Structures) used C, with no explanation of the C language. I had the good fortune of looking at the syllabus in advance and learning C over my summer vacation. The bulk of the students did not do this, and were tripping over C, which was merely being used as a tool to teach data structures.

In retrospect, I wish that I had stuck with the Computer Science program there on some levels. It was the only time that I was offered a chance to really learn Computer Science as a science, as opposed to a vocation. That is the difference between a why oriented education and a how oriented education. Learning the why means that you understand the underlying theories and principles, without necessarily being taught the implementation end of the process. The how based education teaches you to implement something, but you really do not learn how to make the architectural decisions that you need to craft quality code.

Unfortunately, as my college experience showed, this is exactly what students want. Schools such as DeVry and Chubb (I have talked to a number of graduates from their programs, and inspected the coursework and books) are how oriented. They teach students to lay bricks, but not to design the building. The Cognos training was the same way. We were being taught how to have the software do what we wanted it to do, but not how to know what we needed it to do. The end result? "Shake and bake" programmers.

There is nothing necessarily wrong with "shake and bake" programmers; someone needs to do the grunt work. But at the end of the day, it is the grunt work which often has the biggest impact on end user satisfaction. If the person writing the code that parses the data file has a poor understanding of things such as the impact of various file reading techniques on performance, the program will run slowly. If the person designing the interface has no knowledge of usability theory, the application will be difficult to use. The programmer ignorant of proper optimization of SQL statements is going to be writing code that makes the server beg for mercy. And so it goes.

Language documentation is frequently not much better. It serves as a syntax reference, and not much else. Rarely does the documentation for a language offer any suggestions or tips as to where, when, and why you would choose to use one technique as opposed to something else. The Perl documentation is the only significant exception to this that I am aware of. As a result, people using the documentation as their reference rarely learn that there may be a better or easier way of accomplishing a task. For example, a programmer coming from a language without a built-in, optimized file slurper object may simply try to rewrite their standard file slurping code in the new language, without knowing that there is an easier, more optimized method available to them. This is where the Perl documentation shines; the FAQ's are complete why oriented. They are laid out like, "I am trying to accomplish XYZ, what is the best way of doing it?" The answers show not just the best code, but often have different variations with an explanation of which one is best in which situations, and why. More language references need this kind of documentation, as opposed to the current system which is more or less a syntax reference and an explanation of what each parameter is and what the output is.

Over the years, I have held onto my passion for programming. As a result, I am always reading things that are at least indirectly tied to software theory. Even when I am reading something that I do not think is related to computers, I start to wonder how the can be applied to software design and development. I spend time experimenting with languages that are not popular, to see how the methods they use can be applied to my job, and evaluate the suitability for use in projects. And so forth. The why of software is what interests me, not the how.

A properly designed architectural document for a program is the essential blueprint for implementing a software project. Without a good understanding of the why of code, it is very difficult to construct the best possible how. This is why I think that a major shift has to occur in the attitude towards training and documentation.

J.Ja

This is the introduction to a two part series.

Bill Gates recently wrote about the overload of information that people face in one of Microsoft's June 27th 2006 Executive Email ("Bill Gates on the Unified Communications Revolution"). What Mr. Gates understands, and most developers do not seem to understand, is that the proliferation of various electronic communications methods, while resulting in an increase of speed of communications, has a point of declining returns to the point where productivity is harmed. Part of the problem is on the users' lack of bit literacy. And much larger share of the blame falls on the shoulders of programmers and companies putting out software and devices which refuse to interact with anything else. While there are legitimate technical and business reasons for creating these isolated devices and systems, the solutions end up only addressing the users' how and rarely their why.

This series of articles will examine the different types of information overload, and what developers can do to address the problems.

PART 1 ? Too Many Options

Too Many "Inboxes"

Users receive messages through a ton of different methods. To make matters worse, many of these methods require manual intervention. Once data is on the wire, it should find its way to the best destination based upon the receiver's preferences and availability. For example, if I am away from my desk, email should come to my cell phone (and automatically synced with my desktop mail client) and MSN or Skype VoIP contact attempts should ring my cell phone. Once I return to my desk, they should come to my desktop, and through a central piece of software. The average user now is running an email program, checking Webmail periodically, has multiple email clients, three or four phone numbers, and even more inboxes, when they really just need one.

Too Many Outboxes

There are far too many ways to communicate with people. For most of the people in my network, I have the option of using the phone, IM, and email. Some users add in various Web forums, walled garden outboxes, faxes, and even more ways of sending a message. What users really need and want is one method of sending voice, text, and data from their desk, and another way of doing it mobile, with the best outbound path being chosen based upon availability of the data path and the receiver's online status.

Too Many Sign-ons

We all have way too many logins to manage. Over the course of a day, most people have three voicemail passwords (home, work, and cell phones), 2 or more IM usernames, a webmail username for personal email, a work login (and way too many companies have different logins for each system as opposed to just all using the same username/password database), and so on and so on. It is simply unmanageable. A number of years ago, Microsoft pushed Passport, and ABM pushed the Liberty Alliance. The only place I see Passport is on Micrsoft properties, and I do not even know who uses Liberty Alliance. Because of username/password overload, users do not access all of the routes that belong to them.

PART 2 ? Data Difficulties

Walled Gardens

Far too many content and communications systems are walled gardens. What this means is that while working within the system is great, it has no way of communicating with the outside world. MySpace is the biggest example of this. The blogs you post there cannot be accessed via RSS, the songs, videos, and images you post cannot be accessed outside of their system through URLs, and you are unable to send messages outside (or into) MySpace. In a nutshell, it is like having a cell phone that is unable to call any phone except other cell phones with that provider. What this does is force users to spend far too much time accessing and using far too many devices and systems in order to work with everyone in their network.

Lack of Storage Unification

Even for systems that are not walled gardens, there is a complete lack of storage unification. For example, a message sent through Yahoo! Mail does not appear in my Outlook "Sent Items" folder. A file stored in Flikr is completely cut off from my local file system; a change in one requires manual intervention on the other. As a result, users end up with a large number of places for data to be stored and hidden. Users need less storage areas, not more, that seamlessly sync and interact with each other. Even more importantly, simple data such as basic word processor documents need an easy way to appear on Web sites.

The Metadata Problem

Metadata is great. Not only does it provide useful information, but it is a great aid in the data searching process. Unfortunately, very few systems automatically generate metadata. While they offer systems for the manual creation of metadata, rare indeed is the user who tags, marks up, or enters metadata. More systems need to find ways of properly providing the metadata automagically.

Poor Search and Lookup

Most search systems offer simple text search, and sometimes allow the user to refine their search based upon metadata. This is just not enough. For example, if you remember that someone emailed you their phone number, which is better? Searching for all emails from that user, or being able to use a pre-defined regular expression (such as "find all emails from John that contain a phone number")? More systems need to support regular expressions in a way that the average user can understand, and pre-define appropriate searches. In addition, more systems need to search through automagically generated metadata.

Data Format Chaos

Right now, all of our data is in far too many formats. Format incompatibility and conflicts are the kinds of issues that users simply do not care about, only vendors. The user does not care if they are using ODF or OpenXML, they only notice the format if it gives them a problem. Documents need to be able to be seamlessly shifted to the Web or from the Web to an environment where they can be edited. Vendors need to learn that the value they add to the users does not come from a file format, but comes from their business logic. Furthermore, as code increasingly gets developed within managed code with massive standard libraries (Java, .Net) or uses common open source libraries, the urge to develop custom file formats is reduced.

J.Ja

This is the first of a two part series.

Information overload is a condition that many, if not most users face on a daily basis. Email, phone calls, instant messages; this list is endless. While end users are generally not very skilled in bit literacy, developers shoulder much of the blame and it is up to the developer to make sure that their applications and devices actually help the user, not hinder the user. In this initial part of my "Information Overload" series, I will show some examples of how developers can help resolve this problem.

Too Many "Inboxes"

Users receive messages through a ton of different methods. To make matters worse, many of these methods require manual intervention. Once data is on the wire, it should find its way to the best destination based upon the receiver's preferences and availability. For example, if I am away from my desk, email should come to my cell phone (and automatically synced with my desktop mail client) and MSN or Skype VoIP contact attempts should ring my cell phone. Once I return to my desk, they should come to my desktop, and through a central piece of software. The average user now is running an email program, checking Webmail periodically, has multiple email clients, three or four phone numbers, and even more inboxes, when they really just need one.

The easiest way to "deal" with this problem is to simply ignore it. It seems like every developer imagines that their application will become the users' new "home" throughout the day. This is an example of "developer hubris." Chances are, your user will not "live" in your application. In reality, your users have Outlook (or some equivalent) open all day long, and an IM application or too. The last thing your users need is yet one more application sitting in the system tray, popping up "piece of toast" every thirty seconds demanding attention or notifying them of a new message. On the other hand, you do not want them to only check your application once every few hours to see if they need to do anything in it.

The best way to resolve this problem is to acknowledge that your software is probably not the most important thing in the users' lives, and to work with their existing systems. Instead of delivering messages and notifications to your own data store, why not deliver it to Outlook? If the users do not use Outlook, send the messages via RSS or email. This way, the users do not need to monitor yet one more application. In addition, the users are already familiar with their existing applications, and can use that application to organize your information in the manner that suits them best. Another advantage is that it is actually less work to do this for you. If you have to choose between reinventing the wheel (data storage, searching, organizational capabilities, filtering, etc.) and simply working to deliver the information via a standard delivery mechanism, which would you rather do?

Too Many Outboxes

There are far too many ways to communicate with people. For most of the people in my network, I have the option of using the phone, IM, and email. Some users add in various Web forums, walled garden outboxes, faxes, and even more ways of sending a message. What users really need and want is one method of sending voice, text, and data from their desk, and another way of doing it mobile, with the best outbound path being chosen based upon availability of the data path and the receiver's online status.

This is really the same problem as the "Too Many Inboxes" issue, just in reverse. Why not offer the users a way to communicate from their existing application, using your application as a pass through? For example, a document management (or online content management system) that presents itself as a Word plug-in is a heck of a lot easier for a user to use regularly than a system where the user needs to take active action. A system that lets a user work with it from within Outlook to perform online scheduling (or send an email to a special email address) is going to get a lot more use than something that lives in its own space.

The vast majority of the overwhelmed are using Microsoft Office. Modern versions of Office are capable of being pluged-into via COM. The .Net Framework makes it extremely easy and attractive to target an Office application as the wrapper for your code. There is a reason why so many users use Excel and Word to do things that they "shouldn't" do; these applications are familiar and comfortable to them. Instead of writing yet another application that gets dusty and grows mold through disuse, why not leverage the users' existing comfort zones and knowledge base and work within Office?

Too Many Sign-ons

We all have way too many logins to manage. Over the course of a day, most people have three voicemail passwords (home, work, and cell phones), 2 or more IM usernames, a webmail username for personal email, a work login (and way too many companies have different logins for each system as opposed to just all using the same username/password database), and so on and so on. It is simply unmanageable. A number of years ago, Microsoft pushed Passport, and ABM pushed the Liberty Alliance. The only place I see Passport is on Microsoft properties, and I do not even know who uses Liberty Alliance. Because of username/password overload, users do not access all of the routes that belong to them.

Passport seems to involve giving Microsoft money, and who knows what the case is with Liberty Alliance? One alternative is to leverage the users' existing authentication scheme, whether it be LDAP, Active Directory, or something else. The Higgins Trust Framework seems like a great idea as well (essentially on open source version of Passport or Liberty Alliance), and I truly hope it goes somewhere. Unfortunately, most services will not let you authenticate against their servers, at least not the common ones. For example, Yahoo! will not let you authenticate against a Yahoo! ID without some sort of partnership. I think this is a real shame. Most people have a Hotmail or Yahoo! username (Google is pushing their users to get Google IDs as well, but their market penetration is startlingly low in non-search market segments).

However, systems designed for internal usage have no excuse whatsoever. If you are writing a client/server application, you should not ever need to ask for the user to authenticate, unless they need to escalate their login to a more privileged level. Except in a few very rare occasions, the fact that a user is logged into their desktop should be good enough for your application. Reduce the logins, and increase the happiness.

J.Ja

This is the second in a two part series.

In the first part of this series ("Information Overload - Part I: Too Many Options"), I discussed how developers create too many options and data paths for users, creating a situation information overload. In this part, I will review how issues with the data itself contribute to the problem, and how developers can help resolve it.

Walled Gardens

Far too many content and communications systems are walled gardens. What this means is that while working within the system is great, it has no way of communicating with the outside world. MySpace is the biggest example of this. The blogs you post there cannot be accessed via RSS, the songs, videos, and images you post cannot be accessed outside of their system through URLs, and you are unable to send messages outside (or into) MySpace. In a nutshell, it is like having a cell phone that is unable to call any phone except other cell phones with that provider. What this does is force users to spend far too much time accessing and using far too many devices and systems in order to work with everyone in their network.

This is a bad case of developer hubris. Like the issues in Part I, too many developers seem to have the idea that their users will "live" in their application, on their device, or on their Web site. This is completely false. MySpace and Outlook are the only "homes" for people. People really do not need another. Outlook is hardly a "walled garden." MySpace sadly is.

The rational behind a "walled garden" is that you want your users to spend as much time there as possible. It is a problem particular to Web sites that earn money via advertising. MySpace is the perfect example, as stated earlier. If you know enough people on MySpace, you have no almost no need for an email account, a Web site, an RSS reader, or many other applications (or Web applications). However, this comes at a price: you are cut off from anyone not on MySpace.

Users really do not like "walled gardens," they only tolerate them, and only if there is enough in the garden to be satisfactory. As soon as a more open or more attractive option becomes available, the users flee. Look at the failure of every single portal in the late 90's. AOL is another good example of what happens to "walled gardens." Its closed system could not survive the Internet.

At the end of the day, try to avoid creating a "walled garden." Sometimes business dictates that you must, but whenever possible, hook into known, open standards such as SMTP, NNTP, POP3, LDAP, IMAP, etc. The more you allow your system to interact with others, the more likely you are to gain and retain users.

Lack of Storage Unification

Even for systems that are not walled gardens, there is a complete lack of storage unification. For example, a message sent through Yahoo! Mail does not appear in my Outlook "Sent Items" folder. A file stored in Flikr is completely cut off from my local file system; a change in one requires manual intervention on the other. As a result, users end up with a large number of places for data to be stored and hidden. Users need less storage areas, not more, that seamlessly sync and interact with each other. Even more importantly, simple data such as basic word processor documents need an easy way to appear on Web sites.

Where systems and users are headed towards is a new version of thin client computing, but unlike traditional thin client computing, the data is stored all over the place, not on a central server. Because of this, each system has its own rules for data storage and retrieval, and there are higher barriers between them. In the thin client days (as well as the client/server days), you could count on each piece of software having access to the same data, either through the file system or a database connection. Now, with the data locked behind a registration screen and no standard APIs for over-the-Web authentication (do you use HTTP authentication headers? Pass usernames/passwords through GET or POST variables? What are the fields called?), it is much harder for various systems to read or write to each other. At best, one site will provide you with an HTML snippet (YouTube style) to post your content from one site onto another site.

Of course, this is really not different from what the Web was originally conceived to do: allow data from many different sources be put together into one place. But there is now a business model attached to it. A company cannot make a living if you link directly to an image or movie on their site; they need to wrap it in their packaging. Again, look at YouTube. They could provide you with code to allow their movies to appear in the user's media player of choice within the page. Instead, they put a wrapper around it that encourages you to visit their site. In the future, will they be splicing their ads into it as well? Google Maps followed this path. They waited until a critical mass of Web sites were using their API, then suddenly started plastering their ads on those maps.

This would not be as much of a problem, except that data is now scattered all over the Web, with different logins and different methods of retrieving or updating it. This leads back to "too many options." Even with a self-contained system, like Salesforce.com, the result is chaos. Some data is within the SOA provider's cloud, some of it is stored within the LAN. Getting applications (and users) to work with two separate pieces of data like this is not easy at all.

Unfortunately, there is no easy fix for this at this time. The only hope is for a set of standards to be developed which help resolve this situation. Anything short of being able to access a piece of data just like network storage is not good enough.

The Metadata Problem

Metadata is great. Not only does it provide useful information, but it is a great aid in the data searching process. Unfortunately, very few systems automatically generate metadata. While they offer systems for the manual creation of metadata, rare indeed is the user who tags, marks up, or enters metadata. More systems need to find ways of properly providing the metadata automagically.

You, as a developer, need to find a way to determine the applicable metadata within your user's data, and make it available easily and openly. Plain text search just is not good enough either. There needs to be relevant, searchable metadata. It is a shame that WinFS is dead; I loved the idea. In general, file systems do not store any truly relevant metadata.

Poor Search and Lookup

Most search systems offer simple text search, and sometimes allow the user to refine their search based upon metadata. This is just not enough. For example, if you remember that someone emailed you their phone number, which is better? Searching for all emails from that user, or being able to use a pre-defined regular expression (such as "find all emails from John that contain a phone number")? More systems need to support regular expressions in a way that the average user can understand, and pre-define appropriate searches. In addition, more systems need to search through automagically generated metadata.

This is a tall order. One thing I would think would be helpful is if languages (or regex libraries) had some standard regex's built in. It would also be great if SQL supported regex's, for those applications that query against databases. Languages also need to do a better job at supporting regex's. OOP Languages in particular make regex's too much work. I like that Perl brought regex's to the level of string operators. One reason why Perl is "so good" at regex's compared to other languages (despite other languages typically using Perl-like regex syntax) is because a regex is a string operator.

That aside, if you want to support truly useful search/replace in your software, not only do you need to provide regex support, but it has to be done in a user friendly way. Pre-built regex's (email addresses, phone numbers, mailing addresses, other relevant data) helps, but the syntax itself needs to be friendlier too, maybe something along the lines of Office string formatting codes.

Data Format Chaos

Right now, all of our data is in far too many formats. Format incompatibility and conflicts are the kinds of issues that users simply do not care about, only vendors. The user does not care if they are using ODF or OpenXML, they only notice the format if it gives them a problem. Documents need to be able to be seamlessly shifted to the Web or from the Web to an environment where they can be edited. Vendors need to learn that the value they add to the users does not come from a file format, but comes from their business logic. Furthermore, as code increasingly gets developed within managed code with massive standard libraries (Java, .Net) or uses common open source libraries, the urge to develop custom file formats is reduced.

Pick a common file format, whether it is ODF, HTML, XML, OpenXML, CSV, or whatever is applicable to your data needs, and support it natively and seamlessly. This file format should be one that other applications in the same market, as well as related markets support. A good example of a data format that "refuses to die" is the dBase format. Despite the fact that no one has used dBase itself in I do not know how long, the format still is around because dBase was popular enough that everyone allowed importing and exporting from it. As a result, everyone retains that dBase compatibility, and the format itself has developed a second life as a common database data sharing format. Maybe if OpenXML is as open as ODF (or nearly as open) it will achieve the same status. HTML is at that point already, but unfortunately it is not well suited for data transportation, only data display. XML in and of itself is not a very good format for data exchange, because it different vendors do not use the same schema (or interpret the schema in different ways) it becomes useless. XML really relies upon certain standards (such as RSS) to become a useful format.

If you can, get together with other vendors to work out a common data format. Doing business based upon data format lock-in, as opposed to competing on actual feature sets and performance hurts the users. Look at the trouble Microsoft has had over the years because of their various attempts to lock users in based on formats. It does nothing but generate bad press and bad feelings. Reducing the data format mess is a joint effort, and you cannot do it alone unless you are working in a well established market with well established standards.

J.Ja

I think that it is time to fully explain my philosophy of Web site design and development. I call this theory "running water Web design," because it follows the idea of running water: smooth flowing, taking the path of least resistance, and simplicity. This philosophy does not force anything upon the user, adapts to their needs and environment, and degrades gracefully. It makes Web sites that endure the inevitable change, while delivering your content to the end user in a manner that makes it as easy as possible for the end user to consume it, with an emphasis on usability.

To distill the running water Web design philosophy into one simple sentence: make no assumptions about your user, their technical savvy, or their technical capabilities, while at the same time never forcing them to do things your way instead of their way.

The first step in adopting this philosophy is to simply give up on the idea of being able to precisely control the way your Web site appears to the end users. Once you surrender to the reality that users have a wide variety of screen resolutions, operating systems, color depths, and needs, you give yourself the freedom to design great looking, highly usable Web sites.

Use CSS as much as possible, but only use CSS that all browsers will accept and only use CSS that the site will properly work without. CSS is great because it allows you to separate your presentation as much as possible. This also translates to cleaner server side code, because design changes require minimal editing of templates or of code that generates HTML. But CSS has a dangerous side as well, which is that despite the perpetual call for HTML standard compliance, different browsers interpret much of it differently, or have their own specific extensions to it. This means that you should be very careful with your CSS. Unless you are designing an intranet, where you are guaranteed that users have a particular browser with particular settings, you cannot count on your CSS working right if you rely upon a particular browser's interpretation of it. Some users will disable CSS entirely, or have their own browser settings establish that override or ignore your CSS, so make sure that your site is still usable without the CSS as well.

All font sizing should be proportional; any non-standard sizes (such as "smaller") should be percentages of the base size, and the base size should be set in ems, not pixels or points, in order to ensure correct cross platform sizing. Different platforms interpret "pixel" differently, in terms of onscreen DPI. The end result is that a font defined as "14 px" will appear differently on different screens (Mac OSX is one example of an OS that does not follow the norm). Font points are also slightly different from OS to OS, and should be avoided as well. Ems are the only unit of measurement that you can count on being the same on every user's system. Never forget: anything less than the equivalent of 10 point font is going to be unreadable to a substantial portion of users. If you find yourself tempted to use a small font for whatever reason, take a moment to consider if losing the page views of people with less than perfect vision is worth preserving your design.

Using fixed width table or page sizing is acceptable, as long as at least one portion is set to a width of 100% in order to consume the user's window, as well as to scale down properly if their window shrinks. One of the biggest frustrations I have with Web sites is seeing a site where the designer made sure that it worked great at 800 x 600 resolution without allowing it to easily scale upwards for users with higher resolutions. The end result is a design where users at higher resolutions see content floating in a sea of background color or tiled images, causing unnecessary vertical scrolling. Even worse, to compensate for these horrible fixed width designs, the designer often forces a font size that is nearly unreadable to those with less than perfect vision. Your user picked a specific size for their browser window on purpose. They have a particular display resolution for a reason. Forcing the user to scroll unnecessarily will lead them to subconsciously hate your site.

Any design element must be wholly optionally or changeable according to the users' web browser settings without breaking the site or rendering it useless. A great example of this is link color. Over the years, the user has become accustomed to the default link colors of their browser. Trying to change this or using non-default link colors will cause the user to have a more difficult time finding the links. Even worse, if your background is similar or the same as the user's standard link colors and the user has forced their browser to use the default link colors no matter what, the user will not see your links at all. Font size is also in this category. Modern Web browsers all the user to force their own font sizes onto any site. If this completely breaks your design, your design needs to change.

Eliminate the use of any non-HTML compliant tags or code. Do not use HTML compliant tags that major browsers interpret in a radically different way that will break your site design. Again, our goal is to sure that anyone interested in your content is able to consume your content. Any design element which does not accomplish this goal is a waste of time and makes QA a nightmare.

Do not use fancy code that will not work well for users with screen readers as well as content or navigation that requires user input (including mouse gestures) to be visible. Not only is writing this kind of code a lot of additional work, QA testing on it is a royal pain in the neck. Why make your life hard, as well as turning users off to your site for some drop down menu or flying widget? Many users (if not most users) instinctively position their mouse on the vertical scrollbar while reading your site. If the content or navigation requires any kind of input to be visible, many or most users will not see it. This especially applies to links; if the link is not noticeable without a gesture or input from the user, you can bet that you are going to lose a ton of page views.

If user input other than a single left click is required to view or activate content, assume that the user and search engines will not see it or be able to access it. Any content displayed in such a way must not be critical or important information. Ant interface behavior that operates in a desktop application manner as opposed to a Web browser manner is not good either. I know, I have been hammering this point quite hard lately. There is an excellent reason for this. Any user activity other than a single mouse click, or typing into clearly defined text boxes is a total failure of "the Mom test." Users do not do experiment with interfaces, users do not instinctively know that things can be dragged/dropped or right clicked or whatever. To have your design require this type of input (even if there are clear directions) renders your site useless to many users. For example, if clicking on a column header sorts the contents, also provide a drop down list for sorting. Yes, some users know that when the column header is linked that it typically means that clicking it will sort the table on the column. But a huge portion of users don't know this. When your user opens a Web browser, they expect anything within it to act like a Web site, not Excel or Outlook or whatever application you think you are replicating, regardless of how similar to the original it looks like.

Any client side scripting (including, and most importantly input validation) must be replicated server side as well, in order to allow "tightened down" browser installations, downscale installations, and automated site usage (search engines, screen scrapers, etc.) to properly use the site. Malicious users should not be able to circumvent your security by disabling client side scripting. This is an easy trap to fall into; client side scripting, especially input validation is friendlier to the user. Why have a round trip to the server just to tell them, "Sorry, your chosen password does not meet or strength requirements?" On the other hand, the moment you assume that the user has client side scripting enabled or that your client side scripting worked properly on their browser, you are in deep trouble. Many users disable client side scripting, or restrict its use to pre-approved Web sites. Other users may have a browser with a different implementation or interpretation of your client side scripting. And one of the first thing a malicious user will do is try your website without client side scripting, to see if they will be able to pass bad or dangerous values to your backend server. The last thing you need is to have assumed that everything was handled client side, and not re-validate and error trap on the server side.

All images must have ALT text as well as sizing information to ensure proper display. Never assume that the user is able to view the images. You want your users to be able to start consuming your content immediately, not to have to wait for images to download. Users on slow connections may even be done with that particular page view before the images finish downloading. Proper usage of image sizes help the browser properly render the page before the images download and help the browser render them properly (IE, for example, has a bad habit of resizing images as it sees fit). ALT tags make the images useful to the user before they download, as well as provide context to a search engine indexing the site. There is no reason to leave out these critical, but simple elements of HTML design.

Whenever possible, allow the user's browser setting to override your own settings without breaking the site. I have already mentioned this numerous times. Whenever you assume that whatever you have specified the site to look like will be what the user sees, you are going to be in trouble. There will always be users out there who override your link colors, font sizes, window sizes, and so on with their own settings. If your site breaks under these conditions, you will be in trouble. If your navigation consists of transparent images with white text, and you set a dark image to be the background, and the user disables background images, they will not see your navigation. If you use the default link colors for a background, and specify your own link colors, and the user forces the default link colors, your site will not be usable by those users. If your site relies on a particular font size to not overflow itself, and the user forces a particular font size for readability, you are going to be in trouble. Making no assumptions about the user or their browser settings will help you have a grateful audience that is delighted to use your site.

Aim your code towards the current least common denominators: screen resolutions, color depths, HTML levels, plug-ins, etc. If your site requires 1024 x 768 resolution as a minimum or 16 bit color or a plug-in or a JVM or something else that may not be installed, turned on, available, or set, you are limiting who can view your site. Why turn away visitors just because their setup or computer does not meet your definition of how a computer should be set up?

Never, ever use JavaScript to activate a page view, unless yourAJAX (or similarly coded) application requires it. This completely breaks the user's ability to use their browser's tabbed browsing or opening a link into a new window. It also makes the link invisible to a search engine. If you need to open a link in a new window, use "target=_blank" not JavaScript. If you want to try to force that window to be a certain way (no scrollbars, a certain size, etc.) use JavaScript on that new page to do this, do not try to do it within the page link.

Stick to major navigation being placed on the top or left as much as possible. The top and the left (particularly the top left corner of the page) portions of the page are what the users' eyes are drawn to upon the initial page view, and the user spends more time looking there than anywhere else on the screen. If you put your navigation there, the users will be able to quickly and easily navigate your site. Another fact to consider, is that these locations are where users are accustomed to looking for navigation elements, since most Web sites put the navigation there. If you put expected design elements (navigation, site search, "email the Webmaster" link, etc.) where the user is used to seeing them, your users will be able to use your site from the get go. No one likes having to learn to use a site, and that significantly damages usability. You want your users to remember your site as "the one with good prices that I had no problems with," not "the one with great prices, but I could not figure out how to add items to my cart."

Remember that the top left corner is the most important place on the screen; the bottom right corner is the least important. This means that whatever you feel is the most important part of your site design, whether it be the logo for branding, a site search, navigation, or something else, needs to go here. There is no way to change this fact. If you put the site search at the bottom right corner, no one will use it.

All links must obviously be links. This is something else that has been changing since the introduction of CSS. For whatever reason, many Web designers feel that making links look less "ugly" by removing the underlines and/or making them look like the rest of the text is a Good Thing. I suppose on an aesthetic level, they are correct on occasion. In terms of usability, they are dead wrong. If you want to drive users away ASAP, make it impossible for them to find the links they want to follow by making them blend into the rest of the

Offer a "printer friendly" link on every page. This is optional, but as highly recommended as can be. When a Web page is printed, none of the design elements are useful to the person reading the hard copy. They cannot click the links, the styles are useless, frequently it is being printed on a black and white printer (or in black and white mode) and so on. All the user wants is the content. Give the user what they want, and provide a "printer friendly" link that provides the content, plain and simple, in a format and font suitable for printing as opposed to viewing on a screen. If the content is split over multiple page views, the printer friendly version should provide all of the pages as one

URLs should be browser bookmark, email, and search engine friendly. If they are not, the server should be able to determine the correct content to display to the user. If you are passing really long URLs around that may break in email clients, or may get mangled if someone tries to post them to a newsgroup or Web forum, you are missing out on potential page views. If your URLs contain a session ID, you are giving your users a lousy experience, especially if they send "?product_id=50&session_id=541039ASD" to a friend in an URL, and the friend gets a giant warning message about an expired session. Do you really think that the average user is able to correct the URL to see the product? Of course, you also want search engines to spider our site nicely, and present clean URLs to the user. Ugly, long, session specific URLs may make life a touch easier on the backend, but they are going to kill you on the front end. And accepting the session ID from the URL is a dangerous habit, as it allows a malicious user to participate in other user's session with enough automated guessing or reverse engineering of the ID creation system.

Avoid splitting content over multiple pages. Each page should be a discrete item, and not require additional page views to finish reading. If multiple page views are needed for whatever reason, usually due to an ad-driven business model where page views drive revenue, a "print friendly" link must display all pages as one. This serves many purposes. First of all, when someone finds the page through a search engine, and they land on page three out of seven, they may be rather confused, and they may not be able to find their way back to Page 1. Users really do not like having to go through multiple pages. In addition, compressing the content into one page increases its visibility to search engines. And search engines like to only display one or two pages for a site; would you prefer that the users see three separate pieces of content that might help them, or just three portions of the same piece of content? Remember, each time you require your users to use the navigation system, regardless of how well designed it is, you are introducing a potential difficulty or interface failure. Keeping your content on one page minimizes the points of potential failure for your users.

Never break the user's "Back" button or other elements of their Web browser's functionality. If you are using something other than standard anchor tags for navigation, you are doing something wrong. By breaking the "Back" button and other basic functions of the user's browser, you are damaging your usability. The user always knows where their browser's "Back" buttons are; they may not always find the "Back" button that you coded onto the site. The same applies to "Close Window" buttons. If you have a procedure which will cause something bad to happen (such as double charging for a checkout) by a page refresh or form re-submission, recode your backend logic. A simple disclaimer of "refreshing this page will cause you to be charged twice" just does not cut it. These are functions that the user has probably spent years learning how to use as they need to. Breaking them is like building a highway that requires drivers to drive on the opposite side of the road, it just does not work and will cause major problems.

Never alter the browser window's appearance (scrollbar color, title bar color, etc.). This has become very, very common. Between CSS to control appearance, and JavaScript to change the browser's window display, Web designers and developers have gone nuts. For some reason, they seem to think that their particular design is more important than the user's settings. I have been to sites where the scrollbar was forced to be nearly all the same shade of black, including the arrows. Needless to say, scrolling to see content would have been nearly impossible without a wheel mouse or knowing that the keyboard also scrolls the page (many users think that the keyboard is solely for entering input!). The user has a particular color scheme and window layout on their computer for a reason. Some users may require a high-contrast color scheme for visibility reasons. Others may have spent money or effort on a GUI skinning system to fit their own aesthetic tastes. There are a million reasons why the user's browser window looks the way it does. So why break it? Doing so does not help the user, it just confuses them. If your Web site changes their browser to no longer look familiar, you have forced the user to relearn how to use their browser, just for your site. Remember, the goal is to work within the user's environment, not to make the user learn your environment!

To summarize the running water Web design philosophy into a simple checklist:

* Use CSS as much as possible, but only use CSS that all browsers will accept and only use CSS that the site will properly work without.
* All font sizing should be proportional; any non-standard sizes (such as "smaller") should be percentages of the base size, and the base size should be set in ems, not pixels or points, in order to ensure correct cross platform sizing.
* Using fixed width table or page sizing is acceptable, as long as at least one portion is set to a width of 100% in order to consume the user's window, as well as to scale down properly if their window shrinks.
* Any design element must be wholly optionally or changeable according to the users' web browser settings without breaking the site or rendering it useless.
* Eliminate the use of any non-HTML compliant tags or code. Do not use HTML compliant tags that major browsers interpret in a radically different way that will break your site design.
* Do not use fancy code that will not work well for users with screen readers as well as content or navigation that requires user input (including mouse gestures) to be visible.
* If user input other than a single left click is required to view or activate content, assume that the user and search engines will not see it or be able to access it. Any content displayed in such a way must not be critical or important information.
* Any client side scripting (including, and most importantly input validation) must be replicated server side as well, in order to allow "tightened down" browser installations, downscale installations, and automated site usage (search engines, screen scrapers, etc.) to properly use the site. Malicious users should not be able to circumvent your security by disabling client side scripting.
* All images must have ALT text as well as sizing information to ensure proper display. Never assume that the user is able to view the images.
* Whenever possible, allow the user's browser setting to override your own settings without breaking the site.
* Aim your code towards the current least common denominators: screen resolutions, color depths, HTML levels, plug-ins, etc.
* Never, ever use JavaScript to activate a page view.
* Stick to major navigation being placed on the top or left as much as possible.
* Remember that the top left corner is the most important place on the screen; the bottom right corner is the least important.
* All links must obviously be links.
* Offer a "printer friendly" link on every page.
* URLs should be browser bookmark, email, and search engine friendly. If they are not, the server should be able to determine the correct content to display to the user.
* Avoid splitting content over multiple pages. Each page should be a discrete item, and not require additional page views to finish reading. If multiple page views are needed for whatever reason, a "print friendly" link must display all pages as one.
* Never break the user's "Back" button or other elements of their Web browser's functionality.
* Never alter the browser window's appearance (scrollbar color, title bar color, etc.).

J.Ja