Discussion on:

The worst sin in software development

You wrote "Some developers are just too lazy to take time to test their own code." My feeling is that too many of todays developers have too little training. "Senior" developers can get by with only 2-3 years experience. If they have not previously worked for a company which takes software development seriously and has implemented a software development methodology and quality program, they will not understand the reason for quality testing.

I disagree with the article in the sense that practically all the current crop of exploits for hacker/crackers is based on a buffer overflow situation. Code should check to make absolutely certain that overflows cannot occur. Then build in CRC checksumming to make sure the code was not improperly modified. Do that FIRST, then get the proper data for testing. "Let's make it work" should not take a back seat to "Let's make it secure."

Checking for buffer overflows is just a part of normal bounds checking, or it should be. I totally fail to see how 'Lets make it work' is any different from 'Lets make it secure'. Security is a necessary part of every system and must be built in from the ground up. I just wish that the people who make our OS's and tools would adopt a decent programming methodology and employ some team leaders who have 'real world' experience!

They won't understand the reason for quality testing without years of training? What's to understand? Most children have discovered that things don't come out perfect the first time through. Too many people are lazy and don't care that they turn outpoop. I guess training involves forcing them to do good work even though they don't care to, until it gets to be a semi-habit.
IMHO, there isn't any good excuse not to test your program thoroughly before you turn it out as being a finished product.

They won't understand the reason for quality testing without years of training? What's to understand? Most children have discovered that things don't come out perfect the first time through. Too many people are lazy and don't care that they turn outpoop. I guess training involves forcing them to do good work even though they don't care to, until it gets to be a semi-habit.
IMHO, there isn't any good excuse not to test your program thoroughly before you turn it out as being a finished product.

If testers aren't testing and aren't getting training, it seems to me the biggest driver is one we can't solve - pressure to be fast-to-market. That to me is the force behind the profusion of software and behind the loss of quality.

As an EDI coordinator I have to deal with those firms that specialize in setting up electronic commerce for their clients. The testing process is never anything like real life will be when the client takes over the process. If the client doesn?t do additional testing your first sample of good data is a live purchase order.
The answer to that is changing the bad test data to something more reasonable. I often have to create a reasonable data set that will actually go through my order entry system and create something other than error messages. Error testing is good but you have to have enough valid data to progress to the next step in the process.

I agree with you when you say that you need to have enough valid data to test errors. But despite there being enough valid data available, there are instances of no error testing. I was recently working on a similar product (as technical writer writing help again) and found major goof-ups that were unforgivable. Like the Currency field accepting more than 2 decimal places, etc. The point remains that one needs to test before delivering a product.

Testing is always the most difficult part and should be done by the programmer and later by the users more thoroughly. It is very hard for anyone to check on himself.

(As for currencies it is the right way for Euro calculations to use upto 5 decimal places. Obligatory)

As a tech writer and instructional designer, I get slapped with this all the time. It amazes me that developers don't take the time to test their code with realistic data -- and in some cases don't test their code at all.

I haven't written a training program yet (and I've written quite a few) where I didn't end up as a tester simply because the developers didn't do their job. And then no one on the project team can figure out why it takes half-again as long to complete the training program. (Hmmm, I wonder if it could be that they had to rewrite the code because I had to spend my time finding their mistakes, and I had to wait for them to finish doing that before I could finish the training design?) So in the end I end up getting screwed.

Modify your next contract for tech writing to include an additional charge and a time delay for each error you find. Then you get paid as both a writer and a tester.

I'm not a contractor. This is all internal work between departments. Can't charge anybody anything.

A couple of years ago when I was contracting the only job I could get was "an exciting insurance application". It was the first (and will be the last) purely business-type app I've ever worked on. I found it frustrating, because there wasn't a proper test environment so, if I wanted to test bits of code I'd written I had to create my own test data (fair enough), only to find it'd been blown away next time I needed it to do further testing on the same bit of code. Additionally, when questioned on why I'd taken 40 hours to do a job which had been estimated at 3 hours, my supervisor didn't want to accept that I'd had to redo both the business analysis and the technical analysis because the people who should have done that had done a half-assed job, and I'm not a business analyst. (They didn't debug their work.) I only work on technical applications now.

I know what you mean... try time registration and payroll software. I think I know everything about social law by now.

There is a well known comment which says that the fool is always greater than the proof. As a professional developer I always test my work (and foolproof it as much as possible), but there are almost invariably small problems when the application reaches others. There is currently no formal test/QA operation at the location where I work, and although I do my best there are some things I simply don't think of doing. When testing, I have the advantage (or disadvantage) of knowing how the application should be driven. Almost invariably I find that whichever method "makes sense" to the developer works as advertised.

That's not to say that errors with other paths through the application should not be found and fixed before it is released. However it is often very difficult for the designer &/or developer of the application to find such problems - just as it is difficult to proof read a document you have just written. Certainly such skills can be learned, but unfortunately many companiesregard time spent by developers doing "other work" as time wasted.

In my opinion, any company that doesn't have some additional post-developer testing of its software products is asking for trouble. Unfortunately, such testing all too often fallsinto the lap of the tech writer, the sales team, or (in the worst case scenario) the customer. Companies wishing to develop quality products *must* learn that to get a quality product, everyone needs to have quality as their primary goal - even if it does result in longer initial development times.

rfoster@askfoz.com wrote: "When testing, I have the advantage (or disadvantage) of knowing how the application should be driven."

In my opinion, this is a disadvantage. You know what the program does when you use it like you're supposed to use it, but not what happens if you don't use it like you're supposed to.
When I test software, I try to create as much havoc as possible. New screens with right-mouseclick-menus? Ok, I'll turn my mouse upside down, now what key combination shall I use???Implemented new features that can only be reached by using certain keystrokes? OK, I'll disconnect the keyboard.
Fields where you're supposed to enter dates? I'll enter them in text format, in Swahili (I speak it fluently - NOT!)

Suprisingly howmuch bugs you find this way. And more surprisingly how many programs accept crappy input.

And everything that that entails. The other guy and I both function as system analysts, we both work on user interfaces, he develops most code, I do most testing, documentation, help files, and we both do customer support (and both of us function as network administrators and support technicians). Jim is very fond of saying, "Bill really isn't an idiot--but he sure can play one when he's testing!".

You're right--it IS more difficult to test the software when you KNOW what it's supposed to do, but that also gives you a heads-up on what it ISN'T supposed to do. Yes, give it the full idiot-treatment--fill in fields with obviously incorrect information, and nearly-correct information. Enter dates in multiple formats; if two or more fields are dependent upon each other, screw them up one at a time in every conceivable configuration. Generally, a modification that Jim may have programmed in--oh, let's say 15 minutes--it may take me as long as two or three hours to test.

Yeah, I sometimes get a little grouchy when I'm trying to get a software release ready, but the testing pays off in much happier users.

SilverBack (Bill?) wrote: Generally, a modification that Jim may have programmed in--oh, let's say 15 minutes-- may take me as long as two or three hours to test.

And there lies a problem for many companies too. I completely agree with your comments about "playing the idiot". Like you, I try to do so... but all too often I am getting yelled at by management for taking so long. Frequently as far as they are concerned if they don't see the original bug that needed to be fixed, or if they have heard that a desirable feature has been added, then the product is "good to go", even if it hasn't been tested exhaustively. I rarely (if ever) get longer to test the changes than I (or whoever developed it) did to implement the modification.

In the end, it really comes down to the customers - and we are customers too. While we are willing to accept buggy software, and worse pay for it, then that is what will be delivered by many companies. Sooner or later, customers will stop accepting software that crashes, and then testing will start to assume its rightful place in software development, just as it already has in most manufacturing industries.

I've been a developer in many companies and would like to point out that MANAGEMENT is responsible for making everyone do their job. I too have been screwed by other developers who didnt test their code but managment thought they were great becausethey pounded out lots of (buggy) new features. Fixing those great new features was "somebody else's job". Now I do network admin and if code dont work, I dont buy it.

In acheiving my computer science degree, some of the more valuable courses were not programming courses. There was a pair of classes where we developed an application from beginning to end. Unlike other classes, this was a real world application for a non-profit organization. The instructor emphasized the testing even before any code was entered. Our test data ranged from good with unusual but valid parameters, to typical key entry errors, to absolute incorrect data. Too bad many developers are actually just coders.

I find the best test data to use is the client's (that is, when you're working with a client). Trying to come up with real data that comprehensively tests a wide range of possibilties on your own can be time consuming. When specing out a project, I include a section that requires the client to provide real cases. Not only does this provide a good testing sample, it also exposes your code to the 'quirkiness' of real life data.

When working on a more generic project, this can be more difficult. You could try to get real data from a prospective customer/end user in that case, offering them some sort of carrot for their input.

Lastly, it does help to have someone else test the code. It's easier to find issues in an application when you don't know every in-and-out of it, what order you're 'supposed' to do things in etc.

1. Some programmers think they are born for coding and do not indulge in this menial tasks like Testing,putting the code properly in version control and having proper in-line documentations. Little do they know that they might be brilliant but these tasks however uninteresting are essential tasks in software development.

2. Some programmers just do not know.

3. Some programmers are lazy (as the article mentions)

4. General lack of awareness. Just increase the salaries for Testing skill sets then you would see the programmers jumping onto that bandwagon. They would start taking it seriously.

Those "sins" you listed are trully bad news for any project. But, THE first rule of software developmet - and if you don't pay attention to this one you might as well hang it up now cause the project is doomed to be full of bugs - is this:
Never, EVER, under ANY circumstances have the developer test his own code. He'll be able to make it work EVERY time. Not that any developer is intentionally trying to be irresponsible but NO-ONE can successfully test their own code. There ALWAYS has to be an independent tester if you want to make sure the code works right.

I work primarily in the application of PC's to automatic machinery as operator interfaces or data collectors. I cannot imagine not entering valid data during the testing phase of a project. A lack of understanding on my part could easily become a smashed finger on the operator's part due to unanticipated mahine operation. To me it is just pure irresponsibility to not test code. Besides, what fun is writing code that you don't watch work? That's where the magic happens!

I laughed when I read this article. I can't
imagine developing any significant piece
of software without using at least
semi-realistic test data, including testing
your validation. How else do you know it
works?
Not using proper test datamust be a
Microsoft thing.

I'm painting with a very large brush here-- and splashing myself liberally in the process--but really good software testers pretty much fall into the psychological category of anal-rententive-compulsive. And, I'm not saying that like it's a BAD thing....

All levels of testing must include tests that make sure everything works as it should, and nothing does anything that it shouldn't. The first part is simple; the second part is where the hard work comes in (and a slightly warped personality might help, too). My job classification is Computer Programmer, but the biggest chunks of my job involve training, documentation, and testing. I've found that a change that takes the programmers 15 or 20 minutes to code may take two or three hours to test completely, simply because of the numbers of combinations and permutations that may be involved.

Many people are intelligent enough, well enough trained, and perfectly capable of performing adequate testing. But THEY DON'T WANT TO. There is lots of fun in programming, but--for most people--very little fun in testing. And most people don't want to do things that aren't a lot of fun, even if they're being paid to do it.

You're right about the mindset for a good software tester.

Ideally they should be capable of working through extensive lists of every possible input error you can create for a given field. They should also be capable of working through similarlists of every possible input error for combinations of fields too, so you can detect problems that may be masked by other problems.

You also need two additional types that may or may not have the testing mindset. One will be the power user who takes the software way past its design specs. The other is the 'oops' user who can crash a well designed system by signing on and who shouldn't be allowed anywhere near a computer.

Of course, one of the big problems with the above is that it requires extra people dedicated to those tasks. If your organization is operating on a shoestring, you may be limited to developer testing.

At the same time, the developer is often the support person in the same organizations. So he or she will reapthe benefits, or costs, of their testing.

One of the things I have found to be a problem with programmers is compatability. On their desktops, besides their programming software, they also have "personal" apps that are usually not in the corporate standard that have a history of overwriting standard settings to favor their needs. Then they write a program, test it on their machine, and wonder why it looks bad on the user's screen. One example is a programmer that loves 1024 x 768 resolution. He created a software package, tested itand installed it for production claiming all tests were complete. When the first user went to go look at it at 800x600, every field was screwed up. It took him another month to rewrite his application to handle different resolutions. When asked why he didn't test resolution quality, he shrugged his shoulders and claimed he did. He even requested we change everyone's resolution to 1024x768 to fix the problem! And one wonders why support dislikes programming!

Absent in this discussion thread is the concept of automated testing. Once the programmer finishes unit testing the code should be run through an automated test tool with repeatable test data. If the code doesn't make it through integration testing, the reworked code is run again with the testing tool. This accomplishes two goals: 1) it forces the programmer to create a test set of data and 2) it gives a tester a set of data that at a minimum is repeatable.
Check out Mercury Interactive.

I've read about automated testing tools here and there and have thought that they would be nice to have, especially when dealing with critical systems that are frequently modified.

Unfortunately, I haven't been able to convince any of my employers that they are needed. They look at the costs of purchasing, installing and using the systems and cringe.

Of course, most of the organizations I've been with have been small ones. I suspect that a big organization with a separate testing department could easily justify it.

In our company, there are a few ways to get realistic test data. But first let me say what we do: we're in the time registration and access control business. So users put a badge in front of a badge terminal (which is designed, developed, tested andinstalled by our own hardware department)

First method, going on at the moment: we're testing a new terminal which uses fingerprint recognition. We use our own terminals for our own time registration, of course. So we put up the new terminal nextto our regular badge terminal. We hold the badge at the regular terminal, and at the same time use the fingerprint terminal. This way we get 2 sets of data: real data and test data, which we can compare.
In short: use the program (device,...) yourself as if you were a real user. Find a situation where you can do this. For example the programmer of the software in the article could use the school reports of his/her own kids.

2nd method: our support department sometimes needs backups from clients when their database crashes,... to correct it. We repair the database, and we keep a copy. You never know if a client fails to make a backup themselves. Later when we have a new version of our software, we upgrade the (repaired) data of real clients and test this. The only thing we do (for privacy reasons) is garble up the names: switching first names from client A with first names from client B, and last names of client C (or things like that).
This data (and that of method 1) is also used for our manuals.

3rd method (least preferred) is slowly entering data one by one. Imagine how much work this is: manually entering 4 registrations for a small company of 50 people, for only 1 month, this means entering 4000+ registrations...
Weonly do this for fast testing of small parts of the program during development, not during final testing.

i have seen people who do not even do unit testing. they expect the Integration testing to catch even their typos. And often expect the testing team to catch these kind of trivial errors.

for the records i am a developer, i create a basic testingplan - on paper or sometimes mentally - and some times certain bugs do escape my scrutiny, but I can not understand why some one will not do even testing.

i guess they think they are GOD and can do no wrong. even if they are caught they explain its becos of the complexities in the application or the complex nature of software development. i can see them giving excuses.......and there are no excuses in our industry.

This is a testers job not a developers job. A developer will never be able to thoroughly test their own code.

A parallel analogy is Sales vs Customer service. Every Sales person is front line customer service, true, but as a company, you want your sales force selling, not servicing.

You want your developers developing, not testing each possible scenario. (they make too much money for what is essentially data entry)

Every developer needs to basic test their code and products, but that is why testers exist.

I can't believe the errors in programs we recieve. As end users, we find the idea is sound or exactly what we are looking for, but we always find programming errors that become frustrating to the point that we have removed several programs from our systems. It seems that the transition from a good idea to a working product,that doesn't require patch after patch, is impossible and we are paying for the lack of thorough testing...
Thanks for letting me sound off ...ah I feel better already....at least till tomorrow..Brent