Tech & Work

Poll: Are known bugs acceptable in software releases?

Some developers feel comfortable releasing software with known bugs because they know the bugs will be resolved in the next iteration. Let us know what you think about releasing buggy software.

As more development shifts to Agile methodologies, it seems that developers are feeling more comfortable about releasing software with known bugs, especially since the bugs will be fixed in the next iteration. At the same time, bugs are always frustrating to customers and, in many cases, bugs can turn a customer off for good.

I am not a fan of releasing a product with issues, but I understand that some things simply cannot be allowed to hold up a release. What are your thoughts?

Related TechRepublic IT Leadership post: Buggy software: Why do we accept it?

J.Ja

About

Justin James is the Lead Architect for Conigent.

37 comments
xmetal
xmetal

In complex software like what I work on it is impossible to release without some known bugs. What is more important is to answer the questions: How serious is the bug? How many people will it affect? How often will they see it? We break the first question into three basic categories: crashes, data loss, everything else (this last one has sub categories). All bug tracking systems I've worked with allow bugs to be classified based on severity. In one of the applications I work on (a customizable editor) we have over 1200 APIs and a client is free to use these in any combination they like to extend the product in addition to other configuration files used for altering the UI. They can completely change the default behavior of the editor, the UI, connect it to other systems or 3rd party software, and embed 3rd party software, which allows them to do virtually anything imaginable in the context of what the editor's main function is. In our case this means that most of the major bugs we hear about are not found by our QA team because they cannot possibly test everything, just that each individual aspect of the product is functioning as expected. Most complaints come directly from clients extending the product in never anticipated but legitimate ways. It also means that the majority of these types of bugs will only ever affect that client in that specific way which often puts us into the unique situation of deciding which client is most important. There are obvious cases where a bug is found that will affect many people, but our QA team typically finds these. What we don't allow to ship are known bugs that cause a crash or data loss. After that everything else is put into a big pile, ranked according to various criteria (the questions above together with other information like an estimate of how much effort it will take to fix) and then scheduled appropriately. In the world of commercial software it is generally not development that decides release dates, it is either marketing, sales or a combination of both and this means the number of new features and bugs both must always be squeezed into a particular period of time. If a client notifies us of a problem but we have already fully scheduled features and bugs for our next release their bug will remain and they will need to wait for the following version to see a fix. One more note: We are in the enviable position of often being able to offer most clients half a dozen work-arounds for any particular issue they have run into due to the number of different ways a particular feature can be implemented (primarily because we have so many APIs and because we can integrate with or embed other software or scripts).

jim
jim

IMHO Bugs are only acceptable in release if there is a documented workaround available to the user. I currently have released a version of a SW integration solution that has a bug. Prior to release I consulted with the end user, explained the probelm, demonstrated the workaround and provided documentation, and asked if it was acceptable to release with the bug.

bbsinc98
bbsinc98

7/14/2010 The finance departments pressures to release a.s.a.p., the clients want to have it now, the market launch window to establish, all is "Russian Roulette" of our technical reality. Looking back from the first DOS and CPM days on, we all accepted the blue screen of death. If our cars, elevators, production equipment, airplane, ship or other major impact producing equipment would stop because of buggy software, we get mad. In Linux and Unix we let new born child process replace a faulty mother process without any impact for the user. The created event log files help us improving the software. MS does it with their Dr. Watson and other tools the same way. With today?s connected world security bugs opening our files, confidential data and identity buggy software should be an absolute no no. ?It would happen to us? we heard before. It happen to your sub-contractor or connected clients and vendors coming into your system. Why? You had accepted them without a timed access window and instant security check. You spend your money on TV and Internet ads. Image came first security second. Cheap software with bugs to met the budget! Greetings Doc Hacky

dknichol
dknichol

I have to "proof" everything I produce in order to earn my living - why can't software developers?

Tony Hopkinson
Tony Hopkinson

Known bugs may be acceptable. One of our policies is to never release with a known bug where there is no work around and we always document the bug and the work around. Equally important whe the next versions comes out we document any fixes and deprecate the work around. The bugs we address in a version are prioritised by the number of customer contacts we get about them. Let's face it a bug can be a mispelled lable ( :p ) to pressing delete wth no customer selected wipes out all of them. If you were a customer which one would you have us fix? Let's be honest here, we work in a world of good enough, the people who talk bout having no bugs will not resource the effort to achieve it as our customers would be most reluctant to pay for it. Quality is not what drives commercial development, price does. To get no bugs and good sales would mean no profit.

Ocie3
Ocie3

The "Yes, but only if fixing the bugs would cause an unacceptable delay." option sounds reasonable, if not inevitable. Unfortunately, such a policy is too easily adopted by managers who want a revenue stream ASAP regardless of the cautions which the development team is more inclined to heed. But there are "bugs" and there are "bugs". If the program runs without incident or execution error, yet it produces invalid output from valid input, then the problem is not a "bug". It is a conceptual, design, or implementation flaw. Either that absolutely must be remedied else the developer doesn't have a product to market, or perhaps some feature(s) or function(s) of the software can be removed or disabled as a (temporary) remedy. If a bug causes invalid output to be produced from valid input, then it is likely to occur for enough users that it will adversely affect the confidence and opinion that they will have of the software and the developer. Much depends upon how severe the effect of the bug is as much as how many users are likely to be affected by it. The developer should try to find workarounds before releasing the software for distribution, and inform prospective users of the "known issues" and the corresponding workarounds, if any. If the problem is that some software features will not work on some computers which need to have the OS upgraded, then that is another matter entirely, because whether the software functions correctly is not in the control of the developer, but in the control of the prospective customer.

Marc Thibault
Marc Thibault

The question is badly formed. "Bugs" is too ambiguous and misdirecting. Are we talking about programmer errors or product shortcomings (missing or inadequate use case realization)? Shortcomings you disable the use cases and list them as futures. Programmer errors should be listed as such, with the name of the offending programmer or lead as part of the topic. E.g. "Suzie Brown's Memory Leak." When the errors are listed in order of severity and difficulty, decide whether to release or not, with the error list part of the release package. Product quality should improve dramatically. Seriously, the attitude that errors are inevitable so it's ok to release an error-strewn product is pathological. First, fix the errors; then, stop making them. Put some intelligence and care into your code. In any other craft, errors are rightly seen as evidence of incompetence, inattention, or both. I've seen no reason to give programmers a free pass. It's not easy. Software rates among the most complex artifacts we make. But that's why we make the big bucks; if it were easy, anyone could do it. You claim to be an engineer--do some engineering. At the very least, strive to be an expert craftsman. A line of code is a line spent [EWD1036]. An erroneous line of code is spent many times.

rkling
rkling

Accepting errors gives the impression they are acceptable. It lowers levels of expectation. Many years ago computer science courses were teaching "proving code correct? was possible and demonstrated it in the class room. An IBM fellow taught a course on software where grading was determined based on how many compiled were needed before it ran correctly. The student got an A if it ran correctly the first time, a B if it ran correctly the second compile, and a C if it ran correctly on the 3rd compile. More than 3 was an F. This set the expectation level very high and "trainees" were able to write a correct COBOL program that was ?correct? by practicing good software development techniques. IBM had signs "Think" that we had at our desks. We spent more time ?thinking? and less time coding. Now it is trial and error until it doesn't explode, then we ship and let the customer debug with great loses in productivity. How sad that we have "progressed" to acceptable errors. Imagine if it was acceptable when we landed a man on the moon? We probably never would have. Instead we have a giant oil spill in the Gulf!

sethbburgess
sethbburgess

I can live with bugs that are non-critical and will be fixed in the next service release. What drives me up the wall are bugs that survive from one version to the next and are NEVER addressed.

jsaubert
jsaubert

It depends solely on the nature of the bug. If it's going to inhibit the functionality of the //intended use// of the software then it should not be released. However if the software will operate as expected, in a manner that is efficient and the bugs do not cause disruptions then there is no reason not to push it out. The end user will consider everything from a total system failure to the use of a wrong font in a generated report a bug. Yes both //have// to be fixed for the project to be considered finished, but one of those you can live with and one you can't. You have to weigh lost productivity with perfection. The user needs (and deserves) to know of any known bugs, and they need to know that what they have will be fixed by priority.

tkainz
tkainz

Sorry....but sending out a release with known bugs is just unacceptable....it's short-changing your customers. I guess that there could (maybe) be a reason for sending out a release where the design might not quite be right or some functionality might not be fully implemented but I wouldn't consider that a bug. To me a bug is a function of the software that is broken...that doesn't work right; and that is just not right. If you're sending out buggy software it's most likely because too much is being packed into one release and there's inadequate time to fully develope, test and debug the software, or resources are insufficient. In either case, I have yet to see a software developer put a notice on the package that the price is being discounted 20% (or so) because the software is faulty. If full price is being charged for than a fully developed, fully tested and debugged product should be delivered.

eftpotrm
eftpotrm

Surely it depends what's actually the matter? If it's a showstopper in a core function, clearly no. If it's a 'bug' as in a change request from a user who thinks something is 'clearly obviously needed' but never thought to tell you, yes (though with expectations management). Assess the likelihood of the failure being reached, the consequences of the failure, the consequences of delaying, the time / cost to fix and the ease of deploying the fix. Only then do you know if it's acceptable to release software with known issues.

zigrest
zigrest

Id rather not let bugs lie, but im forced to at my job.

taylorstan
taylorstan

On the head, I'm going to say no. There is not any guarentee that the problem will be fixed. Found this out the hard way with some software I bought. Buy the new vesion was their answer. And as software vendors find out this works more and more, they will release increasingly crappy software. Rush to market is the driving force behind everything.

vnthiru
vnthiru

bugs are not acceptible. developers should have bug list, status of fixing them and workarounds. After using a standard version of a s/w, I recently bought a profesional latest version of the same s/w thinking it would be better than what I had. But I am tottaly disappointed. The feature I use is full of bugs and have wasted time and money.

BobNRay
BobNRay

Releasing with bugs can be acceptable under certain conditions: 1. Known bugs are published (I agree with previous poster AnsuGisalas). 2. There is an acceptable workaround and the workaround is published. (Acceptable=impact is known) 3. There is a scheduled 'fix' or, alternatively, there is an agreement that the 'bug/workaround' combination is acceptable until a fix can be identified. I'd be interested in any examples of software that was released only after all the known bugs were closed.

QAonCall
QAonCall

Unless you are paying for it. Then each person will answer differently based on their perception of the risk. Further, the term 'bug' is very nebulous. Many stakeholders will define something as a bug, that they thought they included in requirements, or 'assumed' it would be done that way. If the major functionality is diminished to a significant point that there is loss of productivity...it should never ever be released...unless the customer says they are 'ok' with it! The truth is, there is no perfect software (sorry Apple fans!) ;)

Slayer_
Slayer_

On a program that requires you to purchase the next version, with the bugs fixed, its unacceptable. But if you can meet 90% of the demands now when its needed, and catch that last 10% in a month or two, I think it's worth it.

AnsuGisalas
AnsuGisalas

I think it makes a difference. If a bug is known to the publisher, but not to the public, that's a trust issue, which affects my answer at least. At least insofar as I think I'd say option 2 if documented and 3 if undocumented. Depending on severity.

Tony Hopkinson
Tony Hopkinson

Given that it doesn't make commercial sense to do it right does it?

Marc Thibault
Marc Thibault

"the people who talk bout having no bugs will not resource the effort to achieve it " My experience managing developers and a thousand years of engineering lore tell me that not making mistakes is a lot cheaper than fixing them. Dijkstra said it eloquently in 1972: "Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with."[EWD340] What we need most is to run out of excuses for sloppy code.

Raymond Danner
Raymond Danner

tkains stated, "If full price is being charged for than a fully developed, fully tested and debugged product should be delivered." Amen, but that, frankly, means that a Microsoft OS product would never, ever be released. One show-stopper in Win95 alone could and did ruin HDDs from one particular maker, and it was one of six major problems /known/ by Microsoft (and only Microsoft) to affect the /installer/ on release. Not acceptable at all. Microsoft has far, far too many secrets for the consumer's good, in my opinion. Pity they're not alone.

Justin James
Justin James

I don't mind buggy Web apps hosted by a third party vendor as much as bugs in locally deployed apps, and those aren't as bad as bugs in firmware. The worst are bugs in server-side software, since there are so many things to go wrong on an update. For example, I haven't updated a Drupal install in forever because the upgrade is worse than the original install, even though the updates have big security patches in them. And anything where the update needs to be tested by a third party before being deployed... UGH. My phone is miserable, the next version of Android is supposed to fix the problems... but I have to wait for Motorola to test/deploy it. J.Ja

Sterling chip Camden
Sterling chip Camden

All known bugs should be documented, hopefully with work-around suggestions. One of my clients allows read access to their bug tracking software to all of their customers. Customers can therefore see where any issue rates in priority, and they can make their case for elevating it.

des
des

all releases are tested prior to release. Run time versions are also tested by the client prior to final distribution. There may be design misinterpretations which are upgraded in the next release, but never a bug in that a process dos not perform according to its specifications.

mdbizzarri
mdbizzarri

I work for a software company, and we have a major release every year. We have an extremely thorough QA team, but there is no way to have every type of Oracle environment, network, and VM to test all of the different aspects. All we can do is address bugs as we find them. If it is having a major impact on a customer, we provide a patch ASAP. If it is not major, then we provide a work around until we can create a fix. We only have so many man hours to develop, and we have to make decisions on how many customers are affected, and how to apply the man hours. We do our best to help customers out, and when they see how jump on the issue, they are usually appreciative. It's not a perfect world, but we do try.

Tony Hopkinson
Tony Hopkinson

Ok matey I'm calling an existing routine. It passes it's unit tests, it passes it's integration tests. However during regression testing, it's found to do something else as well. There's a side effect which manaifests in an apparently unrelated area. So Do I Do nothing. Cut and paste the code writing a new version without the the side effect. Amend the existing code taking the side effect out and hope nothing was relying on it? Rewrite it and all the code calls so it looks like it was written by an effective developer? I'm taking a wild guess that most managers would pick options 1 or 2. Given they do who and who isn't effective? Now if you want to say if the original developer had been effective, then there would have been no issue. You might be right, but what pillock employed said incompetent. Who said he could program? Or was that side affect a last minute feature request by someone important, that was going to be 'fixed' next version. Sloppy coding is a symptom of management choosing not to resource effectively.... Not creating bugs in the first place is simplistic to the point of being imbecilic. This is IT, change is a given. You can't have rapid cheap change and no bugs. I donlt care how good your developers are, they don't work in isolation. Twenty + years and counting I've been doing this commercially, I've seen way more deliberate low quality than high, way more...

Sterling chip Camden
Sterling chip Camden

It's not knowing how to design a piece of software that can evolve gracefully. Sloppy code can be fixed. A bad design must be scrapped.

AnsuGisalas
AnsuGisalas

Along with Glasnost, are effective tools at handling the angry masses. I still hear occasional echoes of those infamous "party line statements" of the 90's that "We do not make mistakes". Foreshadowing the state that follows the doom; where indeed mistakes are no longer made.

tallachoo
tallachoo

You must work with our developers - they claim that our product never has "bugs". In fact we are not allowed to say "bug" at work! They are so confident that they make releases, some even one-offs, without testing. Leaves us support folk picking up the pieces when a customer calls and has lost all of their data due to a BUG in the software. It gets even better when they don't accept responsibility. Bugs need to be taken more seriously and more should be done to release software out into the wild calling it a "Full-Release" when it should really fall into either Alpha, Beta, or Release Candidate status. At least then the customer is aware and knows what may potentially happen. STOP this "its good enough" mentality, it causes more harm than good, and costs more money in the end.

tallachoo
tallachoo

I also work for a software company - and the idea is, and I think the question is - do you allow a release with KNOWN bugs. My fight has always been no. Now we all know bugs will appear as there is no way to test everything, but why send a product out if you know that users with Dell printers will crash the software every time they try and print? KNOWN bugs = NO release. New bugs = oops gotta fix them.

Tony Hopkinson
Tony Hopkinson

that quality developers with quality processes will always be cheaper in the long run, just that they are constantly driven by short term goals. I earn my corn navigating through other peoples code, green field opportunities being very rare and not often trusted to mere technical people. I'm good at it, but I'm not cheap. Here's one for you. Telecomms ordering and provisioning application. GUI designed based on spreadsheet that had evolved. No split between ordering and provisioning (the design meant a name change had to be ordered) Only one order per customer (as 1 - 1 link between order and customer) Link to the billing systems was by Surname! Not a sloppy design, but a totally incompetent one. Don't even ask about the implementation I'm still not sure if that was even worse. Oh anbd this was done by a fully qualified state of the art developer with more letters after his name than I've got in mine. It took six months to develop this abortion, then they paid me for five months on contract to fix the leaks, bugs and add in truly radical new feature, a customer who can order more than once! It was still a complete tip when I left, I could have re-written the entire thing in two months, but that woud have wasted their earlier investment. They didn't want me to do it right, and they paid me 30 GBP per hour for five months to do it wrong, instead of admitting that they had f**ked up big time. This is corporatism in IT, short term, reactive, and cya.

Marc Thibault
Marc Thibault

It's a real problem and I sympathise. However. In the end, if you're now responsible for the product, you're responsible for what you make, and what you take. Cleaning up other people's messes is a bitch. You seem to be living in a Dilbertian universe that I've never actually seen in practice. Not to suggest that you're exaggerating, but every manager I've had to deal with could be led to understand that quality up front is the cheap, fast thing to do. The problem I have had, that you don't quite allude to, is with HR when I had to explain that to get programmers who can actually code, you have to pay 20% over the going rate.

Tony Hopkinson
Tony Hopkinson

:D Redefine the problem so your design is no longer sloppy... Saves money, gets it out on time, leaves smileys on the dash board, sells. And hopefully you can get promoted before version 2...

xmetal
xmetal

I agree with this 100%, and the example is exactly pretty much our philosophy as well. Crash = fix it Data loss = fix it Everything else is debatable. Where I work development, QA and tech support have "bug triage" meetings to qualify them.

SnoopDougEDoug
SnoopDougEDoug

I haven't yet seen the discussion of severity. A known user action that leads to a crash should be a S1 (or S0), which means you do not ship with a severe bug. Minor bugs, such as Arial 16 italic renders as Arial 14 on Firefox, is something you can punt to release notes.

Editor's Picks