As the general saying/joke goes, “There are two hard things in computer science: cache invalidation, naming things, and off-by-1 errors.” 😆
“Software companies face serious challenges in testing their products, and these challenges are growing bigger as software grows more complex. The first and most important thing to be done is to recognize the complex nature of testing and take it seriously.” – James Whittaker
Whittaker said that in January 2000, in an IEEE article. He’s right, and software has only become more complex in the last 25 years!
In my experience, software testing is one of the hardest problems in computer science!
11 Reasons Software Testing is Difficult
- Misunderstood. Software testing is often misunderstood. The lack of education, the myths, the judgements — “it’s so boring,” “not creative,” “it’s too easy that anyone can do it,” “it takes zero training since you’re just clicking links and buttons.” To me, it’s exciting, fun, and creative! The original software testers had Ph.D.s in computer science!
- Endless. We can’t test everything exhaustively. There is no upper limit, no ending, no finish line. It’s endless. We need to decide when to stop testing. The Pareto Principle comes into play here, loosely speaking: 80% of bugs come from 20% of the features.
- To Infinity. There are an infinite number of integration points, pathways, and combinations between functions, methods, classes, APIs, events, messages, systems, system of systems, settings, configurations, devices, etc. As Buzz Lightyear said, “To infinity and beyond!”
- No Guarantees. We can’t guarantee we’ve found all the bugs, defects, glitches, or all the ways it works correctly! Are you sure you covered every statement, branch, state, condition, device, and configuration possible? We don’t have the time, the resources, the funding, or even the compute power to test every single permutation possible!
- No Magic Formula. There’s no “one size fits all” solution, technique, formula, or algorithm to assure quality in software. We have a variety of techniques and tools to choose from that are complementary in usage. The more you use, the more you can prove correctness.
- Limited Time. We aren’t given enough time to solve problems. We are always under a time crunch to meet a deadline. Available time often gets removed from testers, and we never get time added.
- Limited Resources. We have limited resources to find all the bugs. Other than time, this comes down to not always having the right tooling, environment, architecture, or support to do the proper testing. We learn to do more with less.
- Communication. We always get the blame if something goes wrong, even when it’s not our fault. This coincides with not getting enough (or any) praise when everything goes right.
- Believability. Also part of communication, testers are often not believed when they identify a problem. Oftentimes, we hear: “Well, it works on my machine,” or “I can’t reproduce it.” We also deliver bad news often. Every bug report may be considered “negative or critical feedback” to the listener. Will they believe us? How will they react and respond to the feedback? Testers will continue to advocate for quality in the face of adversity and build resiliency.
- Lack of Measurement. It’s difficult to measure quality. How do you quantify something qualitative? These are often seen as adversarial attributes. Quality, by definition, is subjective, not objective.
- Repeatability. It’s human nature to seek out novelty! People often get bored if they have to repeat the same task over and over again, which happens when you verify bug fixes or do regression testing. Bugs get missed when you’re bored. This is one reason we use automation!
Conclusions
A lot of these can feel frustrating for engineers and business leaders, especially if you don’t enjoy the process of testing. You want a guarantee, 100% certainty, and finality? These feel good to us as humans! As testers, we want that, too, but we just can’t provide it. However, we can guarantee our knowledge, skills, education, experience, and our overall efforts — in other words, merit!
Again, to quote James Whittaker, “Hire the smartest (most passionate) people you can find, help them get the tools and training they need to learn their craft, and listen to them when they tell you about the quality of your software. Ignoring them might be the most expensive mistake you ever make.”
Do you agree that software testing is one of the hardest topics in computer science? Did I leave out any reasons that make it difficult?