I am changing gears a little in this blog and am discussing a recent paper and presentation (and soon to be book) that was made known to me via a LinkedIn discussion. The topic deals with common testing problems.
Donald Firesmith from the Software Engineering Institute (SEI) at Carnegie Mellon University has put together a comprehensive report on common testing problems. You can read the full text in his two-part blog: “Common Testing Problems: Pitfalls to Prevent and Mitigate” at:
https://blog.sei.cmu.edu/post.cfm/common-testing-problems-pitfalls-to-prevent-and-mitigate
https://blog.sei.cmu.edu/post.cfm/common-testing-problems-pitfalls-to-prevent-and-mitigate-2
In his paper and in his blog, Donald makes the following statement: “A widely cited study for the National Institute of Standards & Technology (NIST) reports that inadequate testing methods and tools annually cost the U.S. economy between $22.2 and $59.5 billion, with roughly half of these costs borne by software developers in the form of extra testing and half by software users in the form of failure avoidance and mitigation efforts. The same study notes that between 25 and 90 percent of software development budgets are often spent on testing. “
I was especially pleased to read about how “requirements” fit into the common problems Donald found. Note: This topic is of special interest to me since I have seen many projects fail because of poor planning for (1) the verification and systems validation phases of projects and (2) the activities required to address those defects that are attributed to requirements.
Donald included defective requirements as one of the categories of common testing problems: “Requirements-related testing problems are related to the requirements that should be driving testing. Often, the requirements are ambiguous, missing, incomplete, incorrect, or unstable. Lower-level requirements may be improperly derived from their higher-level sources. Likewise, verification methods may be unspecified and the tracing between requirements and tests may be lacking.”
Donald lists the following requirement defects as pitfalls to testing:
– Ambiguous Requirements
– Obsolete Requirements
– Missing Requirements
– Incomplete Requirements
– Incorrect Requirements
– Requirements Churn
– Improperly Derived Requirements
– Verification Methods Not Specified
– Lack of Requirements Trace
For each of these, Donald goes into more detail, including: defect description, potential applicability, characteristic symptoms , potential negative consequences, potential causes, and recommendations for mitigating these defects.
We address all of these defects and their mitigation in our training.
Donald’s work is excellent and is sorely needed. Response to his paper and presentation was so great that Donald has turned the paper into a book: Common Testing Pitfalls: Ways to Avoid Them and Lessen Their Impact, to be published by Addison Wesley, as part of SEI’s series in Software Engineering, later this year.
Comments to this blog are welcome. Also, if you are interested, send me an email at info@argondigital.com and I will send you a copy of my paper and presentation on “Thinking Ahead to Verification and Validation”.
If you have any other topics you would like addressed in our blog, feel free to let us know via our “Ask the Experts” page and we will do our best to provide a timely response.
