Save Hours of Rework By Implementing Work Product Inspections

We all face the dilemma of producing higher quality products at lower cost in less time. Achieving all three goals simultaneously seems impossible. This paper describes the proven technique of Work Product Inspection for removing potential problems at the source. Incorporating this technique into your product development process will increase product quality while reducing development cost and time.

Introduction to Work Product Inspections

We live in a world where our customers expect us to produce high-quality products quickly at the lowest possible cost. Yet, typically, product development projects exceed budget and are delivered late. Customer satisfaction seems to be getting harder to realize Although there can be many underlying causes for increased costs and slipped schedules, the main cause is usually traced to rework; the time, often unbudgeted, spent fixing defects or errors detected late in the development or maintenance life cycle.

The Work Product Inspection is a technique aimed at eliminating or dramatically reducing the rework required during a product development or maintenance project. The goal is to catch potential problems early, before they contribute to cost increases, schedule slips, and lower product quality at delivery. One hour spent on a Work Product Inspection can save tens or even hundreds of rework hours later. Applying this technique is a win-win situation. You succeed in reducing development costs while improving product quality and your customers benefit through reduced costs and higher satisfaction.

This paper provides a history of the inspection process, compares inspections to other types of reviews, and explores the relationship of inspections to testing. The paper also explains the roles of the participants, provides an overview of the inspection process, discusses the keys to conducting successful inspections, and provides tips for integrating the inspection process into a product development process.

History of Inspections

In the early 1970’s Michael Fagan, at IBM, was looking for a method to reduce the number of errors in the software designs and code in the project he was managing. He decided to adapt industrial hardware statistical quality methodology to his software project. The success of the resulting technique was documented in “Design and Code Inspections to Reduce Errors in Program Development,” IBM Systems Journal, vol. 15, no. 3, 1976. Use of the technique spread at IBM and was introduced to the rest of the world soon after. Thus the “Software Inspection”, alternatively “Fagan Inspection”, was born.

The original name and purpose of the inspection method has led to the myth that this technique is only for software designs (pseudo code) or source code. As the use and knowledge of the methodology increased, it has proven to be effective on any work product created during product development, software-related or not. A work product is a document, computer program, or other artifact produced during the course of developing an engineering or application product. A work product could be an interim or final project deliverable or a supporting document that enables the project to be completed successfully. Examples include various types of project plans, requirement specifications, design documents, user interface documents, user interface designs, schematics, blueprints, drawings, source code, test documentation, user and system documentation, training material, and process documentation.

Inspection includes activities undertaken to determine whether results conform to requirements. Inspections can be conducted at any level (e.g., the results of a single activity may be inspected, or the final product of the project may be inspected). Inspections are variously called software inspections, Fagan inspections, reviews, peer reviews, product reviews, audits, and walkthroughs. In some application areas, these terms have narrow and specific meanings.

The technique described in this paper is an adaptation of the method developed by Michael Fagan at IBM. I have chosen to call the technique Work Product Inspection. The choice of the term work product is used to help overcome the “software only” myth. The use of the term inspection is to tie the technique described to its rich software process heritage, where it was developed and perfected. Work Product Inspection should conjure a vision of a quality improvement technique applicable to any project.

Throughout the remainder of this paper, I will use the terms Work Product Inspection and Inspection interchangeably, as necessary, to improve readability and understandability.

Inspections Compared to Other Reviews

You’re probably thinking that your organization already does several other reviews on its work products, how is another review going to decrease development time and improve quality? The answer lies in the objectives and focus of the different types of reviews. The table below shows four major types of reviews and their objectives.

Although all four types of reviews will uncover defects, the inspection is the only review whose objective is focused entirely on defect identification and removal. The first three types of reviews are typically held when a work product is completed or at the end of a development phase. An inspection is an in-process review. It is conducted on portions of a work product as they are completed. The intensity of the inspection makes it impractical to conduct one on a fully completed work product. If a completed work product needs a Work Product Inspection, it will need to be “chunked” to meet the inspection rate guidelines detailed in the inspection process section.

Chunking is the logical division of a large work product to facilitate inspection. Thus the inspection of an entire work product is typically a series of inspections conducted on smaller cohesive parts of the work product. In many cases, completion of an inspection is a prerequisite for the other three types of reviews. It should be emphasized the Work Product Inspection is not a substitute for the other types of reviews. Work product inspections complement the other types of reviews and help to make them more efficient.

The number of review participants varies significantly between the different types of reviews. In the management and technical reviews, depending on the scope and size of the development project, group size can run into the hundreds! The significant difference is the organizational stratification of the participants. The attendees of reviews and walkthroughs typically represent a vertical slice of the project. That is to say, they are made up of management, technical leads, individual contributors, and support groups who are stakeholders of the project. Sometimes the customer is included in these types of reviews. The participants in an inspection represent a horizontal slice of the organization. This includes participants from outside the project, but associated with similar product development projects. This is why inspections are sometimes called peer reviews. If the work product to be inspected is a program management plan, the inspectors and the author should all be program managers.

The advanced preparation required for participants vary greatly with the type of review. It stands to reason that a participant responsible for making a presentation must prepare regardless of the type of review. In the management and technical reviews, the participants typically receive a data package when they arrive at the review. This data package allows the participants to follow along with the presenter(s). A detailed agenda is usually published in advance and included in the data package. The participants are not required to prepare in advance. Issues, discussions, and decisions are raised and initiated during the review. Participants use their domain knowledge, issue logs, and understanding of project objectives and goals to keep the review moving and bring it to closure. The data package distributed for a walk-through typically is the work product to be reviewed. The participants of a walk-through are asked to have a basic familiarity with the work product under review, but since the main objective is education, advanced preparation is frequently unchecked. In an inspection, all participants must prepare in advance. They need to review the prerequisite documentation, the standards, and conventions, as well as the work product to record potential problems. The participants must identify their issues and log their potential problems in advance of the Inspection Meeting. In fact, if any of the participants in an inspection come unprepared, that is reason to postpone the Inspection Meeting to give them time to complete their preparations. The pace and intensity of the Inspection Meeting make “real-time” review impossible. Any attempt to accomplish this feat dramatically decreases the efficiency and effectiveness of the inspection.

All of the reviews have some data collection requirements. Meeting minutes are typically kept. There may also be an action item log. The purpose of these data items is to let people who did not attend know what happened and identify who is responsible for resolving any issues generated during the review. In an inspection, there are formal data collection requirements. At a minimum, time spent preparing for and conducting the inspection, documenting potential errors, and correcting actual errors is recorded. Additionally, data such as type of error, severity of error, and error injection point can be recorded. The purpose of collecting this data is to gauge the efficiency and effectiveness of inspections. Efficiency is measured by defects found per hour invested. This is based on time spent discovering errors. Effectiveness is measured by cost to correct discovered defects. Thus based on measured efficiencies and effectiveness, an organization can project an annual cost savings per year for a Work Product Inspection program. A secondary purpose is to create feedback used for process improvement. The actual data remains with the project. It is not released to the general population of the organization. An aggregation of review data is presented for reporting purposes and as feedback for process improvement.

The leader of a management review, technical review, or walk-through is the responsible manager, lead engineer, or author respectively. Although there usually is a standard agenda and maybe a review procedure document, very little training is required to lead these types of reviews. The moderator of an inspection must be trained in the inspection process and as a facilitator if inspections are to succeed. The moderator is a very busy person. During the inspection process, the moderator will act as planner, strategist, mentor, coach, and facilitator. The moderator should be senior, well-respected, and technically competent.

The inspection process is very focused on the following characteristics:

• There are formal “entry” and “exit” conditions

• Participants must prepare in advance

• The author does not present the material

• There is NO discussion of defect resolution or defect validity

• Inspector roles are defined to increase team efficiency

• Customized checklists are used to guide inspectors in locating defects

• Metrics are used to guide presentation time and inspection rate

• Process improvement is an additional goal

The other types of review tend to focus on the question “Are we building the right product?” In contrast, the inspection focuses on the question “Are we building the product right?” This is the same question testing addresses. Consequently, the inspection process has a strong relationship to testing.

Inspection in Relation to Testing

Inspections and tests are both designed to uncover errors and potential problems in work products before the product is given to the customer. Inspection can be applied much earlier than testing. Using inspections, errors in plans, requirements, architectures, and designs can be detected and removed before subsequent investments in the errors can occur. Many errors usually found during testing can be removed much earlier with inspections. But inspection is not a replacement for testing. Tests are conducted in a product’s intended environment. Inspection cannot do that. Inspection is a static process, and testing is dynamic. Inspections and testing are complementary. A good inspection process will improve test effectiveness and significantly decrease the time required to complete a test cycle.

Work Product Inspection Participants

The number of participants in an inspection range from a minimum of three to a maximum of six. The minimum number of participants is derived from the required roles to be assigned. While the maximum number of participants is driven by decreased return on investment, explained by the law of diminishing return. Having more than six participants does not increase the number of defects uncovered, but does increase the cost of effectiveness. Plainly stated, having more than six participants increases the cost per defect discovered to unacceptable levels.

The participants in an inspection are all assigned at least one role. The roles in an inspection are:

• The Moderator *

• The Recorder *

• The Author or Producer *

• The Reader

• Inspectors

* Required roles

Each participant in an inspection is an inspector. Participants can fill multiple roles with two exceptions. The author can be neither the moderator nor the recorder. Likewise, the moderator should not serve as recorder. The author must be free to listen to the comments as they are given and provide clarification if required. The recorder is extremely busy documenting the potential problems and recording the other required data. An explanation of each role is given in the following paragraphs.

The moderator is responsible for ensuring the item to be inspected has met the entry criteria for inspection readiness. If these criteria are met the moderator plans the inspection. This involves selecting the other participants, arranging for meeting rooms, ensuring the inspection data package is prepared and distributed, and ensuring sufficient time for participant preparation. During the preparation phase, the moderator acts as mentor and coach to the participants ensuring that they are prepared. The moderator facilitates the Inspection Meeting, maintaining the inspection focus. The moderator follows up with the author to ensure all potential problems have been properly handled. When these items have been correctly disposed of, the moderator ensures all of the required data has been recorded and signs off that the inspection is complete.

The recorder is responsible for logging the potential defects during the Inspection Meeting. This should be done as rapidly as possible, ensuring the essence of each comment is recorded without logging them verbatim.
The recorder also writes down the other data collected during the meeting. It is important the other participants remember the recorder is not a secretary, but an active participant. The recorder, along with the other participants, is responsible for eliminating duplicate comments.

The author or producer submits their portion of a work product for inspection. Additionally, they provide the rest of the reference material that makes up the inspection package distributed to each inspector. The author assists the moderator in planning the inspection and also acts as an inspector during the preparation phase. During the Inspection Meeting, the author answers questions about the work product and provides clarification as needed. The author is responsible for determining the validity and disposition of all issues recorded during the meeting. After obtaining the moderator’s concurrence, the author modifies the work product.

The reader is the participant responsible for leading the inspection team through the work product during the Inspection Meeting. The moderator may be the reader to help control the pace of the meeting. Alternatively, the recorder may be the reader to ensure all comments get logged quickly. The reader does not actually read the work product, but usually uses paraphrasing or some other technique to get through the work product without confusion. The reader is also responsible for determining the most logical way to present the work product during the meeting.

Each participant, as an inspector, is responsible for objectively examining the work product for potential defects. To aid the inspectors, checklists are provided to focus their efforts. The items on the checklists are derived from data from previous inspections on this type of work product. To increase the inspector’s efficiency, inspectors may be assigned responsibility for certain portions of the work product or assigned an emphasis role. In the former, the inspector is responsible for finding all defects in the portion of the work product assigned. In the latter, the inspector is responsible for finding specific types of problems in the entire work product. In either case, an inspector may record any potential defect discovered even if it is outside their area of responsibility.

All participants should be trained in the inspection process. Additionally, the moderator should receive training in facilitating meetings and leading inspections.

The Inspection Process

There are six phases in the inspection process. They are Inspection Planning, Inspection Overview (optional), Inspection Preparation, Inspection Meeting, Work Product Rework, and Inspection Follow-Up. A brief description of the activities in each phase is provided below.

For the Inspection Planning phase, the moderator, in conjunction with the author, ensures the following activities occur:

• The work product is ready for inspection

• Inspectors are selected

• The inspection is scheduled and planned

• An inspection package is prepared and distributed

• The appropriate project quality assurance person is notified

The inspection package contains the material that the participants need to prepare for the inspection. The inspection package should include:

• Work product to be inspected (line numbered if possible to aid in potential defect logging)

• Supporting documentation (requirements or work product from which the work product is to be inspected was derived)

• Checklists

• Inspection cover sheet to be completed

During planning, the inspectors’ preparation time is estimated. The recommended maximum inspection meeting time is two hours. An organization will determine their own preparation times based on the data collected during inspection meetings, but for an organization just starting there are guidelines. The time for participant preparation and the meeting time are about the same – for a two-hour meeting the preparation time is two hours per participant. Typical preparation rates for inspections are 4 – 8 pages per hour for documentation and 300 – 600 lines of code per hour. From these rates, you can see that you don’t inspect a typical requirement specification in one meeting. The document must be “chunked” into logical pieces that can be completed within the planning guidelines.

The Inspection Overview is an optional phase. The moderator and author should determine the need for an Inspection Overview during the Inspection Planning phase. If the inspection under consideration is a normal review of a work product that is typically produced by the organization and the chosen inspectors are familiar with these types of work products and their background, an Inspection Overview is not required. However, if there are special circumstances or some unique aspect surrounding this inspection, the Inspection Overview is an opportunity for the moderator or author to provide the inspectors with the information they need to ensure the success of the inspection. Examples of circumstances where an overview would be appropriate to include are: a critical work product that will affect all downstream work products, unusual work product complexity, work product uses new or infrequently used technology, or the project is sufficiently small that inspectors must be drawn from outside the project. The Inspection Overview, if used, should not exceed the planned duration of the Inspection Meeting.

During the Inspection Preparation phase, the inspectors individually get ready for the Inspection Meeting. Using the checklists and other reference material provided in the inspection package, they examine the work product ensuring that the item meets its requirements and adheres to the appropriate standards and conventions.

The checklists are crucial to the efficiency of inspections. The checklists allow the inspectors to look for defects typically found in the organization’s work products. These defects are determined historically from data collected during inspections and testing of similar work products. Checklists can be arranged by emphasis role. Emphasis roles allow the inspectors to focus on a specific aspect of the product under review and help further increase the efficiency of the inspection. Although any inspector can record any defect they find, they are only responsible for their assigned portion of the checklists. For example, errors made in requirement specifications are typically incorrect facts, omissions, inconsistencies, and ambiguities. Appropriate emphasis roles for a requirement document would be:

• Review for incorrect facts and ambiguities

• Review for omissions and inconsistencies (trace established)

• Review for testability

The moderator facilitates the Inspection Meeting. The reader leads the inspection team through the work product using an appropriate technique such as paragraph by paragraph, paraphrasing, or through scenarios. The recorder captures the defects identified during the inspection. The team, by consensus, classifies the captured defects by severity, type, and cause. At the conclusion of the Inspection Meeting, the team determines the disposition of the work product by consensus.

It takes great skill on the part of the moderator to keep the meeting on track. It helps if the inspectors adhere to the following guidelines:

• Be prepared (the majority of inspection failures are due to a lack of preparation)

• Review the product and not the author

• Raise issues, don’t resolve them

• Avoid discussions of style

• Avoid discussion about whether an issue raised is a problem

Because of the intensity of the inspection and because the participation of all inspectors is required for complete coverage of the work product, the Inspection Meeting should be canceled and rescheduled for any of the following reasons:

• Key inspector missing

• Lack of preparation by any inspector(preparation time recorded at start of meeting)

• Defect with large impact found (enough rework to dramatically change the work product)

• Ineffective inspection process (defect detection rate much less than expected)

During the Work Product Rework phase, the author resolves all potential defects. Resolution can take one of three forms. One, the defect in the work product is corrected. Two, the correction of the defect is deferred (i.e. a change request was initiated for correction at a later time). Three, a determination is made that the issue is not a defect, and the comment is retired. The effort expended during work product rework must be captured, reflecting time for fixing defects, disposing of defects, and rewriting material for re-inspection as part of the inspection data recorded on the inspection cover sheet. This data is used to gauge inspection effectiveness (cost per defect removed).

During Inspection Follow-up, the moderator examines the rework and the issue dispositions to ensure that all potential defects are disposed of. Any material scheduled for re-inspection is re-inspected. Any remaining information on the inspection cover sheet is completed. The moderator signs and dates the inspection cover sheet to bring inspection to closure.

Keys to Success

To be successful, the inspection process must be ego-less. That is to say, inspectors must review the work product and not the author. Inspectors should refrain from comments that are merely matters of style or personal preference. They should not take ownership of their comments, intending to ensure their comments are incorporated in the work product. The comments are, after all, potential defects until examined and disposed of. Inspectors must trust the moderator and the author to satisfactorily resolve these issues. This means that re-inspect should only be used as a work product disposition when necessary. Re-inspect is not a means to verify that certain comments were included! At the same time, the author must not take umbrage at comments made by inspectors. Pride of authorship can kill the inspection. If the author is asked for clarification, the author should respond freely and frankly, without getting emotional (angry, sarcastic, etc.). There are three keys to making this happen.

First, management and all potential participants in the inspection process must be trained in the inspection process. This training should include the benefits of doing inspections, the inspection process, and the potential roadblocks to conducting inspections. Additionally, participants expected to act as moderators should receive training in work product inspection planning, meeting facilitation, and problem resolution.

Second, checklists are crucial. They provide the focus and contain the knowledge to ensure Work Product Inspections are effective and efficient. It is the responsibility of the moderator to ensure that checklists are included in the inspection package. If checklists don’t exist for the work product under consideration, the moderator must prepare them. Checklists should be living documents. As additional data is collected, checklists should be updated to reflect current experience.

Finally, management support of the inspection process is mandatory. While the savings achieved during the testing phase of a project are substantial, it takes time and effort to conduct inspections. This time must be planned and budgeted. Without management acceptance and support any attempt to implement a Work Product Inspection process will fail.

Conclusions

We are searching for techniques that will lower our product development costs without sacrificing product quality or customer satisfaction. Work Product Inspection is a proven method for achieving that goal. Organizations that have implemented inspection processes have reported significant gains in development productivity, development time reductions, and reductions in test time and cost. Karl Wiegers provides the following examples in his book Peer Reviews in Software, A Practical Guide. Raytheon Electronic Systems reduced its rework level from 41 percent of total project cost to 20 percent in two years. Raytheon also reduced the effort needed to fix code problems found during integration by 80 percent and cut its retesting effort in half. Hewlett-Packard’s inspection program measured a return on investment of 10 to 1, saving an estimated $21.4 million per year. IBM finds up to 90 percent of the defects in a project through inspection. They also report that each hour of inspection saved 20 hours of testing and 82 hours of rework needed if the defects found by inspection had remained in the released product. Notice there are no disclaimers such as, “Results reported are not typical”. Different organizations report remarkably similar results from implementing inspection processes.

Work Product Inspections are a review by a work product producer’s peers. They focus on removing defects and ensuring that each successive phase of the project is based on the highest quality inputs. The inspection process has been proven useful on any work product of a product development effort. Work Product Inspections are complementary to testing and dramatically increase the effectiveness of testing efforts. When Work Product Inspections are used as entry criteria for other types of reviews, the effectiveness of those reviews also improves.

When I teach this technique to an organization, I challenge them to bring their best, most mature requirement specification to use as
the work product for the practice inspection. I also provide a basic checklist appropriate for use in a requirements inspection. During the exercise follow-up, we explore the efficiency and effectiveness of the practice session. The examination includes the total number of potential defects discovered, how many major and minor potential defects, and defects discovered per hour invested. After looking at the numbers, I encourage the students to discuss what worked well and what didn’t work. At one session, the client had chosen the recently completed requirement specification for a new product offering as the artifact. This new product was intended to sustain this organization into the foreseeable future. The organization was under considerable time-to-market pressure to attain this goal and felt that they had done a much better job of specifying the product than usual. When I asked the what worked well question, the development manager responded, “We uncovered two major defects that if we had proceeded would have been absolute showstoppers. The project would have failed. Now we can recover and still meet our goals.” Since the entire development team was in the room, any residual skepticism evaporated immediately. Instantly, I was in a room of believers! Rarely are the results of the exercise this dramatic, but they are always convincing.

Integrating a Work Product Inspection process with a product development process is relatively straightforward. First, obtain management support and buy-in for inspections. This support includes the budget, schedule, and resources to support the inspection process. Second, proper implementation of Work Product Inspections requires all participants and managers to be trained in the benefits, roles, and conduct of inspections. Additionally, moderators should receive training in inspection planning techniques and facilitation skills. Third, implement the process on one or more projects. Adjust the process to the organization’s culture and product development environment. Finally, collect and analyze the data. Once implemented, the data collected during each inspection will show the return on investment inspections are providing.

Work product inspections are not a panacea for all of a development project’s ills. However, if your organization is involved in product development or maintenance and you are interested in improving quality while lowering cost, this technique deserves a closer look.

About the Author

Larry A. Fellows

Larry has over twenty-five years of international experience in project management, process definition and improvement, product development, and consulting. He has extensive experience in many aspects of requirements engineering and management, a key product of CAI.

Larry has an in-depth knowledge of the Software Engineering Institute’s Capability Maturity Model (SEI CMM). The model is a globally accepted model for software process. He is an Authorized Lead Assessor in the SEI’s Assessment Program. Larry also has training in the System Engineering Capability Model and the Capability Maturity Model Integrated, which combines the software and system engineering models.

His product development experience includes systems engineering, software development, software and systems test, system integration, and operational evaluations. Products he has worked on include communications systems, inertial and satellite navigation systems, submarine combat systems, and aircraft landing systems.

You can reach Larry Fellows at info@argondigital.com. If you want additional information about CAI products or services, please contact us via e-mail: info@argondigital.com.

Copyright ArgonDigital © Compliance Automation, Inc., All rights reserved.

Print or Save to PDF