I am excited to see that the Government Accountability Office (GAO) just released a draft of a new document: “Technology Readiness Assessment Guide” (TRA Guide). As the title suggests, the focus of the TRA Guide is on assessing the technical readiness of technologies critical to the success of a project. In this blog, I start by doing a bit of level setting by discussing a few key concepts covered in previous blogs before I move to the main discussion wherein I provide an overview of what is covered in the TRA Guide.
In a previous blog “Using Technology Readiness Levels to Manage Risk”, I discussed the use of Technology Readiness Levels (TRLs) as a concept to help mitigate risks to projects associated with the inclusion of new technologies in a product.
Most recently, I have posted a multipart blog titled “Concept Maturity Levels” (CMLs). In this blog I discussed how CMLs can be used by management and development teams as an approach to measuring the maturity of a project’s system concept for all gate reviews defined by an organization. CMLs provide a standardized method that allows management to:
– determine how much work (funding, resources, and effort) has been placed into the definition and maturation of a system concept;
– compare competing project system concepts in terms of relevance to meeting the organization’s strategic goals, objectives, and Return on Investment (ROI) with acceptable risk (ROI includes benefits and value);
– determine which system concepts have had the same level of work and can be compared on the same terms;
– understand the maturity of technologies needed to meet the project’s goals and objectives,
– understand how much future work a system concept will require to get to a subsequent level of maturity; and
– have the information needed to determine when a proposed project’s system concept is mature enough to proceed to the next stage of system development.
Advancing from an CML to the next and moving from one lifecycle stage to another involves a gate review where the development team presents evidence to management that the system concept is at the CML needed to proceed to the next lifecycle stage. A key consideration during a gate review is an evaluation of the maturity of critical technologies being proposed as part of the system concept needed to meet the goals and objectives of the project.
Previously, the GAO has shown that using effective management practices and processes to assess how far a technology has matured is critical to evaluating its readiness to be integrated into a system and managing the risk associated with the integration of the new technology into a system. They have also shown repeatedly that understanding the maturity of technologies at the start of system development has a direct affect on the schedule and cost of developing a product stating that “technologies that were included in a product development before they were mature later contributed to cost increases and schedule delays in those products.”
How do you determine the maturity level of a technology?
A key question many have is how to determine the maturity level of a technology. As stated in the GAO TRA Guide: “TRLs are a scale of levels from 1-9 that represents a measure for systematically communicating the readiness of new technologies or new applications of existing technologies to be incorporated into a product. Technology development is a continuous discovery and development process reflecting close collaboration between the science and technology community, the user, and the system developer. It is iterative, designed to assess the viability of technologies while refining user requirements.” The TRL is dependent on the assessment of the maturity of the technologies used in all parts that make up the system. The system TRL is the lowest TRL of the parts that make up the system.
In the past, the main assessment tool was the TRL Calculator – a spreadsheet that allowed management and developers to answer key questions by checking off boxes, and the spreadsheet would indicate the TRL level of the parts of the system and the overall TRL of the system.
A key problem with developers and project managers using the TRL Calculator is that they tend to be over optimistic. As stated in the GAO TRA Guide “Program managers may believe that lessons learned from past programs will benefit their program and assumptions about the maturity of certain technologies may not be closely scrutinized. Or, they may be more willing to take on greater risk and accept immature technology because their promised performance is vital to obtaining funding and stakeholder buy-in. In addition, in today’s competitive environment, contractor program managers may be overly optimistic about the maturity of critical technologies, especially prior to contract award.”
In the GAO TRA Guide, the point is made that this may not be the best approach and a more formal approach is needed in order to objectively determine and communicate the TRL of the system. This more formal approach is called the Technology Readiness Assessment (TRA) and is the focus of GAO’s TRA guide.
What follows is a summary of what is covered in the GAO TRA Guide. ” ” paragraphs represent text copies from the GAO TRA Guide.
Technology Readiness Assessment
“A TRA is an evaluation of the maturity of critical elements of a product’s technologies, often called critical technologies. It is a normal outgrowth of the system engineering process and relies on data generated during the course of technology or system development. A TRA is a systematic, evidence-based process that evaluates the maturity of hardware and software technologies critical to the performance of a larger system or the fulfillment of the key objectives of an acquisition program. TRAs, which measure the technical maturity of a technology or system at a specific point in time, do not eliminate technology risk, but when done well, can illuminate concerns and serve as the basis for realistic discussions on how to mitigate potential risks as programs move from the early stages of technology development, where resource requirements are relatively modest, to system development and beyond, where resource requirements are often substantial.”
“Technology elements are considered critical if they are new, novel, and the system being developed or acquired depends on them to meet its performance requirements within defined cost and schedule parameters. All critical technologies must be identified to achieve a comprehensive evaluation of technological risk. Critical technologies can be subcomponents, components, subsystems, or systems composed of hardware, software, or both.”
TRA Report
The results of an TRA are documented in a TRA report. TRA reports provide technology developers, program managers, and governance bodies with credible, objective and useful information about the maturity of technology, its state of development, and potential areas of risk.
TRA reports are prepared so the project team can report the progress of their technology maturation efforts and to certify the readiness of critical technologies to support the system concept at gate reviews or key decision points. They may also document the results of discovery learning needed to advance a piece or component technology to understand what is needed to enable the greater product to advance to the next lifecycle stage or gather evidence whether or not to continue development efforts or initiate steps toward using an alternative or backup technology.
TRA Reports are also an important part of knowledge-building exercises that document the maturity of technologies as part of a project whose purpose is to advance the TRL level of a technology. These projects are maturing the technology independent of any specific project or program with the intent of enabling a future project to integrate the technology into their system once the technology has reached the desired TRL appropriate for the project’s stage in the development process. For NASA, these types of technology maturation projects are managed per the requirements in NPR 7120.8. For commercial enterprises, these types of projects would be managed within their research and development (R&D) organization.
TRAs conducted as part of knowledge-building exercises for project managers and technology developers are conducted with a focus on accessing the progress made in maturing technologies (advancing the TRL). The resulting TRA reports can be used to
– communicate what was learned about specific aspects of technology development (that is areas that may be challenging or areas where our knowledge is limited and further work is needed to fill in gaps in knowledge),
– document assessment findings on whether critical technologies are ready for integration in to a project.
The information in the TRA reports can be used by technology developers and program managers to assist them in their day-to-day responsibilities for maturing critical technologies, systems, and sub-systems, i.e., what areas to focus on, gaps in knowledge, etc.
The information communicated in the TRA reports can also be used as the basis for developing or updating the project’s Technology Maturation Plan (TMP) —a planning tool that lays out the steps and actions to bring immature critical technologies to a designated TRL or higher maturity.
Technology Maturation Plan
A key part of program/project management is the inclusion of TRAs as part of the overall project planning. Conducting TRAs involves the identification of the technologies critical to project success and developing a plan to evaluate the maturity of these technologies and to advance the maturity of these technologies. “This planning typically begins where analytical and experimental critical function or characteristic proof of concept occur at TRL 3 through the completion of the system and its qualified test and demonstration at TRL 8.” This plan is referred to as the TMP.
As stated in the GAO TRA Guide, “The TMP is a management planning tool to help ensure critical technologies are evaluated and that evidence of the evaluation and the results are retained both for proof that the processes were undertaken and for evidence of progress toward maturity. The TMP is developed for critical technologies identified in the TRA report that do not meet specific TRL goals or expectations where gaps exist that require further evaluation, testing or engineering work in order to bring the immature technology to the appropriate maturity level.”
“The TMP lays out the necessary steps, actions, and resources needed for maturing critical technologies that have been assessed as less mature than desired or are lagging in maturity compared to other critical technologies. The purpose of the plan is to bring them to a higher or acceptable maturity or readiness level. The TMP uses TRA results and other information for establishing a road map with the necessary development and engineering activities to mature technologies. As such, it provides an effective gauge of the overall progress of technology maturation.”
“The TMP is also useful as a key reference document at gate reviews to verify that sufficient progress has been made in closing the maturity gaps appropriate to the lifecycle stage that is the subject of the review. For these gate reviews, TRAs and the resulting TRA report provide management evidence that the product’s technical development is progressing as desired and that technologies are mature enough to move to the next phase of development. TRAs can be used to protect program managers from unknowingly accepting or being forced to accept immature technologies into their programs.”
To summarize, the steps involved in TRAs are:
- Identity a candidate system architecture
- Identify critical technologies
- Perform TRA
- Document the result in a TRA report
- Based on the assessment, develop a TMP
- Update the TRA, resulting report and TMP during each major product development lifecycle and present the current status at each gate review
Several key points made in the GAO TRA Guide include:
– TRAs must have the following characteristics: They must be credible in both their design and execution, objective in their evaluation of credible evidence, reliable in the process used to conduct them, and useful to technology developers, program managers, and governance bodies.
– The TRA is conducted and executed by an assessment team of knowledgeable individuals, often outside of the program/project office, who have expertise with the technologies to be evaluated and bring objectivity and independence to the activities.
– For funded projects, TRAs need to be included in the project planning documents such as the systems engineering management plan (SEMP), identified in the project’s master schedule, and included in the project manager’s budget.
– For technology maturation projects, TRAs need to be included in the project’s project plan and TMP.
– TRAs are conducted and the resulting TRA Reports updated as part of the activities to conduct a gate review. Part of the entrance criteria to have a gate review needs to include a completion of an TRA or update to the TRA. Part of the exit criteria for the gate review needs to include proof that the critical technologies discussed in the TRA Report is mature enough to proceed to the next development stage.
– The TMP is a “living” document that needs to be updated as progress is made, new information comes to light, or conditions that materially affect the plan ensue. The program/project manager or designated lead is responsible for monitoring, tracking and making adjustments to the TMP as necessary. If a subsequent TRA triggers an update to the TMP, the program manager needs to establish a schedule to ensure the update and its completion before the next TRA and gate review. The updated TMP serves as a source document as part of a TRA for the assessment team.
