Concept Maturity Levels Defined (tailored to be domain independent)
– In this first part of this blog, I introduced the benefits and need for organizations to include CMLs in their systems engineering processes.
In this second part, I go into more detail defining and explaining the activities and outcomes for each CML.
The CML rating scale shown in Figure 1 is expanded as shown below. The idea for a CML scale is to address the common path of progression through the project initiation phase from initial idea through Critical Design Review (CDR) or comparable gate review. Varying levels of concept maturity may entail differing levels of fidelity of project planning and engineering analysis needed for broader vs. more localized trade studies, and varying techniques for cost, schedule, and risk analysis.
Figure 1: CML summary
CML 1 – “Cocktail Napkin”– Overview and Advocacy – used to launch a new concept. In the overview, the problem (or opportunity) is defined, communicating why this project is worthwhile. (For more details on defining the problem, see my blog: “What problem are you trying to solve?”) The essence of what makes the concept unique and meaningful is clearly stated. The project vision and “Need” statement is defined showing the top level reason for developing the system and providing a focus for the team. The overview lays out a broad outline of the intention of the project, high level benefits, overall advantages, but provides limited details of cost, schedule, or other programmatic or design information. Think of the overview as the “30 second” elevator speech addressing what the problem being solved, or opportunity being pursued, is and what the Need is – in other words, the desired end result at the end of the project once the system is operational.
From an advocacy viewpoint, information needed to sell the idea is documented. Key stakeholders are identified, the benefits, advantages, and ROI that result from pursuing this concept are defined for the stakeholders and for the organization. Needed capabilities are defined in terms of people, process, and products along with what needs to be done to provide these capabilities. A rudimentary “back of the napkin” sketch of the system concept is developed.
The advocacy perspective expands on the overview, going into more detail as to both what needs to be done to meet the Need and how management will know if the project is successful. Preliminary goals and objectives are defined including preliminary measures of effectiveness (MOEs) and Key Performance Parameters (KPPs) including preliminary key figures of merit concerning safety, reliability, sustainability, affordability, extensibility/evolvability, agility/robustness, effectiveness, and innovation.
Assumptions are documented and system level drivers and constraints are identified including cost, schedule, standards, regulations, technology, resource availability, interaction with existing systems, and higher level requirements allocated to the system of interest.
CML 2 – Initial Feasibility: The system concept is expanded and assessed on the basis of feasibility (cost, schedule, technology) from ROI, technical, and programmatic viewpoints. Preliminary system life-cycle concepts are documented including concepts for development, procurement, system verification, system validation, manufacturing, transportation, deployment, operations, maintenance, upgrades, and disposal. Enabling systems required during development and operations are identified.
The system concept is expanded to address how the proposed system will fit within the macro system of which it is a part. A preliminary functional architecture of the macro system and system of interest is developed. How the systems that make up the macro system are related (connectivity, interaction, and flow of information) to each other and to your system of interest is defined. External interfaces are identified. High-level schedule, resource, and cost estimates are developed. Key risks are identified.
Outside influences the system needs to take into account are addressed. Need, Goals, and Objectives (NGOs), MOEs, and KPPs are refined based on the information gained at this level.
CML 3 – Trade Space: The system functional architecture is transformed into an initial physical architecture. This physical architecture is used to access the options of the system to be developed meeting the NGOs within the defined drivers and constraints.
The study/design team will perform architecture trades between the system, subsystems, support systems, critical technologies, and enabling systems with elaboration to better understand the relationship between the parts of the architecture and impacts to ROI, cost, schedule, and risk.
The system (product, device, application, etc.) concept is demonstrated to be viable within the defined drivers and constraints, technology maturity, proposed macro architecture, and operating environment.
Objectives, MOEs, and KPPs are refined based on the information gained at this level. Measures of Performance (MOPs) are defined. Key risks are investigated, updated, and possible mitigation strategies outlined.
A key consideration at this stage is understanding the partials, that is how ROI changes as a function of some system parameter (e.g., mass, data volume, communication bandwidth, computing power, operating environment, technologies used, etc.). How will the ability of the system to meet the NGOs, MOEs, MOPs, and KPPs be affected as a function of changes in cost, schedule, technology, or other key parameter? How will the cost, schedule, technology, and schedule be effected by a change in an KPP?
For complex systems where many of the parameters across the system architecture have a dependency (directly proportional or inversely proportional) it is very hard to track and understand the partials “manually”. For complex systems, we recommend the study/design team use Systems Engineering (SE) software tools to develop a model of the system including the subsystems that make up the system as well as the macro environment in which the system will operate. With such a model, the team can make changes to the MOPs and KPPs as well as external constraint parameters to access the impact of the change to meeting the NGOs and other key system parameters. This knowledge will be of great benefit in choosing or optimizing the system architecture within the defined drivers and constraints. (See my blog on Model Based Systems Engineering (MBSE): MBSE and Requirements.)
Another key consideration when performing architectural trade studies is the TRL of the candidate parts of the architecture. Trade studies will guide which technologies are needed to best meet the project’s objectives.
Critical technologies needed to meet the goals and objectives, MOEs, and KPPs are identified. An initial Technology Readiness Assessment (TRA) is performed per the processes outlined in the GAO Technology Readiness Assessment Guide and the resulting TRA Report is generated.
If the current capability to meet an MOP or KPP is beyond current state-of-the-art for that part, then new parts/technology with the potential to provide the desired capability will have to be considered. Best practices and guidance from the Government Accountability Office (GAO) state that the system architecture should not rely on parts/technology with a TRL less than 3 at the time of scope baseline (CML 5) and a plan needs to be in place to mature the technology to at least TRL 6 by the time of the Preliminary Design Review (CML 8) and TRL 7 by the time of the Critical Design Review (CML 9). For more information on TRLs, see my blog “Using Technology Readiness Levels to Manage Risk”.)
CML 4 – Point Design – Architecture selected within Trade Space: Candidate system architectures are identified that will implement the system concept selected to meet the project NGOs and achieve the desired ROI providing the needed capabilities, functionality, performance, and quality within the specified drivers and constraints within the trade space. These architectures are defined to the level of major subsystems with acceptable cost, schedule, risk, and resource margins and reserves. (For a detailed discussion on the importance of defining development and operational margins and management reserves at this stage of the development lifecycle, see my four-part blog: Resource Margins and Reserves.) Subsystems’ trades are performed. Descope and backup options are defined. The objectives, MOEs, MOPs, KPPs, cost, schedule, and risks are further refined based on the information gained at this level.
Operational scenarios and/or use cases are defined (nominal, alternate nominal, and off-nominal) and the system concept and candidate architectures are assessed based on their ability to successfully address each of the operational scenarios and/or use cases. Stakeholder roles and functions are defined and how various stakeholders (operators, maintenance and software update personnel, etc.) will interact with the system during operations are documented. The focus in on roles and functions – what the stakeholders do with and how they will use the system once it is operational. A “day-in-the-life” of the system from the stakeholder’s viewpoint is addressed. How does data and information flow between architectural elements? How are the capabilities of the overall system used to accomplish its intended purpose in the operational environment?
Organizations may use “rapid prototyping” of potential physical solutions of a single component, subsystem, or system. With advances in 3D printing, study/design teams can develop prototypes quickly to be used as part of the architectural trade activities. When prototypes are developed, various configurations can be made available to the actual users. These prototypes can be functional, non-functional, or just “form and fit”. A prototype is useful to help the study/design team understand the user needs and expectations which will drive the system requirements. Prototypes are developed based on the currently known requirements. By using prototypes, the users can get an “actual feel” of the system in representative operational environments.
Also referred to as “discovery learning”, user interactions with the prototypes can enable both the user and study/design team to better understand the requirements for the desired system. Rapid prototyping allows for user feedback much earlier in the development lifecycle concerning errors and problems, such as missing functions, substandard performance, safety issues, and confusing or difficult user interfaces. This is an interactive process that allows users to understand, modify, and eventually approve a working physical model of the system that meets their needs.
From the operational scenarios and user interactions with system prototypes, a draft set of system requirements are developed including core functional requirements and associated performance requirements, requirements to implement defined objectives and MOPs, requirements addressing defined drivers and constraints, and system level interface requirements.
The TRA and resultant TRA report are updated. Based on the results of the TRA a Technology Maturation Plan (TMP) is created per the processes outlined in the GAO Technology Readiness Assessment Guide to mature the critical technologies needed to meet the goals and objectives, MOEs, and KPPs.
Based on the knowledge gained to this point and the activities completed, the study/design team needs to determine whether or not they are ready to proceed with the activities associated with CML 5 or go back and repeat activities associated with CML 2-4 to further refine the system concept.
Note: it is a myth that design work shouldn’t start until after the scope and requirements are baselined. A key outcome of the Scope Review (CML 5) is that there is at least one feasible concept that will result in the NGOs being achieved within the given drivers and constraints with acceptable risk. Lessons learned from key programs indicate that design work during the early concept maturation phases can identify important issues that are difficult to determine until a design concept is of sufficient maturity to evaluate a more complete set of development, test, operations, and evaluation considerations. Using the knowledge gained from prototypes and trade studies, a design concept should be developed, as early as possible in the project formulation phase, to allow a more complete understanding of programmatic implications and development challenges and risks. The design work at this stage enables validation of the draft requirements, providing analysis to determine where something is missing or perhaps not needed. This validation is a key part in accessing feasibility of the concept. Prototyping results in a system concept that is much more than just an artist rendering or a parametric model.
CML 5 – Scope/Concept Review or analogous gate review Baseline: The implementation approach is defined including project management, systems engineering, contracting mode, integration, system verification and system validation approaches, cost and schedule. Relationships and dependencies, partnering, key risks, mitigation plans, and system “buy (from an external source), build/code (make internally), reuse/modify (existing systems)” or even buy/try/decide approaches are defined. The TRA is completed verifying a minimum of TRL 3 for all parts of the architecture and a plan for advancing the TRLs to at least TRL 6 by PDR is complete, for systems going operational. (The purpose of some projects may be to advance the TRL. In this case advancing to TRL 6, may be the goal of the project.) . A viable, feasible, and stable system concept is defined that will meet the NGOs, MOEs, MOPs, and KPPs within the defined drivers and constraints with acceptable risk.
The proposed system concept along with operational scenarios/and or use cases is documented in a system concept document or model. Project management and systems engineering documentation is developed to the maturity required by the organization for this lifecycle stage of the product development. The TMP is baselined. (For a more detailed discussion on baselining scope, the entrance criteria, exit criteria, and risk identification, see my blog: Baseline Your Scope Before Writing Requirements.) For a two-step project funding process, this is the maturity that the system concept needs to be at for a Step 1 funding proposal as discussed earlier. By baselining scope, the project has demonstrated a system concept that is mature enough to proceed to the next lifecycle stage and document its requirements.
Note: In the above text I referred to a “system concept document” rather than an Operational Concept or Concept of Operations Document. See my blog: ConOps vs OpsCon – What’s the Difference?) For those moving away from a document-centric approach and towards a data-centric systems engineering approach, the system concept may be documented in a model developed using a systems engineering modeling tool. Using this approach, requirements and other systems engineering lifecycle artifacts can be linked to the model. With a data-centric systems engineering approach, documents are generated from the data in the form of “reports’, where the “ground truth” data is maintained and managed within the model. These documents can then be shared with other internal and external organizations. For more details on data-centric systems engineering and selection SE tools to implement this approach, see my blog: Features an SE Tool Set Should Have.
Traditionally, projects focus on developing an operational concept or concept of operations document that is baselined as part of the scope review. Too often the focus is on writing this document. This is a mistake. The focus shouldn’t be on the document but on maturing your system concept to the point you can document a feasible concept that has reached CML 5 and is ready to be baselined. Developing and documenting a system concept is not an exercise in writing, rather it is an exercise in systems engineering!
CML 6 – System Requirements Review (SRR) or analogous gate review: Requirements are finalized. Design and planning commensurate for a SRR is complete. From the baselined system concept, stakeholder needs and expectations are transformed into “design-to” requirements. In this context, requirements resulting from this transformation represent an agree-to obligation of what is expected by the customer and communicates clearly these expectations to the design/development team. For more details on the transformation process, see my blog: Going from system concepts to requirements.
Using SE tools, requirements are traced to their source and allocated to the subsystem level, schedules for the subsystem development defined, and system verification and validation approach defined. Interfaces are identified and reflected in interface requirements documented in the set of requirements being baselined. Expanded details on the technical, management, cost, risks, and other elements of the system concept have been defined and documented.
Project management and systems engineering documentation is developed to the maturity required by the organization for this lifecycle stage of the product development. The system concept document or model is updated as necessary. The TRA, resultant TRA report, and TMP are updated. The requirement set is baselined. (For a more detailed discussion on baselining requirements, the entrance criteria, exit criteria, and risk identification, see my blog: Why Do I Need to Baseline My Requirements?)
CML 7 – System Design Review (SDR) or analogous gate review: Preliminary Implementation Baseline. Design and planning commensurate for a MDR is complete. System architecture trades have been completed and a design approach to meet the baselined system requirements has been defined and is ready for baseline. Preliminary subsystem-level requirements and analyses, demonstrated (and acceptable) margins and reserves, prototyping and technology demonstrations, risk assessments and mitigation plans are completed. Preliminary integrated cost-schedule-design is baselined.
The system concept document or model is updated as necessary. The Preliminary Project Plan is ready for review. Other project management and systems engineering documentation is developed to the maturity required by the organization for this lifecycle stage of the product development. The TRA, resultant TRA report, and TMP are updated. For a two-step project funding process, this is the maturity that your concept needs be at for a Step 2 project funding proposal as discussed earlier.
Note: Some organizations do not have a standalone SDR. Some may combine the SDR with the SRR or with the PDR. For those cases, criteria for determining whether or not your concept is at CML 7 would combined with criteria for either CML 6 or CML 8 depending on how your organization defines their gate reviews.
CML 8 – Preliminary Design review (PDR) or analogous gate review: Design and planning commensurate for a PDR is complete. System design documentation, “build-to” requirements and drawings are 10%-20% complete. Interfaces are defined in Interface Control Documents (ICDs). Final integrated cost-schedule-design is baselined. All parts/technologies have reached an TRL of at least 6. The system concept document or model is updated as necessary. The Project Plan is baselined including budget and schedule. Other project management and systems engineering documentation is developed to the maturity required by the organization for this lifecycle stage of the product development. The TRA, resultant TRA report, and TMP are updated.
CML 9 – Critical Design Review (CDR) or analogous gate review: Design and planning commensurate for a CDR is complete. Detailed system design documentation, “build-to” requirements and drawings are 80%-90% complete. ICDs are complete. The system concept document or model and other project management and systems engineering documentation is updated as necessary.
Notice that the CMLs apply to the left side of the Systems Engineering “Vee” Model. Ensuring all the activities are completed successfully on the left side of the SE Vee Model go a long way in minimizing problems that often occur on the right side of the SE Vee Model during system integration, verification, and validation. This is a major reason, and benefit for using CMLs to manage your system development activities. The TRA, resultant TRA report, and TMP are updated.
In the next part of this blog, I show how CMLs are an implementation of an older concept referred to as “The Doctrine of Successive Refinement”.
– Government Accountability Office, Technology Readiness Assessment Guide, draft, August 2016, http://www.gao.gov/assets/680/679006.pdf
NASA, Systems Engineering Processes and Requirements, NPR7123.1B, April 2013, NASA Online Directives Information System (ODIS) Library: http://nodis3.gsfc.nasa.gov
– Wessen, Randii R., Borden, Chester, Ziemer, John, Kwok, Johnny, Space mission concept development using concept maturity levels, NASA JPL, AIAA SPACE 2013 Conference & Exposition, San Diego, California, September 10-12, 2013 http://hdl.handle.net/2014/44299
– Wikipedia, Rapid application development, https://en.wikipedia.org/wiki/Rapid_application_development