This is part 4 of our discussion on margins and reserves.
In Part 1 of this blog we introduced resource margins and reserves during development and operations.
In Part 2 of this blog we addressed the demand side of the resource balance equation by going into more detail concerning the terms associated with resource margins and reserves for mass and consumables.
In Part 3 of this blogwe addressed the supply side of the resource balance equation by going into more detail concerning the terms associated with resource margins and reserves for resource production.
In this final part of this blog, the focus is “how” to manage resource growth margins and reserves over the development life of your system.
Why is the management of growth margins and reserves such an issue?
This is a very important question that needs to be addressed. There are several reasons:
- In Gentry Lee’s talks, he tells a story where he was approached by a young engineer who asked him how it was possible for intelligent engineers to be off the mass by 50%. Gentry replied that “engineers and scientists tend to be overly optimistic at the beginning of a project.”
- As I discussed in my previous blog on technology readiness levels (TRLs), there is often an over-reliance on technologies that are at a low maturity level – a low TRL. Because of this over-reliance, resource demand and supply requirements are often defined that are “best case” or “theoretical” values based on approximations and assumptions for an “average or nominal” case.
- The competitive bidding (contracting) process results in a bidder underestimating or being overly optimistic when developing their bid and not including margins and reserves as a result. This results in the lowest bid the bidder thinks is needed to win the contract with the intent to recoup cost overruns through contract mods. A real dilemma – bid what it will really cost, or bid low and address cost overruns and schedule slips after the contract is awarded?
- A final reason is politics! Similar to the previous reason, by including realistic margins for growth, management reserves, and operational margins at the beginning of a program, the projected cost will most likely be much higher than if margins and reserves were understated or left out of the proposed budget estimates. Trying to get Congress to approve the real cost of a project can be very hard given all the other budget constraints. Some feel it is better to address these costs later, after the initial funding has been obtained. It is perceived (often wrongly) that once the initial funding has been granted, it will be easier to get additional funding from congress to address these concerns.
The failure to appreciate the need to define margins and reserves at the beginning of the project to mitigate the risk often results in cost and schedule overruns (due to the need to fix problems later in development) or failure of the system or product to meet stakeholder expectations in the operational environment.
The good news is that in order to mitigate these risks, the trend is to place more emphasis on Earned Value Management (EVM) to track actual progress as a function of cost and schedule. In the contracting world, there is also an increased emphasis on “best value” rather that “lowest cost”, with the selection committees grading risk mitigation strategies higher as a part of the scoring process for selecting a bidder.
How big should the margins and reserves be?
When a program begins, during the scope definition phase, the program needs to determine what they are going to base the size of their resource margins and reserves on. If the project is basing them on a general rule of thumb from a design guidelines document, there may be a problem. It is important to understand that design guidelines are for an “average or generic” project.
It is also important to understand that resource margin and reserve values change over the project lifecycle. As the design matures and parts become a reality, the Base Value will be updated to reflect your best knowledge of the actual resource. Thus the Growth Allowance Margin + Development Margin values can be decreased as you progress through the product development lifecycle. General guidelines for mass and power margins are addressed in NASA Goddard Space Flight Center Gold Rules, GSFC-STD-1000 and NASA Jet Propulsion Laboratory (JPL) Design Principles, D-17868.
As a rule of thumb, for an average or generic system development project, NASA recommends that at the scope or mission concept review (MCR) %Growth Allowance Margin + %Development Margin for mass should be 25%-40% for an average or generic project but as the design matures these values could be lowered to 20-25% at Systems Requirements Review (SRR), 15%-20% at Preliminary Design Review (PDR), 15% at Critical Design Review (CDR), and 5%-10% at Test Readiness Review (TRR). For power and other resources, these values can be slightly (~5%) lower. Likewise, Management Reserve can be “released” at these gate reviews as the uncertainties decrease and the design matures.
Typical %Growth Allowance + %Development Margin guidelines assume an average level of uncertainty; adjust %Growth Allowance + %Development Margin upward for items with higher uncertainty and downward for items with lower uncertainty. Another approach is to allocate increased dollar reserves to offset lower margins in some areas, e.g., technical performance or unknown development schedules. In order not to go over-budget, %Growth Allowance + %Development Margin may be applied individually to portions of the system and then summed to define the total System Growth Allowance for each resource.
%Growth Allowance Margin + %Development Margin values greater than those shown for an average or generic system development project may be required by project-specific circumstances. For example, highly complex mission and/or system designs, relying on maturation of a low TRL technology, uncertainty of heritage designs, small performance tolerances, and budget and schedule constraints result in a higher degree of uncertainty. With this higher uncertainty, a higher than the %Growth Allowance Margin + %Development Margin described above is warranted.
Likewise, %Growth Allowance + %Development Margin less than those shown may be acceptable when the following conditions apply:
– reuse of a known system design with a heritage system used in a previous mission application and environment that is close to the current application;
– other circumstances where the unknown factors are fewer and/or mature hardware is, by project policy, not to be changed; and
– where ample margins in other technical and programmatic resources exist, the degree of risk is low.
To define your actual resource margins and reserves, Gentry Lee recommends you base them on the answers to the following questions, and, based on the answer, define your resource margins and reserves accordingly:
- “How many things are being done for the first time?
- “How many things have been done once, but in a slightly different application in a slightly different operational environment?”
- “How many things being done have been done before in the exactly in the same manner and in the same operational environment?”
If everything falls into the last category, the risk is low and the use of average or generic margins are probably safe to use. However, if much of the system falls into the first or second categories, the system likely has a low TRL and your margins and reserves at the beginning of the project need to be much larger (>50%).
A similar approach can be used when defining the Management Reserve for each resource both on the demand and supply side of the resource balance equation.
How do you determine the maturity level of a technology?
A key question is how to determine the maturity level of a technology. The TRL is dependent on the assessment of the maturity of the technologies used in all parts that make up the system. Conducting this assessment is the subject of a new document released by the Government Accountability Office (GAO): “Technology Readiness Assessment Guide”. As the title suggests the focus of the Guide is on assessing the technical readiness of technologies critical to the success of a project. To learn more about Technology Readiness Assessments (TRA) see my blog: Technology Readiness Assessment Guide – GAO.
Managing resource growth, margins, and reserves over the life of your project
Once the program has identified and defined the resource requirements, margins, and reserves, these parameters should be managed closely throughout the system develop lifecycle. Each major review will include a status of these parameters. Due to the importance of parameters to mission success, they need to be treated as measures of performance (MOP) and measures of effectiveness (MOE) to provide technical and programmatic leadership with necessary information to make informed decisions.
Because these requirements are considered critical to successful development and operations, they are also referred to as Key Performance Parameters (KPPs). Each resource requirement needs to be classified as a KPP as failure to meet a KPP requirement will put the project at risk of cost and/or schedule overruns, or at risk of performance shortfalls. KPPs indicate the requirement is high priority. For a human system to Mars, the systems will be based on new technologies that have not been proven in the actual operational environment, adding additional development and operational risk.
It is safe to say that for human missions to Mars, all the resource MOEs can be considered KPPs because the success of the mission depends on the ability of the systems to meet the resource demand and supply requirements. Because they are also high risk, they are also tracked by management as TPMs.
KPPs that are tracked by management as TPMs will be monitored closely during implementation by comparing trends for the current actual achievement of the parameters with the values that were anticipated for the current time and projected for future dates.
KPPs and TPMs tracked in this manner are also referred to as Leading Indicators (LIs). LIs are measures that provide information for evaluating effectiveness of how a specific activity is applied on a project. LIs provide information about impacts likely to affect the system performance objectives. LIs may be an individual measure, or collection of measures, which can be used to plot trends (predict future state) before the actual performance is realized. The goal is to provide insight into potential future states to allow management to take action before problems are realized. It is safe to say that all resource KPPs and TPMs should also be treated as LIs. For NASA space missions, mass is to always be treated as an LI.
Many of these terms may seem overly complex, but these terms are commonly used in the Systems Engineering world. Their purpose is to provide management with measures and metrics that needed to be watched and tracked closely throughout the development lifecycle (and operations) to help ensure a successful program and mission. The importance of identifying and managing these types of parameters is emphasized by the number of documents that the International Council of Systems Engineering (INCOSE) has available in their store:
– Metrics Guidebook for Integrated Systems and Product Development
– Systems Engineering Measurement Primer
– Technical Measurement Guide
– Systems Engineering Leading Indicators Guide
– Project Managers Guide to SE Measurement for Project Success
Once the project has identified and defined the metrics for their resource margins and reserves, these metrics need to be managed closely throughout the system development lifecycles.
Concept Maturity Levels
Another concept to help manage projects that are heavily dependent on resource demand and supply and the technologies needed is Concept Maturity Levels (CMLs). I have published a multipart blog titled “Concept Maturity Levels” (CMLs). In this blog I discussed how CMLs can be used by management and development teams as an approach to measuring the maturity of a project’s system concept for all gate reviews defined by an organization. CMLs provide a standardized method to allow management to determine the maturity of the system concept including the maturity of technologies needed to meet the project’s goals and objectives.
The need for a High-fidelity System-of-System Model
The overall architecture of a human base on Mars will involve a number of separate systems, all operating together as a system-of-systems. Even though the individual systems can operate separately for their unique tasks, they are dependent on each other. For example, the system to produce fuel for the Mars Ascent Vehcile, will need hydrogen to combine with carbon extracted from the Martian atmosphere (CO2) to produce methane (CH4) and oxygen (LOX). This hydrogen and oxygen will be produced from the water extracted by the Mars Regolith Water Extraction system. Water produced by the Mars Regolith Water Extraction system will also be used to produce oxygen for breathing and water for drinking, washing, and growing food..
It is a major role of the systems engineer to make adjustments to the resource requirements, margins, and reserves as the project matures. Due to the complexities of managing a systems-of-systems of this magnitude, a complex, high-fidelity model of the system-of-systems will be needed in order run simulations to deterine the “optimum” value for each of the resource demand and supply requirements (MOEs). As the design matures and uncertainties decrease, the changes discussed previously to the parameters concerning requirements, margins, reserves can be input into the model and the effects of these changes can be assessed throughout the system-of-systems.
It is advised that threshold values be established to distinguish green (high probability of Program/project compliance – Action: none), yellow (moderate risk of Program/project compliance – Action: watch), and red (high risk of Program/project non-compliance – Action: immediate mitigation plan is necessary). These threshold values should be included in the system-of-systems model. (For more information on thresholds see my blog: Threshold vs. Objective requirements.) Current SE modeling tools can then show “dashboards” that summarize the MOPs, MOEs, KPPs, TPMs, and LIs status. These dashboards will enable managers to quickly see potential problem areas and take appropriate action. Today’s SE tools should allow a manager to “click” on a parameter on the dashboad that is showing yellow or red and view trends and other information needed.
Summary
The early establishment of proper Growth Allowance Margins, Development Margins, Operational Margins, Management Reserves and the effective management of them throughout the project’s life cycle play a critical role in the overall success of your program.
From the beginning of a program, program managers and systems engineers are strongly encouraged to define Growth Allowance Margins, Development Margins, and Management Reserves — especially for programs with significant complexity and high risk. Failing to define these resource metrics for your program places your program at great risk of cost overruns and schedule slips.
Additionally, if a program fails to define the proper Operational Margins, the mission operations and the astronauts are placed at risk by compromising on the Operational Margin and not allowing enough contingency capability to address the unknowns or change that can occur during operations.
Projects that start a project with a resource Base Value that is equal to the Resource Limit with no Growth Allowance Margins, Development Margins, Operational Margins, nor Management Reserves are doomed for failure from the beginning!
Comments to this blog are welcome.
If you have any other topics you would like addressed in our blog, feel free to let us know via our “Ask the Experts” page and we will do our best to provide a timely response.
