Objective measures are needed to determine the success and failure of commercial software development efforts. The feedback for companies that sell software as a product or as service is immediate and determined by the marketplace – they either have revenue or they do not. However, for software developed to use internally by companies, there is no such clear cut external feedback mechanism that is available in all cases.
In practice, using revenue as an objective measurement seems like a perfectly reasonable and sensible way of determining success – if targeted revenue goals are achieved after the software is deployed, then the project is successful, else it will have failed. However, this approach has a couple of problems associated with it that I will describe below.
1. Ascribing Causality
This is perhaps the biggest problem with using revenue to measure the success, or lack thereof, of a software development project. Software projects supporting revenue generating efforts do not operate in a vacuum. They need to be supported by proper marketing, sales, support and pricing policies, that all have as important a role to play in final success or failure of a project, as the software itself. If any of the supporting functions fail, it can have an extremely negative impact on revenue that is completely out of control of the software project team. Conversely, favorable market conditions can result in revenue targets for the project being achieved even if the delivered software was substandard. In either case, we will be left with a measure that does not really shed any meaningful light on the real business impact of the delivered software.
2. Problems with Measuring Raw Numbers
Companies have a very hard time measuring slices of revenue that can be attributed to functional improvements. When a company introduces an online ordering system for the first time, it is relatively simple to measure online sales and use that as the raw revenue numbers. However, for very mature web sites or applications that have been operational for many years, measuring the revenue impact of changes that are incremental in nature is extremely difficult to do. Mature systems have a lot of noise in them and pinpointing revenue additions or drops to specific functional changes can be inexact, at best, and downright misleading in other scenarios.
Does this mean then that we should do away with using revenue numbers to measure project success? The answer is no.
The following two techniques can go a long way towards overcoming the deficiencies of simply measuring raw revenue numbers without context as a measure of project success.
1. Use A/B Testing to Measure Impact on Revenue.
If there are no practical or policy reasons preventing running the old and new system in parallel, then performing an A/B test scenario is perhaps the best way of isolating the revenue impact of the new software. In an A/B measurement scenario, all other external factors will be identical for all practical purposes. The only variable impacting revenue will be the software being used. Any change in revenue between the old and new software environments can be directly attributable to the project.
The control group that is still on the old environment will give an estimate of what would have happened if the new software were not available. The measurements from the group using the new software will show a relative increase or decrease in revenue numbers. This measured increase or decrease when extrapolated out across the entire user population will give a true measure of the efficacy of the new software.
2. Smooth out external factors when using revenue numbers.
When it is not possible to perform A/B testing, subjective adjustments need be made to revenue in order to remove external influences that either positively or negatively impact revenue numbers. For example, if sales have been growing 10% a quarter for the last four quarters, and sales go up by 10% in the quarter that a new software is released, then once we factor out historical trends, it one could reasonably surmise that the net impact on revenue due to the new software is 0. However, if in the quarter where the measurement was made, industry wide sales dropped by 5%, then maintaining a constant sales growth rate could be attributed in some part to the new software that helped to prevent a downturn.
Done properly, revenue measurements are an excellent indicator of software project success. Companies can learn valuable lessons on the true Return on Investment and impact to the top line. However, if it is determined that A/B testing is not feasible and there can be no consensus on how to subjectively smooth out external influences on revenue, then it is better to use a different measure of project success.
