This paper is one in a sequence defining Evaluated Product Maturity (EPM). EPM is designed to assist in assessing the risk of breaking or diminishing the functionality of a system-of-systems functional thread by modifying, changing, or replacing one or more systems that make up that thread. The sequence of this paper provides the basis for computing EPM using BTI’s automated systems and software data science capabilities.
This first document defines the purpose, background, and general approach for computing the EPM score. Future installments will provide the details of calculations, how EPM is automatically calculated from the system of system’s development and maintenance environment, and how information is reported on a near-real-time dashboard.
This document provides guidance on the process and procedure to quantitatively evaluate the Health and Status of Products and Threads through the use of key measures and metrics. This health and Status is reported as a single identifier; one of three colors (Red, Yellow, Green) that indicates the degree to which a Product or Thread is operating against expectations. As such this identifier is integrally related to the effective maturity of the Product/Thread and can be used as an evaluation characteristic to compare Products against one another for purposes of determining risk levels, allocating process improvements and resources and the like.
In the context of this paper, a Product can be a project, tool, or any entity that provides a capability and is managed as a project. A Thread is a collection of one or more products that provides a capability. Threads are very similar in nature to systems and system-of-systems (SoS). For purposes of this paper, the term ‘product’ will be used unless the context requires a distinction between thread and product.
Note: This document focuses on non-service-based products only.
All products can provide operating measures that when analyzed and evaluated can provide insight into the characteristics of that product, both from an operational and developmental view. A consistent, standardized method of monitoring key measures of these characteristics can provide a quantifiable evaluation of not only the health and status of the product but a good estimation of the maturity of that product (and even, albeit indirectly, the maturity of the process used in the development of that product). Product maturity, with respect to established industry standards, is thereby evaluated by meeting measurement thresholds that are captured directly from a product’s development and maintenance tool environment in place of executing a formal process/product assessment such as the CMMI model or ISO 9001. The final maturity ‘score’ is an enumerated quantity, called the Evaluated Product Maturity (EPM) which takes on a value of either Green, Yellow or Red, providing an indication of how successful the product is with respect to the goals and requirements of its mission (see Table 1). Note that the decision to use this simple three-valued enumeration for the product maturity removes the focus from trying to achieve a higher maturity value, or a desire to comparing one product to another, to evaluating whether or not a product is meeting customer/user expectations against requirements and goals.
| Table 1 – Evaluated Product Maturity Status
|
The process by which this evaluation is achieved is dependent on a number of prerequisites:
Products that wish to levy additional requirements deemed critical to the success of their goals are encouraged to do so and details of these ‘extended’ evaluations are included as part of this discussion. Metrics derived from these additional requirements however, are not used in the determination of the overall health and status, i.e., maturity calculations.
Figure 1 illustrates the main process in the EPM evaluation. The process of determining the EPM of a product involves four steps:
The performance of a product against each metric is then enumerated by assigning a Metric Status Indicator (MSI) of Green (3), Yellow (2), or Red (1). The final, aggregate value, the EPM, is determined by a weighted average of all MSIs. Products with a Green EPM are considered to be operating at a maturity level consistent with the operation/task/mission they were designed to accomplish.
Additional steps are included in the process to handle the downstream activities. The complete process is outlined in Figure 2 with the process steps relevant to this discussion highlighted in green.The performance of a product against each metric is then enumerated by assigning a Metric Status Indicator (MSI) of Green (3), Yellow (2), or Red (1). The final, aggregate value, the EPM, is determined by a weighted average of all MSIs. Products with a Green EPM are considered to be operating at a maturity level consistent with the operation/task/mission they were designed to accomplish.
The first step in the process is the creation of a Product Profile. This artifact provides the characterization of the product and provides elements for use when determining the final health and status value. The product profile includes the following:
Product Category is determined by evaluating the relative risk and criticality to the functionality it supports.
The following is a list of major groups of categorizations, or factors. These factors are then used to calculate the Product Type.
Project Size. Project team size influences both the quality and availability of products as larger teams require more management, oversight and direction, and as size grows so do the communication paths within the group. Increased staff turnover negatively impacts team cohesion; additional mentoring and training become necessary aspects of the team’s development program.
Product Group. Products that are stand-alone and/or are not part of a larger capability (system, system-of-systems, threads, etc.) are generally less risky and easier to develop and sustain.
Team Experience (average). Elements considered in this evaluation include programming language experience, application domain experience, familiarity with the toolset, project management experience and skills.
Application User Base (Size in Users). Products that have a large user base are more difficult to maintain, as both the development and repair cycles may be longer and more resource-constrained due to the usual larger numbers of feature requests and bug fixes.
Process Maturity. Higher maturity positively impacts both development quality and increased productivity.
Technical Uncertainty. Uncertainty in the product development process and toolset can lead to unpredictable results in the final product both in terms of quality and deployment frequency.
Critical Application (as determined by Reliability Class)
By: Mike Mangieri
Senior Principal Process Engineer
Business Transformation Institute, Inc.
