Blog

Evaluating the Health and Status of Products and Threads

Background

This paper is one in a sequence defining Evaluated Product Maturity (EPM).  EPM is designed to assist in assessing the risk of breaking or diminishing the functionality of a system-of-systems functional thread by modifying, changing, or replacing one or more systems that make up that thread.  The sequence of this paper provides the basis for computing EPM using BTI’s automated systems and software data science capabilities.

This first document defines the purpose, background, and general approach for computing the EPM score.  Future installments will provide the details of calculations, how EPM is automatically calculated from the system of system’s development and maintenance environment, and how information is reported on a near-real-time dashboard.

Purpose

This document provides guidance on the process and procedure to quantitatively evaluate the Health and Status of Products and Threads through the use of key measures and metrics. This health and Status is reported as a single identifier; one of three colors (Red, Yellow, Green) that indicates the degree to which a Product or Thread is operating against expectations. As such this identifier is integrally related to the effective maturity of the Product/Thread and can be used as an evaluation characteristic to compare Products against one another for purposes of determining risk levels, allocating process improvements and resources and the like.

In the context of this paper, a Product can be a project, tool, or any entity that provides a capability and is managed as a project. A Thread is a collection of one or more products that provides a capability. Threads are very similar in nature to systems and system-of-systems (SoS). For purposes of this paper, the term ‘product’ will be used unless the context requires a distinction between thread and product.

Note: This document focuses on non-service-based products only.

Overview

All products can provide operating measures that when analyzed and evaluated can provide insight into the characteristics of that product, both from an operational and developmental view. A consistent, standardized method of monitoring key measures of these characteristics can provide a quantifiable evaluation of not only the health and status of the product but a good estimation of the maturity of that product (and even, albeit indirectly, the maturity of the process used in the development of that product). Product maturity, with respect to established industry standards, is thereby evaluated by meeting measurement thresholds that are captured directly from a product’s development and maintenance tool environment in place of executing a formal process/product assessment such as CMMI or ISO 9001. The final maturity ‘score’ is an enumerated quantity, called the Evaluated Product Maturity (EPM) which takes on a value of either Green, Yellow or Red, providing an indication of how successful the product is with respect to the goals and requirements of its mission (see Table 1). Note that the decision to use this simple three-valued enumeration for the product maturity removes the focus from trying to achieve a higher maturity value, or a desire to comparing one product to another, to evaluating whether or not a product is meeting customer/user expectations against requirements and goals.

 

Table 1 – Evaluated Product Maturity Status

EPM Description
Green Product is operating at the expected level of operational maturity against the key elements of Performance, Quality and Software Vulnerabilities.
Yellow Product is at risk or requires special attention to one or more elements of Performance, Quality or Software Vulnerabilities.
Red Product is failing at one or more key elements of Performance, Quality or Software Vulnerabilities.

The process by which this evaluation is achieved is dependent on a number of prerequisites: 

  1. That a standardized set of metrics is established for all products
  2. That measurements supporting those metrics are reliably captured in a consistent and repeatable manner; and,
  3. That products are characterized in terms of relative risk/importance to mission/thread based on the standardized formula. 

Products that wish to levy additional requirements deemed critical to the success of their goals are encouraged to do so and details of these ‘extended’ evaluations are included as part of this discussion. Metrics derived from these additional requirements however, are not used in the determination of the overall health and status, i.e., maturity calculations.  

General Approach

Figure 1 illustrates the main process in the EPM evaluation. The process of determining the EPM of a product involves four steps:

  1. Profiling the product;
  2. Selection of the appropriate measures/metrics;
  3. Assigning thresholds to each metric; and,
  4. Monitoring data against those thresholds. 

The performance of a product against each metric is then enumerated by assigning a Metric Status Indicator (MSI) of Green (3), Yellow (2), or Red (1). The final, aggregate value, the EPM, is determined by a weighted average of all MSIs. Products with a Green EPM are considered to be operating at a maturity level consistent with the operation/task/mission they were designed to accomplish.

Additional steps are included in the process to handle the downstream activities. The complete process is outlined in Figure 2 with the process steps relevant to this discussion highlighted in green.The performance of a product against each metric is then enumerated by assigning a Metric Status Indicator (MSI) of Green (3), Yellow (2), or Red (1). The final, aggregate value, the EPM, is determined by a weighted average of all MSIs. Products with a Green EPM are considered to be operating at a maturity level consistent with the operation/task/mission they were designed to accomplish.

Product Profile

The first step in the process is the creation of a Product Profile. This artifact provides the characterization of the product and provides elements for use when determining the final health and status value. The product profile includes the following:

  • Product/Thread Name and Organization
  • Key personnel and POCs
  • Development Processes Used
  • Development Environment
  • Product Development Life Cycle Model (Scheduled Releases/Sprint Cycles, Continuous Integration and Deployment)
  • Product Category/Type
  • Quantitative Goals
  • Value Statements (if any)

Product Category is determined by evaluating the relative risk and criticality to the functionality it supports. 

Product Categories/Factors

The following is a list of major groups of categorizations, or factors. These factors are then used to calculate the Product Type.

Project Size. Project team size influences both the quality and availability of products as larger teams require more management, oversight and direction, and as size grows so do the communication paths within the group. Increased staff turnover negatively impacts team cohesion; additional mentoring and training become necessary aspects of the team’s development program. 

  • Small – 1 to 5 full-time equivalent (FTE) staff
  • Medium – 6 to 15 FTEs
  • Large – 16 to 50 FTEs
  • Very Large – greater than 50 FTEs

Product Group. Products that are stand-alone and/or are not part of a larger capability (system, system-of-systems, threads, etc.) are generally less risky and easier to develop and sustain.

  • Single Product
  • Thread (multiple products and/or cross-product development)

Team Experience (average). Elements considered in this evaluation include programming language experience, application domain experience, familiarity with the toolset, project management experience and skills.

  • Low – less than 2 years
  • Medium – 2 to 5 years
  • High – greater than 5 years

Application User Base (Size in Users). Products that have a large user base are more difficult to maintain, as both the development and repair cycles may be longer and more resource-constrained due to the usual larger numbers of feature requests and bug fixes. 

  • Small – 1 to 50 users
  • Medium – 51 to 200 users
  • Large – greater than 200 users

Process Maturity. Higher maturity positively impacts both development quality and increased productivity.

  • Stage 1 – Minimal project management; little or no process definition
  • Stage 2 – Overall project management by the numbers; defined processes
  • Stage 3 – Processes measured, with continuous improvement
  • Stage 4 – Processes optimized, with consistent, excellent performance 

Technical Uncertainty. Uncertainty in the product development process and toolset can lead to unpredictable results in the final product both in terms of quality and deployment frequency. 

  • Low – Well known, mature technology
  • Medium – Adaption of familiar technology; limited new technology
  • High – First use of new technology, previously developed
  • Super High – Development of new technology

Critical Application (as determined by Reliability Class)

  • Class 0 – 1 not critical to the organization or its customers
  • Class 2 – 5 critical to the organization or its customers (2 lowest to 5 highest)

 

By: Mike Mangieri
Senior Principal Process Engineer
Business Transformation Institute, Inc.

Share:

Partners with

Supporter of