What Is EPM? Evaluating the Health and Status of Products and Threads (Part 3)

1. BACKGROUND

This paper is the third in a sequence defining Evaluated Product Maturity (EPM).  EPM is designed to assist in assessing the risk of breaking or diminishing the functionality of a system-of-systems functional thread by modifying, changing, or replacing one or more systems that make up that thread.  The sequence of paper provides the basis for computing EPM using BTI’s automated systems and software data science capabilities.

This first document defined the purpose, background, and general approach for computing the EPM score.  

The second installment defined the factors that the data collected from a wide array of systems has indicated is critical in evaluating the risk for changes to a system to break or diminish the functionality of that system.

The third installment, which you are currently reading, discusses automatically capturing data to be processed by the EPM tool and how calculations are done.

The next (also the last) installment provides an example use case for EPM.

For Sections 1 through 4, see 

• https://www.biztransform.net/evaluating-the-health-and-status-of-products-and-threads/
• https://www.biztransform.net/evaluating-the-health-and-status-of-products-and-threads-2/ 

5. Capturing Measurement Data

Of the sixteen Fundamental Metrics identified, only 12 are used in the calculation of the product’s EPM. They are listed below. Eight of the metrics are reported on a per release basis. The others are reported as most recent captured.

• Mean Deployment Frequency (MDF) – per release
• Mean Change Lead Time (MCLT) – per release
• Mean Time To Repair (MTTR) – per release
• Test Coverage (TC) – per release
• Code Complexity (Comp) – per release
• Defect Density (DD) – per release
• Vulnerability Density (VD) – per release
• Code Smell Density (CSD) – per release
• New Tickets Received (NTR) – current date/time
• Ticket Closure Rate (TCR) – per release
• Mean Active User Count (MAUC) – current date/time
• Plan vs Actual Progress (PvAP) – current date/time

Most of these metrics are available directly through the use of existing tools. In those cases, no detailed formula (or discussion) is needed. For the others, the tools may provide the base measures that are used to create the composite metric, or additional processing may be needed to get the actual metrics. For example, DF, and hence MDF, is not a metric that is generally reported. But the base measure used to calculate DF is (Time of Release). A simple calculation giving the difference between the last two releases provides a value for DF. In the following sections detailed formulas for calculating the metrics from base measures are provided. If no actual calculation is necessary, a detailed description is not provided.

Throughout the discussion of data capture and analysis a release is defined to be a released version of the software. This is the version where a project plans on releasing new features, or defect repairs, or both, to end users/customers. 

Since release names can be non-specific, even within a single project, statistics need to be captured based on the release type that contains the most records, although the raw data for each release group is retained. For example, project A has decided to track new requirements on a version numbering scheme that follows the x.y.z notation, where the versions are identified numerically with the numbers assigned in increasing order. The ‘x’ represents a major version, ‘y’ a minor version and ‘z’ a patch or bug fix. Releases then follow in time like the following:

1.0.0,  1.1.0,  1.2.0,  1.2.1,  1.2.2,  1.3.0,  2.0.0,  …

However, during that release string there were also some priority bug fixes which were identified using a different nomenclature for release name. These were interspersed with the normal releases as shown below:

1.0.0,  1.1.0,  1.2.0,  1.2.1,  BugFix_1,  BugFix_2,  1.2.2,  1.3.0,  2.0.0,  …

An argument is made that the x.y.z series (based on the SemVer versioning scheme) is the primary release pattern and so the TCR is calculated based on that series of data. Data files containing the two types of releases are maintained if offline analysis is desired.

Although there are a number of industry standard tools that can be used to capture these metrics, information is provided based on the use of the most commonly used tools, namely, Atlassian’s Jira, SonarQube™, Git variants (such as Git, GitLab, GitHub) and Microsoft Project, and outlines a prototype environment for use in initial piloting of the product maturity effort. The data is being captured and analyzed using the R programming language together with a Shiny Web interface and Excel for dashboard prototyping. 

The following sections describe the requirements for the last step in the overall process, monitoring the data, and contains the guidance needed to develop a data capture and analysis system to accomplish the goal of evaluating the EPM of monitored products and threads. The relevant section of the complete process is shown in Figure 7 with the pertinent areas discussed highlighted in green.

Figure 7 – Main Process

6. Fundamental Metrics

The twelve fundamental metrics needed for the calculation of the EPM are listed in Table 21. These metrics are categorized into one of three major Metric Groups: Quality, Performance and Software Vulnerabilities. Standard Tools used to capture these metrics are also listed in the table. Each metric is captured and evaluated at every release of a product.

Table 21 – Fundamental Metrics with Weights

7. Detailed Requirements For Data Capture

 Because of the need to provide a consistent, standardized method of capturing the required metrics, many of the measures need to be adjusted, or normalized. We have found that many products have taken liberty in the way Jira allows customer customization to include many user-defined fields. Since there was no apparent standard way of standing up an instance of Jira, there is a lack of uniformity in the selection of key fields contained, used and propagated within a product’s Jira environment. Each section below describes the mechanisms, procedures and processed of normalizing the data captured so that a consistent representation of the data, and subsequent analysis, can be realized. 

7.1 Quality Metric Group

The metrics for measuring product quality come completely from SonarQube™, via the Nymbul Treasure instance.

7.1.1 Data Capture

Each of the four Quality Metrics are easily captured via the RESTful interface to SonarQube™. However, they change whenever a product executes a SonarQube™ analysis (as new code is developed, changes are made, etc.). Since data must be captured at product release, there needs to be a determination of when a release is conducted and an appropriate trigger to run the data capture for the SonarQube™ metrics. 

Test Coverage (TC) is a singular metric and will be reported as the percentage of code covered by unit tests. This is sometimes known (and reported) as overall coverage.

Code Complexity (Comp) is a singular metric and will be reported as the average cyclomatic complexity per file.

Defect Density (DD) and Code Smell Density (CSD) are composite metrics. Since SonarQube™ provides five levels of severity for Bugs and CS, these metrics will be reported as described in the mapping below:

• Trivial:  Sum of ‘Info’ and ‘Minor’ per KLOC as reported by SonarQube™. Issues that have minor or unknown impact on productivity.
• Average:  ‘Major’ per KLOC as reported by SonarQube™. Issues that might have substantial impact on productivity.
• Critical:  Sum of ‘Critical’ and ‘Blocker’ per KLOC as reported by SonarQube™. Issues that are of operational risks.

 7.2 Performance Metric Group

Performance is centered on the ability of a product team to provide their product to end users and customers with a high degree of quality and available in a timely manner. It is not a measure of how well the product performs in terms of reliability and availability, although these measures can be captured in future instantiations of the EPM model. Most of the data needed is readily available in Jira.

7.2.1 Data Capture

The majority of the required base measures needed to calculate the Performance Metrics are captured via the RESTful interface to Jira. The Mean Active User Count and Plan vs. Actual are available via other mechanisms.

Additional processing is required to combine the base measures obtained via Jira into the metrics reported on the dashboard and used in the calculation of the EPM. A sample size is used to provide normalization across the metrics and projects. Typically, a sample size (ss) of 10% of the most recent releases is adequate providing a rolling average for the final mean calculation. For measures where the number of releases is less than a specified minimum value (defaulted to 8), the whole set of data is used to calculate the mean.

7.2.1.1 Mean Deployment Frequency (MDF)

The Deployment Frequency is measured as the rate at which deployments are dropped to the customer. It is a time value that measures the number of hours between product releases. The Mean Deployment Frequency (MDF) is the mean of the DF values across all product releases.

  The Release Identifier (name) and Release Date (releaseDate) are both captured via the following RESTful call to Jira:

https://location-of-jira-db/jira/rest/api/latest /project/<projectname>/versions

For each new release, r, calculate the Deployment Frequency (DF) and the Mean Deployment Frequency (MDF). The Release Identifier is captured for reference and is required in the following section.