How the quality of test execution should be measured?

As a test manager, I always wanted to assess at the end of a testing project whether the quality of testing was good enough to ensure quality of the product. Literature of software testing suggests hundreds of metrics that can be used by the managers. However, not all of them are intended to measure the performance of the testing team.

Various test metrics can be divided into three primary categories: Product metrics (intended for Application Managers), Project metrics (intended for Project Managers) and Process metrics (intended for Test Managers). Here, our primary interest is Process metrics, which specifically measures the quality of testing, or in other words, the performance of the testers. Let's see which serves this purpose.

Is it total number of defects found?

No. This is neither of Product, Project or Process metric. This is because it doesn't tell you a story. Let's take an example where the testers found 200 defects. We can't infer anything from this number. The reason is, we don't know whether the testers did a good job or a bad job because we don't know how many defects are still unidentified.

Is it Test Execution Productivity?

This Process metric do measure the tester's performance, but in terms of speed, not quality. So, if a tester executes test cases in lightning speed but with errors, then it doesn't serve the purpose of testing in the first place.

Then what are the metrics that assesses quality of test execution? Well, in order to assess this, we should answer two following questions:

(A) Did the testers identify all the valid defects?
(B) Did the testers spend too much time and effort on invalid defects?

And, these two questions can be answered very easily with the following two metrics:

(A) Defect Leakage (to the next upper environment) = Total number of defects identified in next upper environment / Total number of defects identified (in lower + next upper environments)
(B) Defect Rejection Ratio = Total number of defects rejected / Total number of defects (valid + rejected)
​
Note that both the above metrics are "lesser the better". This is how I was measuring the quality of test execution for a long time. Request the readers to share thoughts on this as well.