Blogs

"Fail Probability of test case #7 is 69%!"

12/26/2016

During execution of test cases, wouldn't a statement like the one above, for each pending test cases, be helpful?

Also, wouldn't it be even more helpful if these probabilities are revised automatically every time a test case is executed and its outcome is recorded?

Well, I tried to summarize some of the benefits below.

1. Test Prioritization: Fail probabilities assigned to each test case would enable testers to determine the order of execution in such a way that defects (especially the critical and high defects) are identified at the earliest. This will give the developers enough time to fix those defects.

2. Stopping Rule: Creation of rules like "stop testing if all remaining test cases have fail probabilities < 10%!" before the test execution starts would be possible. This would help the testing team to avoid over-testing, by objectively and quantitatively deciding when to stop. This would also help in determining the extent of regression testing for a release.

3. Effort Estimation: At the time of estimating the testing effort for a project and the number of resources required, usually a fixed percentage (e.g., 15%) of the total testing effort is assumed for defect re-testing. However, most of the time we under-estimate it and thus experience a tremendous amount of time-crunch towards the end of testing. With these fail probabilities; defect re-testing effort could be determined more accurately.

Makes sense?

Now, the question is, it is possible to calculate these fail probabilities?

And if so, how?

Very recently I helped a friend in the analysis of a completely different problem (not even related to software testing). An organization shared a list of their employees. Our task was to calculate the probability of attrition for each employees based on their demographic information, salary information and survey responses so that the organization can take necessary steps to prevent attrition.

This analysis was done using some statistical models and the accuracy of the models were very high in terms of predictability.

And, while doing this analysis, I discovered something else that would be help in software quality assurance!

I found that, at the time of test execution, the determination of fail probabilities for test cases, based on various attributes of the test cases, is the exact same problem!

We, at Testing Algorithms, are working on creating a framework where the fail probability of test cases (generated by our patent-pending automated requirement analysis and test case design solution) can be automatically calculated and revised during test execution.

If you are interested to know more, feel free to contact us. We would be happy to talk to you about this.

How the quality of test execution should be measured?

4/23/2016

As a test manager, I always wanted to assess at the end of a testing project whether the quality of testing was good enough to ensure quality of the product. Literature of software testing suggests hundreds of metrics that can be used by the managers. However, not all of them are intended to measure the performance of the testing team.

Various test metrics can be divided into three primary categories: Product metrics (intended for Application Managers), Project metrics (intended for Project Managers) and Process metrics (intended for Test Managers). Here, our primary interest is Process metrics, which specifically measures the quality of testing, or in other words, the performance of the testers. Let's see which serves this purpose.

Is it total number of defects found?

No. This is neither of Product, Project or Process metric. This is because it doesn't tell you a story. Let's take an example where the testers found 200 defects. We can't infer anything from this number. The reason is, we don't know whether the testers did a good job or a bad job because we don't know how many defects are still unidentified.

Is it Test Execution Productivity?

This Process metric do measure the tester's performance, but in terms of speed, not quality. So, if a tester executes test cases in lightning speed but with errors, then it doesn't serve the purpose of testing in the first place.

Then what are the metrics that assesses quality of test execution? Well, in order to assess this, we should answer two following questions:

(A) Did the testers identify all the valid defects?
(B) Did the testers spend too much time and effort on invalid defects?

And, these two questions can be answered very easily with the following two metrics:

(A) Defect Leakage (to the next upper environment) = Total number of defects identified in next upper environment / Total number of defects identified (in lower + next upper environments)
(B) Defect Rejection Ratio = Total number of defects rejected / Total number of defects (valid + rejected)

Note that both the above metrics are "lesser the better". This is how I was measuring the quality of test execution for a long time. Request the readers to share thoughts on this as well.