I have studied Statistics as a student and then spent most of my professional life doing software testing! It's always been my curiosity to find out how analytics can help in identifying various patterns while testing a software under development. One of the specific areas that always attracted me was the test management where I thought inferences about the performances of the development and testing teams could be drawn based on statistical data analysis. And, as part of that investigation, I found out that the information contained in the logged defects tell us many stories.
In one of my previous projects, we were using a standard defect management tool where individual defects were following the following workflow:
While looking at the defect metrics, I always made the following assumptions about existing defects:
1. Defects staying in New status for a long time indicates that there might be a problem with the defect triage process. Either triage should be performed more frequently, or there are many requirement gaps that are causing the confusion about whether an anomaly is a defect or not.
2. Defects staying in Open or Reopen status for a long time indicates that there might be a problem with the performance of the development team. Clearly, they are not able to fix defects effectively and efficiently. Either the development team is understaffed, or they need more technical training.
3. Defects staying in Fixed status for a long time indicates that there might be a problem with the performance of the code deployment team. It is an indication that the testers can't retest the defect yet because the updated code is not available in the test environment.
4. Defects staying in Retest status for a long time indicates that there might be a problem with the performance of the testing team. Clearly, they are not able to retest defects effectively and efficiently. Either the testing team is understaffed, or they need more training on testing processes or the application under test.
So I found out a way to measure how long the defects are staying in these statuses so that inferences can be made about the root cause and appropriate actions can be taken to improve the speed and/or quality of testing.
I noticed that the defect management tool captures and maintains the history of status changes for all defects. And also, a report containing that history can be exported to excel format. So, using my little coding knowledge, I was able to write a small program that parsed through that excel report and captured the duration of stays of all defects in various statuses. The output of my program looked something like this:
New: 1, 5, 3, 7, 5, 2, 3, 1, 7. (Note: There was total 9 defects)
Open/Reopen: 5, 8, 4, 2, 6, 8, 7, 8, 9, 4, 2. (Note: 1 defect got reopened 2 times)
Fixed: 4, 5, 2, 7, 4, 6, 8, 3, 2, 9, 8. (Note: 9 defects were fixed 11 times)
Retest: 3, 6, 7, 2, 6, 8, 4, 6, 3, 9, 8. (Note: 9 defects were retested 11 times)
I assumed a Gaussian or Normal distribution for all the duration data. A Normal distribution looks like this:
So, while doing testing, I constructed normal distribution for each of the 4 sets of data (i.e., for New, Open/Reopen, Fixed & Retest) on a daily basis to understand how this duration is increasing or decreasing over time. And to to do this, I used box plot diagrams. Following is a example where box plots are constructed on a daily basis:
RSS Feed
