Testing Algorithms, LLC.
  • Home
  • About Us
  • Solutions
  • Case Study
    • Job Portal
  • FAQ
  • Blog
  • Video
  • Tutorial
  • Contact Us

"Fail Probability of test case #7 is 69%!"

12/26/2016

2 Comments

 
Picture


​During execution of test cases, wouldn't a statement like the one above, for each pending test cases, be helpful?

Also, wouldn't it be even more helpful if these probabilities are revised automatically every time a test case is executed and its outcome is recorded?

Well, I tried to summarize some of the benefits below.

1. Test Prioritization: Fail probabilities assigned to each test case would enable testers to determine the order of execution in such a way that defects (especially the critical and high defects) are identified at the earliest. This will give the developers enough time to fix those defects.

2. Stopping Rule: Creation of rules like "stop testing if all remaining test cases have fail probabilities < 10%!" before the test execution starts would be possible. This would help the testing team to avoid over-testing, by objectively and quantitatively deciding when to stop. This would also help in determining the extent of regression testing for a release.

3. Effort Estimation: At the time of estimating the testing effort for a project and the number of resources required, usually a fixed percentage (e.g., 15%) of the total testing effort is assumed for defect re-testing. However, most of the time we under-estimate it and thus experience a tremendous amount of time-crunch towards the end of testing. With these fail probabilities; defect re-testing effort could be determined more accurately.

Makes sense?

Now, the question is, it is possible to calculate these fail probabilities?

And if so, how?

Very recently I helped a friend in the analysis of a completely different problem (not even related to software testing). An organization shared a list of their employees. Our task was to calculate the probability of attrition for each employees based on their demographic information, salary information and survey responses so that the organization can take necessary steps to prevent attrition.

This analysis was done using some statistical models and the accuracy of the models were very high in terms of predictability.

And, while doing this analysis, I discovered something else that would be help in software quality assurance!

I found that, at the time of test execution, the determination of fail probabilities for test cases, based on various attributes of the test cases, is the exact same problem!

We, at Testing Algorithms, are working on creating a framework where the fail probability of test cases (generated by our patent-pending automated requirement analysis and test case design solution) can be automatically calculated and revised during test execution.

If you are interested to know more, feel free to contact us. We would be happy to talk to you about this.

2 Comments

How the quality of test execution should be measured?

4/23/2016

0 Comments

 
Picture


As a test manager, I always wanted to assess at the end of a testing project whether the quality of testing was good enough to ensure quality of the product. Literature of software testing suggests hundreds of metrics that can be used by the managers. However, not all of them are intended to measure the performance of the testing team.

Various test metrics can be divided into three primary categories: Product metrics (intended for Application Managers), Project metrics (intended for Project Managers) and Process metrics (intended for Test Managers). Here, our primary interest is Process metrics, which specifically measures the quality of testing, or in other words, the performance of the testers. Let's see which serves this purpose.

Is it total number of defects found?

No. This is neither of Product, Project or Process metric. This is because it doesn't tell you a story. Let's take an example where the testers found 200 defects. We can't infer anything from this number. The reason is, we don't know whether the testers did a good job or a bad job because we don't know how many defects are still unidentified.

Is it Test Execution Productivity?

This Process metric do measure the tester's performance, but in terms of speed, not quality. So, if a tester executes test cases in lightning speed but with errors, then it doesn't serve the purpose of testing in the first place.

Then what are the metrics that assesses quality of test execution? Well, in order to assess this, we should answer two following questions:

(A) Did the testers identify all the valid defects?
(B) Did the testers spend too much time and effort on invalid defects?

And, these two questions can be answered very easily with the following two metrics:

(A) Defect Leakage (to the next upper environment) = Total number of defects identified in next upper environment / Total number of defects identified (in lower + next upper environments)
(B) Defect Rejection Ratio = Total number of defects rejected / Total number of defects (valid + rejected)
​

Note that both the above metrics are "lesser the better". This is how I was measuring the quality of test execution for a long time. Request the readers to share thoughts on this as well.

0 Comments

    RSS Feed

    Author

    Abhimanyu Gupta is the co-founder & President of Testing Algorithms. His areas of interest are innovating new algorithms and processes to make software testing more effective & efficient.

    Archives

    April 2017
    March 2017
    January 2017
    December 2016
    November 2016
    October 2016
    August 2016
    July 2016
    June 2016
    May 2016
    April 2016
    March 2016

    Categories

    All
    Agile Testing
    Analytics In Software Testing
    Automated Test Case Design
    Business Model
    Defect Prediction
    Model Based Testing
    Outsourcing
    Quality Assurance
    Requirement Analysis
    Requirement Traceability
    Return Gift
    Status Report
    Test Approach
    Test Automation
    Test Coverage
    Test Efficiency
    Testing Algorithms
    Testing Survey
    Test Management
    Test Metrics
    Test Strategy
    Training
    Types Of Testing
    User Story

    View my profile on LinkedIn
© 2015 - 2018 Testing Algorithms, LLC.
​All rights reserved.
​
support@testingalgorithms.com
  • Home
  • About Us
  • Solutions
  • Case Study
    • Job Portal
  • FAQ
  • Blog
  • Video
  • Tutorial
  • Contact Us