Blogs

Can quality of Quality Assurance be assured?

3/26/2016

In a typical software development project, Quality Assurance (QA) process ensures that the quality of the code, written by developers, is compliant with the requirements defined by business users. The IT QA team, as part of their test design, makes sure that all requirements are covered by the test cases, and thus all possible defects can be identified and fixed before the application is exposed to the business users for User Acceptance Testing (UAT).

But who measures the quality of Quality Assurance? And, how?

Let me share what I experienced so far in my career. At the end of a project, QA is considered to be well-performed if no (or may be a very few cosmetic) defects are percolated to the upper environments (like UAT and/or production). But this is a very rare event. In real-life projects, with almost certainty, some moderately critical (if not very critical) defects are always uncovered by the User Acceptance testers (in UAT environment) or business users (in production environment). And, when they are uncovered, the IT QA team had already missed the bus!

But, wouldn't it be helpful if the quality of test cases could be measured before even executing them in lower environments? This way the IT QA team would have had a chance to make corrections and adjustments to their test cases, instead of waiting for the UAT team to discover their mistakes!

Some organizations thought about this problem and came up with an solution where IT QA team reviews the test cases with Business Analysts to make sure all test conditions are covered. But the big problem with this approach is, still the quality of test cases is not measurable. And, because it is not measurable, it is not optimized. So, the set of test cases often grow exponentially in order to ensure that all possible combination of requirements are covered.

Testing Algorithms proposes a solution by which the quality of test cases can be measured and optimized, even before the test execution begins. This solution ensures the coverage of maximum requirement combinations by minimum number of test cases. And, that too, with at least 90% less test design effort.

Is Requirement Traceability Matrix overrated?

3/17/2016

A few years back I was involved in managing testing activities for a bank's ATM application. The application was being developed using latest technologies to replace their old system. A couple of Business Analysts spent hundreds of hours understanding and documenting business requirements and then they passed that knowledge on to the development team (for application design) and testing team (for test design).

One of the requirements, I remember, was that the maximum cash amount that could be withdrawn in a single transaction was $2000. Therefore, our intelligent testers applied the theory of boundary value conditions in this case and created 3 test cases for withdrawal involving the following amounts: $1999, $2000 and $2001. They also created the Requirement Traceability Matrix and linked this particular requirement with those 3 test cases.

Very reasonable, right?

After it was deployed in production, it was found that this functionality was not working for some Premium Debit Cards. A critical defect had been logged and a Root Cause Analysis initiated. It was suspected that this particular functionality has never been tested. However, when the Requirement Traceability Matrix was checked to confirm the hypothesis, it indicated otherwise.

What was the problem then?

Well, the problem was that the Requirement Traceability Matrix was incorrect, and thus, misleading. In this example, even if the 3 test cases pretended to cover the requirement, it actually didn't, as it didn't take into consideration all possible permutations and combinations of various attributes of the card. This requirement should have been tested with multiple types of Credit and Debit cards. But in reality, unfortunately, it was tested with only one type of card.
Testing Algorithms recommends that test coverage should be measured against all possible permutations and combinations of the various parameters involved in the process, not just against some requirement ids.

1140 test cases in 10 hours? Believe it!

3/9/2016

Testing Algorithms create high quality test cases using a proprietary automated process by analyzing the application behavior. This results in:

1. Complete overlap of test design with requirement gathering process, eliminating the need of having a separate test design phase
2. Reduction of test execution time and effort, as minimum number of test cases cover maximum requirements

In one of our recent projects, we were supposed to create complete set of test cases for an eCommerce website. The conventional approach of test design (used by other vendors) estimated a few thousands of test cases and more than a thousand hours just for manual test case creation.

Testing Algorithms created a total of 1140 test cases in Quality Center format (i.e., with detailed test steps, expected results and requirement traceability) covering all pairwise interactions. In addition to that, all business models (e.g., UML format Use Cases, Swim-lane Diagrams and Process Flows Diagrams including primary and alternate paths) were also created and shared with the customer in no additional effort, time and cost.

Most interesting part of the project was, those 1140 test cases (and business models) were created in just 10 hours.