In one of our recent case studies, we applied our methodology for an online book store that was being developed using Agile methodology. Below is one of the user stories that was in the scope of the first sprint:

User Story:
As a customer, I want to add books to my shopping cart, so that I can build a list of books I want to buy

Acceptance Criteria:
a. I can add books to my cart from search results
b. I can edit the quantity of a specified book
c. I can remove a book from my shopping cart
d. I can proceed to check out from my shopping cart

Using Testing Algorithms' solutions, following test cases were created for add book, edit quantity and remove book functionalities.

Test Cases - Add Book:

Test Cases - Edit Quantity of Book:

Test Cases - Remove Book:

Now the question is, are the given-when-then statements (i.e., test cases) better 'definitions of done'? Should the original acceptance criteria be replaced by them?

There are various ambiguities in the traditional acceptance criteria that were identified while creating the model using Testing Algorithms' approach, as follows:

1. Can a negative quantity be selected while adding a book into shopping cart?
2. How many ways a customer can land on the Search Page?
3. Are cancel buttons present when a book is being added, edited or removed to shopping cart?

Clearly, the traditional acceptance criteria were incomplete. Also, as a matter of fact, creation of the test cases (i.e., Gherkin statements) didn't take any additional time to create. In the product backlog grooming session, before the sprint started, Testing Algorithms' solution was instantly used to create not only these test cases, but various other documents including use case diagrams, process flow diagrams and manual test cases with expected results.

Visit https://www.testingalgorithms.com/online-book-store-agile.html or contact support@testingalgorithms.com for additional details of Testing Algorithms' solution.

Original: I am a fisherman. I mostly work in waterfall.
Translation: I am a software tester. I mostly work in waterfall.

Original: My goal is to catch all fishes in a pond.
Translation: My goal is to identify all defects in a software application.

Original: I aim to catch the bigger fishes first.
Translation: I aim to identify the critical defects first.

Original: I plan to navigate through all geographical coordinates of the pond.
Translation: I plan to cover all business functions of the application as part of my testing.

Original: I don't know how deep the pond is at various geographical coordinates.
Translation: I don't know how critical the various business functions are.

Original: I met with Department of Natural Resources and created a depth map of the pond.
Translation: I met with the Business Users/Analysts and created a criticality map of the business functions.

Original: I looked at the list of fish species present in the pond to determine my fishing strategy.
Translation: I looked at the various requirements of the application to determine my testing strategy.

Original: I have created a plan on how deep I will place my bait in each coordinate.
Translation: I have created all my test cases with expected results.

Original: I have collected various baits required for fishing.
Translation: I have arranged for various test data required for fishing.

Original: I have reached in front of the pond to start fishing.
Translation: The application under test has been deployed to the test environment.

Original: I made sure that my boat is in working condition.
Translation: I made sure that I have all required access to the application.

Original: During test execution, I often change my directions to coordinates where fishes are more probable.
Translation: During test execution, I often re-prioritize my testing to focus on defect-prone business functions.

Original: I inform my captain about the number of fishes caught on an hourly basis.
Translation: I publish daily/weekly test execution status reports during test execution.

Original: Sometimes fishes move from one coordinate to another coordinate that has been covered by me earlier.
Translation: Sometimes one defect fix breaks a functionality that has been tested by me earlier.

Original: So, I keep an eye on coordinates that I covered earlier while moving to new coordinates.
Translation: So, I assess the need for regression testing of functionalities tested earlier after each defect fix.

Original: Sometimes I catch turtles by mistake that I eventually return to the pond and thus waste my time.
Translation: Sometimes I waste my time on non-defects that eventually get rejected.
​
Original: When I am done, I move to another environment.
Translation: When I am done, I move to another environment.

Based on a recent survey, 68% of software development projects are considered 'failed' by the respective project sponsors. Moreover, as per the root cause analysis, 53% of these failures are attributed to poor functional requirements.

In the context of quality, the difference between quality assurance and quality control is that one is defect prevention and the other is defect remediation. However, no structured process is followed in the current industry to assure the quality of functional requirements before it is used by the developers.

Some organizations follow strong approval processes for the functional requirement document. Some others perform ambiguity review of the requirements as well. But, how objective, complete and thus reliable these processes are, is the question.

Testing Algorithms, LLC. have come up with a research based approach to perform quality assurance of the functional requirements first. This process uses various business modelling concepts to identify requirement gaps during the requirement gathering phase itself. Moreover, once the requirements are captured, they are processed through a proprietary algorithm to automatically convert to test cases with expected results and 100% requirement traceability. These test cases can be integrated with standard test management and test automation tools.

Struggling with software quality? Not having a dedicated testing team? Testing processes are not standardized? Many requirement defects found during testing?

Testing Algorithms is not a QA outsourcing company.

It is a research based organization that aims to improve requirement gathering and test design processes across industry, in collaboration with top Universities of USA.

We help you create the most effective & efficient testing approach that drastically reduces overall testing time and effort and increases test coverage using optimization algorithms!

Share your requirement documents to try out our innovative service for free.

Want us to sign an NDA? Contact support@testingalgorithms.com.

"Its a machine, and you know that it is stupid!"

These were the exact words that Gary Kasparov said before his games with Deep Blue!

Do you think the same way about requirement analysis and automated test case design?
​
Try our unique solution at www.testingalgorithms.com before answering the question!

As per the ISTQB study material, the definition of software testing is "a process of executing a program or application with the intent of finding the software bugs."

But to me, software testing is a different poker game where a tester 'pokes' an application under test using various input actions and data combinations and compares the observed behavior of the application with its expected behavior. The primary dilemma in determining a test strategy is 'how to intelligently poke the application considering limited resources and time?’ To me, this sounds more like an optimization problem.

I have seen many testing teams come up with test cases for an application without giving much attention to the optimization aspect of the problem. Traditionally, the complete set of functionalities of the application under test is broken down into transactions, then into test scenarios and then into test cases with steps and expected results. However, this manual exercise often leads to a big number of repetitions and missed requirements in the test suite created. This is because a structured thought process and an effective optimization technique is usually missing in the process.

If we look at software testing as a special case of the poking game mentioned above, a mathematical optimization problem can be formulated that will generate minimum number of test cases that eliminates all gaps in test coverage.
​
We, at www.testingalgorithms.com, have come up with a process that makes software design process much faster, better and cheaper.

As a test manager, I always wanted to assess at the end of a testing project whether the quality of testing was good enough to ensure quality of the product. Literature of software testing suggests hundreds of metrics that can be used by the managers. However, not all of them are intended to measure the performance of the testing team.

Various test metrics can be divided into three primary categories: Product metrics (intended for Application Managers), Project metrics (intended for Project Managers) and Process metrics (intended for Test Managers). Here, our primary interest is Process metrics, which specifically measures the quality of testing, or in other words, the performance of the testers. Let's see which serves this purpose.

Is it total number of defects found?

No. This is neither of Product, Project or Process metric. This is because it doesn't tell you a story. Let's take an example where the testers found 200 defects. We can't infer anything from this number. The reason is, we don't know whether the testers did a good job or a bad job because we don't know how many defects are still unidentified.

Is it Test Execution Productivity?

This Process metric do measure the tester's performance, but in terms of speed, not quality. So, if a tester executes test cases in lightning speed but with errors, then it doesn't serve the purpose of testing in the first place.

Then what are the metrics that assesses quality of test execution? Well, in order to assess this, we should answer two following questions:

(A) Did the testers identify all the valid defects?
(B) Did the testers spend too much time and effort on invalid defects?

And, these two questions can be answered very easily with the following two metrics:

(A) Defect Leakage (to the next upper environment) = Total number of defects identified in next upper environment / Total number of defects identified (in lower + next upper environments)
(B) Defect Rejection Ratio = Total number of defects rejected / Total number of defects (valid + rejected)
​
Note that both the above metrics are "lesser the better". This is how I was measuring the quality of test execution for a long time. Request the readers to share thoughts on this as well.

Because they focus on how much has been accomplished till date, as opposed to how much needs to be accomplished in the available time and what is stopping or might stop the progress towards that target.

Here, I am primarily talking about waterfall projects that have dedicated test execution phases (separately for System Testing, User Acceptance Testing, etc.)with planned start and end dates. As a Test Manager, I always used to create and publish daily and weekly status reports (for ongoing projects) that were expected to be used by senior management to identify issues and risks, if any. However, in spite of publishing status reports on a regular basis, in almost every project we gave a surprise (and not a good one!) to the senior management in the last week of test execution that it can't be completed in time because of this, this and this reason. (Mostly, "it's developer's fault!")

A typical status report that I used to publish contained information around total number of test cases planned, executed, passed and failed till date, grouped by modules or functionalities. It also had information on the total number of defects identified, how many were valid defects among those, how many were in open, being retested and closed status etc., grouped by severities and priorities of defects. 

But what story does this status report tell us? Can the situation be quantitatively assessed in terms of how far ahead or behind the progress of test execution is and why?

In my experience as a Test manager, I have figured out a way to handle this situation. My status reports do not brag about what has been done till date. Instead, it focuses on the following things:

•    How much needs to be done from today till the planned execution end date? 
•    How feasible and practical that plan is in terms of the workload? Do we need more people to meet the target? 
•    If we are behind the schedule, what are the root causes? 
•    How can we measure the impacts of the root causes? And, how can we improve on them?

So, my test execution status report has the following metrics:​

•    Expected Test Execution Productivity (as of date), i.e., total number of test cases remaining to be executed or retested versus total number of days remaining per person per day. 
•    Percentage increase (or decrease) in Expected Test Execution Productivity with respect to the productivity assumed originally (i.e., at the beginning of test execution). A cut-off value needs to be defined at the beginning of the execution that, if this is +20% or more, then we need either more people or more time.

Now, if the progress is behind schedule, there are primarily four possible areas of root cause, and the corresponding metrics, as follows:

•    Testers are either slow or under-staffed. A good measure of this is traditional Test Execution Productivity, i.e., how many test cases were executed per person per day. If this productivity looks good, then definitely the testing team is under-staffed and there was a problem during resource planning at the beginning of the project. Another measure of this is Defect Rejection Rate, i.e., what percentage of identified defects are invalid. This is a good indicator of how much time the testers are wasting in identifying and analyzing non-defects. 

•    Developers are not able to fix defects quickly, which is blocking a number of test cases and stopping the testers to proceed. Two good measures of this is Defect Aging (i.e., average number of days in which defects are closed) and Defect Reopen Rate (i.e., what percentage of defects are being reopened by testers after retest). There can be two similar reasons why this could happen: either the developers are under-skilled or they are under-staffed. 

•    Requirements are changing very frequently, which is shifting the focus of the testers to go back and adjust the test design. A standard measure of this is Requirement Traceability Index, i.e., what percentage of the requirements are added, deleted or modified in the life cycle of the project. 

•    Test Environment is not stable. Environment Stability Index is a good measure for this in terms of how many days during test execution the environment was down.

If you have already noticed, the four root causes are related to four different teams participating in the project: testing team, development team, Business Analysts and Environment Support team.

To summarize, the test execution status reports have much more information and usefulness if it contains the various metrics described above.

Contact Testing Algorithms at support@testingalgorithms.com for refining your testing processes to make them more meaningful for all project stakeholders.

In a typical software development project, Quality Assurance (QA) process ensures that the quality of the code, written by developers, is compliant with the requirements defined by business users. The IT QA team, as part of their test design, makes sure that all requirements are covered by the test cases, and thus all possible defects can be identified and fixed before the application is exposed to the business users for User Acceptance Testing (UAT).

But who measures the quality of Quality Assurance? And, how?

Let me share what I experienced so far in my career. At the end of a project, QA is considered to be well-performed if no (or may be a very few cosmetic) defects are percolated to the upper environments (like UAT and/or production). But this is a very rare event. In real-life projects, with almost certainty, some moderately critical (if not very critical) defects are always uncovered by the User Acceptance testers (in UAT environment) or business users (in production environment). And, when they are uncovered, the IT QA team had already missed the bus!

But, wouldn't it be helpful if the quality of test cases could be measured before even executing them in lower environments? This way the IT QA team would have had a chance to make corrections and adjustments to their test cases, instead of waiting for the UAT team to discover their mistakes!

Some organizations thought about this problem and came up with an solution where IT QA team reviews the test cases with Business Analysts to make sure all test conditions are covered. But the big problem with this approach is, still the quality of test cases is not measurable. And, because it is not measurable, it is not optimized. So, the set of test cases often grow exponentially in order to ensure that all possible combination of requirements are covered.

Testing Algorithms proposes a solution by which the quality of test cases can be measured and optimized, even before the test execution begins. This solution ensures the coverage of maximum requirement combinations by minimum number of test cases. And, that too, with at least 90% less test design effort.

A few years back I was involved in managing testing activities for a bank's ATM application. The application was being developed using latest technologies to replace their old system. A couple of Business Analysts spent hundreds of hours understanding and documenting business requirements and then they passed that knowledge on to the development team (for application design) and testing team (for test design).

One of the requirements, I remember, was that the maximum cash amount that could be withdrawn in a single transaction was $2000. Therefore, our intelligent testers applied the theory of boundary value conditions in this case and created 3 test cases for withdrawal involving the following amounts: $1999, $2000 and $2001. They also created the Requirement Traceability Matrix and linked this particular requirement with those 3 test cases.

Very reasonable, right?

After it was deployed in production, it was found that this functionality was not working for some Premium Debit Cards. A critical defect had been logged and a Root Cause Analysis initiated. It was suspected that this particular functionality has never been tested. However, when the Requirement Traceability Matrix was checked to confirm the hypothesis, it indicated otherwise.

What was the problem then?

Well, the problem was that the Requirement Traceability Matrix was incorrect, and thus, misleading. In this example, even if the 3 test cases pretended to cover the requirement, it actually didn't, as it didn't take into consideration all possible permutations and combinations of various attributes of the card. This requirement should have been tested with multiple types of Credit and Debit cards. But in reality, unfortunately, it was tested with only one type of card. 
Testing Algorithms recommends that test coverage should be measured against all possible permutations and combinations of the various parameters involved in the process, not just against some requirement ids.

Testing Algorithms create high quality test cases using a proprietary automated process by analyzing the application behavior. This results in:

1. Complete overlap of test design with requirement gathering process, eliminating the need of having a separate test design phase
2. Reduction of test execution time and effort, as minimum number of test cases cover maximum requirements

In one of our recent projects, we were supposed to create complete set of test cases for an eCommerce website. The conventional approach of test design (used by other vendors) estimated a few thousands of test cases and more than a thousand hours just for manual test case creation.

Testing Algorithms created a total of 1140 test cases in Quality Center format (i.e., with detailed test steps, expected results and requirement traceability) covering all pairwise interactions. In addition to that, all business models (e.g., UML format Use Cases, Swim-lane Diagrams and Process Flows Diagrams including primary and alternate paths) were also created and shared with the customer in no additional effort, time and cost.

Most interesting part of the project was, those 1140 test cases (and business models) were created in just 10 hours.