Blogs

Test design is a science, not an art…

7/25/2016

My brother was obsessed with cars when he was a kid. He built a huge collection of model cars of various colors, shapes and categories over many years of his childhood. At some point of time, he even started designing newer models and thought of selling them to the toy companies so that they can actually make those cars!

I must admit, he did an awesome job as a designer when he was a third grader!

However, when we both look back now, we realize that there was one fundamental problem with those designs. They were merely sketches, not designs. They were more arts than science. They were about how he sees the cars, instead of how the cars work. There were absolutely no measurements about anything that were specified in the design.

Based on my experience of interacting with various software testing teams, I feel the same way about the test cases creation process. The creation and review of test cases are still very subjective and thus qualitative. They are more art than science. No one cares to quantitatively measure the quality of test cases, as part of the review process, before they are actually executed. The only time the quality of test cases are assessed, unfortunately, is when there is a bug found in production and management has doubt about how much was the test coverage when the application was tested in lower environments.

Very recently I co-founded Testing Algorithms, LLC. with a friend to help organizations with an automated test case design algorithm that makes the test case creation a scientific, logical, quantitative and automated process, as opposed to artistic, subjective, qualitative and manual one.

Inferences from Testing Challenges Survey

7/11/2016

Participants:

Participants of the survey were mostly:
1. Part of dedicated testing teams (92% of the organizations)
2. Following Agile methodology (89% of the organizations)
3. People who are not in managerial positions (77% are Test Analysts/Leads)
4. Located in Asia & Europe (76% of the participants)
5. From bigger organizations (67% has >200 employees, 59% has >500 employees)

Observations:

6. Requirement Analysis (64%) is a bigger area of improvement than Automated Test Execution (61%).

7. Test Data Creation (45%) is a bigger area of improvement than Status Reporting (38%).

8. Test Case Creation (42%) is a bigger area of improvement than Manual Test Execution (32%).

9. Defect Management (26%) is not a big problem in most of the organizations.

10. Combining points 6-9, it looks like the test execution, defect management and status reporting process has been streamlined quite a bit. However, requirement analysis & test planning is emerging as the primary pain area in many organizations.

11. Only 41% organizations use Model Based Testing tools for test case creation and only 36% use a Test Data Management tool. This confirms our hypothesis in point 10.

12. Automated Regression Testing has been adopted by most organizations (80%), but still 87% participants feel that testing takes more time. This also confirms our hypothesis in point 10.

13. Even if 80% organizations are using Automated Regression Testing, 74% says that testing phase exceeds their budget. This indicates either or both of the following:

a. Automated Regression Testing didn’t reach its break-even point;
b. Automated Regression Testing is not cost effective.

14. Most importantly, 92% of organizations fail to prevent defect leakage to the next testing environment, in spite of spending more time, spending more money and utilizing strong test management, test management and automated testing processes.

Conclusion:

The primary problems are in requirement analysis and test design processes. These continue to be manual processes since time began and its quality is believe to be dependent on human skills vs. automated techniques. There are various tools and techniques that are currently available that can make requirement analysis and test design much faster and better. It’s high time testing organizations start moving towards the direction of automated test case design. In fact, I predict automated test case design to be one of the most demanding topics in software testing area in the next decade.

Losing jobs to machines is inevitable...

7/11/2016

...unless we become artists or inventors!

Let's look at the history of corporate jobs.

In 1960's and 1970’s, typewriting and shorthand was one of the niche skills worldwide. This is because the job of a secretary was in high demand in all industries. Microsoft Office killed that job in 1990's.

The story is similar for the telephone operators.

Factory workers were replaced by machines across the world as well. A couple of years back we visited Hershey's and took a tour inside the chocolate factory. We were amazed by how commercially packaged chocolate bars are being produced starting from cocoa beans without a single human intervention.

The latest inclusion to the list is the self-driven cars. I wonder if drivers all over the world are going to lose their jobs in next few years or decades.

I am a selfish person and have a habit of connecting everything I say with what I do. So, please bear with me...

If we look at the historical pattern, any mechanical, logical and analytical jobs have slowly been taken over by machines over the years.

But, fortunately, machines have two big limitations.

1. They can't connect with human emotions to produce arts; and
2. They can't invent other machines.

Now, what does this mean to the software testing world? (Here you go!)

Very simple. Manual testing jobs are going to go away in a few years or decades. And this is because this is a mechanical, logical and analytical job.

What is the way out then?

Be an artist or an inventor (or both). Keep in mind the 'survival of the fittest' theory of the great Charles Darwin.

You might argue that not everyone is an artist or an inventor.

If I could borrow some thoughts from the great author and public speaker Seth Godin, I would ask you, how do you know that you are not an artist or an inventor? Did you try to be one? Thomas Alva Edison was not an inventor until he invented the light bulb! Mark Twain was not a writer before he wrote Tom Sawyer!

You don't have to be Thomas Alva Edison or Mark Twain. But you can start making small differences in the work that you do, however insignificant that might be. A difference that only a human can make. That will make you an artist. That will make you an inventor.

Automation is a journey, not a destination...

7/6/2016

1602:

It is a great day in William Shakespeare's life! He just finished writing “Hamlet”! Now it's time to start rehearsing!

But he needs so many copies of the script for all the actors!

So, everyone sat down for months to have a copy of the entire play for themselves.

Some of them were really struggling to read the author's handwriting...

1921:

Charlie Chaplin just finalized the screenplay of “The Kid” a few days back.

He is considering buying a typewriter to copy the script so that it is legible.

It is much faster too. He only has a few weeks before the shooting starts.

But a typewriter is very expensive these days and he also needs to find someone who can type...

1952:

Mr. Chaplin found a better way!

A typewriter uses something called Carbon Paper that makes multiple copies in one round of typing.

He is both excited and nervous about “Great Dictator”! The shooting is starting next week.

He wishes he could start before the Fuehrer knows about it!

He is all ready, except for the scripts...

1982:

Steven Spielberg just handed over the full screenplay of “E.T.” to his office boy.

Available copies of the script can't be used because he had to make a few changes in the last minute.

He needs the photocopies positively by tomorrow morning!

John Barrymore's 6-year old granddaughter is auditioning tomorrow after lunch...

2004:

Fortunately, this time Steven wrote the entire screenplay for “Catch Me If You Can” in Microsoft Word!

Both Tom and Leonardo are very demanding actors! Why on earth do they need the script in next half hour??!!

Oops!!! The printer is out of paper...

...a verb, a subject noun and an object noun!

7/3/2016

A verb is a word in a sentence that expresses an action, an occurrence, or a state of being.

A noun is a word that is the name of something and is typically used in a sentence as subject or object of a verb.

A subject noun is a noun that performs the action of the verb in the sentence.

An object noun is a noun that receives the action of the verb.

A few years back I learned this technique from a great Business Analyst I worked very closely with. It was very early in my career as a tester at that time. She taught me that whenever we gather, analyze, interpret or elaborate business requirements of a software applications, we should always start with these three words (or phrases): a verb, a subject noun and an object noun, to identify all the business functions that the application is supposed to perform. Once we have the list of all the functions documented in this form, we should then elaborate on the individual functions to come up with the details of the requirements and/or test cases.

Since then, I has been tremendously helpful for me in writing good quality test cases!

Let's take the example of the deposit function in an ATM machine.

Here, the verb is 'deposit'.

Now, who deposits and what is deposited?

Clearly, an account holder deposits.

But which account holders? Checking Account? Credit Cards? Mortgage? Domestic? International?

Here comes the need for elaboration. We should document very explicitly what types of account holders can use this function.

Similarly, what is being deposited? The answer is simple. Money.

But, in which form? Cash? Check? This again needs elaboration.

This process can be extended to identify all business functions of the ATM machine.

Note that, this technique is not specific to any methodology and can be used in Agile, Waterfall, Iterative or anything similar.

Here are a few examples:

Account holder withdraws money
Account holder views balance
Guest opens account
Administrator prints report, etc.

Over the years, this technique has been of great help to me as a tester to identify all positive and negative test conditions.