Inferences from Testing Challenges Survey

​Participants:
 
Participants of the survey were mostly:
1.      Part of dedicated testing teams (92% of the organizations)
2.      Following Agile methodology (89% of the organizations)
3.      People who are not in managerial positions (77% are Test Analysts/Leads)
4.      Located in Asia & Europe (76% of the participants)
5.      From bigger organizations (67% has >200 employees, 59% has >500 employees)

Observations:
 
6.      Requirement Analysis (64%) is a bigger area of improvement than Automated Test Execution (61%).

7.      Test Data Creation (45%) is a bigger area of improvement than Status Reporting (38%).

8.      Test Case Creation (42%) is a bigger area of improvement than Manual Test Execution (32%).

9.      Defect Management (26%) is not a big problem in most of the organizations.

10.   Combining points 6-9, it looks like the test execution, defect management and status reporting process has been streamlined quite a bit. However, requirement analysis & test planning is emerging as the primary pain area in many organizations.

11.   Only 41% organizations use Model Based Testing tools for test case creation and only 36% use a Test Data Management tool. This confirms our hypothesis in point 10.

12.   Automated Regression Testing has been adopted by most organizations (80%), but still 87% participants feel that testing takes more time. This also confirms our hypothesis in point 10.

13.   Even if 80% organizations are using Automated Regression Testing, 74% says that testing phase exceeds their budget. This indicates either or both of the following:

a.      Automated Regression Testing didn’t reach its break-even point;
b.      Automated Regression Testing is not cost effective.

14.   Most importantly, 92% of organizations fail to prevent defect leakage to the next testing environment, in spite of spending more time, spending more money and utilizing strong test management, test management and automated testing processes.

Conclusion:
 
The primary problems are in requirement analysis and test design processes. These continue to be manual processes since time began and its quality is believe to be dependent on human skills vs. automated techniques. There are various tools and techniques that are currently available that can make requirement analysis and test design much faster and better. It’s high time testing organizations start moving towards the direction of automated test case design. In fact, I predict automated test case design to be one of the most demanding topics in software testing area in the next decade.