First off, I want to extend a big “thank you” to the many participants in our Open Source Testing Survey! I truly appreciate you taking the time to share your experience with the community. We are proud to announce that, thanks to your help, this turned out to be THE BIGGEST global testing survey in 2020!
Some of the survey findings confirm our own observations and emphasize the expected. However, there are also some pretty surprising stats that change our understanding of open source tool adoption in testing…and even the software testing discipline in general. It will certainly be exciting to see what the future brings.
Even the demographics part of the survey yielded some interesting insights. Most of the respondents (62%) have been in the testing discipline for five years or longer. It seems that software testing is a profession one can fall in love with. 😉
Let’s take a deeper look at the findings related to the three areas of open source testing that the survey focused on: functional testing, BDD, and load and performance testing.
Not surprisingly, the need for technical skills and the lack of product support were confirmed as top challenges for open source testing adoption (with almost 45% of the total votes). This matches what we’ve seen in the field: Open source tool adoption requires skilled resources and a strong commitment to making it work. Those who expect that it will be simple to deploy and scale open source testing frameworks are usually disappointed. Although open source test automation offers strong support for web and mobile technologies, it’s hard to achieve end-to-end process coverage when numerous integration needs must be resolved (e.g., with complex enterprise apps).
Based on this need for technical skills, I personally expected SDETs (software development engineers in test) and even developers to be the dominant users of OS testing tools. However, this wasn’t the case here. Only 8% of overall survey participants were developers, and functional testing was carried out by a distinct QA function 84% of the time! It’s not so surprising that testing still isn’t a focus area for devs. However, even though so many people predicted that Agile development and DevOps would kill off testing, the erosion of software testing as a discipline does not seem to be happening.
The main reasons cited for selecting open source testing tools were cost and the promised flexibility regarding integration and customization (since the code is fully under your control). In response to the unprecedented cost pressure resulting from the COVID-19 crisis, we have seen a considerable spike in demand for the free community solution, TestProject — confirming that cost is an important consideration in tool usage.
After people select an open source testing tool and invest in customizing it and integrating it into their process, they heavily rely on that tool. 92% report that the tool they have selected is important or even very important. Most use the tool as part of a Continuous Testing process: more than 75% run their functional test automation frequently or as part of their CI/CT/CD toolchain.
It’s interesting to note that time is the #1 impediment to functional testing in general, but cost is the #1 overall motivator for selecting open source testing tools. This could be an interesting point to explore in further detail.
In general, most participants find it useful to specify application behavior using examples in some form (specification by example). Given-When-Then (GWT) is being used more and more, but it is not the sole standard yet.
Cucumber and SpecFlow are the dominant players in the BDD tool area (accounting for 94% of responses). The choice of tools is highly correlated to the programming language being used; Java shops go for Cucumber, .NET ones use SpecFlow. While many customers are just at the beginning of their BDD journey (79% beginners or proficient vs. expert), they are already experiencing significant (49%) increases in development efficiency.
As we expected (and have written about), scaling BDD is still a huge issue: when you get into thousands of tests, maintenance becomes tricky. Stay tuned for our ideas on how to enable BDD at enterprise scale!
Load & performance testing
Although the majority (56%) of participants run load tests frequently, it’s not part of their CI/CT/CD toolchain. In other words, Continuous Load Testing is not a reality…not yet, at least. Surprisingly, QA and testers cover the biggest portion (47%) of load testing. Very few respondents reported that load and performance testing was conducted by dedicated performance engineers.
Many of the responses provided for the question, “What’s your greatest challenge with open source load testing?” touched upon the interpretation of results. This is a topic that Tricentis is currently researching heavily and will improve with the help of AI.
To learn more, watch this on-demand webinar, in which we review key results and their implications for the testing community.