• share on
State of

Open source testing

Discover the Findings

Why open source is
important

There is no doubt about it - open source testing tools have revolutionized the way testing is performed. Through community collaboration and transparency, tools are developed at a rapid pace with consistent alignment to customer requirements.

Furthermore, the major projects such as Selenium and Appium have become a sort of global standard, allowing for interoperability of best of breed tools, free or commercial.

Tricentis commitment
to open source

Unlike some other commercial tool vendors which reject the importance of open source, Tricentis fully embraces open source as a crucial part of our product strategy.

Our commitment to the open source Flood, SpecFlow, and TestProject communities is unwavering. We are investing millions of dollars per year into open source testing and related initiatives.

Size of Tricentis open
source communities

Tricentis is proud to claim one of the largest open source testing communities in the world. Between Flood, SpecFlow, and TestProject, Tricentis supports 100,000’s of users around the world.

The wide array of features across the three products supports roles including business analysts, functional testers, performance testers, and more.

Demographics

1

In what region do you work?

We were surprised, but not shocked, to see the overwhelming support of open source tools from Asia. Specifically, countries we surveyed including India, Vietnam, Philippines, and China showed fantastic interest and engagement with our open source testing initiatives.

Map
2

What is your role?

As we would expect, open source testing tools are most often used by those that identify as Testers/QA. However, what was surprising was the low level of engagement from developers and engineers with our survey. We were also comforted to find the growing number of responses for Software Development Engineer in Test - a specialized, technically focused testing role.

Chart 52% 26% 8% 14% QA Developer/Engineer Software Development
Engineer in Test Other
3

How many years of experience in this field do you have?

Testing is a complex field, especially working with open source testing tools which can require a specific and technical skillset. We were proud to see that our survey respondents averaged almost 9 years of experience in the field. Clearly, many who choose the path of becoming a testing expert choose to remain in that field for good.

0-2
years 11%
2-5
years 23%
5-10
years 29%
10-15
years 19%
15+
years 14%
4

What programming languages does your team use?

Java was the clear favorite in terms of programming languages, which is no surprise as it is the default language to many popular open source testing tools like JMeter, Selenium and TestNG. Runner ups were JavaScript and Python, which we believe lends itself to a wider audience of contributors with an accelerated learning curve.

  • Java
    67%
  • Javascript
    40%
  • Python
    30%
  • C#
    22%
  • C
    13%
  • php
    12%
  • Other
    14%
5

What is the biggest roadblock to open source tool adoption in your organization?

Technical skills remain the most valuable commodity, and a key to success in adopting open source technologies. Open source testing tools seem to be gaining support as viable options for even the largest and most security focused enterprises, but some respondents still faced headwinds related to open source security and support.

Road
6

What is the biggest benefit of open source testing tools for your organization?

Cash is king. We believe that the current economic climate has increased focus even more on the cost saving benefits of open source technologies. Still, more than half of the users cited reasons beyond cost as their biggest realized benefits of using open source tools. A robust community, wealth of integration options, and limitless customization pushed open source tools above commercial options.

39% Cost
15% Integration to
Tools/Workflows
7% Freedom from
Vendor Locking
6img
16% Community
support
14% Ease of
Customization
6% Scalability

Functional Testing

Brought to you by

7

How often is functional test automation carried out in your organization?

Approximately 37% of organizations had functional test automation integrated into their build pipeline, to assume some functional testing was completed with each and every build. When paired with the number doing functional test automation with each sprint (40%), more than 75% of organizations were doing functional automation testing at least every 2 weeks.

Clock

4% Never

40% Frequently e.g. linked
to agile methodologies

37% Very frequently e.g. linked
to continuous
integration

10% Infrequently e.g. linked
to scheduled test
efforts

9% Rarely e.g. only in
response to particular
issues

Frequently e.g. linked to agile methodologies Very frequently e.g. linked to continuous integration Infrequently e.g. linked to scheduled test efforts Rarely e.g. only in response to particular issues Never
8

Who is responsible for functional test automation in your organization?

We hear time and time again that QA or Testers are responsible for creating functional test automation, which held true in 80%+ of organizations surveyed. Developers are still lagging in the push to create functional automation, with only 8% of organizations requiring their developers to lead this effort.

Chart

84%
Quality Assurance/Testing

8% Development

3% Business Analysts

2% Operations

2.5% Nobody

Quality Assurance/Testing Development Business Analysts Nobody Operations
9

How important is an open-source solution for functional testing to you?

90%+ of organizations surveyed said that having an open source functional testing solution was important or very important to them. This supports the recent trends we are seeing towards organizations wanting to work with vendors that support open source approaches to test automation.

Speed Arrow

58%
Very important

34%
Important

4%
Important only in regards
to certain conditions

4%
Not important

10

Which free/open-source functional automation tools do you prefer or are already familiar with?

Selenium is still king of the web, and Appium rules the mobile domain. Cypress is beginning to take some of the Selenium web automation market share, but still trails with 11% to Selenium’s 81%. TestProject and Katalon, both free tools built on top of Selenium and Appium, were found to be used in roughly 17% of organizations surveyed.

  • 81%
  • 50%
  • 35%
  • 32%
  • 25%
  • 19%
  • 17%
  • 11%
  • 10%
  • Other
    8%
  • Coded UI (Microsoft)
    8%
  • Internally developed
    7%
  • 6%
  • 5%
11

What are the biggest impediments to functional testing in your organization?

Time is money, and with agile development, time is an even more precious commodity. 50% of organizations surveyed stated Time was one of the biggest impediments to their functional testing efforts. Technical training and test environments were not far behind, which might echo a shifting popularity towards easier to use and deploy functional automation tools in the future.

Road
12

What's the greatest challenge to you when choosing to adopt or actively using open source functional testing tools?

Convince the customer there will be enough support and stability in development.
Lack of documentation.
To align the entire team towards it. To prove that it will be a long term solution for the team and cost effective at the same time. Scalability and customisation are another aspect while choosing an open source tool. Bigger community is also an important factor.
The tool's ability to integrate other open source tools in order to build an end-end framework.
Different staff have varying background and skill levels in different tools they used in the past. New tools keep coming up and they try out different things and increases the options to be evaluated at every decision point.
A small change in an input makes a big change on the output that needs to be checked An app gets updated, and locators and formatting change Lots of locators exist on the app page.

Behavior Driven Development

Brought to you by

13

Are you using Behavior Driven Development (BDD) and/or Examples Given When Then as part of your specification?

Many of our customers have shared with us that the use of the Given When Then format is widespread and established. Most companies surveyed are actively using this to increase automation rates and reuse. However, the use of Given When Then to align teams on requirements and streamline documentation is still lagging.

Clock Clock
18% Examples in a
different format
20% Documentation
based on
Scenarios
46% Automation of
Scenarios
50% All aspects of BDD
51% Scenarios in Given
When Then format
14

Rate your team's expertise with BDD

Most surveyed claim to be in the proficient category of users, but have not yet reached the expert status. Given the healthy number of respondents who claim only beginner level expertise, this may point to BDD’s current growth, with many organizations still picking up the finer details of it.

24% Beginner
55% Proficient
21% Expert
15

What automation approaches do you use?

The trend towards automation is clear: more companies reported using functional test automation than manual testing approaches. We were surprised to see the low number of respondents performing unit testing of their code. Will BDD tests replace the need for extensive unit test coverage?

Clock
16

Does any automation or exploratory/manual testing take place outside of your CI/CD pipeline?

A majority of companies are still doing some form of testing outside of their build pipelines. We expect to see this number continue to drop, as the adoption of CI/CD will demand that testing be closely integrated within the pipeline, to automate deployment decisions.

35% No

Clock

65% Yes

17

Which BDD tools are you using?

With the high proportion of Java fans responding to the survey, it is no surprise that Cucumber is the #1 tool reported for BDD. SpecFlow is the closest second, with a distinct correlation between C#/.NET coders and an affinity for that tool.

  • 70%
  • 24%
  • 10%
  • 9%
  • 9%
  • 4%
18

What is your biggest challenge when using BDD today?

Test management. We have thousands of tests and steps and maintaining a common domain language is difficult when it has developed organically over years.
Convincing team members and management to use BDD. Team members missing skills and not being interested in learning BDD.
It takes too much time to write and automate scenarios. We spend more efforts building tests than actual code.
When it comes to maintaining automation using BDD, feature files seems to me as an unnecessary layer above underlying code that performs the actual execution.
New BDD users take long to adjust. And sometimes it’s hard to fit the test steps in a Given When Then format.
Organize Tags based on Automation Execution (eg: for Smoke, Regression, Different Builds of Regression, etc).
19

How did BDD increase the efficiency of your software development?

BDD can have massive benefits when it comes to increasing software development efficiency. Though many respondents are still in their journey to fully implementing BDD, they are already seeing nearly a 50% increase in efficiency, on average.

49%

Speed

Load & Performance Testing

Brought to you by

20

How often is performance and/or load testing carried out in your organization?

In comparison to functional testing, performance testing was carried out slightly less often, with 57% doing it at least every sprint (vs. 80% for functional testing). Nevertheless, it was good to see that 95% of all organizations commit to doing some form of performance testing throughout the year.

Clock

5% Never

28% Very frequently e.g.
linked to continuous
integration

28% Frequently e.g. linked
to agile methodologies

22% Infrequently e.g. linked
to scheduled test
efforts

15% Rarely e.g. only in
response to
performance issues

Frequently e.g. linked to agile methodologies Very frequently e.g. linked to continuous integration Infrequently e.g. linked to scheduled test efforts Rarely e.g. only in response to particular issues Never
21

Who is responsible for performance and/or load testing in your organization?

Interestingly, only 33% of companies surveyed had specifically tasked performance engineers (or similar Operations, Site Reliability Engineering) with doing performance testing. More often, Quality Assurance and Testers were responsible for functional testing as well as performance testing. A higher proportion of developers were reported to be responsible for performance testing than for functional testing, which was a large surprise.

Chart

47%
Quality
Assurance/Testing

3%
Site Reliability
Engineering

22%
Performance
Engineering

12%
Development

8% Operations

7% Nobody

1% Other

Quality Assurance/Testing Development Business Analysts Nobody Operations
22

How important is an open-source solution for performance and/or load testing to you?

As with functional testing, 90%+ of respondents found it important or very important to have an open source solution for performance and/or load testing.

Speed

54% Very important

38% Important

3% Important only in regards
to certain conditions

5% Not important

23

Which open-source performance and/or load testing tools do you prefer or are already familiar with?

JMeter is king when it comes to open source performance testing, with 53% of companies reporting some use of it. Surprisingly, Selenium was the 2nd tool with 39% of organizations using the functional testing tool to spawn browser based load tests. However, Flood Element showed a strong presence with 12% of respondents using it, coming to market as the first purpose built tool for browser based load testing.

  • 53%
  • 39%
  • 20%
  • 12%
  • 10%
  • 8%
  • 7%
  • 5%
  • 5%
  • 5%
  • Other
    4%
24

What are the biggest impediments to performance and/or load testing in your organization?

As with functional testing, one of the scarcest resources in performance testing efforts is time, with 38% of respondents wanting more of it. However, unlike functional testing, the larger roadblock was actually technical training and skills, with 39% of respondents noting its negative impact.

Road
25

What's the greatest challenge to you when choosing to adopt or actively using open source performance and/or load testing tools?

Metrics and reporting. Comparison of test results with the benchmark. Interpretation and presentation of the results. Forming actions.
Learning curve for new resources.
The tool has some limitations in terms of the feature they provide. This can be a challenge trying to load test a protocol which is not supported by the tool in use.
Network and Security system enablement prior deploying tool and integration with existing interfaces.
Moving away from Commercial test tools.
From an enterprise perspective, the major challenge is related to the confidentiality that the tool provides related to audits.

About the survey

Darts

Aim of the survey

We conducted this survey to highlight the major trends and developments in the open source community. Specifically, we wanted to make sure to include and compare the global developments with those specific to the BDD, Functional Testing, and Performance Testing communities.

Distribution of results

Results are distributed on the
websites of the participating survey sponsors:

1,822

Number of survey responses

Gift

Incentives provided to participants

To incentivize respondents to spend the time to complete a survey response, we provided 5 separate $100 Amazon e-gift cards. These gift cards were provided to 5 randomly selected survey participants, who were contacted via email to claim their prize.

About the Sponsors

Checkmark Watch the Webinar

Key trends from our 2020 survey

Listen to leaders of Tricentis’ open source communities discuss key findings and what the results reveal about the future of BDD, functional testing, performance and load testing, and more.