Using a general-purpose, ubiquitous language for testing has its limits. It may work for specific teams, but does it scale to your whole organization and enterprise portfolio?
Reason for Topic
Code-only testing may be fine if most of the tests (not just unit tests) are written and troubleshot by developers, but in larger teams and organizations it’s more common for testing subject-matter experts to take on the more elaborate (i.e. user experience, performance, security, etc.) testing activities, and this is where code-only testing approaches can often contribute to brittle and flakey test suites.
Introduction / Definition
Using a general-purpose programming language for domain-specific problems ultimately:
- Provides flexibility, but forces inherent complexity on users (which is like a leaky abstraction)
- Saves the tool/framework developers lots of time, since they can’t account for every future need, but not necessarily the users (which in this case is also developers and testers)
- Overshoots the skills requirement for grey-to-black box level testing; at some point, you shouldn’t have to be a (or the) dev to test what was developed, certainly by the time we’re testing the UI and business validity of features and apps.
- Forces future implementation options, typically perpetuating the same language/framework even when it grows stale or out-of-favor with teams
So. Many. People. Know. It.
The choice of language for code-based tests really must depend on the technological ecosystem and appetite of a team, as well as the likelihood of the test assets being handled by other less-skilled practitioners. Also, how mature the organization’s testing practices are impact the velocity at which all-code testing approaches produce useful feedback to developers and ultimately decision-makers.
But just as a thought experiment, would you do the same thing with C? With Java? Some have. Does all testing activity benefit from being manifested as code?
So. Many. Options.
[likelihood of complexity] = [all the framework options] + [all the plugins] + [all the styles]
+ [all the teams in an organization] + [all the variations of skill over time]
At the core, a great testing framework accelerates the busy work of constructing, proofing, executing, and maintaining tests. Frameworks often include additional supportive capabilities such as mocking, data connectivity, element selector or object repository mapping, extensibility for customization, and thorough data export and reporting options. A great testing framework should also come with thorough documentation and tutorials, a community (or at least forum) of support, and examples
It’s unfair to judge a automated testing framework on cultural and people issues, yes? Maybe a little, but without thinking about the impact on the people and process components of your software delivery strategy, blindly introducing or perpetuating a technological choice can lead to unexpected outcomes.
Drawbacks / Gotchas
A key problem with using a general-purpose programming language in domain-specific contexts such as in functional, performance, or security testing is that it’s very difficult for a framework author to provide the proper amount of ‘safety guardrails’ that might otherwise be easier to implement with Domain Specific Languages (DSLs). With great power comes great responsibility, and the immense flexibility of general-purpose programming languages when applied to specialized problem domains often allows new or novice practitioners to end up either incorrectly addressing their domain,…