Automated acceptance testing: A guide to get started

There are so many types of automated software testing that learning about all of them and keeping them straight in your head is a challenge.

In this post, we’ll provide a guide on automated acceptance testing, which is essential if you want your applications to meet users’ requirements and stay that way.

Let’s dive in.

What is acceptance testing?

TL;DR: Acceptance testing confirms that software meets user requirements and acceptance criteria before release.

Definition

You’ve just read that there are lots of types of software testing. That’s because there are lots of reasons why people test software, so each type of testing caters to a specific need.

One such need is ensuring that the application actually does what it should do, meets all users’ requirements, and remains fit for use.

Another way to think about acceptance testing is that it verifies whether the system under test meets all of its acceptance criteria, hence the naming.

Here’s a formal definition for acceptance testing, straight out of the ISTQB Glossary:

Formal testing with respect to user needs, requirements, and business processes is conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers, or other authorized entity to determine whether or not to accept the system.

Motivation

What’s the motivation behind acceptance testing? Here’s a thought exercise: Let’s say you built an application that is robust, fast, has a great UI, and is easy and intuitive to use. It might even be fun to use.

Awesome. Then it turns out that your software solves problems A, B, and C, and what the user really needed was a solution for problems C, D, E, and F.

That means the value of your software is effectively zero, at least to that user, regardless of how objectively great it is across a number of factors.

I’ll say it again: the value of software is zero if it does not solve the problems real users have.

This is the value that acceptance testing brings to the table: it prevents the scenario above from happening. Having stakeholders—preferably real end-users—verify the application against its acceptance criteria ensures it won’t be accepted until those criteria are met.

The ISTQB Glossary also explains its importance:

Acceptance testing validates user and business needs, reduces post-launch risks, acts as a final verification before deployment, and identifies requirement gaps. It ensures user satisfaction, mitigates risks, assures quality, ensures compliance, and builds stakeholder confidence.

What is automated acceptance testing?

TL;DR: Automated acceptance testing uses automation to verify that features meet acceptance criteria and continue working after changes.

Automated acceptance testing refers, unsurprisingly, to the use of automation applied to acceptance testing. In terms of definition, this is mostly it.

However, companies can achieve automated acceptance testing in many ways, and teams often vary the specifics based on their expertise, available tools, and organizational practices.

You’ll learn more details in a bit.

Why is automated acceptance testing important?

TL;DR: Automation ensures validated functionality keeps working as the application evolves and new changes are introduced.

The question of why automated acceptance testing is important blurs with the question of why any automated testing at all is important.

The way I usually like to frame why automated testing is crucial goes like this:

1. You test the current state of your application and declare it to be working.
2. Changes are added to the application.
3. You test the new changes, but you also need to retest the first batch of features, because any change to a codebase has the potential of breaking what was done before.
4. As the software increases in size and complexity, this quickly becomes unfeasible if you’re relying on manual testing alone.

In short, automated testing is crucial because it’s the only economically viable way to ensure that once some piece of functionality works, it stays that way.

It’s not hard to apply the same reasoning to automated acceptance testing. Once you validate that the system meets its acceptance criteria, you ensure it stays that way even when you add more changes.

The acceptance criteria must be a black box, in the sense that whoever specifies them shouldn’t care about implementation details but only how the application behaves from the end-user’s point of view.

Core building blocks of automated acceptance testing

TL;DR: Automated acceptance testing relies on acceptance criteria, test scenarios derived from those criteria, and tools that execute automated tests.

Let’s now turn to the building blocks of automated acceptance testing. How does this work in practice?

Acceptance criteria

The first and probably most important component of automated acceptance testing is the acceptance criteria.

One of the cornerstones of Agile development, acceptance criteria are a set of requirements that need to be met for a given piece of functionality or user story to be accepted.

Who defines acceptance criteria? Ideally, the customer or end-users. More pragmatically, it’s often a product owner, business analyst, or some other professional working on behalf of the end-user, with their best interest in mind.

Regardless of who does this, this person must be knowledgeable about the business domain, but not necessarily be a tech expert.

The acceptance criteria must be a black box, in the sense that whoever specifies them shouldn’t care about implementation details but only how the application behaves from the end-user’s point of view.

Test cases/scenarios

Someone needs to turn acceptance criteria into actual test cases or scenarios. How this is done varies by company, but it’s often a QA analyst, tester, software engineer, or even the end-user or customer themselves, even though that’s the rarer option.

This professional can write the test scenarios using a variety of approaches: using BDD/Gherkin syntax, some DSL (domain-specific language), a scenario outline with example tables, or even a simple checklist of scenarios.

Automated testing tools

Once you have test cases that map out to the acceptance criteria, the next step is turning those test cases into automated tests.

For that, you need an automated testing framework or automated tool, with which you can create the automated test cases, run them, and have the results reported back to you. Typically, you’ll integrate these into your CI/CD pipeline, so they run with every change to the codebase.

How AI can elevate automated acceptance testing

TL;DR: AI accelerates test creation, reduces maintenance through self-healing mechanisms, and helps prioritize relevant tests after changes.

Up until this point, we haven’t mentioned AI, and I know you’re wondering whether AI can improve automated acceptance testing.

As you’ll see, the answer to that is a resounding yes. AI-driven testing alleviates—or outright fixes—many of the pains teams have with automated testing, acceptance and otherwise.

Here are just some of the ways in which that happens:

1. Accelerating test case creation

Test creation is something that used to take a long time. But AI-driven testing enables teams to describe what they need to test in plain English and get high-quality AI-generated test cases at a fraction of that time.

2. Reducing the burden of test maintenance

Changes in UIs traditionally created a heavy maintenance burden for teams. Tests were flaky and often broke due to locator changes.

AI-driven testing tools use self-healing locators that adapt to those changes. This results in much more robust test suites.

3. Prioritizing tests after code changes

AI can highlight the tests that touch code modified by a change, so only those tests are executed rather than the entire test suite, resulting in faster testing times.

Automated acceptance testing, specifically, is vital because you really want to ensure your application works correctly from the point of view of end-users and business requirements.

AI-driven automated testing is here to stay

Automated testing is crucial for delivering high-quality software at the pace users and the market demand.

Automated acceptance testing, specifically, is vital because you really want to ensure your application works correctly from the point of view of end-users and business requirements.

Nowadays, AI-powered automated testing is such a powerful help that it simply makes no economic sense not to leverage it.

I’ll say it again: not leveraging AI-driven automated testing can be a bad business decision. Make a great business decision instead and give Tricentis a try today.

This post was written by Carlos Schults. Carlos is a skilled software engineer and an accomplished technical writer for various clients. His passion is to get to the bottom (the original source) of things and captivate readers with approachable and informative technical content.