Agile Methodology: The Complete Guide to Understanding Agile Testing

Over the past several years, a new way of creating software has taken the software development and testing world by storm: Agile.

In fact, according to VersionOne’s State of Agile Report, as of 2018, 97% of organizations practice agile in some form. However, respondents report that this adoption is not always widespread within their organizations, which means there’s still a long way to go in terms of adoption and maturity.

So what exactly is agile and why has it become so popular so quickly? Let’s explore exactly what agile methodologies entail and how to introduce it in your organization in more detail. Specifically, we’ll cover:

• How does testing fit into agile methodologies?
• What are the different ways to test on an agile team?
• What’s next for the agile movement?

Ready to dive in? Let’s go! (Or download our agile testing ebook for offline access.)

About Agile Methodology

Agile methodology has taken the software development world by storm and rapidly cemented its place as “the gold standard.” Agile methodologies all started based on four core principles as outlined in the Agile Manifesto. These methodologies are rooted in adaptive planning, early delivery and continuous improvement, all with an eye toward being able to respond to change quickly and easily. As a result, it’s no surprise that 88% of respondents in VersionOne’s 2017 State of Agile Report ranked “ability to adapt to change” as the number one benefit of embracing agile.

However, as more and more development teams adopt an agile philosophy, testers have struggled to keep pace. That’s because the widespread adoption of Agile has led teams to issue releases and totally undocumented software on a more frequent basis. This frequency has forced testers to shift when they conduct testing, how they work with developers and BAs and even what tests they conduct, all while maintaining quality standards.

What Does it Mean to Test on an Agile Team?

Agile principles are all about being collaborative, flexible and adaptive. It’s built on the premise that the world now changes regularly, and that means software teams no longer have years to bring new products to market. In that time, competitor offerings or customer expectations can change, and the team risks irrelevance. Agile minimizes this risk by helping teams collaborate together more by adapting to what the team needs to be successful. It does this by encouraging teams to regularly show off their work and gather feedback so that they can adapt to change quickly.

Jump-start your agile collaboration: Read our article, “The Secret to Alignment Between Developers and Testers.”

Narrowing in on testing, the fast pace of agile development has led to several imperatives for testers:

1. Prioritizing requirements based on risk since it’s not possible to test everything
2. Automating tests to increase efficiency
3. Increasing the use of exploratory testing to accelerate the time from code delivery to test completion and to emphasize the need to create code that works
4. Adapting to changes from sprint to sprint

 

The fourth imperative — adaptability — is particularly important because it requires testers to have broader, cross-functional testing skills, which represents a departure from the narrower testing skills often needed in a waterfall environment. Unlike in a waterfall environment, testers following an agile methodology need to stay in close contact with developers to collaborate on testing throughout the entire software development lifecycle. In waterfall methodologies, there is often a large requirements document that testers test against. That document doesn’t change frequently, so testers have been able to exist fairly independently of developers. However, most Agile methodologies are light on documentation and the requirements for a new feature may only be in a ticket in a requirements tracking system without all the edge cases listed. The testers in these scenarios need to be highly communicative with the development and business teams, as tests they wrote a few weeks ago may become obsolete quickly. To succeed, testers need to be flexible and able to adapt to moving targets.

In general, there are four central tenets of the Agile Manifesto that are important for testers to remember:

1. Collaborating with customers over contract negotiation
2. Working software over comprehensive documentation
3. Responding to change over following a plan
4. Collaborating with customers over contract negotiation

 

The bottom line in all of this is that everyone — testers, developers and beyond — must evolve in order to embrace an agile style of working.

Agile is Not One-Size-Fits-All

Every organization is unique and faces different internal factors (i.e. organization size and stakeholders) and external factors (i.e. customers and regulations). To help meet the varying needs of different organizations, there are various agile methodologies and several different types of testing you can do while working within one of those agile methodologies. Which mix is right for your team will depend on your internal and external factors, needs and goals. Let’s take a look at what some of the most popular agile methodologies and testing methods entail, including:

• Agile Methodologies
  • Scrum
  • Kanban

• Testing Methods
  • Behavior Driven Development (BDD)
  • Acceptance Test Driven Development (ATDD)
  • Exploratory Testing
  • Session Based Testing

2 Agile Methodology Types

1) Scrum

What is it? One of the most popular software testing methodologies (used by 58% of organizations that have embraced agile according to VersionOne), Scrum takes a highly iterative approach that focuses on defining key features and objectives prior to each sprint. It is designed to reduce risk while providing value quickly.

Scrum starts with a requirement or user story that outlines how features should perform and be tested. The team then cycles through a series of sprints to provide small bursts of value quickly. To help the team work in this flexible way and avoid shifting priorities, Scrum requires that questions be answered from the very start.

How is it different from Waterfall? Whereas Waterfall includes several testing and bug fixing cycles before releasing a product, Scrum is far more collaborative and iterative. One of the biggest differences is that Waterfall calls for heavy documentation early on. This documentation makes it harder to change features as the process goes on, which may be a negative in some environments (such as consumer-grade software) and a positive in others (such as those where the team is trying to launch a rocket, since no one wants requirements for something dangerous shifting frequently). That said, you might think of Scrum like many “mini waterfalls,” since requirements are well defined at the start of each sprint and shouldn’t shift within it. The difference is that detailed requirements for the next sprint are not set months in advance.

Diving deeper, Scrum calls for more regular collaboration between testers, developers and BAs, typically in the form of daily standups and sprint retrospectives, to ensure proper communication and alignment. Additionally, there is a Scrum Master who helps keep the project on task by removing blockers from the team to make sure they can be most effective. The Scrum Master can be anyone on the team, such as a developer or a tester.

What does adoption entail? Scrum offers one of the easiest transitions for teams coming from a Waterfall environment because it’s time-based with sprints and releases can still be planned out in advance. That said, it does call for faster iterations and stronger collaboration.

Who is it for? Because of its fast iterations, Scrum is best suited for teams whose customers and stakeholders want to be actively involved by regularly seeing working products at showcase meetings. This collaboration allows the team to make changes for upcoming showcases. Key team members who should be involved when taking a Scrum approach include:

• Product Owner
• Scrum Master
• Developers
• Automation Engineers
• Testers
• Stakeholders

What are best practices? In addition to strong communication, collaboration and adaptability, other best practices for testers following a Scrum methodology include:

• Determining acceptance criteria based on communication (typically in the form of a user story) from a sales representative or customer (note: this direct connection should help reduce miscommunications)
• Using the acceptance criteria to develop code and ensuring team approval of that code
• Testing the code in sandbox-like environments as well as production-like environments prior to deploying it to production

2) Kanban

What is it? Kanban is a very simple Agile based methodology rooted in manufacturing (it was developed by Toyota to help increase productivity in factories). At its core, Kanban can be thought of as a large, prioritized to-do list. Like with Scrum, requirements in Kanban are tracked by their current stage in the process (to-do, in development, in test, done).

Unlike Scrum, Kanban is not time-based. Rather, it is based solely on priority. When a developer is ready for the next task, he/she pulls it from the to-do list. Since there are fewer planning meetings, this approach means the team needs to be extremely close. In this type of environment, if developers work much faster than the testers, bottlenecks will crop up. In these situations, anyone on the team should jump in and help in different areas. Of course meeting this need requires a great deal of flexibility and adaptability.

How is it different from Waterfall? Kanban still has requirements like Waterfall, but the requirements can change as the testing team doesn’t start to think about testing each requirement until the developer selects it from the top of the backlog. In contrast, Waterfall is heavily time-based with a lot of overhead in planning. The heavy planning that comes in a Waterfall environment is great in some cases, like when building expensive things, but it’s not always necessary. With Kanban, releases still get planned, but teams usually don’t promise anyone features by certain dates unless the item in question is near the top of the backlog.

What does adoption entail? Kanban offers a simple transition for the right teams. To make a smooth transition to Kanban, business analysts, developers, testers and stakeholders should sit close together and communicate regularly. When transitioning to Kanban, it’s important to remember that this methodology offers the quickest way to bring code to production, but the code is likely to have some technical debt. That’s because developing without always knowing what’s next doesn’t necessarily lend itself to producing the most reusable code.

Who is it for? Kanban is best suited for small teams or teams that don’t produce features for the public and/or promise certain dates for releases. Additionally, it is a top methodology of choice for any products or teams focused primarily on maintenance work since bugs are not always straightforward and often require research to resolve, which makes time management challenging. Teams that cannot minimize the amount of planning for issues are likely better off following a Scrum or Waterfall methodology.

Key team members who should be involved in a Kanban environment include:

• Product Owner
• Project Manager
• Developers
• Automation Engineers
• Testers

What are best practices? In addition to maintaining visibility and prioritizing collaboration, best practices for testers following a Kanban methodology include:

• Keeping very open lines of communication between the business owners, developers and testers
• Ensuring the team has the flexibility to take on other roles outside of their core responsibilities in order to help clear bottlenecks
• Making everyone an owner of the product so that they are fully invested in the result

4 Agile Testing Methods

1) Behavior Driven Development (BDD)

What is it? Many people have heard of or used Test Driven Development (TDD). For example, developers use TDD to write unit tests to fail before code has been written. BDD is based on the same principles as TDD, but instead of unit tests, it calls for higher-level tests at the business level. Instead of starting with a technical-facing unit test as TDD does, BDD starts with an initial requirement based on end-user behavior and calls for tests that are “human readable” and can even replace some requirements documentation. This requirement is based on behaviors that the product should exhibit, creating an airtight guide for engineers to use as they develop tests.

For more information, read our article: “Why BDD is the Secret Sauce for Testing in DevOps.”

Specifically, BDD starts with a functional specification using the Gherkin Given/When/Then syntax. This specification then guides developers, testers and product owners who move across features. As they do, they use automated test functions to determine completeness, refining code until it passes the test, much like in the TDD approach except at the team level. To ensure the test passes (and it typically requires multiple tries), the developer should only refactor code, not add any new functionality.

In sum, BDD requires a “smart” automation strategy that drives a high level of efficiency. This strategy sets BDD apart from other Agile methodologies.

How is it different from standard Waterfall testing? BDD is extremely different from standard Waterfall testing because the former requires test cases to be written early against requirements and calls for those tests to be executed toward the end of the development cycle. With BDD in an Agile environment though, tests are not based on requirements and testing happens in conjunction with development of the features.

Additionally, within a Waterfall methodology, testers are the ones writing the test cases. On the other hand a BDD approach lends itself to the business owners writing the tests. This switch reduces communication (or miscommunication) between the business analysts, developers and testers.

What does adoption entail? Changing to a BDD methodology can be challenging when the team is used to a traditional style of testing. It requires a BA or tester to write tests upfront and for the developers to write the test specification in code to match. It’s a new type of coordination within the team, but it’s extremely positive in that the team works together as one unit, business users included.

Who is it for? The BDD methodology is ideal for teams that work on feature-focused software and/or teams that put the user experience first. Key team members who should be involved in a BDD environment include:

• Product Owner/Business Analyst
• Project Manager
• Developers
• Automation Engineer/Testers

What are best practices? Best practices for testers following a BDD methodology include:

• Streamlining documentation in order to keep the entire process lean
• Embracing a “three amigos” model where the product owner, developer and tester form a cohesive team
• Using a test framework like Cucumber to define criteria
• Building automated tests in a way that make them easy to reuse as much as possible
• Having business analysts learn the Gherkin syntax and write test cases directly

2) Acceptance Test Driven Development (ATDD)

What is it? ATDD is like BDD in that it requires tests to be created first and calls for the code to be written to pass those tests. However, unlike in TDD where the tests are typically technical-facing unit tests, in ATDD the tests are typically customer-facing acceptance tests.

The idea behind ATDD is that user perception of the product is just as important as functionality, so this perception should drive product performance in order to help increase adoption. To bring this idea to life, ATDD collects input from customers, uses that input to develop acceptance criteria, translates that criteria into manual or automated acceptance tests and then develops code against those tests. Like TDD and BDD, ATDD is a test-first methodology, not a requirements driven process.

Also, like the TDD and BDD methodologies, ATDD helps eliminate potential areas for misunderstanding by removing the need for developers to interpret how the product will be used. ATDD goes one step further than TDD and BDD though because it goes directly to the source (aka the customer) to understand how the product will be used. Ideally, this direct connection should help minimize the need to re-design features in new releases.

How is it different from standard Waterfall testing? ATDD is different from standard Waterfall testing because it is a test-first methodology. Standard Waterfall testing calls for test cases to be written upfront based on requirements, whereas ATDD is not a requirements driven testing process.

What does adoption entail? Because ATDD represents such a departure from traditional methods, going from one to the other isn’t easy for teams to do. To be in the best position to adopt an ATDD methodology, teams need to get stakeholder buy-in, which can prove challenging at times.

Who is it for? Due to its emphasis on user perception, ATDD is best suited for teams that are focused on the user experience, have goals around high rates of adoption and want to minimize the number of feature changes in future releases. Key team members who should be involved in an ATDD environment include:

• Customer/Customer Advocate
• Developer
• Product Owner/Business Analyst
• Automation Engineer/Testers
• Project Manager

What are best practices? Best practices for testers following an ATDD Agile methodology include:

• Interacting closely with customers, for example through focus groups, in order to determine expectations
• Leaning on customer-facing team members, such as sales representative, customer service agents and account managers, to understand customer expectations
• Developing acceptance criteria based on customer expectations
• Prioritizing two questions:
  • Will customers use the system if it does X?
  • How can we validate if the system does X?

3) Exploratory Testing

What is it? Next we have exploratory testing, which is actually a type of functional testing but is important in Agile environments. Exploratory testing gives testers ownership over the code to test it in an organized, chaotic way. In this case, testers are not following test steps, but rather using the software in standard or clever ways to try to break it. Testers will document defects as usual, but detailed documentation of what and how the application was tested is not always provided.

Exploratory testing is not scripted. Rather, it’s about developing the best tests based on each unique piece of software. Because of its unscripted approach, exploratory testing often mimics how users will interact with the software in real life.

Across the board, exploratory testing follows four key principles:

1. Parallel test planning, test design and test execution
2. Specific yet flexible
3. Aligned toward investigation of potential opportunities
4. Knowledge sharing

 

How is it different from standard Waterfall testing? Exploratory testing can actually be done in both Waterfall and Agile environments, but the tight integration between testers and developers in an Agile environment helps ease any bottlenecks that might surface while running exploratory testing in a Waterfall environment.

Furthermore, in order to run exploratory testing in a Waterfall environment, documentation on test results is a must and this documentation should be easy to trace back to requirements. Of course, this type of documentation is always useful in any environment.

What does adoption entail? Embracing exploratory testing is relatively easy, as it is quick to launch (and scale), simple to learn and provides benefits for the entire team. That said, it’s important to keep in mind that it should not be the only form of testing (rather, it should inform what type of testing happens next). Additionally, even though it’s unscripted, exploratory testing should not be unstructured (testers still need to set a goal, log your activities and take on the mindset of a specific user persona).

Who is it for? Exploratory testing can help reduce time spent testing, find more defects and improve code coverage. As a result, exploratory testing is best suited for teams that are under time constraints, teams that need help identifying the best types of tests to run (especially in cases where there are no specifications from developers) and teams that want to make sure they didn’t miss anything in previous tests. Key team members who should be involved in exploratory testing include:

• Testers (although everyone on the team should participate in some way)

What are best practices? Best practices for testers using exploratory testing include:

• Organizing the functionality in the application using something like a Mindmap or a spreadsheet
• Focusing on certain areas or certain scenarios
• Tracking what gets tested to help reproduce any errors
• Documenting results in a tool like qTest Explorer so there is some accountability for what got tested

4) Session Based Testing

What is it? Finally, let’s review session based testing. Session based testing builds on exploratory testing by providing more structure. Because exploratory testing is completely unscripted, it makes accountability difficult and relies heavily on the skills and experience of the tester(s) involved. Session based testing aims to ease some of these shortcomings by bringing more structure to exploratory testing without taking away from the benefits that exploratory testing provides, such as the ability to better mimic the user experience and get creative with testing.

Session based testing provides this structure by conducting testing during time-boxed, uninterrupted sessions, testing against a charter and requiring testers to report on the testing that took place during each session. Additionally, session based testing should be capped off with a “debrief” between the tester(s) and the manager that covers the five PROOF points: What happened (Past), what was achieved (Results), what got in the way (Obstacles), what still needs to be done (Outlook) and how does the tester feel about it (Feelings).

How is it different from standard Waterfall testing? The same as exploratory testing, session based testing can be run in both Agile and Waterfall environments, but it is more conducive to the tight collaboration between testers and developers that is typically found in Agile environments.

What does adoption entail? Much like with exploratory testing, adopting session based testing proves relatively easy because it is easy to pick up and launch quickly. For testers already accustomed to exploratory testing, the biggest hurdle is embracing the additional structure for which session based testing calls. Also, like with exploratory testing, teams running session based testing should remember that it is not a final stop but rather a method to help determine the best type of testing to conduct next.

Who is it for? Session based testing can help reduce testing time while increasing defect discovery and code coverage, making it ideal for teams that face time constraints and need more guidance to determine what types of tests to run. It is also ideal for teams that have found benefits in exploratory testing but need to improve accountability throughout the process. Key team members who should be involved in session based testing include:

• Testers
• Managers

What are best practices? Best practices for testers using session based testing include:

• Outlining a mission so that testers are clear about the software they’re testing
• Developing a clear charter that indicates the mission, the areas of the software to test, which tester(s) will run the session, when the session will take place and for how long, the tests designed and executed, the bugs found and overall notes (like with exploratory testing, a documentation tool like qTest Explorer can help here)
• Running the testing session without any interruptions
• Clearly documenting activities and notes during the session in a session report
• Holding a debrief between the tester(s) and the manager to review findings from the session and discuss next steps for testing

How to Align Testing with an Agile Delivery Process

Once you determine which testing methodology is right for your organization, you’re not quite done yet. You still need to align testing with the delivery. To achieve this goal we recommend a three-pronged approach:

1) Get involved in the development process as early as possible

The sooner testers can get involved, the better. Ideally, testers should be present from day one. That’s because giving testers a seat at the table every step of the way provides a higher level of insight into requirements and goals, encourages collaboration and helps hammer home the need to conduct frequent (if not continuous) testing.

2) Test frequently, but thoughtfully

As more and more teams adopt Agile methodologies, efficiency is everything. This need for speed has led teams to embrace DevOps and continuous integration as well in order to keep things moving, and that requires testing more frequently. But in the midst of an efficiency and frequency focused shakeup, testers need to remain thoughtful so as not to create more overhead and run unnecessary tests that actually slow down the process.

3) Hit the ground running with test creation

Keeping in mind the need for speed in today’s Agile, DevOps driven world, testers need to hit the ground running when it comes to getting tests created. Specifically, the more testers can reduce the time from requirements gathering to test creation, the better. Having a seat at the table for all conversations from the very beginning should help in this regard.

Agile Testing?

Although Agile has already made significant inroads into the software development lifecycle, there’s still a long way to go, especially among testing teams.

Going forward, more widespread adoption and greater maturity of Agile methodologies will require testers to go beyond test creation and execution and begin to focus on code delivery and integration too. At the same time, testers will need to hone their automation skills, become more involved in the entire software development process and continue to develop a collaborative relationship with developers. Ultimately, these changes will also require testers to become experts at development and product usage in order to provide more holistic testing strategies and take on the role of “quality champions.”

In the future, three key tenets will become particularly important for testers working in Agile environments:

1) Communication

Agile requires a tight collaboration between testers and developers, and that collaboration makes communication a top priority for testers. Additionally, in a world where quality becomes everyone’s responsibility, testers will become “quality champions” that serve as internal experts, which will put their ability to clearly communicate testing needs and reasoning under the spotlight.

2) Skill Diversity

In an Agile environment, everything can change on a dime, and that requires testers to be adaptable. Part of this adaptability is having a diverse skill set so that testers can change course as needed. For instance, functional testers need to expand their skills beyond manual scripted execution. This diverse skill set will be a must as different sprints require different types of testing to be executed in a short timeframe.

3) Business Mindset

Finally, Agile takes on a very customer-centric approach in order to ensure customers receive as much value as possible as quickly and as early on as possible. Testers have a big role to play in delivering this value, but it requires them to take on a business mindset so that they can understand customer expectations, desires and concerns and develop their testing strategies accordingly.

Why Leading Companies are Embracing Agile with qTest

Over 300 leading companies have chosen to improve their software testing processes and embrace Agile with qTest. Here’s what a few of these leaders had to say about why they made the choice to go with qTest:

“We were able to quickly mitigate all of our test cases from HP Quality Center to Tricentis qTest and get the team up and running in just a few weeks. The implementation process was excellent.”

-Radka Iordanova, Director, E-Commerce, Office Depot, Inc.

“We’ve seen a large reduction in tester errors and now we have history on our defects. We can see where the problems are. The partnership between American Equity and Tricentis has been wonderful.”

-Dennis Young, Assistant Vice President of QA, American Equity

“We evaluated a lot of other testing platforms out there and found Tricentis qTest to be the best and by far the most intuitive…we’ve improved efficiency by at least 50% since first implementing qTest.”

-Jesse Reynosa, Senior Quality Engineer, Zappos