Functional testing reviews each aspect of a piece of a software to make sure it works (aka functions) correctly. Quite simply, functional testing looks at what software is supposed to do and makes sure it actually does that. So while functional testing looks at an application’s ability to execute, non-functional testing looks at its overall performance (e.g. by testing scalability, reliability, security, and compatibility).
When conducting functional tests, you typically need to follow a process that looks something like this:
- Use test data to identify inputs
- Determine what the expected outcome should be based on those inputs
- Run the test cases with the proper inputs
- Compare the expected results to the actual results
Following this method, if the expected results and the actual results match, then you can conclude that the software functions properly and that the test has passed. If they do not match (assuming you properly understand what the outcome should have been and used the correct input), then there is an issue with the software.
Common functional tests include:
- Unit tests: Tests an individual unit of the software to make sure it performs appropriately.
- Integration tests: Takes multiple individual units of the software and tests them as a group to make sure they interact appropriately.
- Smoke tests: Tests the major pieces of the software in a non-comprehensive manner to ensure the software works well enough (or is not riddled with too many issues) to move on to additional tests.
- Regression tests: Tests that code changes do not have a negative effect on the functionality of the software.
- Sanity tests: Tests the major pieces of the software in a non-comprehensive manner after code changes have been made to ensure the changes didn’t create any serious issues. Just like smoke tests take a wide and shallow view versus unit tests’ more in depth view, sanity tests take a wide and shallow view following new builds versus regression tests’ more in depth view.
- User acceptance tests: Often the last step before software goes live, user acceptance tests make sure the software meets user needs. End users typically perform these tests during a Beta period.
In addition to confirming that the main areas of the application function correctly, these tests also aim to review the usability and accessibility of the software as well as what happens when an error occurs.
Use cases, tips & best practices for the most popular functional testing types
Regardless of what type of functional tests you’re running, there are several different methods you can use to execute those tests and organize your efforts. To help you get started, we’ve compiled a list of the five most popular functional testing methods, including what they are, when to use them, and tips and best practices for doing them right.
Manual scripted testing
- What is manual scripted testing?
- When should you use manual scripted testing?
It may seem like an ideal option to forgo manual testing in favor of automation, but not all tests can be automated. We still need manual testing in a variety of cases. Some common examples for when it makes sense to take a manual testing approach include when you’re dealing with legacy systems that don’t easily support automation, when you need to adhere to strict regulations and require documentation as a result, and when you need to run complex tests. This last point on complexity is a bit open-ended, but the idea is that when dealing with complex test cases, having a human at the steering wheel can make it easier and more likely to catch issues. In general, manual testing is better in cases where you need to closely mimic real-world scenarios.
Manual scripted testing best practices, tips and tricks
- Write reusable test cases (and then reuse them!): Developing test cases takes time, so the more reusable you can make test cases, the better (as long as you then remember to reuse them when appropriate). To write reusable test cases, make sure that you write in simple language that’s easy to understand and that the test steps are short and easy to execute. Then, to help reuse test cases when appropriate rather than reinventing the wheel each time, be sure to stay organized by managing your test cases in a single source of truth like a test case management tool.
- Design tests driven by test data and configurations: Relying on test data and configurations to develop your test steps will make sure you have everything you need to complete the tests and fully understand the results. For example, you need to make sure you have the right login credentials so that you have access to all of the functionality you are supposed to test and ensure you have the right input data, as this will impact whether a test passes or fails. Ultimately, you need to pay close attention to test data and configurations at every step of the way, as these details are also important when informing developers and product owners about issues and helping them understand the root causes of those issues.
- Prioritize high risk and complex test cases: The fact that you can’t have 100% test coverage combined with the time-consuming nature of manual testing makes it imperative to prioritize your test cases. When it comes to manual testing, you need to prioritize high risk and complex test cases, as these are the ones that most need a human at the steering wheel.
- Consider opportunities for automation (without going overboard): Although you shouldn’t automate all of your testing, you should always look for opportunities to embrace automation in order to save time and make it easier to run certain tests more often (especially as testing becomes more integrated in the entire software development process). Remember to take an automated first approach when considering which tests are best off executed manually and which tests are good candidates for automation. You should also keep complexity top of mind. While highly complex test cases should remain manual, automating simple smoke tests can add significant value.
- Don’t overlook the importance of documentation: Documentation is a critical piece of manual testing, especially when it comes to tracking issues through reports. A good report is essential to helping others on the team (testers and developers alike) understand what issues exist and how to find them. To write a strong report, you need a straightforward title that clearly specifies the issue, an ordered list of steps to recreate the issue that are easy to understand and follow, details on the severity and priority of the issue, and insight on what should happen if the issue is resolved.
Exploratory testing
What is exploratory testing?
Exploratory testing is a time-boxed approach in which testers set out a certain amount of time to “explore” the software. Although this is also a form of manual testing, exploratory testing deviates from a strict workflow of planning, designing, and executing test case steps. During this session, testers aim to understand how the software works and identify different tests to run based on that understanding. Since exploratory testing is not scripted, it often mirrors how users will interact with software in real life.
This testing method is complementary to context-driven testing, which contends that there is no “one best way” to conduct testing. Rather, it argues that testing needs to be handled differently for each project based on the context in which the software will be used. Exploratory testing complements context-driven testing because both take the uniqueness of each piece of software into account rather than supporting a standard, one-size-fits-all approach to testing.
When should you use exploratory testing?
The best instances in which to use exploratory testing include those when you are under a time constraint, as they require minimal preparation and allow for fast feedback, those when you do not have any specifications from developers, those when you need help determining what types of tests to run, and those when you want to conduct a good conscience catch-all test to make sure you didn’t miss anything when executing previous tests.
Exploratory testing best practices, tips and tricks
- Always include exploratory testing: Regardless of what other testing methodologies you plan to use for a given project, you should always include exploratory testing as well. This inclusion of exploratory testing helps you determine what other tests you might need to run or which methodologies makes the most sense to use and can serve as a good check on the status of your results from tests already executed. Best of all, you can insert exploratory testing at any phase (or even multiple phases) of the project.
- Develop a clear charter: You need to develop a clear charter for every exploratory testing session you conduct. This charter should include a mission statement, a list of the areas you’re testing, the names of all the testers involved, the date and time of the session, a list of the tasks involved (including the duration, the number of tests designed and executed and the number of bugs investigated and reported), notes on your testing activities and details on any bugs and issues you identified. An exploratory testing and documentation tool like Tricentis qTest can make this process easier and more efficient.
- Monitor your time: Because exploratory testing is time-boxed, it’s important to monitor your time. Specifically, you should track your time spent creating tests versus executing tests, finding bugs versus reporting bugs and setting up the session. But perhaps most important is the actual time you spent on an exploratory testing session versus the time you planned to spend on an exploratory testing session. Overall, these comparisons can help you determine how much more testing a piece of software might need as well as which areas of the software need the most testing.
- Prioritize exploratory tests over low value scripted tests: If you’re running scripted tests that end up providing little to no value, it’s a clear sign to switch over to an exploratory model. Exploratory testing can help you correct course and result in more value for the same time spent by helping you determine a new testing strategy that will uncover issues and result in a higher quality end product. When it comes to exploratory tests versus low value scripted tests, the former wins every time because it can breathe in new ideas by more closely mimicking real world user experiences and by removing the limitations posed by predetermined, scripted tests.
- Make documentation your best friend: Exploratory testing should be different every time you do it, but that doesn’t mean you can’t learn from what worked in the past and what didn’t. Although the actual actions you take within different sessions will vary, your high level approach can certainly be similar time and again. To understand what works and what doesn’t, you need to make documentation your best friend. You can do so in your charter by detailing what you did, how you did it, and what you found in order to determine the overall effectiveness of your efforts. As a bonus, the more details you have, the clearer the roadmap you’ll have when outlining next steps based on your exploratory session.
UI/automation testing
What is UI/automation testing?
UI/Automation testing is the act of conducting specific tests via automation (as opposed to conducting them manually). While it’s true that when you actually run a test via automation it will run on its own, automation testing is not just “set it and forget it.” With automation, you need to develop and maintain source code for testing scripts, which includes updating the code as you update the application. Additionally, you can’t rely on automated testing tools alone, since you need people with the skills to operate those tools and maintain the source code. All of that said, automation can help you run tests more quickly and do so automatically based on certain conditions being met.
Automation can help you run tests more quickly and do so automatically based on certain conditions being met.
When should you use UI/automation testing?
It’s tempting to try to use automation for all of your testing needs, but doing so would be a mistake. For example, too much automation in testing can lead to maintenance challenges when trying to keep up with changing source code needs. It also means you lose the human aspect of testing, which is critical to catching flaws in real-world scenarios. As Michael Bolton puts it, automation is more about checking or confirming that something is true than it is about testing or recognizing unanticipated problems.
With that in mind, the scenarios best suited for UI/automation testing include high-risk scenarios that require a simple smoke test (such as confirming that user logins go through), simple tests that you need to run often (automation can improve efficiency here) and tests that often fail due to human error. If you’re not sure what to automate, running an exploratory testing session can help you make more informed decisions. And although you should never hit 100% automation in testing, the number of instances in which automation makes sense to use will likely increase as you move to a continuous testing model, since this model requires more testing to be completed more often throughout the development cycle.
UI/Automation testing best practices, tips and tricks
- Test early and often, especially in high risk situations: One of the biggest benefits of automation in testing is the ability to execute tests quickly and repeatedly without manual effort. It’s important to take advantage of this benefit by running automated tests early (to uncover issues as soon as possible) and often (to catch issues as soon as they arise). This “test early and often” mentality is particularly important in high risk situations, as the sooner and more regularly you test, the better position you’ll be in to catch and mitigate the risks associated with potentially damaging issues.
- Make maintenance a priority and aim for sustainable source code: Automated testing is only as good as its source code, and source code tends to require regular maintenance. Specifically, you typically need to update source code every time developers make changes to an application because the source code needs to reflect those changes or the test will produce inaccurate results when executed. Therefore, you need to make maintenance a priority. That said, you should aim to make your source code as sustainable as possible so that the changes you need to make alongside application updates will be minimal. Even with a sustainable approach, remember that you still need to revisit the source code each time the software is updated.
- Consider the best tool for your needs (but remember your people!): When selecting an automated testing tool, consider your needs in terms of development language, operating systems and platforms as well as each tool’s capabilities around creating feature-rich tests. Tool selection is most definitely important when it comes to automation testing, but it’s also not the only piece of the puzzle that matters — so too do the people who will develop the automated tests. As a result, you also need to consider ease of managing tests and source code within the tool and your team’s relevant skills. You can learn more about software testing tools, including the most popular automation tools, here.
- Regularly evaluate results to avoid running low-value tests: Just because you can run a test quickly and easily if it’s automated, doesn’t mean that you should. That’s because automated tests still require effort in the form of maintaining source code. Therefore, you need to analyze the results of your automation testing to determine if you’re running tests that aren’t providing any value, such as those that regularly miss issues that surface later on or those that pass 100% of the time (which likely means you don’t need that test period unless it’s a high risk situation). If you do identify low-value tests, you need to decide if you should change the test, change the approach (e.g. by taking a manual approach instead), or eliminate the test altogether.
- Recognize the limitations of automation testing: For all the benefits that automation delivers, it is not the be-all-end-all for testing. In fact, even if you could automate all of your tests, you wouldn’t want to do that because automation has its limitations. Plain and simple, there are some instances where you need a human perspective. For example, with UI testing, automation can determine if an element renders, but only a human can determine if it appears in the right place and looks good. In general, automated testing does not mirror the user experience, so you need to keep the goals of your test cases in mind when determining whether or not to take an automated approach.
Behavior-Driven Development (BDD) testing
What is BDD testing?
Behavior-driven development is a test-first methodology that is a subset of test-driven development (TDD). BDD encourages collaboration between all stakeholders — testers, developers, product owners, etc. — to remove the silos between these different groups that often create a “broken telephone” scenario in which developers must translate requirements from product owners, testers must translate requirements from developers and so on. By bringing everyone together, BDD “creates a shared understanding and surfaces uncertainty,” according to its founder Dan North. In other words, it gets everyone on the same page and makes it easy for each party to ask questions that get to the root of the business case, ultimately helping everyone work together to create a better end product to fit user needs. The end result of BDD is requirements defined in plain English and scenarios that typically follow a Given-When-Then format that testers and developers can use when determining testing needs. In general, BDD is a very user-focused approach that delivers high levels of efficiency and effectiveness when it comes to user experience testing.
When should you use BDD testing?
You should employ a BDD approach when writing tests and specifications. BDD is most helpful when it comes to unit tests, which often require tedious changes every time software gets updated. BDD can simplify this process because it requires testing based on behaviors, not based on code.
BDD testing best practices, tips and tricks
- Make your scenarios declarative (not imperative): Writing scenarios in a declarative format as opposed to an imperative format leads to simpler scenarios that better mimic the user’s perspective. For example, a declarative scenario for making an online purchase would be written as:
Given I am in my shopping cart
When I place my order
Then I should see an order confirmation
The imperative version of the scenario would be written as:
Given I am in my shopping cart
When I click the “place my order” button
And I fill in my billing information
And I fill in my shipping information
And I click the “submit my order” button
Then I should see an order confirmation
As these examples illustrate, the declarative version is simpler, easier to follow and maintain and takes on the perspective of a user, not a programmer.
- Understand the Gherkin syntax: The Gherkin syntax of Given-When-Then (as used in the scenarios above) is critical to BDD testing. This syntax should describe the context (Given), the action (When), and the outcome (Then). Each item in this syntax is important, as is the order in which the items appear. Forgoing an item or placing them out of order can change the entire meaning of the syntax and negatively impact future testing and development efforts.
- Be mindful when writing step definitions: With BDD, you need to write step definitions that translate your Gherkin scenario into actions the system will take during testing. As you develop your steps, you should write unique definitions to avoid issues in which the system does not know which step is the match for a given scenario. You should also make sure each step only includes one action, as having one action per step makes it more likely that you can reuse steps across scenarios.
- Recycle your steps: If you’re mindful when writing your step definitions and write with reusability in mind, then you should also recycle your steps as often as you can. Recycling steps not only saves time and effort associated with creating new steps, but it also simplifies maintenance, as any time you need to change a step you can change it once and have that change apply across multiple scenarios.
- Don’t overlook the importance of and interest in collaboration: Many testers and developers overlook the importance of collaboration or fail to gauge the other side’s interest in collaborating. When this happens, the end result suffers. Whether you’re a developer, a tester, or in any other role, you’re all on the same team and have the same end goal: to deliver high quality software that solves a problem for the end user. And collaboration is key to achieving this goal in any situation, especially when taking a BDD testing approach.
Risk-based testing
What is risk-based testing?
Risk-based testing is an organizational approach that prioritizes testing for high-risk areas of software. It contends that high-risk areas need to be tested often and have a high level of test coverage in order to mitigate the risk.
When should you use risk-based testing?
Risk-based testing is best used when facing time, budget and/or resource limitations. You can determine which areas of your software are high-risk in numerous ways. For example, risk might be due to high impact (e.g. if an issue will affect 90% of users), the presence of sensitive information (e.g. if users need to share credit card details or personally identifiable information), or a high level of complexity.
Risk-based testing best practices, tips and tricks
- Assess risk at the project level: In order to properly understand all of the potential risks associated with a piece of software, you need to assess risk at the project level to get a more comprehensive picture. This project-level assessment should include everyone from testers and developers to business stakeholders and should outline what potential risks exist, the impact of those risks, plans to test and mitigate each risk, likely causes for each risk, and a contingency plan. Taking on this risk assessment should help you properly scope testing requirements and provide everyone involved with a clear picture of the potential risk and how it will be reduced.
- Develop a contingency plan: Even though the goal of risk-based testing is to mitigate potential risks by paying particular attention to the most potent risks, sometimes things slip through the cracks. Whether you miss a potential risk during testing, the issue isn’t properly resolved, or anything else, it’s always important to have a contingency plan in place that dictates what actions your company needs to take should any high-risk issues materialize.
- Identify risks with a heuristic approach: James Bach recommends taking a heuristic approach to risk identification. This approach requires you as the tester to sit down with developers and ask questions about potential risks as the developers review how the software works. Doing so allows you to question the source directly and ask follow up questions as needed. During these discussions, you should ask questions about potential weaknesses/vulnerabilities, threats, and victims of failures. You can take this heuristic approach a step further by going through the same process with product owners, stakeholders, and even end users. Extending your questions to these groups will paint an even more complete picture and help identify risks about which the developers were not even aware.
- Prioritize risks using a statistical analysis: The best way to prioritize risks is to use a statistical analysis that weighs the severity of impact for a particular risk against the probability that it will happen. In terms of severity of impact, you should consider the criticality of the issue (e.g. insecure credit card data is far more critical than a login glitch), the visibility of it, including the number of users who will encounter the issue (e.g. is it an issue at login that users are very likely to encounter or an issue on some deep page that users are far less likely to visit), and the number of users it will impact (e.g. does it apply to all users or only a small subset of users). As you consider severity of impact, you can rank risks on a sliding scale from highly critical to negligible. In terms of probability, you should consider the likelihood of failure. As you consider probability, you can rank risks on a sliding scale from inevitable to unlikely.
- Regularly monitor risks: As both the software and its end users/environment change, so too will the associated risks. As a result, it’s important to regularly monitor risks in order to identify changes in their criticality (due to changes in severity of impact or probability) and to pinpoint new risks and/or risk triggers. Just like with the initial risk assessment, this ongoing monitoring should take place at the project level and involve stakeholders beyond the testing team in order to paint the most comprehensive picture of all potential risks.
Delivering the software experience users expect with rock solid testing
Today’s users are tech-savvy: They use software in nearly every aspect of their daily lives and they have high expectations for how this software should work in order to help them go about their everyday needs better, smarter and faster. As a result of these high expectations, software testing has become increasingly important to and intertwined with the entire software development process.
And just as the bar has been raised on the practice of software testing, it’s also been raised on software testers, who are now a more strategic part of the team than ever before. These changes have created boundless opportunities for software testers, but capitalizing on those opportunities all starts with building a strong foundation of skills, including developing a deep understanding of the various approaches to testing and mastering the execution of them.
If you’re ready to build that foundation by learning even more about the ins and outs of software testing, Tricentis has you covered.
We’ll help you keep a pulse on all things testing through our:
- Blog: Updated multiple times a week, our blog shares quick insights and actionable tips from our team of experts
- Extensive resource library: With infographics, webinars, eBooks, data sheets, videos and more, you could get lost in here
- Quality Jam: Our annual user conference brings together the brightest minds in the industry to talk about the future of software quality, help you brush up on some skills and of course have some fun along the way