be_ixf;ym_201912 d_06; ct_50
gamifying your test automation

Test Automation

Gamifying Your Test Automation Maturity Journey: Snakes to Avoid and Ladders to Look Out For

By Romil Chennupati and Aditya Hosangadi, Wipro

Test automation is a key focus area for every IT organization’s application strategy. Who wouldn’t want top-notch quality coupled with blazing fast test cycle times, reduced overall effort and the added bonus of reducing human error? For an organization just starting its test automation journey, test automation can seem like a magical, foolproof strategy, guaranteed to be successful and reap quick benefits. It is just an incremental version of basic manual functional testing after all. What could go wrong?

Now comes the interesting part. Ask any organization which is knee-deep into their enterprise-wide test automation journey, and they will tell you that the reality on the ground is very different. Test automation is like an iceberg. Most organizations only account for the visible part above the surface, and seem to either delay addressing, or be unaware of the part underneath the surface.

Of the many reasons for disappointing test automation journeys, the most common and important one is organizations believing that by just investing in a tool or platform, they will be taken to the promised land. However, test automation is a multi-faceted journey, and though the tool or the platform is the key component, how you use it to traverse this journey is equally important. It is a long one, and it requires a patient, multi-year approach to be able to see benefits across the enterprise.

To understand and appreciate this better, we have gamified this journey by comparing it to the ‘Snakes and Ladders’ board game. Below are the correlations we have considered:

  • The board in its entirety is the automation journey, peppered with snakes (pitfalls) and ladders (best practices).
  • Each square/ block on the board is an important step in the journey toward your test automation goal
  • The ladders are the key steps you should take to see nonlinear benefits in the future
  • The snakes are the roadblocks you encounter during the journey which have the potential to bring you back to square one because of shortcuts you may have taken, or key steps which were overlooked or not considered

Figure 1. Gamified view of Automation Journeys

For a successful and seamless automation journey, here are some of the key aspects to consider while blueprinting your strategy:

Automation Platform Costs

• ‘Snake’ to avoid

Every organization’s least favorite topic! Often, organizations seem to take a ‘stone age’ approach to factoring tool cost in their strategy. What we mean by this is that they consider the high initial tool cost of superior automation platforms to be a roadblock, as they may not have carried out a comprehensive Business Value Assessment (BVA) factoring in the ROI benefits down the line.

From our experience, organizations tend to move toward creating a framework of their own, integrating with either existing or new automation tools (often across different technologies), and selecting the more cost-effective options (open source or commercial). The high number of integrations needed and varied protocols involved make the framework extremely brittle and subject to frequent crashes, making automation more of a hindrance than a blessing.

• ‘Ladder’ to help you climb the board faster

Rather than investing in highly skilled talent (difficult to get in today’s extremely competitive markets) to build a framework from scratch, focus on end-to-end technology compatible automation platforms which can augment existing manual test engineers. This makes automation easy to adopt enterprise-wide, irrespective of the technology, be it custom web applications, Commercially Off the Shelf (COTS) products, or cloud-native applications. The ROI in such scenarios is easier to realize, and therefore automation will be perceived as a boon rather than a necessary evil.

Script maintenance and related overheads

• ‘Snake’ to avoid

Let us take desktop and mobile web apps in digital projects to illustrate this point. The most common test automation tool options for such projects are open source in nature. Yes, these tools do bring down costs in a major way. But they do have their drawbacks, which can impact ROI and schedule adherence severely. Dynamic web pages, for example, cause all sorts of issues when using open source tools, as the backlog of scripts to be developed and maintained can keep increasing, leading to behemoth testing efforts. Eventually, one will hit a roadblock and realize that that further scripting efforts are unable to reduce overall testing efforts!

• ‘Ladder’ to help you climb the board faster

A comprehensive tool fit analysis framework is a good start to any automation journey. If we continue with the previous example of dynamic web pages in agile/pseudo-agile projects, the tool’s ongoing maintenance efforts are a critical factor to be considered during ROI calculations. If traditional Open Source or COTS tools prove inefficient for such scenarios, there are AI/ intelligence-driven platforms available in the market, which have the ability to update their object repositories automatically, thereby eliminating the need to maintain scripts at all.

Optimum level of automation coverage. 50%, 75%, 100%?

• ‘Snake’ to avoid

To keep it short and simple, no, 100% automation is NOT an organization’s nirvana state. Many organizations believe in the notion that, “If I my automation coverage is 60%, I am still 40% inefficient!and end up moving toward a scenario where they have automated almost every test case out there. This is not always the right approach because as one moves toward extremely high levels of automation, there is often a linear increase in money spent on tool licenses, as well as the people needed to carry out scripting.

We once had a client who learned this the hard way. They had spent a large amount of effort and money to move from 70% to 90% automation. Ironically, the amount to reach 90% automation was more than what they had spent to achieve 70% automation – from scratch – in the first place. They eventually ended up realizing negative ROI.

Always keep in mind that higher levels of automation may not always result in positive ROI. It is okay to have low automation numbers as long as your ROI keeps increasing and one achieves that magical maximum coverage percentage.

• ‘Ladder’ to help you climb the board faster

Dynamically tracking ROI as well as technical debt, coupled with techniques such as impact-based testing is a key way to avoid over-automating one’s test suites. Always remember Brooks’ law and the law of diminishing returns when investing in any IT-related activity.

Another dimension organizations should explore is identifying the automatable test suite set at the start, to enhance coverage. Using historical analysis, the target number of artifacts to be automated can be further reduced, thereby enabling one to achieve a high degree of automation (considering only the new subset) at a much lower effort and consequent cost than before.

There are also algorithms available in the market today which can perform impact assessment by analyzing code level coverage and identify new changes. The intelligence provided can help with smarter automation coverage – only targeting the impacted code and leveraging model-based test cases, providing high confidence without compromising risk coverage.

To conclude, test automation is a multifaceted journey, which an organization needs to tackle diligently, keeping in mind the long-term benefits, the future technology road map of IT and changing business needs. Remember, one benefit of the test automation board is that there is no rolling of the dice required to traverse the board – you get to pick the number every turn, and the smaller the number the better!

About the Authors

Aditya is a lead consultant with Wipro’s Digital Assurance practice, a part of the Modern Application Services group. He works collaboratively with clients embarking on their digital transformation journey, to blueprint their multiyear quality engineering transformation strategies. His key areas of strength include the use of disruptive technologies revolving around Crowd, Mobile and User Experience in the sphere of application quality.

Romil is a senior consultant with Wipro’s Quality Engineering Consulting Crew, a part of the Modern Application Services group. He works extensively with clients to define and overhaul quality-engineering journeys with domain centric platform solutions. His key areas of strength include Quality Engineering for Cloud Native applications and next-gen technologies like Blockchain and Bots.