Contributed Articles

Achieving the Holy Grail of Software Testing: What CIOs Need to Know

How Reinventing Software Testing can Transform your Business — and Change the World

By Terry Milholland, former CTO & CIO for the IRS; CTO for Visa Inc., CIO and CTO for EDS; and CIO for Boeing

Do we really know the business risks associated with the new software we design, build, test, and implement?  Will the new capabilities we invested so much time and resources in actually function and perform and expected, or will they draw the ire of customer? Getting answers to these questions is the last “Holy Grail” of IT.   Unfortunately, most IT stakeholders focus on the number of tests that are successfully run, not whether the tests are providing insight into critical business risks. 

It is a rare enterprise CIO who likes to dive into software testing methodologies. But, IT leaders at all levels might be intrigued to learn what an archaic and ineffective process most enterprises are using to test the innovations driving their digital transformation initiatives. The way it’s commonly done, testing is an expensive activity that holds back your innovation—without providing clear insight into the business risks of your applications.

As an industry veteran with experience across a broad swath of industries, I can tell you that transforming testing is emerging as a stealth strategy to take your digital transformation initiatives to the next level.  Many organizations are quietly transforming their testing process to deliver instant insight into the business risks of each release candidate. At the same time, they’re slashing testing costs, which frees up additional resources to dedicate to innovation.  With this new competitive edge, they’re surging ahead.  

How this Process Holds Us Back

What might surprise IT leaders—and especially CIOs—is that amount of time and money that testing requires vs. the business value that it actually delivers.

Most testing focuses on the number of tests that are run, not whether the tests are addressing the risks that are most important to the business. Often the last stage of testing is conducted by a dedicated independent testing organization with inadequate understanding of their company’s business risks or objectives with the new software system.  To make matters a bit worse, defects (or even false positives) discovered late are triaged for impact, and perhaps not addressed because of mounting pressure to release.  And load testing – streaming transactions and queries against a system to make sure the system can handle full production volumes – usually gets short shrift. 

I understand that many IT leaders find software testing boring. However, upon learning that this process often consumes 40% of an IT application budget, people tend to find it more intriguing.

Why is testing so ineffective yet expensive? Thanks to agile methodologies and DevOps, software development cycle times have been reduced significantly, yet today’s software testing continues to operate at a slower pace—creating a process cadence mismatch. This is because testing is dominated by manual tests. In fact, 70-80% of software testing is manual, tedious, and time-intensive.  At the end of the day, we do not really know whether the right tests were done, nor do we know if the software system and databases can perform well in production.

Transforming Testing for a Competitive Edge

So, should a CIO care about software testing?  Yes – but not about how many tests were completed.  What is far more important is whether the tests that were completed actually address the business risks of the enterprise in a way that gives confidence in test results…