It’s the end of the year (finally). Instead of writing more, we wanted to highlight some of the most interesting, inspiring, and contentious software testing quotes that have crossed our path over the past year.

***

The testing mindset is ‘I’m going to figure out how to break this.’ This is great because it gets developers thinking: ‘I’m not going to let them break this’—and the software is much stronger as a result.

Anders Wallgren, CTO of Electric CloudProfessional Testers vs SDETs: The Next Great Testing Debate

 

In previous years we found that test automation had a significant impact on continuous delivery. This year, we built upon prior years’ research and found that continuous testing positively impacts continuous delivery.

Dr. Nicole Forsgren, Gene Kim, and Jez Humble at DORA  | 2018 Accelerate: State of DevOps Report

 

Most firms recognize that their test suites do not always give them a good indication of business risk. In fact, just 15% of respondents say this is the case today — and nearly 40% say that they have a good indication sometimes or less often. Even more advanced Agile+DevOps firms see the limitations here: Fewer than one-third of them say their test suite always gives them a good indication of business risk.

Diego lo Giudice, Forrester VP, Principal Analyst | Forrester’s Agile Metrics and DevOps Metrics for Software Quality

 

A lot of clients are looking for SDETS in the market. Let’s just unpack what that looks like as a job spec. Typically, clients are insisting on deep technical skills. They want to see a developer history and they want to see deep skills in automation. We even have clients insist on a 4- hour exam to pass their onboarding. So, I think the clients are getting very serious about technical skills in testing. Why is this? It’s to try and correct the automation problem.

So, is that what you do if you have challenges in automation, find some SDETS, drop them in, sit back, and it all gets fixed? No, because automation is pretty complex. A lot of things have got to come together. You’ve got to have the right tooling. You’ve got to have the right strategy. You’ve got to have environments that work. But SDETs can help and I think, as the direction of travel, testing having more technical skills in its armory, is a good thing, but we need balance. It’s not the only game in town.

Jason Steele, Senior Account Executive Accenture | The Great Debate: Testers’ Role in DevOps Continuous Testing

 

If you’re doing test automation, what’s the biggest pain in your ass? It’s test data…The developers are done, the testers are getting ready to test, and guess what? The tester has no test data. Not only does he not have test data, but he doesn’t have time to go create that test data now. It’s too late.

Renee Tillet, Manager of DevOps Verify at Duke Energy |Making Continuous Testing for DevOps a Reality

 

I’ve seen testers reject automation because it totally goes against the way that they’ve always done things, and it’s not very comfortable. They’re not very receptive of change, but they also worry about, “Well, if I’m automating this, I’m automating myself out of the job.” That’s a mindset that people really need to change because what they don’t realize is they should automate themselves out of a job.

If you automate yourself out of that job, you’re going to show your expertise and you’re going to get promoted, or you’re going to do more interesting things rather than, “Oh, here comes another build.” Click, click, click. “I’ve got to do this same thing over and over again. It’s boring.”

Adam “the Automator” Bertram | Continuous Testing Live: Automate Everything in DevOps? Not so Fast.

 

Even today, 80 percent of enterprise testing is done manually. This is crazy. We’re getting ready with autonomous cars on the one hand, but, on the other hand, we’re manually testing the software that’s so fundamental to every enterprise. If you move to Agile development but your testing cycle is still 6 to 12 weeks due to manual testing, you’ll fall right back into a Waterfall mode.

Sandeep Johri, Tricentis CEO | DevOps Chat: Continuous Testing with Sandeep Johri, CEO, Tricentis

 

You can’t continue testing manually, the way that it always had been done, when you have an enterprise that’s moving at the speed of light. As a CIO, you do not want to be the bottleneck that’s slowing everything down. This means that you have to address testing and ensure it advances your digital transformation initiatives rather than holds them back.

Jennifer Sepull, former CIO USAA, Kimberly Clark, American Honda |  CIOs Share Why Software Testing Matters

 

I do wish more people would pop their heads up a little bit more and think of two kinds of wider worlds. One is that there’s a huge worldwide testing community and it always blows my mind how few testers know about it. And the other dimension is, there are so many things that are useful to a testing mindset that are outside the field of computer programing and software engineering and software development altogether. That’s where many of the best ideas come from: outside a discipline.

Michael Bolton | Learning to Embrace Confusion in Exploratory Testing

 

Like Lucy and Ethel struggling to keep pace at the chocolate factory, many software testers have been scrambling to keep pace with accelerated processes—then along comes the supervisor proclaiming, “You’re doing splendidly…Speed it up!”

As expectations associated with test are changing, legacy testing platforms aren’t keeping up. Legacy testing platforms take a “heavy” approach to testing. They rely on brittle scripts, deliver slow end-to-end regression test execution, and produce an overwhelming level of false positives. As a result, they’ve achieved limited success with test automation. The overall test automation rate is 18%, on average—8% for enterprises. In a polling question asked at industry webinars and trade shows, respondents overwhelming reported that the results of test automation to date have been “So-So.”

Wayne Ariola, Tricentis | What Testers Need to Know about Continuous Testing

 

I think quality is core. You need to treat quality as a product internally, and you do not outsource a core product.

Mik Kersten, Tasktop Founder and CEO, author of Project to Product |  The Great Debate: Testers’ Role in DevOps Continuous Testing

 

I know that software testers can make the company more profitable, make it safer, and help it grow faster. If they make your testing faster and get your new apps out there, you can be more competitive. And if they can do that while lowering costs, that’s remarkable.

Robert Webb, CIO Etihad | CIOs Share Why Software Testing Matters

 

If you were to ask me how to define automation, it’s basically every way in which you can use tools to enhance or support your testing activities. Executing functional regression tests or checks is part of that and is a big part of that, but it’s definitely not all of it. If you have some kind of tool that helps you generate lots of test data that you can then use for the testing that you’re going to carry out, that’s automation. Performance testing is automation. Things like service virtualization, for me, that’s all part of automation.

If we start to look at automation in a broader sense, then I think it would quickly become more clear to people that there’s lots more that can be done through automation where instead of just replacing those tedious and expensive regression checks that need to be done at the end of every development cycle.

Bas Dijkstra, Test automation trainer and consultant | Continuous Testing Live: Test Automation’s What, Why, and How

 

The digital transformation sea change is impacting virtually every department at every business today, and few QA departments will remain untouched. The resulting wave will likely wipe out the “boring as shit” manual verification many people call testing. But it could also elevate testing to a sexy discipline where testers become the primary stewards of the customer experience.

Wolfgang Platz, Tricentis Founder and CSO How the Role of Software Testing is Evolving

 

I’d like to give you a few perspectives, being in India, and having been a consultant, and actually touring services companies. What I think is a lot of companies claim this is the way they test software. Yes, that’s the way they test software, but what they are not actually telling, is that they do outsource a bunch of “checking” jobs to some services companies in India.

I have seen books published that cover how Google tests software, or how Microsoft tests software. I was consulting for a team whose job was to open the top 5,000 links of a particular browser, and report saying “Yes, every page element loads.” But, when they publish a book or a blog of, “This is how we test software,” they don’t tell that checking is a part of testing, and that is outsourced to a company in India or some other place. But they project an image saying everything is automated, and there are no humans. Yes, there are no “visible” humans, is what I would say.

Pradeep Soundararajan, Founder CEO of Moolya Testing & App Achhi Continuous Testing Live: Will AI Make the Leap to Exploratory Testing?

 

If we really want to maximize the impact of innovation, we need to make testing faster, better and cheaper. As long as I’ve been in the profession, the joke has always been “faster, better, cheaper — pick two.” However, if we open our minds to reinventing testing, we’ll discover that these trade-offs are no longer necessary. In fact, once you start making software testing better and faster, it organically becomes cheaper.

Todd Pierce,  Former Chief Digital Officer for the Bill & Melinda Gates FoundationHow Reinventing Software Testing Can Transform Your Business…and Change the World

 

At Accenture, do we want to have everybody be trained engineers from the best engineering schools in the world to come in and do quality engineering, do development, do pattern analysis, do whatever it is that we do from a technological perspective? We are a technology consulting company after all.

Well, the answer that we have decided on is no, we don’t. We want to have people with accounting backgrounds, people with a history backgrounds, people with philosophy backgrounds, people with engineering backgrounds, mathematical backgrounds, right? We want to have a mix of people that have different skillsets. And you can extend that to demographics. We want to have people from different demographic backgrounds, different races, different ethnicities, sexes, sexual orientations. By doing that, we’re getting an optimal mix of people without optimal skills that bring different ideas to the table on a consistent basis.

Jeff Wilkinson, Managing Director at Accenture | Does Diversity Matter in Software Testing?

 

Think of the School House Rock on how a bill becomes a law. This is how we need to think about our code. We need to consider all the different things that happen to our code as it passes along the conveyer belt of our software delivery factory, all the different people that are involved in the process. Once we have this mapped out, that’s when we can really start optimizing the process. If a bug escapes through that process and reaches the customer, this means that there’s a larger organizational failure that must be identified, analyzed, and addressed.

Anders Wallgren, CTO of Electric Cloud | Professional Testers vs SDETs: The Next Great Testing Debate

 

When people ask me how to be successful at load testing, or, how do you guarantee performance, I always tell them that’s a massive trap to get yourself into. And, I think that’s because maybe the attitude comes from, say, people come to performance testing with an automation background and are heavily into seeing “Red/Green” kind of results. I always say to just forget that. It’s not “Pass/Fail.” To me, performance testing is more like a risk management activity. You can never get to the end of a test effort and say, “We’re done, ”because you only know as much as you know, and what you’ve observed during testing. What you can provide is a risk assessment. You can say, “I’ve identified these types of risks to production based on impact to users, or the likelihood of occurring. We’ve tested some mitigation around that.”

Tim Koopmans, Flood co-founder | Continuous Testing Live: Why Load Testing = Exploratory Testing

 

A friend of mine once said that venture capitalists don’t invest in dots, they invest in lines. They begin by taking conversations with a lot of different companies until they can create a picture of what that company’s doing, and they can be confident that they are making the right decision and putting money into it. The same is true of measuring performance. You can’t measure performance by just taking a few tests here and there and hoping for the best, which sadly is still what a lot of people do with load testing.

The reality is load testing, like any sort of measurement, any sort of monitoring, is about connecting a lot of dots to get a picture of how your application is going to perform in production.

Ivan Vanderbyl, Flood co-founder | Load Testing at the Tricentis Accelerate 2018 Conference

 

If you have a look at the Agile test pyramid, what you’re going to see is these different test stages. Where is the perspective of an Agile team? The perspective of Agile teams is usually limited. It’s going to cover the unit level, it’s going to cover the integration level to some regard, but it’s not going to go beyond. It’s not going to go for system integration, it’s not going to go for end to end. These things are simply missing, because they are behind the horizon of the Agile teams.

This is actually also the horizon of where the open source tools are usually working, because open source is from devs, for devs. Who’s taking care of the end-to-end testing, integration testing, and regression testing? Testers—and Tricentis— are, and this is crucial.

If you don’t do it, your software is going to hit production in a bad way. You’re going to have a gap between what you deliver and what you want to set into production. Your operations guys will say, “Oh, don’t do it to me, don’t do it to me.” What we really need to do, is we need to establish what we call a staging area that is in between the delivery, at the end of the CICTCD, and goes into production.

This staging area better have—I’m sorry for the other bad word—a quality gate. Make sure that quality is here. What we’re going to see, is that on one hand yes, testing is going to shift to the left into the teams, but on the other hand, it’s going to shift more to the right as a quality gate that is very, very close to the operation side.

Wolfgang Platz, Tricentis Founder and CSO |  Digital Transformation, the Future of Testing and Navigating the Path Forward

 

Other scripting tools for test automation were not well adopted well because they didn’t really get traction within each of the teams. We’ve had success democratizing citizen data scientists and citizen integrators with applications like SnapLogic. Now we’re taking that same approach to test automation, using model-based test automation. This allows our business analysts to start test automation in an easy, fast way that will get us away from what we had before, which was a lot of scripting. Our goal is to get heavy, heavy adoption in the test automation space.

Say you’ve got Team A over here, and the Team B over there. Team B’s leaving at a decent hour of the night, and Team A is working all night. Team A asks, ‘Why are you leaving so early? Don’t you have more testing to do?’ Team B responds, ‘Well we’ve got all our testing automated. I’m going to push a button and I’m going go home for the night.’ That gets teams to adopt test automation.”

Amber Woods, VP of IT Enterprise Applications and Platforms at Tyson Foods | Making Continuous Testing for DevOps a Reality

 

What warmed my heart is that about six months after we really started getting into test automation, one of the business COE managers called me up and said ‘Wow, where did this come from? I want to put it the hands of all of my business process experts. For the first time, we can control our own test automation. Test automation helps us ensure that, over and over again, business critical functionality works after each application change.’ That actually started a competition amongst different business units—everybody wanted to get on that bandwagon.

Ann Lewis, Quality Manager at ExxonMobil |  Making Continuous Testing for DevOps a Reality

 

In an Agile environment where various developers and testers use different technologies and platforms, and sometimes with remote teams, having a stable test environment to base all quality activities on is a key for success. It is not just about matching the test lab and test environment to the tools of the team, but also matching these to the team’s skills and software methodologies, such as BDD, ATDD, TDD or others.

Test labs that don’t seamlessly fit into development CI processes cannot serve the purpose of DevOps and CT. The stability and reliability of CI relies on the lab as its backbone.

Finally, especially in the era of DevOps and CT, there is much more test data being generated per execution. Having the ability to get great insight into large piles of test data and drill down into specific issues can be a key enabler to maturing CT. Executives are unable to reach a “go, no-go” decision without executive dashboards; developers and release managers cannot see whether their CI processes and builds are moving in the right direction and remaining within time constraints. In many cases, pinpointing an issue feels like finding a needle in a haystack; test engineers and test managers can lack the necessary test artifacts to provide proper feedback to developers.

Eran Kinsbruner, Director, Author, Lead Software Evangelist at Perfecto |  Continuous Testing for DevOps Professionals

 

I think there’s been a significant misunderstanding of the role of test, and the way that companies try to emulate what startups do, and what the tech giants have done. I’ve seen a lot of enterprise IT organization who are trying to take this very lightweight approach and bring as much testing as possible into development, and leverage on the opensource frameworks that you’ve heard about. But if you actually look under the covers of what’s happening in a tech startup or a growing tech company, or what’s happening in some of the tech giants, the amount of investment in what we all think of as testing is actually tremendous. It’s just done a bit differently.

Mik Kersten, Tasktop Founder and CEO, author of Project to Product | The Great Debate: Testers’ Role in DevOps Continuous Testing

 

It’s true: Everybody can test. If that happens to be your developer, so be it. But everybody can also test badly. So you probably want someone on your team who can test more reliably, more deeply, and more efficiently than anyone else on your team. In the same way, you probably want someone on your team who can develop software more reliably, more sustainably, and more efficiently than anyone else on your team.

So, for the same reason most companies have dedicated developers, dedicated product owners, dedicated product managers, dedicated support engineers, dedicated marketing strategists, dedicated documentation experts, dedicated UI designers, dedicated UX experts, and so on, they also have dedicated testers on their teams.

These companies understand the value of professional testing. They don’t reduce the act of testing to the number of test cases that have been created, in the same way they don’t reduce the act of development to the amount of code that has been written, or they don’t reduce the work of a product owner to the number of user stories that have been created.

These are the companies that realize it requires specialists in each field to succeed with their software along the line.

Ingo Philipp, Tricentis Evangelist | The Cost of Software Testing

 

Bonus: Our Top Software Testing Blogs of the Year

If you’re still reading—and you really want to read more—here’s quick list of our top 18 blogs of 2018:

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

X
X