image

Selenium Testing

A realistic look at Selenium adoption for software testing

By Matthew Heusser matt@xndev.com for Tricentis

The story I am about to share with you actually happened.

It is not a joke.

I am not exaggerating.

It was not that long ago.

A potential client brought me in to discuss a test tool project. The team was something like thirty staff members, with perhaps another sixty business folks part-time. They were implementing something like the Scaled Agile Framework, and shipping a change to production every eight weeks or so.

I asked what happened to the last test automation effort.

Awkward laughter followed.

“Well, we sort of made a half-hearted attempt to build a Cucumber BDD framework a while back,” one of the managers said. “But, nothing ever came of it.”

No, not that. The big one.

Three years prior, while I was consulting, a different part of the test organization spent a few hundred thousand dollars implementing a test tool strategy in Java, with Selenium and JUnit.

Two different contractors, forty hours a week, flown in from California, for maybe a year-and-a-half.

They didn’t know about it.

It turns out the entire management team had turned over since the initial eCommerce project began three years ago. None of the people on the maintenance team pushed code because they did not know where it was stored.

Besides, those jokers didn’t know what they were doing. I am the great Matt Heusser, you see. If I put the project on the right footing, it will go fine.

So, we started over again.

All this reminds me of the writer George Santayana, whose great lines include the quote, “Those who cannot remember the past are condemned to repeat it.”

In other words, I may be the great Matt Heusser today. However, unless something changes, I’m pretty sure that in three years, I’m going to be the joker who didn’t know what he was doing.

System forces

On one level, software is pretty simple. You set up your inputs, apply a set of transformations, and you get outputs. Usually a human is involved. Perhaps there are many systems and many humans, but the pieces are always the same: input, transformations, and output.

When you stop working on getting this widget to interact with that widget and pick your head up, you see that is what teams are doing. You input work, transform it through some process, and out pops software. Do that for ten or fifteen years while paying attention and you start to understand how the algorithm that creates the software is working.

Jerry Seinfeld once said that he would never exchange old age for youth because with experience, he started to “See what is going on here.”

With enough experience on enough projects, you can predict the outcome, including when a team will fail with a test tool.

Selenium, by itself, isn’t even really a test tool. It is a class library to drive a browser object. It is the open source assertion packages, frameworks, and formatters that are written on top of it that make it a test tool.

Now, allow me to ask the painful question, “How many of those tools, libraries, frameworks, and assertion packages were created by a real student of testing?” By that, I mean someone who spent ten to fifteen years actually studying which bugs humans find, how they find them, what a computer is capable of, and the difference between the two.

Not enough.

Will they be implemented by someone who has spent that kind of life energy devoted to the subject, with a variety of companies?

Most people aren’t willing to make that sort of sacrifice and I don’t blame them. Still, you shouldn’t need to hire a savant to implement test tooling and you don’t have to.

What goes wrong

There are a dozen things that can go wrong when implementing Selenium and a dozen more when trying to scale it to multiple teams. I’ve already mentioned one of the problems—that the test reporting or dashboarding frameworks are not there. And I have implied a bit more in stating that the team lacks the skills or experience to add a new specialty, which is—hint, hint—not really test automation.

No, it is not. Perhaps the first time a traditional Selenium test tool runs, it is testing the software. Of course, by the time everything works and that build passes, it works. That won’t find any bugs. Once the code changes, it ceases to be test automation and becomes change detection.

The one thing we know about programmers is that it is their job to create change.

Thus, the test tool becomes a boat anchor, throwing “errors” that need to be investigated and corrected by someone. Instead of speeding you up, it slows you down, until eventually the entire test effort is spent maintaining code that generates errors much more often than not.

Yuck.

Inputs. Transformations. Output.

Those that have not learned from the past are condemned to repeat it.

A better way

What if you knew about all the mistakes other people have made and could use those mistakes to plan your Selenium implementation? What if you have something like a checklist that talked you through the key issues before deciding?

I had the privilege of participating as a team member in one of the longest ever, long-term successful Selenium implementations at Socialtext. To do it, Socialtext hired the programmer who wrote the Selenium Perl Driver, Luke Closs, to build a framework deeply integrated into the product. Socialtext also hired Ken Pier, Chris McMahon, and myself as part of the team. One year at the Selenium Conference, there was a competition to create a Selenium test to automate sending a tweet with the conference hashtag—Ken won that competition in minutes using WikiQTests.

I left Socialtext in 2011 to take a full-time position at Excelon. Chris went on to become the test lead at Wikipedia. Shortly thereafter, my good friend Michael Larsen stepped in as a test lead. Then, we lost Ken Pier to cancer. The framework stayed under development for ten years and was covered as a case study by O’Reilly. Recently, Michael told me they were eliminating the old test code and replacing it using a strangler pattern. They had to…mostly because the open source tools it was built from have become outdated and unsupported.

Let me tell you about Michael Larsen.

Michael is probably the single most conscientious technical person with whom I have ever worked. A few years ago, he had a deliverable that was for me and broke his leg in a skateboarding accident. As they were hauling Michael into the ambulance, he was frantically telling his wife how to get the files to me. He was on-deadline, and he was going to deliver—no matter what.

I am incredibly proud of the work we did at Socialtext. Sometimes, I wonder if, instead of a success story, we were the exception. You shouldn’t need a half dozen Ken Piers and Michael Larsens to keep a test framework running.

It should be easier than that.

How to get there

After I left Socialtext, I have had the privilege of working on a variety of delivery projects with a focus on testing. Those range from long-term embedded contributor to two-week assessment, from a small programming team with no test role to twenty teams on three continents with four first languages. Along the way, I wrote and reviewed a whole lot of Selenium code.

The folks at Tricentis have asked me to take my experience and to write something to help make things easier. In response, I wrote two papers. The first paper is on understanding and overcoming the challenges of Selenium. The second is on 17 ways to scale Selenium test automation across multiple teams. And yes, I had full creative freedom in the work with my name signed at the bottom. The project is more like a partnership between Excelon and Tricentis. If you don’t want to reinvent the wheel yourself, take a look. The white papers are complementary.

If you’ve been around the block a time or two, then I am really interested in hearing what I missed.

Let’s start talking.