increase in automated tests over 6 months
reduction in test creation time
Outbrain is a leading global native advertising platform. You may not have heard of Outbrain, but you’ve undoubtedly used their service. Outbrain helps over one billion people discover content, products, and services that may be of interest to them. They specialize in native advertising — serving written content and videos that are incorporated into the platform on which it is displayed.
If you’ve visited news sites like CNN, BBC, or LeMonde and read an article, you’ve likely seen related articles or advertisements that were curated by Outbrain. Outbrain helps connect publishers and advertisers with people who wish to engage with their content.
We spoke with Tsachi Shushan, Head of Frontend Guild at Outbrain about their use of Tricentis Testim.
Outbrain’s software is critical to its success, connecting content publishers with content consumers. The Outbrain native content engine provides users with interesting and relevant content on premium publishers’ sites. Testing is a challenge because the application manifests as feeds on over 100,000 websites that it doesn’t own. Yet, Outbrain’s success depends on readers clicking on the content displayed in these feeds and getting the proper outcome — taking readers to the right place and recording the transaction.
Outbrain is a modern development company in a microservices environment with a continuous delivery pipeline all the way through production. It releases code to production several times a day and relies heavily on tests. “We use unit tests, integration tests, and full in-app workflow integration tests. The relevant tests run on every change to the related code,” said Tsachi, “and the web application testing was always the most painful part.” Outbrain was using Selenium and Protractor, but not every team could keep up.
“Many teams wanted to do in-app workflow tests, but not every team had an automation engineer who could support them.” — Tsachi Shushan, Head of Frontend Guild, Outbrain
Even the teams with automation engineers (AE) were having difficulties. Tests took a long time to author, they failed often, and took a lot of time to debug. Even the runs were configured differently for each team. The tests took a long time to run and were unstable. Tsachi estimated that they were spending 10-15% of their web engineering hours on automation. For an organization of over 50 engineers working on web applications, that is equivalent to five to seven full-time engineers. They were getting feedback from the team that they were spending too much time on automation tasks. They needed a better solution.
Outbrain knew that as its application grew it would want to spend more time on regression tests and the current model with Selenium and Protractor couldn’t effectively scale.
Outbrain did a traditional POC evaluation process that lasted around a month, comparing many solutions including TestCafe, TestCraft, Cypress, Endtest, Ghost Inspector, and more. Tricentis Testim came out on top as the best fit.
Tsachi said that their criteria included ease of use, the ability to quickly author tests, the completeness of functions, and the responsiveness of the support team. The product was easy to use and also worked well for their application. Tsachi emphasized,
“The recorder was at least one level above every other solution we tried. It worked in every scenario including drag and drop, iframes, mouse gestures, and more.” — Tsachi Shushan, Head of Frontend Guild, Outbrain
He was also impressed by the stability provided by the Smart Locators. “They made the tests very robust. They didn’t fail on minor code changes. It’s pretty unique among the products we tested,” he added.
Finally, they also looked at the support experience. Tsachi said, “The Testim support team was very responsive and helpful.”
Outbrain chose Tricentis Testim for in-app workflow functional UI testing for its applications. In total there are about 20 applications that it tests with Testim including the feeds embedded in over 100,000 websites across the world. Its applications include its content advertising platform that allows customers to create campaigns as well as the publisher application that helps publishers manage content and settings for the Outbrain feed.
Outbrain has incorporated Testim into its CI pipelines and processes. This includes running functional tests on new feature commits and regression tests on larger releases, all triggered from a Jenkins CI build. Outbrain runs its Testim created tests on its own Selenium grid.
One of the goals of the test automation project was to get as many teams as possible involved in end-to-end testing. Before Testim, there were only six teams doing in-app workflow testing with Selenium and Protractor and now there are 14 teams involved, largely due to Testim’s ease of use.
Developers are now able to switch context more easily which makes writing tests more acceptable. They didn’t want to write coded tests, and Testim allows them to record and configure tests. “Before Testim, new tests in a large application took one to two days to write and make stable. The same tests in Testim take between 20 and 30 minutes,” said Tsachi.
Tsachi talked about the overall change, “We’ve received lots of good feedback about the process and writing and fixing tests. The developers like that they don’t have to write code, commit code, wait for code review, etc.”
“Developers can just record tests, it makes the context switching for developers much easier.” — Tsachi Shushan, Head of Frontend Guild, Outbrain