How Outbrain reduced test authoring time by 95% with custom web app testing

Outbrain’s software is critical to its success, connecting content publishers with content consumers. The Outbrain native content engine provides users with interesting and relevant content on premium publishers’ sites. Testing is a challenge because the application manifests as feeds on over 100,000 websites that it doesn’t own. Yet, Outbrain’s success depends on readers clicking on the content displayed in these feeds and getting the proper outcome — taking readers to the right place and recording the transaction.

Outbrain is a modern development company in a microservices environment with a continuous delivery pipeline all the way through production. It releases code to production several times a day and relies heavily on tests. “We use unit tests, integration tests, and full in-app workflow integration tests. The relevant tests run on every change to the related code,” said Tsachi, “and the web application testing was always the most painful part.” Outbrain was using Selenium and Protractor, but not every team could keep up.

“Many teams wanted to do in-app workflow tests, but not every team had an automation engineer who could support them.” — Tsachi Shushan, Head of Frontend Guild, Outbrain

Even the teams with automation engineers (AE) were having difficulties. Tests took a long time to author, they failed often, and took a lot of time to debug. Even the runs were configured differently for each team. The tests took a long time to run and were unstable. Tsachi estimated that they were spending 10-15% of their web engineering hours on automation. For an organization of over 50 engineers working on web applications, that is equivalent to five to seven full-time engineers. They were getting feedback from the team that they were spending too much time on automation tasks. They needed a better solution.

Outbrain knew that as its application grew it would want to spend more time on regression tests and the current model with Selenium and Protractor couldn’t effectively scale.

Outbrain chose Tricentis Testim for in-app workflow functional UI testing for its applications. In total there are about 20 applications that it tests with Testim including the feeds embedded in over 100,000 websites across the world. Its applications include its content advertising platform that allows customers to create campaigns as well as the publisher application that helps publishers manage content and settings for the Outbrain feed.

Outbrain has incorporated Testim into its CI pipelines and processes. This includes running functional tests on new feature commits and regression tests on larger releases, all triggered from a Jenkins CI build. Outbrain runs its Testim created tests on its own Selenium grid.

One of the goals of the test automation project was to get as many teams as possible involved in end-to-end testing. Before Testim, there were only six teams doing in-app workflow testing with Selenium and Protractor and now there are 14 teams involved, largely due to Testim’s ease of use.

Developers are now able to switch context more easily which makes writing tests more acceptable. They didn’t want to write coded tests, and Testim allows them to record and configure tests. “Before Testim, new tests in a large application took one to two days to write and make stable. The same tests in Testim take between 20 and 30 minutes,” said Tsachi.

Tsachi talked about the overall change, “We’ve received lots of good feedback about the process and writing and fixing tests. The developers like that they don’t have to write code, commit code, wait for code review, etc.”

“Developers can just record tests, it makes the context switching for developers much easier.” — Tsachi Shushan, Head of Frontend Guild, Outbrain