minutes average test creation time
Microsoft Defender Advanced Threat Protection (ATP) is a platform designed to help enterprise networks prevent, detect, investigate, and respond to advanced threats. Thousands of companies and organizations depend on Microsoft Defender ATP to keep networks and critical business systems safe from cybersecurity threats.
Microsoft Defender ATP’s cloud-based portal is actively used by security operations personnel on a daily basis. Defects can have immediate visibility and impact customer satisfaction, or potentially compromise security. Ensuring that it’s working as intended and that updates don’t break existing functionality is a critical concern.
We talked with Ran Mizrachi, Software Engineering Manager, in Microsoft’s Windows Cyber Defense group.
The Defender ATP product is only a few years old. When the project started, Microsoft had about 25 software engineers on the project working at breakneck speed. They were checking in code daily, but there was very little deployment automation. Testing was manual — primarily sanity checks. The team was on an unsustainable path.
“We didn’t have the manpower to add coverage to any existing code. We allowed a culture to develop that wasn’t sustainable. We needed to make testing part of our core processes, and that required more than a tool change.” — Ran Mizrachi, Software Engineering Manager, Microsoft
The team wanted to implement functional end-to-end test automation but didn’t want to spend a lot of time building test coverage for existing functionality. At the same time, they recognized the need to test for regression of existing code as they released new content.
The team considered Selenium but felt that it would be a steep learning curve to ramp up skills. They also considered Applitools and liked the visual validation aspect, but it lacked the functional validation capabilities needed.
Microsoft Cyber Defense Group chose Tricentis Testim for automated functional and in-app workflow tests.
The onboarding process was straightforward and fast, according to Ran. They conducted a workshop with about 35 of the team’s developers with some support from Testim. Each team prepared a list of scenarios that they wanted to validate in the workshop. The first part of the onboarding process focused on learning about Testim — its features, how to use Testim, and some best practices to help structure tests. The second part of the workshop was applying this knowledge to create tests and build coverage.
“After the six-hour workshop, the teams had created 120 tests including 80 in-app workflow tests that are still running today.” — Ran Mizrachi, Software Engineering Manager, Microsoft
The Microsoft Defender ATP team has since integrated Testim into its development pipelines on Azure, running various tests at specific process gates such as pull requests, pre-deployment, releases, post-deployment, and nightly. Developers are responsible for creating Testim tests for their code. “Every new piece of content that gets released should have proper test coverage in Testim,” said Ran.
The new (nearly) continuous testing strategy is paying off. According to Ran, it is helping to identify a lot of bugs. As they build up test coverage for the existing codebase, if something stops working, the teams add tests so that going forward, they can test that area for regression.
One of the nice features of Testim is the ability to troubleshoot failed tests. Ran emphasized, “This is really important because we have many developers working on the same portal. Being able to quickly determine why a test failed, even when someone else wrote it, saves a lot of time.”
Automated testing helps Microsoft get closer to its goal of continuous delivery.
“Testim is helping to make the CI/CD dream possible — you can’t get to continuous delivery without proper test coverage.” — Ran Mizrachi, Software Engineering Manager, Microsoft