Considerations to advance compliance maturity through a digital validation shift
Kick-start software quality and compliance across the enterprise, and learn the most important elements to include on your roadmap for CSV modernization.
What do we want? Performance Feedback! When do we want it? It depends.
Feedback loops become increasingly critical the faster you deliver software. Effectively implementing automated approaches to feedback loops is all about right-fitting the who/what/when/where aspects. Typically, automated feedback loops live in pipelines and CI. But what about performance testing? Isn’t it too complicated to fit into DevOps and automated contexts?
The short answer is, of course not. But it takes a little engineering thought and setting the right goals first. Why do performance engineers need pipeline-driven performance feedback sooner? A few solid reasons:
Traditional load testing and performance testing has focused on large, complex end-to-end testing of fully baked pre-release systems. Most automated build and packaging cycles take minutes, not hours, so traditional performance test cycles don’t fit well into delivery cycles from a time perspective. Is end-to-end testing still necessary? Yes. Is it necessary for this process to inherit flaws, faults, anti-patterns, and bad assumptions that we could have addressed earlier? Absolutely not.
An intelligent approach requires “right-fitting” load and performance testing into automated pipelines so that we get early signals that are actionable (i.e., feedback loops). From an engineer’s perspective, in order to avoid a mess of end-to-end late-cycle validation, we need to break down the performance verification process into meaningful smaller feedback loops that fit into shorter work cycles. This way, developers and teams can receive performance feedback about their changes earlier and more often so that by the time we test end-to-end, it’s not a dumpster fire.
A proper performance testing tool belt includes a number of practices that inherit these challenges: load testing (i.e., simulating realistic conditions), monitoring, mocking/virtualization, test data management/sanitization, experience sampling, among others. Most enterprises I work with have either partially or fully automated many of these components with the help of tools like Delphix, Mockiato, Tricentis NeoLoad, Grafana, and Selenium. The good news is that there is nothing insurmountable to any of these approaches when automating them; having a solid vision and an approach aligns and organizes our efforts.
It’s easy to say that the goal of test automation is to “go faster” or “accelerate delivery.” That applies when it works reliably and produces actionable outcomes, but getting to “reliable” and “actionable” takes more than load testing tools. It’s like the DevOps “people, process, technology” Venn diagram. If you’re only solving for one of these aspects of the problem, you’re not really solving the whole problem. So how do we solve for the “people” and “process” elements involved in continuous load and performance testing?
Enter automation. Automating our processes puts us up to a challenge to articulate our goals, requirements, and activities in a way that machines can execute on our behalf. It shines light on gaps in our processes, technologies, and skills . . . but that’s a good thing. We need to know our gaps in order to properly address them.
To automate performance tests that produce meaningful, actionable outcomes, people need to communicate about goals and outcomes. Discussing SLAs, SLOs, and SLIs up front is an important part of “left shifting” performance practices. There should be some kind of performance criteria intake process. It could be something as simple as a templatized form or questionnaire about the systems and timeframes that performance testing should target. The goal is to help produce baseline automation artifacts like SLA definitions and API test details.
As you combine these artifacts into a pipeline (i.e., our “process”), you will also have to automate the load infrastructure provisioning/deprovisioning process so that you can run your load tests properly isolated from the Systems Under Test (SUTs). These days, containers and Kubernetes have come a long way to providing a standards-based approach to automating infrastructure, but there are always a million other ways to manage these resources too. Whatever provisioning strategy you employ, it shouldn’t be complicated for teams to run a test. Autonomy to obtain feedback when necessary is a key component to accelerating delivery.
Finally, automation shouldn’t just be in the service of a single team. There’s no force-multiplier to that kind of work in larger organizations when it’s just a local efficiency (for a Performance CoE team). Conversely, automation that many teams can use, even if they aren’t the experts that set it up, is what really helps organizations go faster.
In DevOps, the work of performance and reliability engineering isn’t just running tests or analyzing/consolidating results; it’s providing product teams a way to do these things themselves, while also providing safety guardrails and “good practices” around these processes so that teams can grow their own performance competencies over time.
I see many mature large-scale performance practices that have transitioned from the “performance team as a siloed service” mindset to a “performance processes as self-service” offering. One reason is that there’s just no way to scale “people” performance expertise to hundreds of teams by adding more full-time bodies. But mainly it’s because we can now formalize (i.e., automate) our performance testing practices into work where machines help with the heavy lifting.
In many ways, automating your load testing practices appropriately “brings the pain forward” and puts the forward tension on organizational aspirations to accelerate delivery. Rather than the reflexive approach to leave behind “traditional practices,” we need to re-think them, inherit their useful points of wisdom, and adjust to the constraints and challenges of today.
Running performance tests early and often at smaller volumes is the key to establishing “right-fit” automated performance feedback loops into pipelines. Start small, targeting APIs, to build confidence and competence.
Practical how-to guidance on laying the foundation for a successful transition to an automated Continuous Testing approach — including strategies for prioritizing what to automate, best practices for developing dedicated performance pipelines, overcoming test infrastructure obstacles, and ensuring trustworthy go/no-go decisions — is detailed in this Practical Guide to Continuous Performance Testing.