Agile testing

You cannot be serious! How agile can go wrong

This is a true story about a tester called Frank. Frank works for a large enterprise that claims to work in a truly agile way. Frank revealed some shocking secrets about his company that show how “Agile” can go wrong. Note that Frank is not his real name. I just call him “Frank”, because Frank was really frank with me.

First, let’s understand some aspects of the organizational structure of Frank’s company. About three years ago (Feb 2015) Frank’s company started their “Agile Transformation”. They decided to manage their software delivery process with the Spotify Engineering Model.

  • On the atomic level of this organizational structure you find squads. A squad is essentially a Scrum team. On average, each squad is composed of about 18 people (14 developers, 1 product owner, 1 documentation expert, 1 support engineer, 1 tester). In Frank’s company there is no dedicated agile coach per squad. One agile coach coaches at least 2, but not more than 3 squads. Each squad has end-to-end responsibility. This means that a squad is designed to act like a mini-startup. They are supposed to have all the skills and tools needed to design, develop, test, and release their software to production. The squad’s product owner is regarded as the squad leader.

Now then, that’s already a truly “lightweight” structure (19 people per squad!) on the atomic level, but that’s another story. Let’s just look at one aspect. On average, one tester tests the outcome produced by 14 developers. I already have great respect for these testers. I wouldn’t be able to handle that huge workload.

Please note that the testers are not considered as supporting elements that help their squad members to deliver better software by doing better testing. By “better testing”, I mean testing that is done deeply and reliably. So, testing and development are completely separated in each squad. The sole responsibility for software quality is imposed upon the testers, not upon the squad. Well, I think this narrow-minded view deserves a “You Cannot Be Serious!”. Please watch this to literally see what I feel when I say “You Cannot Be Serious!”.

It seems that the people in Frank’s company have realized that they have the same finish time, but they haven’t realized yet that they also have the same finish line (i.e. high software quality at high speed of delivery). According to Frank, they still value “speed” over “quality” in a way that it is not healthy for their software products.

  • Interestingly, Frank’s company decided just this year (March 2018) to have dedicated testers in each squad. The main reason for this was that they faced severe quality issues in the past. Well, what a surprise! In simple terms, their “makers” (developers) turned out to be worse “breakers” (testers). No worries, I know that we don’t “break” software through testing, the software is already broken.

Anyways, the bottom line is that in the past testing and development were two completely separated entities with a big “wall of confusion” in between them. Well, obviously this didn’t change at all. This is perfect example of a “right” decision that goes in the *wrong* direction. Honestly, I don’t know why this company has even tagged their transformation initiative as “agile” for three years already. Frank revealed the secret: Developers are regarded as “value centers”, while testers are still regarded as “cost centers”. For that reason, Frank’s company started first restructuring their shiny diamonds (developers) without paying much attention to the testers. Now, trust me: Frank’s company is not the only company on this planet that sees testing like that. That’s the hard truth about software testing. So, here is another “You Cannot Be Serious!”.

  • So, what’s the main reason behind this mentality? Well, I think it’s not just one reason, it’s a vast variety of reasons that effect the way testing is perceived. Frank claims that the main reason is that most “testers” in his company simply don’t understand the act of testing. Most “testers” don’t know, and so aren’t able to communicate, the goal, purpose, and value of testing in a way that is understandable by others. Most “testers” simply reduce testing to coding (e.g. regression test automation). Therefore, here is another “You Cannot Be Serious!”. That one now goes straight to the testers. Testers, you don’t just need to know something about testing, you need to understand the game (testing) you are playing deeply. Otherwise, your profession will never be understood or valued by others.
  • Next, the squads in Frank’s company are grouped into tribes. A tribe is a collection of squads within the same business area (e.g. mobile apps). Each tribe has a tribe leader. The tribe leader is the owner of the corresponding business area. In Frank’s company, the tribe leaders are business people (e.g. business analysts). On average there are 6 squads in each tribe. In total there are 18 tribes in Frank’s company. So, there are roughly 1512 developers, and about 108 testers. These are impressive numbers. Therefore, Frank’s company can be considered as a “large enterprise” (at least) from a development perspective.
  • Then, there are chapters. A chapter is a group of people within a single tribe that work within a special area (e.g. testing, front-end development, back-end development). Each chapter has a chapter leader. So, there is also a chapter leader for software testing for tribes with at least 6 squads. This person is called the “chapter testing leader”. A chapter testing leader is Frank’s line manager. In total there are 14 chapter testing leaders in Frank’s company. You might now think that the testers in the remaining 4 tribes are allowed to “organize” themselves. Far from it! These testers are “managed” by other chapter testing leaders. Frank says that chapter testing leaders are supposed to provide what Frank calls “test discipline expertise” to all testers of that chapter for the specific business area. Unfortunately, this doesn’t happen. These chapter testing leaders just manage testers, they do not lead testers by providing the required “test discipline expertise”. According to Frank, the chapter testing leaders are more like project managers with little or no knowledge about software testing. This deserves another “You Cannot Be Serious!”.

Don’t get me wrong. I absolutely see the need for project managers because they do an incredible amount of valuable administrative work. We need them. But then, we shouldn’t call them testing leads. This just defeats the purpose of the role.

  • Next, there are guilds. A guild can be understood as the glue that keeps the company together. A guild usually is a wide-reaching “community of interest”, a group of people that want knowledge, tools, code, and practices [Henrik Kniberg]. So, chapters are always local to a tribe, while a guild cuts across all tribes. In Frank’s company, the situation is “slightly” different. There is one guild for software testing. This guild has a “guild testing leader” who (again) just manages (not leads) the 14 chapter testing leaders. So in Frank’s case, the testing guild is just another layer to manage software testing in his company. It’s easy to imagine what happens in the other chapters and guilds.Now then, all this is just another clear sign that control is valued over trust, that structure is valued over community, that processes are valued over communication. The Spotify Engineering Model has the goal to minimize standardization. The intent of tribes, chapters, and guilds is to make people loosely coupled but tightly aligned. The keyword here is “self-organized teams” – that is what should be fostered by the “leaders”. Well, Frank says that this doesn’t happen in his company. They simply misuse this engineering approach. The reason seems to be obvious. It seems that the people in Frank’s company do not understand the real meaning of the term “Agile”. Ergo, “You Cannot Be Serious!”.

Now, here is what really drives me crazy. Here is the real reason for the latter two “You Cannot Be Serious!”. It is something totally absurd, but unfortunately it is true. Now, this is what happens when Frank (our tester) finds a bug in the software.

  • Frank doesn’t communicate his findings to the squad’s developers or product owner. No, that would be way too simple. Frank has to assign the bug reports to his chapter testing leader. The chapter testing leader then usually analyzes all the bug reports from all the testers in that chapter on the next day. So, in the morning of the next day, the chapter testing leader then decides which of these bugs are “relevant” and which are “not relevant” to share with the members of the squads. Frank couldn’t explain to me what “relevant” actually means. The consequence of this filtering process is that all “not relevant” bug reports are never seen by anybody else (e.g. developer, product owner) in the squad. Boom! That’s already beyond all reason. I could write a whole other article on why this is such a bad idea.

Anyways, here is the reason for this weird process: Frank told me that the developers (company’s diamonds) complained about being distracted too much by the bug reports raised by the testers. The developers argued that this “distraction” prevents them from focusing on product development. Well, I would love to understand what these developers mean by “product development”, but that’s a whole other story. This just shows that these developers don’t understand their work, and they don’t understand the importance of testing.

The squad leaders (product owners) don’t even care about bug reports in Frank’s company, since their development “principles” state that bugs are a type of “technical debt” that needs to be handled by development. Well, that’s what I call “teamwork”.

  • Now then, let’s proceed with the process. At this point, the chapter testing leader has analyzed the bug reports. It is worth noting that this analysis is sometimes carried out in collaboration with the guild testing leader, especially when bugs cut across two or more tribes. So, it might happen that the guild testing leader also puts in his two cents to make sure that only the “really relevant” bugs reports are shared.
  • The chapter testing leader then assigns these “really relevant” bugs reports to the corresponding developers. According to Frank, this process takes about two days in the best case. As bad luck would have it, the developers miss important information in the bug reports, and then reassign the bug reports to the corresponding tester. After the tester managed to recreate the entire test environment to reproduce the bug, the tester then provides the missing information to the developers. This might take another day. Then, the tester is “allowed” to talk to the developer about the bugs. I hope you agree that this process is simply ridiculous. So, here it is: “You Cannot Be Serious!”.

Having said all this, I’ll spare you a summary of this ludicrous process and cut straight to my concluding remarks. Our business is about exchanges values with our customers. We provide value to our customers by solving their problems in our software. Our customers provide value in return to us by paying for what we have built. As such, the goal is to accelerate this value exchange by closing the feedback loop between us and our customers as fast as possible.

When we aren’t able to close our internal feedback loop fast, then we aren’t able to serve our customers fast enough. So, please note: If your customers perceive bad performance (low delivery speed at low software quality) from your company, their next click will be most likely on