Continuous Testing at AI Telekom

Agile testing

Continuous Testing at Austria’s Largest Telecom

Automated tests have enabled A1 Telekom Austria AG to deploy new features faster and with less risk, leading to shortened release cycles and more agility. While it is still an ongoing transformation, it has already become a huge success story. Watch this talk as Christian Fischl and Walter Pintover talk about A1 Telekom Austria AG’s journey to continuous testing  – starting with the first automation and onwards towards an automated system of regression tests, and a look at their future plans.

Here’s the full transcript

Hello everybody. Ladies and gentlemen, welcome to the next dream session. How A1 Telecom Austria beats up their delivery with continuous testing. I’m very proud to announce this session now. A1 is the biggest Austrian telecom provider, and the testing project is a real light house project within Austria area. Welcome with you Mr. Christian Fischel. Christian is head of release and change management within A1, and Walter Pintover. He is the Chief solution architect. They will tell about the solution, how it works, what they have done. Give him a warm hand.

Christian F: Thank you very much. Welcome. My name is Christian Fischel. I stepped for a colleague who is leading the quality management team on a very short notice, but as I’m responsible for the change in release management I can say we definitely have to need out of our releases, so I’ll think I’m properly prepared to tell you why we need test automation, why we also want to invest in that. First, I want to give a brief introduction where we come from. Telecom Austria Group. A1 is a member of the Telecom Austria Group. We have companies in several south eastern European countries, and also a bit in Russia. You see in terms of sales it’s still a bit more than half of the group sales are made in Austria, this is definitely the main part.

Telecom Austria Group itself is a member of America Mobile, which is one of the biggest telecom providers in the world. But in terms of branding, we are independent, and we do mainly the business in Europe. Where we come from in Austria in terms of business we are quite long in the business, so we have to deal with a lot of, from the IT side, a lot of legacy system. With it comes a lot of challenges. When we look what we have, how we have worked before, we invested in automation. We saw that we were dealing in the IT with approximately 200 projects. We had to deal with about 60 most important systems where we did testing, we did manual testing only, we had mainly three, sometimes four major releases a year, and we took the time to really manually test until it was good enough we roll it out to production.

There is a lot of effort. You see a lot of numbers here, how many tests were executed. More than 10,000 in a year, how many deployments we had on the integrated test environment, and that means a lot of manual effort. You see there’s a big team behind 31 test managers on average last year, and 57 testers working Austria, in Germany, and also in Egypt to make possible that we get a quality into production that we need for out business.


What we wanted to achieve with test automation is…

First of all we wanted to be able to increase the quality, and to measure the quality. That by itself is not the … Let’s say the end of the story. We want to be faster. We have to be faster in the market, so we want to increase the time to market, we wanted to release more often, and this is also where I’m directly involved is we introduced monthly releases this year before we had three to four major releases here. Now, we want to release 11 times a year except for December, every month a release, and that requires that we do quality assurance in a different way than in the past.


We simply don’t have the time and resources anymore to test everything manually

We have to improve the quality of the integration. What we have to achieve at the end is that all the systems that have to work together in our end to end process is you need to work together in this in a very short time, and one of the key factors is the quality we get into the integration has to be better than in the past. We want to see earlier results. We don’t want to wait until a few days before the launch until we know what we can expect. We need more predictability, and of course even though we still know that we will not have zero defects in production. We want to reduce the number, at least the number of critical defects that will differ our business.

Last but not least we want to reduce the testing costs. What we have achieved so far is we see first results of test automation that we get earlier feedback, errors are more and more identified already in the development cycle, and not in the integration phase, we have introduce a delivery quality gate. For the first time we are really capable of measuring the quality that we get for a few systems already, and this is of course to be increased in the future that we can really say:

“This is a quality that we can [or cannot] accept into our integration test environment”

We automate not only as we will see later on in the development phase where we have automation on the system level, but we also have other approaches to do sanity tests on the integration via meant to have right after the installation to do daily checks of the software quality on a small scale. But end to end level as well to see which systems are up and running where we have problems on the base processes already because then we don’t have to test specific features when the base doesn’t work. As I said before, we release monthly now, and that forces us to be faster also in quality assurance and be better there, and we see we are growing. We didn’t take the big bang approach, we follow an approach, we want to increase it slightly, and we see this is a very good approach because we don’t have the big volume investment at the beginning with the hope of big savings in the end.

We said we want to keep more or less a stable financial investment there, and to do whatever is possible, and increase automation therefore reduce either costs, or increase the number of test cases we can cover by that. That requires that we standardize also the way we deploy to system. The first step is on virtual systems as we will hear, but this is also the approach to go further on the integration environment to have automated deployments, and also in the future in the production. The first step here is done out of the automation in the development that we standardize the way we deploy systems.

Very important also to make sure that we have the long breath that is necessary to take this. We need transparency right from the start. Transparency is important for everybody involved. It’s important to say “We can accept this software deliverable, to test the nulls, what is working, what is not working before doing tests.” We need from release management an overview how good the quality is, what we can predict, what will be ready for production soon, and also management needs that. A lot players involved, and all of that get transparency from day one, and with an increasing scope. It’s very important that you still get financial support within the company

One important thing is when you have a test environment of about 100 systems, you can imagine that there’s always some part that’s not working. If you have the approach you want to test end to end, and also show to business users how the systems are working.

You need a working end to end

If there is one block between by virtualization you can overcome this, so this will also be a future use case tool to make all the time testing possible.

The key thing is just to summarize. When you start this, you really have to invest in people. You need the people with the right know-how in terms of how to automate this.

You do not need the best testers, you do not need the best developers, you need the people who know how to bring all that together, and make an automation.

The costs do decrease, but not right from the start. You have to be patient. It will come. The results will come, but they need some time, and very important you always have to stick to it. You have to keep it up and running. If you do not maintain the automation all the time, you will be out of the business there very soon.

We figured out the stepwise approaches better than the big bang as I said before. Making step by step one application after the other instead of having the big target you maybe never reach, and stick to it. Stick to it, the results will come. They need time, but they will come, and then you can really gain from it. One part of the automation, that’s why I take on a percent how this is working … Thank you very much.

Walter Pintover: Hello everyone, and a warm welcome from my side as well. I’d like to give you a quick introduction into testing 360, maybe also a bit of a deep dive. Testing 360. Maybe the most important or at least biggest testing project in A1, it’s a project implemented by V-Pro, and I as a test architect, and consultant supported A1 and V-Pro in getting the approach right, in doing the initial set up right, so that’s why I’m standing here today, and let’s see the vision of A1 behind testing 360. It’s not a big surprise, right?

You want to increase quality, reduce costs, speed delivery, and use automated tests for it.

You will know that this is possible with automated tests, that’s why we usually automate, but it also depends a bit on the IT landscape, and the implications in it. We took a look at it, and here is what we found. This is a usual enterprise IT landscape I’d say. You have everything there from mainframe applications, all legacy systems up to the newest mobile apps, and you need to test everything, and everything is interconnected. There are a lot of lines, a lot of connected boxes there, and when I thought about it to automate tests there here is the thought that crossed my mind. You don’t simply automate tests in such a landscape, right? It’s not that easy actually.

What we did, we picked our first application somewhere in the middle, and we analyzed it on the integration environment. An environment where all the applications are actually running, which is usually for acceptance tests, and manual tests, and we analyzed it, and try to automate the first test cases there, and what we saw is that if we want to automate the one in the middle, we have dependencies with all the blue applications. Either direct or indirect dependencies, and all those applications they need not only to work, and run, you need to have the right test data in the database. They need to have the right version so that our testing would actually work.

This is not the right approach. It will interfere with manual testing, it won’t really become stable at all, so we did some thinking again, and here is the thing. We need to create a separate testing 360 environment first. We took our application on a new server, we started to set it up, and we came across some dependencies. For our application for the one we wanted to test we need to actually set up five more applications, and some services, and some databases. Only to set up an environment for one application we have now a couple of systems there, and it will take weeks, if not months to set this actually up. Again, this doesn’t seem to be the right approach.

We did some thinking, and here is what we came up to. It’s not a big surprise.

Whenever you have too many dependencies and when it gets too complicated – you need to set virtualizing services, and the very important first step you have to think about are the test cases.

You cannot just simply create a new environment, and the service virtualization, and just start testing. The first step is always think about the test cases. What do you want to test? That’s influencing the requirements you have for your dependencies to the other applications, so the virtualization, you need the test data, everything.

Once we have the first set of test cases for application we started to execute this on integration environment. The environment was ready, so we could manually execute it. We used Tosca Recorder to record our actions, and even more important we used Tosca Orchestrated Service Virtualization to record every request, and every response in the system. The blinking lines you can see those other direct dependencies between our application, and other applications, and services, and for each request we recorded also the response, and then created a basic set of test automation service virtualization out of it. With the service virtualization, we were able to replace all the other systems by simple virtualized systems.

In the end, we had no more applications around it. We only had the Tosca server, and it knows how to react as the application we just virtualized. Giving … We sent a request to the application right up corner, then the Tosca server will know how to respond to it. With this approach, we were able to create our first environment. We were able to create some test automation, and we basically finished our first application, and we did pretty much the same for the next, and the next, and the next. It sounds pretty easy, but it isn’t because each of those applications come with different challenges. Each application is different, each interfaces are different. It’s basically the same approach, but you will have a lot of challenges, and troubles, but you can solve them one at a time instead of having to solve all of them because of the dependencies.

Our approach now was step one: create virtual services and automations. Step two: Based on the separate T460 environment, and step 3, and this is probably the most important one, but also one of the easiest, we need to integrate our tests. They need to be part of the continuous integration, we need to run tests fully automated, we need to add them to CI as soon as possible, and we need to run tests often. Every often. In the case of testing 360 every day, at least once in order to check if the application is still working, and if the tests are still good. If you don’t do it, people will start to ignore your automation. This is one of the most important things.

Fast forward to today, we are now 15 months into this project. Around 30 core team members, most of them from V-Pro worked on this project, and more than 100 people of A1 are actually are very involved in this project in various roles of operation, business specialists, testers, developers, all of them were involved, and we managed to create, or to finish 20 applications in 15 months. What does it mean to finish it? We created a lot of test automation, a lot of virtualization. We created over 100 new continuous integration jobs, which do not only execute the testing, they execute to build, to test, and deployment for all those 20 for the 20 applications, and we created the dashboard. A dashboard with the most important KPIs so that the managers and leaders of A1 know about the quality of their applications.

At anytime, they can take a look at the dashboard, and know is the quality increasing? Do we have a good risk coverage? What about the technical depth? That’s all you can see now in the dashboard. Something that was missing before the project. Let’s go back to the actual vision, and see whether we were able to need it or not. The increased quality part, not only do we find parts now earlier, we have now a way to continuously check the quality, and to make decisions based on the quality, on the state of our application. Should we really go live with this version? Or is it still buggy? Should we take some more developers to fix the technical depth? Stuff like that. It wasn’t previously really possible.

Regarding reduced costs, Christian already mentioned it. It is a goal to reduce costs, but it doesn’t come in the beginning because first of all you need to create all this stuff, second of all you need to operate, and maintain it. Every test case you create, every virtualization, every environment you need to operate, and maintain. This comes with cost. But on the other hand, you have much more bang for the buck. You can replace a lot of manual testing effort with automation, and you will get the money back later once you can use your power to create more, and more automation instead of simply manually testing everything again.

Perhaps the most important success of our project is that we now have faster development cycles.

Initially when we started it was three months, now it’s one more for most of the applications, so three times faster, and this is very important for a company like A1 that wants to be the innovation leader, and we’re not done yet. There are still 20 applications are yet to come for this project, and I think there’s much more later on when we finished with testing 360, and yeah. That’s basically it. If you’re now thinking that, then come visit me at the Qualisoft Booth, and let’s get into some further discussions because I think the explanation of service virtualization was pretty high level because there was no time to really go into detail, but we could do this later in some talks. Thanks a lot.