Tricentis, as the continuous testing company, takes a keen interest in software bugs. Most of the software bugs a company experiences never become public knowledge, but those that do can have a severe impact. In an effort to track this phenomenon, we decided to catalogue the software bugs and failures that were big enough to gain media attention in 2015.
To do this we set news alerts for the phrases “software glitch“, “software bug“, and several common variants. All of the alerts were first sorted for relevance and uniqueness, then placed into one of six broad industry categories. Any statistics published in the news article were noted alongside each story, as well as external factors, such as whether the bug originiated in a mobile app or posed the threat of physical injury. Stories that were covered by multiple sources and stayed in the news cycle for an extended period of time (days or weeks) were marked as having a high level of brand damage.
To the uninitiated, realizing the ubiquity of the software bug is an eye opening experience. The process of collecting nearly a year‘s worth of software bugs revealed not only the sheer volume of bugs we encounter on a day-to-day basis, but also strong patterns in where and how they occur.
‘Software Fail Watch: 2015 in Review’ makes a strong case for how vital software testing is, and continues to be, despite those who proclaim the “death of testing“. As technology advances and enterprises race to develop the newest groundbreaking app, the demand for fast, efficient, and comprehensive testing only grows.
Comprehensive testing within short product sprints is not always possible however, making risk based testing the best way forward. Ultimately, testing harder isn’t the answer here – testing smarter is.
Ultimately this report serves as a crucial reminder of how easily a software bug can destroy enterprise value – making the role of testing as much about brand protection as it is about quality assurance.
Founder and CPO of Tricentis
Software fails per industry
The software fail stories were sorted into six broad industry categories as displayed on the left. Of the categories, software bugs found in government were the most common, but also generated the least media coverage as a whole. Dare we say it seems that we expect bugs in government systems? Stories within the transportation and retail categories regularly received more attention and general consumer ire, though transportation earned the added distinction of posing the greatest possibility of physical injury as a direct result of their bugs. The finance sector appeared uniquely skilled at burying their software bugs; their stories tending to be particularly vague on the facts. Several of the finance related stories that did come to light however, blew up to be among the biggest stories of the year.
The latter half of the year saw a significant up-tick in software bug frequency, which appear to match the travel and consumer purchasing trends of the year
Bugs in government
Is Agility as critical for Systems of Record as for Systems of Engagement?
Agility in both development and testing is just as critical for Systems of Record as it is for Systems of Engagement. Agility on enterprise level however, doesn’t necessarily imply continuous delivery – agility can be delivered at different speeds. Rather, it’s about applying the core essentials of agility (creating ownership, improving collaboration, enhancing transparency, increasing flexibility, amplifying feedback) to all software development regardless of the system’s character. Being a part of the “Digital Disruption”, businesses must leverage digital technologies to create new sources of customer value and increase operational agility. Agility in development, testing, and operations, therefore, is necessary to meet these needs. – Ingo Philipp, Product Manager at Tricentis.
Top 3 Government Bugs of 2015
Bugs in transportation
Top 3 Travel Bugs of 2015
- Fiat Chrysler scrambles to increase security after hackers hijack a Jeep Cherokee via bluetooth
- Newly discovered Boeing 787 software bug capable of shutting down plane generator while in flight
- 360,000 passengers left stranded for over 4 hours after software glitch paralyzes Tokyo-Yokohama Tokyu trains
Why is risk-based testing important?
As J. Bach has said, “Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous”. This implies that no matter how much you test, there is always an element of risk left over. You will simply never have enough time, budget, or resources to test everything as exhaustively as possible. Testing, therefore, must be selective.
As a result, releasing a product is always directly related to the level of risk you are willing to run. Risk-based testing supports you in making that decision. It allows you to plan thorough testing in high risk areas, ordinary testing in medium risk ones, and light testing in low risk ones. It allows you to remove as much of the risk out of the product as early as possible, making testing one of the best investments you can make. – Ingo Philipp, Product Manager at Tricentis.
Bugs in retail
The Growing Influence of Mobile Apps
With the increasing demand for instantaneous and easily accessible information, many analysts predict that tablets will surpass desktops and laptops within the next few years.
With this in mind, the “Mobile First” strategy has become more and more popular as it focuses on first designing super lean applications for constrained mobile environments before developing that app out for use on desktop or laptop computers.
Mobile users however are very demanding in terms of quality – your app, and therefore your brand, will not recover from bad reviews in the app store. You have only one chance to get it right. Testing early in the development cycle therefore, is critical for releasing the great app you always wanted in the fastest possible time. – Georg Thurner, Product Manager at Tricentis.