Software failures

Software Fail Watch 2016, Quarter Three

UPDATE: For the complete wrap up of 2016’s software fails, read Software Fail Report: 2016 in Review.

This past August, the world celebrated the 25th anniversary of the World Wide Web. It is mind boggling to think how far we have come since that first, simple, hyperlinked text page launched from Switzerland’s CERN by Sir Tim Berners-Lee.

At one time, software was just a way of getting things done. It was nothing more than the convenience of using a calculator over doing your sums by hand. Berners-Lee’s original website was simply a tool for making information more easily accessible.

You don’t need me to tell you that software has made tremendous leaps in the last 25 years. We have gone from 1.2 MB floppy disks to 1 Terabyte flash drives (the storage equivalent of about 711,000 floppy disks). We have gone from cell phones the size and weight of a brick, to smart phones with as much processing power as a desktop computer.

More has evolved than just hardware and software however. The change in our perspective on tech is arguably one of the most extreme societal transformations of the past 25 years. On a personal level, we think about things today that would have never crossed our minds 25 years ago: How do I market myself authentically over social media? How do I maintain a clean digital footprint? How do I teach my children not to look at a stranger’s smart phone screen?

On a corporate level, we face similarly new questions: How do we create an engaging brand on social media? How do we protect our digital assets from hacking? How do we make our software fail-proof?

Ultimately it doesn’t matter if you are a restaurant, a legal firm, a plastic manufacturer, or an investment bank: your software is your brand. As such, every unexpected error message, forced restart, or failed update is a ding against your brand’s shiny reputation. We’ve said it before and we’ll say it again: software may come and go, but software testing is here to stay.

Statistically there is a very good chance that you have experienced a software failure this year – probably even in the last quarter, or week. So far we have collected over 380 individual software fail stories, 114 of which are from Quarter 3 alone.

As per usual (particularly since the United States is in an election year), the first place for most-software-bugs in Quarter 3 goes to the Government sector, with 41 stories. Transportation comes in second with 20 incidents, another not-so-surprising figure given how travel related bugs always seem to emerge just in time for vacation.

The surprise this quarter has been the uptick in finance related software fails. In our experience, software fails in the finance industry are hard to come by. It is not that the industry does not have software fails – rather, they simply seem to be reported less.

As opposed to some other industries, much of a financial institution’s software lies behind the scenes. This makes it easier for bugs to be quickly patched before their repercussions can snowball into a major news story. The finance stories that do come to light either take place in public-facing venues (such as a stock market), or feature bugs so catastrophic that the story cannot be buried.

Here are some of the highlights (finance and otherwise) from Quarter 3:

“Details Emerge Of $400 Million Error In SolarCity Corp (NASDAQ:SCTY) Valuation”

SolarCity Corp retained an investment bank to assist in the sale of the company to Tesla Motors Inc. After the $2.6 billion dollar agreement had been signed however, the investment bank, Lazard Ltd., discovered that they had under-valued SolarCity Corp. by roughly $400 million. Whoops. Unfortunately the error was discovered too late for SolarCity’s shareholders, but Tesla did offer to make up some of the difference in stock.  Source. 

“More than 1500 BNZ Customer’s Confidential Information Compromised in Data Glitch”

In late August, a woman reported to the local news that she had received an email from the of Bank of New Zealand disclosing a stranger’s bank account names, balances, available credit, and more. After first being brushed off by a bank representative, Bank of New Zealand followed up with the woman, letting her know that she wasn’t the only one to have experienced this: they had already received over 1000 calls. In the end, Bank of New Zealand confirmed that over 1500 customers had experience the glitch, though it’s unclear how many of their customers should be wondering if their information was leaked. Source.

“Concern Donors Hit by Multiple Debit Error”

The UK based charity, Concern Worldwide, put a number of their donor’s bank accounts in jeopardy when a software glitch caused their accounts to be debited up to 100 times. Thousands of supporters who had committed to a fixed monthly donation (say $5), found the money automatically withdrawn from their accounts up to a hundred times in one month. Investigations into the issue revealed that “sufficient testing had not been carried out” (surprise, surprise), and that “staff at the charity had not correctly followed internal control procedures”. Thankfully refunds have been issued, though it was a hard-won lesson on QA for the organization. Source.

“Kansas Family Sues Mapping Company for Years of ‘Digital Hell'”

A family in Kansas has filed a lawsuit against MaxMind for mapping over 600 million IP addresses onto their rural farmhouse. While MaxMind specializes in mapping IP addresses with specific geographic locations, IP addresses that could not be placed were given a “default location” due to a software error. The result? Police knocking on their door day and night, looking for missing persons, hackers, identity thieves, or stolen vehicles, as well as receiving allegations of making child abuse films, kidnapping, or suicide threats. It wasn’t until the local police investigated why this particular family seemed to be involved in so much wrong-doing, that they discovered the tie to the IP addresses. Given that this has been going on for 5 years, it is surprising that the family isn’t asking for more in their lawsuit. Source.

“GM Recalls 4 Million Cars over Airbag Defect”

General Motors has recalled 4.3 million cars after discovering an airbag defect that has already resulted in one death and three known injuries. The software error effects the airbag’s sensors, preventing it from deploying during a crash. Unfortunately this story is just another in a long line of airbag defects, mostly supplied by Takata, a large global leader in vehicle safety equipment. The problems with Takata airbags have been known for a while, and have only escalated as their malfunctions have directly resulted in several deaths over the years. Source.

“Software Glitch Delays Elections”

Local election results in Lake County, Florida, were delayed by several hours in August after the vote tabulation software failed. Rather than automatically transmitting the vote totals, election volunteers had to drive to each polling place, collect the vote counts on a thumb drive from each machine, and bring them back to the election headquarters to be counted. Two hours after the polls closed, the votes had just begun to trickle in. Though the incident occurred on a small scale, the glitch brings into sharp relief the massive problems that could occur during the upcoming presidential elections on November 8th. Source. 

“Apple Admits iPhone 7 Has A Problem”

Apple has already received their fair share of criticism over ditching the headphone jack in their new iPhone 7. It does not, unfortunately, help their case that major glitches have already been discovered with their new lightning jack-based “EarPods”. Users have complained of their EarPods crashing, leaving the music playing while the controls and corresponding shortcuts freeze. Apple has already confirmed the bug and announced that a fix is in the works. Source.

More on Software Failures

Software Fail Watch 2016, Quarter Two
Software Fail Watch 2016, Quarter One
Software Fail Watch: 2015 in Review
Software Failures of 2015: Quarter Three
Software Failures of 2015: Quarter Two
Software Failures of 2015: Quarter One