This week, Verizon experienced a massive nationwide outage that lasted for about 10 hours. What can we learn from this?
Quantifying Impacts of Quality
It can be difficult to fully quantify the impacts of bugs, glitches, flaws in software. Quality by definition is qualitative, not quantitative. But when a major company reports a massive, nationwide outage, we can see the numbers in near real-time.
The Numbers
- User impact: ~2 million subscribers
- Length of outage: 10 hours
- User Compensation: $20
- Company Cost: $40 million
- Internal Cost: unknown
Reports indicate around 2 million customers were affected, and now, Verizon is giving out $20 credits as compensation. You can do the math! That’s roughly $40 million in reputation repair — all triggered by a “software issue.” And that number doesn’t include the internal cost of diagnosing, fixing, and stabilizing the system. Those internal costs come from salaries of all the people that helped to fix the issues. That’s likely software developers, software testers, DevOps engineers, engineering leadership, etc. That quickly adds up if there were dozens of people involved!
This is a real cost of poor quality software! In a single moment, $40 million+ is spent to fix an issue that likely could have been prevented! While we don’t know what the exact “software issue” is, we do know how expensive failures can be once they hit production!
Prevention vs Insurance
Software testers know what Benjamin Franklin knew over 250+ years ago, that:
“An ounce of prevention is worth a pound of cure.”
Imagine if even a fraction of that $40+ million had been invested before release — in shifting left, shifting down, acceptance testing, quality engineering, targeted test automation, and testing techniques that are utilized to prevent failures!
This chart, from the IBM Systems Science Institute, shows the Relative Cost of Fixing Defects. It’s from a study published in 2002! Even back then, we knew that fixing a bug in the design phase or the requirements phase costs 1x. Once the code is created, a bug fix now costs 6.5x. During testing? That’s 15x! Finally, if the bug makes its way to production (or maintenance), it’s a 100X cost to fix!
As an example, let’s say that it took $2 million of internal costs to fix the “software issue.” This is a guess as I couldn’t find any reports that indicate the cost. Let’s break this down by SDLC phase:
- Design/Requirements phase (1x): $20,000
- Implementation/Coding phase (6.5x): $130,000
- Testing/QA phase (15x): $300,000
- Maintenance/Production phase (100x): $2 million
As you can see, it saves a lot of money to find bugs earlier!
Today, QA isn’t just about finding bugs. It’s about:
Reducing Risk, Protecting Reputation, and Building Resilience!
As an industry, we are moving away from traditional QA — which can be thought of as an insurance policy. Instead, Quality Engineering is about Quality Intelligence, recognizing patterns, building systems, to create leverage!
This is why software testers, your QA team, your quality engineers aren’t a cost center! They’re your competitive advantage!
Outages like these aren’t just technical failures. They remind us that quality is a business strategy! Software testers will save you money!
Failures will happen. But we can and should limit their impact!