The cost of

downtime and
how to use visuals
to mitigate loss

v20.08.31-20.06

Table of
contents
03 The true costs of downtime
04 How to prevent downtime
07 How to resolve incidents faster
11 Reign over your cloud with
tangible next steps
The cost of downtime and how to use visuals to mitigate loss
Introduction
I
n March of 2015, Apple lost $25
million in 12 hours. That same year,
trading at the New York Stock
Exchange was suspended for four
hours, costing the Intercontinental
Exchange $14 million. And more recently,
in March of 2019, Facebook lost an
estimated $90 million in 14 hours.
The common denominator between each
of these events? Network outages. One
outage can lead to security breaches,
customer dissatisfaction, and lost
revenue. As with any other crisis, the best
way to combat the threat of downtime
is to carefully prepare response plans
ahead of time. However, to make your
plan actionable and agile, you must take
the next step and design the necessary
documentation to align your teams and
guide your decision-making.
In this paper, we’ll cover how to prevent
costly downtimes by doing the following:
• Following an architecture framework
• Visualizing your cloud implementation
or migration
• Maintaining visibility into your existing
environments
• Keeping teams aligned with
architecture diagrams
• Setting and enforcing internal best
practices
Finally, while there are many effective
tools available that are dedicated to
incident management, this paper will
not discuss these tools in great detail,
but instead review critical strategies for
resolving incidents efficiently: investing
in key pre-work; documenting your
troubleshooting processes; collaborating
on incident response in a shared visual
workspace; and holding post-mortems
after incidents to solidify learnings.
sales@lucidchart.com 650-733-6172 2
 The true costs of downtime
A
lthough Apple, Facebook, and
the NYSE were large enough to
absorb the high monetary costs,
their outages also cost them
time and customer trust. And
for smaller companies, similar damages could
be catastrophic. Understanding these costs, and
creating a plan to prevent and resolve outages,
is critical.
Tangible
costs
So what, exactly, are the costs of network outages
and downtime? Let’s start with financial costs.
According to Gartner, network downtime costs
an average of $5,600 per minute or $300,000
per hour. And that might be a conservative
estimate. In their 2016 report, “Cost of Data
Center Outages,” the Ponemon Institute raised
that cost estimate to $9,000 per minute. Even
if the downtime only lasts one hour, that is still a
sizable cost, and most outages last much longer.
One simple error—a failed software upgrade or
changes in device configuration—can lead to
millions of dollars in losses.
And the costs don’t stop there. An outage can
result in a costly lack of productivity from your
teams. When servers are down, employees are
unable to perform their jobs. Since their salaries
are likely a fixed cost, they will continue to get
paid regardless of their inability to work during
an outage. You’ll end up paying them for the time
they couldn’t work and then paying them to catch
up once you’re back online.
The cost of downtime and how to use visuals to mitigate loss
Beyond that loss of productivity, consider how
much time and attention outages demand from your
IT team. Any other projects they were working on,
any other tasks in the queue, have to be set aside
during an outage emergency, and that translates
into backlogs of work and missed deadlines.
You can actually calculate this
loss of productivity with this
formula from Atlassian:
Lost productivity =
Employee salary/hr
x utilization %
x number of employees
Finally, you might be hit with litigation costs. Take
Robinhood, for example. The online brokerage
experienced outages and trading glitches in March
of 2020. Within two weeks of this outage, they faced
three separate lawsuits from angry customers.
Intangible
costs
The effects of unplanned downtime and outages
extend beyond monetary considerations. For
instance, if your company needs to maintain
regulatory compliance (think GDPR for protecting
personal user data or PCI DSS if you handle
credit card data), your standing can be severely
compromised by an outage if it leads to or was
caused by a data breach.
Any outage is an opportunity for sensitive data
loss or exposure, so you can expect compliance
auditors and other stakeholders to express
concern when you experience an outage. This
concern can potentially lead to more audits with
each audit eating up more of your company’s
time and resources on top of what you lost to the
downtime itself.
However, it’s not just about your standing with
regulatory bodies. It’s about your standing with
your customers. Outages are a surefire way to lose
customer trust. This is the ultimate breach in the
provider client service contract and can quickly
damage your customer satisfaction and retention.
And with so many service provider options
available to them, it’s easier than ever for
customers to take their business elsewhere. This
is the case whether you’re a financial service
provider or a travel agent. A single unsatisfied
customer might share their grievances on social
media and soon that one negative experience
can snowball into a larger problem. What may
have started as an intangible cost is now a very
tangible hit to your bottom line.
sales@lucidchart.com 650-733-6172 3
 How to prevent downtime
T
he immediate and residual costs
of downtime pose a danger to
businesses and their customers.
And although downtime is virtually
impossible to eliminate entirely,
there are preventative measures businesses can
take to reduce the number of incidents and the
associated time and costs it takes to fully recover.
Even the latest AWS user agreement suggests
that “you are responsible for … taking appropriate
action to secure, protect and backup your
accounts.”
So, what can you do to prevent downtime and
protect and secure your accounts?
The cost of downtime and how to use visuals to mitigate loss
Follow an architecture framework
Correctly designing and building your architecture
is a crucial first step in preventing downtime.
Best practices for architecting aren’t always clear,
especially because the needs of each business
differ. What may be right for one organization may
not be for the next.
Luckily, many cloud providers, such as Amazon
Web Services (AWS), Microsoft Azure, and Google
Cloud Provider, offer nuanced versions of a
well-architected framework, which outline best
practices for how to architect and monitor your
cloud environment.
These frameworks provide customers with core
strategies that are categorized into five pillars and
a friendly mnemonic: CROPS.
• Cost optimization
• Reliability
• Operational excellence
• Performance efficiency
• Security
Essentially, these five pillars help you plan,
build, and optimize your architecture with extra
attentiveness and consistency.
With these pillars in mind, you can take extra
precautions to protect any associated information
and assets, verify that computing resources
efficiently meet system requirements and present-
day technology demands, and guarantee that
everything is running at its best, at the lowest
possible price point.
When you design carefully from the start, you
can mitigate the risk of downtime in the future.
If you’re looking for a more comprehensive
architecture framework, there are also many
certifications, such as SOC 2 and ISO 27000
series, that provide subsequent guidelines and
best practices.
Not only should these frameworks be used by
architects as they architect, or by coders as
they code, but these frameworks should also be
referenced during internal architecture reviews.
Ideally, reviews should be conducted on a
regular basis.
When supplemented with a well-architected
framework, architecture reviews provide extra
opportunities to prevent the risks of downtime.
During architecture reviews, you should inspect
your architecture design to ensure it is scalable
and reliable, while also evaluating risks to ensure
proper security measures have been taken.
With your architecture diagram on display, your
teams can easily run through risk assessments
and mitigation strategies together, flagging
resources such as unencrypted databases.
sales@lucidchart.com 650-733-6172 4
 Visualize your cloud implementation
or migration to ensure it is
done correctly
The world’s best architecture design means
nothing if what was designed isn’t actually what’s
implemented.
Deploying something into the cloud often
requires the work and input of several teams.
Unfortunately, it’s possible for your plans to be
lost in translation. For example, breakdowns
can happen as a cloud architect handoffs their
design for review by security teams. Or maybe
as the architecture is passed to a cloud engineer
to code. By the time your code arrives in the
hands of your quality assurance teams, there may
already be multiple small (or big) breakdowns
that put your infrastructure at a greater risk for
downtime.
The cost of downtime and how to use visuals to mitigate loss
If designs are not implemented as planned, how
can you be certain you’re not at risk for downtime?
All it takes is one bad game of telephone to result
in a botched deployment, potentially leading to
multiple single points of failure, scaling limits,
incorrect communication between your inputs
and outputs, or any of the other common reasons
for downtime.
With all of these risks in mind, if you design
based on your cloud provider’s version of a well-
architected framework, you then need to tackle
two very important tasks during implementation:
• Verify that migration and implementation
were done correctly.
• Understand why migration and
implementation were not done correctly if
you encounter issues.
Essentially, you need to quickly cross-check plans
with what your engineers actually build.
Automatically generating an architecture diagram
based on your deployed infrastructure helps
you compare your “blueprint” diagram with your
current state diagram. If the diagrams are not
identical, direct stakeholders to the exact location
of misalignment and prompt conversations about
potential engineering mistakes that need to be
corrected or possible technical limitations in the
architect’s design that may require alterations to
your architecture templates for future use.
Maintain visibility into your
existing environments
So you’ve done all of the work to design your
cloud architecture by following a framework
to design and review your architecture and by
verifying that design was implemented correctly.
Now what?
It’s also important to understand what your
environment looks like at all times, especially as
you iterate and deploy in smaller batches.
Creating an architecture diagram is one easy way
to maintain visibility into what your environment
looks like. You’ll score bonus points if you can
visualize it in real time or by simply refreshing your
cloud provider data.
Visualizing your infrastructure is a powerful way to
create a bird’s-eye view of a complicated system.
Doing so can help you—and your teams and
stakeholders—more clearly understand exactly
where certain elements live, why they matter, and
how they can affect other systems and processes
downstream should something go awry.
With accurate and up-to-date visibility, your
employees will be more equipped to:
• Conduct thorough security gap analysis.
• Complete detailed network evaluations.
• Provide informed recommendations.
• Make smarter network decisions and
changes.
sales@lucidchart.com 650-733-6172 5
 Keep teams
aligned
At the end of the day, your employees are
responsible for preventing downtime. And they
can only be successful if they’re on the same
page from start to end.
From the moment an employee is hired and
throughout the onboarding process, you can use
your cloud visuals to help onboard new team
members quickly and effectively with accurate
documentation. Then, these diagrams can keep
new and seasoned team members aligned.
As mentioned earlier, architecture passes through
many hands and teams can easily be misaligned if
you’re not careful.
Whether you’re migrating to the cloud, managing
a hybrid cloud, or optimizing your current cloud
infrastructure, keeping architecture diagrams
up to date ensures that teams can more easily
collaborate and make decisions together about
the current and future states of your cloud
environment.
Use accurate and up-to-date cloud diagrams
to keep architects and implementation teams
informed and on the same page when it comes to:
• How things currently work vs. how they
should work
• What limitations exist
• What changes or updates have been made
• When changes were made
• Who owns the changes or is responsible
for them
The cost of downtime and how to use visuals to mitigate loss
• How ideas, processes, and initiatives should
be implemented
• What the purpose behind these decisions is
• How the cloud impacts the business as a
whole
While it’s important that current employees
are aligned while they manage your cloud
together, it’s equally as important that you don’t
allow critical tribal knowledge to leave with an
employee if they leave your company. A well-
maintained architecture diagram mitigates the
risk of critical knowledge loss, ensuring current
employees have access to the information they
need, regardless of employee turnover.
As you leverage a well-architected framework to
streamline alignment on designing and reviewing
existing and proposed architecture, how can you
be sure employees follow through on the small—
yet high-impact—details of designing and coding
your infrastructure?
Set and enforce
internal best practices
Often, downtime is the result of poorly enforced • Ensure continued access to critical data in the
internal best practices and standards, or lack event of errors and failures.
thereof, in regards to your CROPS. Occasionally,
• Conduct penetration testing.
you will discover knowledge gaps that need to
be addressed and will require you to update your • Reduce interdependencies by loose coupling.
best practices and standards.
After you develop your internal best practices,
Standardizing internal best practices (and how do you ensure they’re enforced and that
keeping them up to date) instills confidence teams follow through? Regularly conducted
in your teams and ensures a cloud-enabled architecture reviews.
workforce that is properly evaluating the current
An effective architecture review should identify
state of your infrastructure and securely planning
and highlight all security weaknesses or critical
for future states.
issues in your applications.
IT leadership will likely develop these best
Rather than dive into lines of spreadsheet data
practices, which should be documented and
or code, present your architecture diagram and
centralized for easy access, so architects and
focus on the information that matters most so
engineers know exactly where to turn should
you can quickly arrive on a set of actions that
questions arise.
should improve the performance of your cloud
While internal best practices may vary by environment and achieve your internal standards.
organization, they could be a combination of the
If possible, your architecture diagrams could be
following on this non-exhaustive list:
automatically generated based on a simple data
• Secure your endpoints. import from your cloud provider. This way, you’ll
know you’re always looking at your current state,
• Encrypt data in storage and in transit.
not a version from six (or 60) deployments ago.
• Use multi-factor authentication to reduce the
risk of credential compromise.
• Always use auto-scaling.
• Manage infrastructure in a source code
control system.
sales@lucidchart.com 650-733-6172 6
 How to resolve incidents faster

W
hile you may do everything
in your power to mitigate
the risks of downtime, it’s
impossible to completely
eradicate. As we’ve seen, the
costs of downtime can be costly. And while you
may not be able to prevent downtime entirely, you
can certainly mitigate the impact.
You may use dedicated incident response and
management tools, such as Splunk, IBM QRadar,
or Demisto that may effectively help monitor
networks, provide issue alerts, etc. But applying
best practices and using visuals alongside these
tools can better equip your teams to plan and
prepare for downtime.
Think ahead,
make a plan
When your network goes down, there’s no time to
lag—the cloud moves quickly. It may seem cliché,
but the most effective way to resolve incidents
faster is to have a plan already in place. If you
don’t have a plan, your teams won’t know where
to begin or how to work together.
When you think about how to approach incident
response in the cloud before an event happens,
it reduces response time. An incident that could
last days or hours could be reduced to minutes
or even seconds, saving you hundreds (if not
thousands) of dollars.
Keep in mind—this should be more than a set-and-
sit kind of plan. You should review your response
plan regularly and iterate alongside changes
made to your cloud environment.
Document your
troubleshooting processes
It’s one thing to agree upon a troubleshooting
process, but you need to ensure that each
scenario you imagine during your pre-work
phases has an associated process to follow during
an incident. Luckily, there are multiple resources
and existing processes that you can follow, but
you should always tailor the processes to fit the
needs and resources of your business.
As you determine which processes you need, try
to cover the following bases.

Consider making a flowchart of all the possible

downtime scenarios so you can visualize the areas
of your cloud environment that are most at risk.

The cost of downtime and how to use visuals to mitigate loss sales@lucidchart.com 650-733-6172 7

Get to the root of the incident
When your network goes down, your first instinct
will likely be to identify the issue. And while you
need to get to the root of the incident, utilizing
efficient tools and processes can help you identify
the cause for downtime faster.
As mentioned earlier, you may already be using
tools such as Splunk, IBM QRadar, or Demisto. But
without a process in place, you may not be using
these tools to their fullest potential.
Consider popular processes outlined in the
Information Technology Infrastructure Library
(ITIL), such as incident management. Incident
management includes a set of steps to follow
when there is an unplanned interruption that
impacts something down the line, such as another
system.
From there, you’d follow another set of protocols
to discover the underlying cause of the incident—
problem management. While these ITIL processes
are effective, it’s useful to create a process flow
diagram so your employees can more easily follow
along during downtime, rather than poring over
chapters of text.
The cost of downtime and how to use visuals to mitigate loss
Understand relative priority
Just as you take steps to determine what the
issues are, you should also document a decision
process that helps incident management teams
understand the priority of events or incidents and
when to escalate a problem.
Because your systems and data flows are in
constant states of change, a decision tree diagram
can help new and seasoned team members alike
quickly determine the next best course of action
when it comes to identification, contaminants,
eradication, and recovery.
Here are a few examples of questions you could
include:
• Did you review recent network configurations?
• Is the application and all services operational
from end to end?
• Can the incident be isolated?
• Have backups been created to protect critical
data?
By creating a decision guide, employees can
resolve incidents quicker because they won’t
waste costly downtime time searching for answers
or the person who may hold the right answer.
Collaborate during incident response
Say you have multiple dispersed offices, and your
network goes down at the end of the workday
in the United States. Are you going to ask your
employees in the U.S. to work through the night,
or do you have a process in place that helps
your global team coordinate handoffs during an
outage?
In this instance, you could adopt a follow-the-sun
model that can be used to coordinate in real time
across teams and time zones. Just as you would
document the flow from team to team, you could
include a step for the coordinating teams to review
an up-to-date architecture diagram that accurately
depicts your current cloud environment, and
together they can determine what’s already been
discovered and what still needs to be done.
These diagrams can also help teams coordinate
and answer these questions:
• What subnet is it in?
• What VPC is it in?
• How is routing set up?
• What’s the IP of the actual box? How is it
routing?
• Where’s it going?
• Where’s it getting stuck?
Ideally, your cloud diagram will be hosted on an
application with advanced collaboration features
to further support these efforts. Imagine if incident
response teams could comment directly on a
specific component of your architecture diagram,
quickly directing each other to the areas where
action is needed.
sales@lucidchart.com 650-733-6172 8
 Focus on the information that
really matters
When you experience an outage, you first need
to determine exactly what is happening. Because
cloud environments are often large and complex,
you need to quickly drill into the information
and components of your architecture that really
matter.
When you maintain architecture diagrams that
accurately depict your current environment, your
incident response teams can quickly understand
and determine:
• Where a problem is occurring
• What the issue is
• Why a problem is occurring
• How the associated systems are connected
Putting a network diagram at the forefront of your
incident response processes and further weeding
out distracting details can speed up how quickly
response teams locate issues.
The cost of downtime and how to use visuals to mitigate loss
From there, they can direct support engineers
to exactly where the issue is, highlighting where
action needs to be taken to resolve the incident.
However, given how quickly teams need to react
during incident response, there often isn’t time to
create a new architecture diagram. So you must
always have an updated and accessible diagram
available. This way, your incident response teams
can focus on resolving incidents, not searching
for documents.
Centralize documents
How you craft your troubleshooting processes If you host your architecture diagrams in the
is important, but it’s just as important that your cloud—and keep them up to date with simple data
documentation is accessible. refreshes—then you can rest assured that your
teams will have access to the documents they
Unfortunately, when you really need your
need when they need them, restoring uptime as
architecture diagrams and process flows, they’re
quickly as possible.
often not available. This challenge may be a
result of only one or two people managing your And, while it’s important to keep your diagrams
architecture diagrams or using a desktop-only centralized, don’t forget about the other apps that
solution like Visio. So when your network goes may include your diagrams, such as Confluence
down, the doc is stuck on one person’s desktop, or Jira. Invest in a diagramming solution that
unavailable to your incident response teams and offers automated integrations, so when you
support engineers. make changes to your architecture diagram, the
changes will be automatically reflected anywhere
Or, given how quickly your environment changes,
else, like your Confluence wiki pages.
you may run into versioning issues, unable to
identify the correct visual, partially due to non-
standardized titling and slight nuances in dozens
of PNGs and JPEGs scattered across your teams.
Luckily, it doesn’t have to be this way—but only if
you centralize your documents.
sales@lucidchart.com 650-733-6172 9
 Focus on the information that
really matters
When you experience an outage, you first need
to determine exactly what is happening. Because
cloud environments are often large and complex,
you need to quickly drill into the information
and components of your architecture that really
matter.
When you maintain architecture diagrams that
accurately depict your current environment, your
incident response teams can quickly understand
and determine:
• Where a problem is occurring
• What the issue is
• Why a problem is occurring
• How the associated systems are connected
Putting a network diagram at the forefront of your
incident response processes and further weeding
out distracting details can speed up how quickly
response teams locate issues.
From there, they can direct support engineers
to exactly where the issue is, highlighting where
action needs to be taken to resolve the incident.
However, given how quickly teams need to react
during incident response, there often isn’t time to
create a new architecture diagram. So you must
always have an updated and accessible diagram
available. This way, your incident response teams
can focus on resolving incidents, not searching
for documents.
The cost of downtime and how to use visuals to mitigate loss
Centralize documents
How you craft your troubleshooting processes
is important, but it’s just as important that your
documentation is accessible.
Unfortunately, when you really need your
architecture diagrams and process flows, they’re
often not available. This challenge may be a
result of only one or two people managing your
architecture diagrams or using a desktop-only
solution like Visio. So when your network goes
down, the doc is stuck on one person’s desktop,
unavailable to your incident response teams and
support engineers.
Or, given how quickly your environment changes,
you may run into versioning issues, unable to
identify the correct visual, partially due to non-
standardized titling and slight nuances in dozens
of PNGs and JPEGs scattered across your teams.
Luckily, it doesn’t have to be this way—but only if
you centralize your documents.
If you host your architecture diagrams in the
cloud—and keep them up to date with simple data
refreshes—then you can rest assured that your
teams will have access to the documents they
need when they need them, restoring uptime as
quickly as possible.
And, while it’s important to keep your diagrams
centralized, don’t forget about the other apps that
may include your diagrams, such as Confluence
or Jira. Invest in a diagramming solution that
offers automated integrations, so when you
make changes to your architecture diagram, the
changes will be automatically reflected anywhere
else, like your Confluence wiki pages.
Host a
post-mortem
After an incident is resolved and uptime has been
restored, rather than indulge in a culture of blame,
create a culture of improvement. Treat every
minute of downtime as a learning experience and
opportunity to revise troubleshooting processes
and architecting best practices.
Gather everyone involved for a comprehensive
post-mortem, where fear of punishment or
retribution is left at the door. An effective post-
mortem holds individuals, teams, and departments
accountable while encouraging improvement.
With your architecture diagram at the ready, invite
engineers and response teams to give a detailed
account of where mistakes were made, why, and
what assumptions were made and then evaluate
what could have been done differently.
You might even consider periodically hosting a
“game day” where you replay past incidents—but
this time, you’ll be on the winning team.
By thoroughly reliving these experiences, you can
better prevent downtime in the future and apply
new knowledge gleaned to better, quicker incident
response the next time around (and there will be a
next time).
sales@lucidchart.com 650-733-6172 10
 Reign over your cloud
with tangible next steps Lucidchart Cloud
Insights helps you

Y
our cloud environment is always on automatically visualize
your architecture so your
the move—code is being written,
deployments are being prepared,
and incidents are lurking.

organization can better

Dedicate the time and resources to developing
a truly comprehensive downtime response understand, optimize, and
govern the cloud through
plan—one that starts with developing state of the
art cloud architecture and results in dwindling
incidents, fast uptime restoration, and avoiding
the ghastly costs of outages.
accurate, up-to-date, and
interactive diagrams that
With the proper processes and diagrams in place,
together your cloud teams can mitigate the risks
and losses of downtime, using new cloud insights
to come out on top and deliver uninterrupted
service to customers worldwide. are consistent across
your org.

Learn more today.

The cost of downtime and how to use visuals to mitigate loss sales@lucidchart.com 650-733-6172 11
Lucidchart is a visual workspace that combines diagramming, data
visualization, and collaboration to accelerate understanding and drive
innovation. With this intuitive, cloud-based solution, everyone can work
visually and collaborate in real time while building flowcharts, mockups, UML
diagrams, and more. Lucidchart is utilized in over 180 countries by more than
15 million users, from sales managers mapping out target organizations to
IT directors visualizing their network infrastructure. Ninety-nine percent of
the Fortune 500 use Lucidchart, and customers include Google, GE, NBC
Universal and Johnson & Johnson. Since the Utah-based company’s founding
in 2010, it has received numerous awards for its product, business and
workplace culture. For more information, visit lucidchart.com.

sales@lucidchart.com 650-733-6172