Addressing The Office 365 Monitoring Gap (7.2)

ADDRESSING THE
MONITORING GAPS
IN OFFICE 365
MICHAEL VAN HORENBEECK JAAP WESSELIUS JUSTIN HARRIS
Meet the Authors
Micheal Van Horenbeeck
Managing Partner at the Collective | MVP | MCSM
Over the past 18 years , Michael has specialized in a variety of products

and services from the Microsoft 365 product stack; more specifically within
Enterprise Mobility + Security, Azure Active Directory, Office 365, etc. Michael
is one of a handful of experts in the world to hold both the Microsoft Certified
Solutions Masters for Messaging certification and the Office Apps & Services
MVP award at the same time.
Jaap Wesselius
Independent Consultant | MVP
Jaap is an independent consultant in the Netherlands focusing on Microsoft

business productivity solutions with Microsoft Exchange server, Lync server
and Office 365 and related Microsoft online solutions. On top of being
an MVP, Jaap is also the author of the books “Exchange 2010 – A Practical
Approach” and “Pro Exchange 2013 Administration.”
Justin Harris
Microsoft MCM | Chief Technology Officer, ENow
Justin has over 22 years of experience in the Microsoft identity and

collaboration stack. Additionally, he has experience helping organizations
shift to a cloud delivery model. Justin is one of a handful of technical
professionals globally that have been recognized with the Microsoft MVP
award for evangelism efforts and obtained the Microsoft Certified Master
certification.
At ENow, Justin is responsible for the entire product stack. Today, ENow’s
solutions are deployed in over 130 countries and trusted by global
organizations like Facebook, Experian, and VMware.
2
ENow Software ADDRESSING THE OFFICE 365
400 Spectrum Center Dr. · Suite 200 Irvine, CA 92618 USA
www.ENowSoftware.com · 877 879 3669 MONITORING GAPS
The Office 365 Success Story
Office 365 is a major success story for Microsoft. Although Microsoft still does not disclose exact numbers of
active users or Office 365-specific revenue, in the April 2019 earnings call they reported that Office 365 had
grown with around 4,15 million active users per month since November 2018. This brings bringing the total
amount of monthly active users to a whopping 180 million.
Office 365 has a wide variety of application available for users (dependent on license type), in particular
Microsoft Teams has seen adoption rates skyrocket. Microsoft Teams was launched just two short years ago in
July 2017, and by July 2019 Microsoft reported over 13 million active daily users, and 19 million active weekly
users.
Another interesting fact we learned from that earnings call was the success of the Enterprise Mobility and
Security suite (EM+S). An install base of approximately 100 million are now leveraging the value that comes
from the EM+S SKU. While this figure could mean several things, it clearly shows a movement in the market
towards a more security-centric approach.
The fact that Office 365 has been so successful across such a wide range of customers is a testament to the
technical and operational excellence of its products. The value proposition Microsoft has established for
hybrid operations is still a big part of its success. Hybrid configurations ease the on-boarding process, making
the (technical) decision to migrate to Office 365 a lot easier –especially for large and complex environments.
However, as customers start their individual journeys to Office 365, there are some gaps in the portfolio of
monitoring and management tools that the service offers, and it is important to know and understand what
these are in order to make one’s own experience as smooth as possible.
3
New Challenges in a Cloud
World
Traditional monitoring solutions focus almost entirely on the infrastructure supporting a specific service or
application. While this approach proves relatively successful in a pure on-premises scenario, it is much less
effective with cloud-based services. In an on-premises scenario, you typically have a full understanding of what
each piece of the infrastructure is responsible for, what the relation is between components, and what “normal”
behavior looks like for each component. When one component breaks or underperforms, you know what the
impact will be on other components and the services depending on it.
4
Cloud-based systems are very different. First, you do not have visibility into the supporting infrastructure, let
alone an understanding of how components relate to one another. As such, even if you would have access to
all the metrics of the infrastructure, the information wouldn’t be useful. For example, consider the servers that
Microsoft uses to implement the Office Graph or Search; outsiders have no way to know how many or what
kind of servers are handling that work, what they’re doing, or what a normal performance baseline looks like.
Even if this information were to be available: should customers care? Or is this solely Microsoft’s responsibility
as part of the service agreement? The answer is simple: some metrics are just relevant to Microsoft alone.
As a customer, you are purchasing a service and an experience. How Microsoft delivers that service, or how
much load their systems are currently under doesn’t really matter. That is, at least, if the delivered service and
functionality continues to live up to the expectations and SLA’s!
The massive scale of services, like Office 365 with users distributed across several datacenters and hundreds
of thousands of servers, make it nearly impossible to maintain the same monitoring paradigm. Within a sea of
information, you can’t distinguish what information is relevant to you and what isn’t, in part because you don’t
have complete information about all the components and in part because of the scale and complexity of the
environment. This problem is worsened by the fact that the cloud service provider has little information about
your network.
The fundamental differences between how an on-premises application and cloud-based system are managed
created a need for a different monitoring approach. This approach must focus on the service and functionality
that is being delivered, as well as the tasks one executes to capture the true user experience. For example, in
Exchange Online, one of the core tasks is the ability to send and receive email, whereas in SharePoint Online,
being able to upload, download and edit documents is the most frequent task. The way you provide the
monitoring results must also account for the changing responsibilities of IT staff. While previous technologies
may have been managed by separate, specialized teams, we see that these teams tend to converge, forcing
administrators to be more up-to-date on technologies they may not be very familiar with.
Cloud-based systems, such as Office 365, enable organizations and users to work from virtually anywhere.
Because of this, monitoring a service from a single location, typically the organization’s datacenter, no longer
represents how applications are used in the real world. As users roam between various locations and connect
from both within or outside the boundaries of the corporate network, it is important for an organization to
understand if service issues are confined to a single location, or if an outage is affecting its operations at a larger
scale –potentially even service-wide.
Without proper monitoring, and by relying solely on the built-in capabilities, administrators are often left to
wait for feedback from their users to understand when something is wrong. This is far from ideal. Being able to
pick up on early warnings of outages in any of the components within the infrastructure, including Office 365,
is vital.
5
Office 365 Service Outages
While service outages don’t happen frequently, they do happen from time to time. Every outage teaches
us something new, they highlight the areas of improvement for Microsoft, but also exposes monitoring
gaps and possible mitigation steps to prevent or reduce the impact of similar events in the future.
The MFA outages in 2018 painfully demonstrated that MFA can be the Achilles’ heel of your deployment. It is
by far the most effective (and necessary!) countermeasure against password spray attacks, but also renders it
impossible for your users to access the Office 365 services in case of an issue. Many organizations that followed
Microsoft’s recommendation to enable MFA for everyone, including their administrative accounts... They soon
found themselves locked out of their tenant, preventing them from making changes, such as temporarily
disabling MFA to allow users to access their services. Organizations affected by the outage quickly learned that
having a break-glass account is a must.
Another example is the May 2019 outage Microsoft suffered after a faulty DNS update. According to the post-
incident-report, Microsoft’s own monitoring solutions did not pick up on the issue until after customers started
reporting it. While the service itself was likely performing just fine, a faulty DNS entry, led to customers being
unable to connect.
The Service Health Dashboard (SHD) is quite vague. All you really know is that there is some sort of problem,
but you have no idea of the scope of the problem, or the impact on your users (until they start screaming!)
6
Some outages may not be on
Microsoft...
It was once reported that around 52% of support calls into Microsoft end up being issues on the client side.
Hybrid configurations prove particularly problematic in terms of monitoring. Just as you don’t have any
visibility into the infrastructure of a service provider, the same is true the other way around— Microsoft has no
idea if or when there are problems affecting your on-premises servers. However, in some scenarios on-premises
components are crucial to the operations of the cloud- based service.
There are many scenarios in which on-premises components become an important, if not critical, part in the
end-to-end user experience of Office 365. This would be the case when you use components such as Active
Directory Federation Services, Pass-Through Authentication, Exchange Hybrid, etc. When an organization
decides to use these services, additional components are deployed on-premises and become the customer’s
responsibility. In some cases, these components may be new to the environment and not managed to the
same standard, or with the same skill, as the existing mature on-premises systems. If one of these services fail,
the consequences for the service can be substantial. For example, when the AD FS infrastructure becomes
unavailable, no users will be able to log on to any Office 365 service.
7
The same is true for certificates: they are crucial to many solutions like, Exchange or AD FS. Although trivial,
expiring certificates have caused many issues in the past, even for cloud solution providers like Microsoft. Being
able to effectively monitor the status of your certificates and when they expire – preferably all in one overview
– will help you to avoid expiration and the issues resulting from that.
These are just a few examples of issues that fall solely within the responsibility of the customer to detect and
solve. But it doesn’t stop there. Often times the responsibility stretches over many other components, some
more obvious than others, like basic network connectivity, routing, DNS, and hybrid Exchange functionality or
mail flow and free/busy, etc.
Microsoft’s Monitoring Native

Tools
Microsoft’s internal monitoring team performs numerous monitoring tests on the Office 365 service. However,
not all of the tools they use internally are made available to administrators. In this section we will provide an
overview of the two main tools available to administrators: the Service Health Dashboard and AAD Connect
Health.
Service Health Dashboard
8
Today, Microsoft exposes health information about Office 365 services through the Service Health Dashboard
(SHD). The information that is provided through the dashboard is only of limited use, as it focuses primarily
on the overall service health instead of tenant-specific or user-specific problems. One of the limitations of the
SHD is that it only gives you part of the end-to-end service view; it provides information on components that
Microsoft is responsible for, but fails to monitor and report on outages caused by other components, such as
your local network, Internet connection or hybrid infrastructure such as Directory Synchronization, federation
health, mail flow or AD FS.
Due to the massive scale of Office 365, the dashboard almost always reports some type of issue in one of its
services as, logically, there is always a problem going on somewhere. A warning in the dashboard does not
necessarily mean that your tenant is affected or that some of your users are experiencing problems. Often,
service issues are accompanied with vague descriptions of who might be affected, leaving the customer
wondering whether an issue impacts them or not. This creates a new challenge for the administrator, as they
are left with the question of whether an issue is relevant and, if it is, to what extent.
The SHD also does not send automated alert notifications. One must purposefully log on to the SHD in order
to view the latest health information. As a matter of fact, Microsoft uses the number of users reading an SHD
alert to help determine the scope of impact for that issue. In the past, some issues in Office 365 were directly
related to an outage in Azure Active Directory, which prevented access to the SHD. This also creates a Catch-22
situation for the Service Health Dashboard: the inability to authenticate to Azure AD prevents users from
getting up-to-date Service Health Information. What good is a health dashboard if you risk being locked out of
it?
Microsoft has become better and faster in term of posting outages to the SHD, unlike during a 2015 outage,
when it took them nearly eight hours to acknowledge the issue in the SHD. However, it remains extremely
hard for Microsoft to close the gap with external monitoring solutions, as they cannot simply post messages
to the SHD prior to assessing the issue and making sure that 1) customers are affected, and 2) the appropriate
message is sent, as to not create unnecessary confusion. The time between the outage happening and
Microsoft being able to confidently assess the issue so that they can craft an appropriate response to the SHD
creates a void for many customers. During this time, they are left to wonder whether there was an outage and,
if so, whether it was affecting them. One of the most common complaints we see on Twitter is, “All my users are
affected. Why isn’t this in the Service Health Dashboard?”
Azure AD Connect Health
Another useful tool Microsoft provides to help you monitor your hybrid Office 365 deployment is Azure AD
Connect Health. This tool is available if you have an Azure AD Premium subscription. It is an agent-based
monitoring solution which helps you gain visibility in both Azure AD Connect synchronization, AD FS and on-
premises Active Directory.
9
It supports AD FS 2.0 on Windows Server 2008 R2, Windows Server 2012, Windows Server 2012 R2 and
Windows Server 2016. It also supports monitoring the AD FS proxy or web application proxy servers.
The main benefits are the usage reports, and that it will notify you if the directory sync engine stops working, or
users are unable to authenticate to AD FS. The information is presented in the Azure AD Connect Health portal.
The installation is simple and does not take much time.
Although Azure AD Health covers part of the components which are not included in the SHD, there are some
limitations that will leave gaps in your monitoring strategy. First, it is not integrated with the SHD, requiring
an admin to have permissions to access the Azure Portal to view the monitoring results. This also means you
do not have a single location to look at all the areas that can affect your users. It also creates a dependency on
Azure AD which – in case of an outage –can render access to the portal impossible. Secondly, while it does
provide the ability to alert on events, it does not offer any capabilities to integrate into 3rd party monitoring
systems, which is a typical requirement for enterprise companies.
Azure AD Connect Health only provides limited insights into the components. While it does perform synthetic
transactions, it doesn’t monitor the network, the Office 365 service, hybrid server health, connectivity and
functionality from remote locations. This makes it very difficult for an administrator to figure out what caused
an outage and if it is something they can fix. Imagine if you think the issue is on Microsoft’s side and you
wait for something that will never be fixed...The lack of visibility hinders an organization’s ability to respond
appropriately to reported issues. The ability to detect where an outage stems from is crucial, as it ultimately
allows you to drive down the Mean-Time-To-Resolution (MTTR) of reported issues. For instance, if a network
issue prevented a remote location from accessing the AD FS servers, you would likely hear from users
complaining they could not access the Office 365 service, but what would you do next? The SHD would not
show a service outage, and AAD Connect Health would not have visibility into the users at the remote location.
The ultimate resolution would be to resolve the network issue, which would be your responsibility.
10
The way you handle an outage is obviously different when the issue occurs within Microsoft’s datacenters. In
these cases, the challenge shifts to first quickly understanding conclusively, that the issue is on Microsoft’s side.
The ability to do this as soon as possible is crucial because you do not want your management or users asking
you if there is an issue. Finally, you can open up a ticket with Microsoft and keep your users up-to-date as to
when the issue will be resolved.
ENow’s Office 365 Platform Fills

the Gaps
Monitoring has undoubtedly changed from the purely on-premises days. Back then, IT departments had
complete visibility into their infrastructure and the interdependencies. As organizations migrate to Office
365, they baton passes to Microsoft to handle the maintenance of their service, per their SLA agreement.
However, Microsoft is only responsible for the cloud service, and has very little visibility into the infrastructure
on your side. This creates some blind spots, as you have no visibility into Microsoft’s infrastructure. In a hybrid
environment, there are added complexities, creating more places to look when an outage occurs.
The best monitoring solutions don’t depend on a single source of data to make assessments. Logically, the
more information that is available, the clearer the picture and the better a decision can be made as to what to
do should an outage occur. In addition, monitoring solutions that extract data on an ongoing basis can report
availability and outage trends on a historical basis on a monthly, quarterly, yearly, or long-term basis. Analysis of
this data can identify any weaknesses that may exist in a configuration and allow administrators to rectify issues
before they become a real problem.
ENow’s Office 365 platform provides this type of visibility so that users remain productive. It monitors Office
365, your on-premises hybrid services, and remote locations, on a single pane of glass. It provides a single,
easy-to-use, view of everything that you need to check. You don’t need separate tools to monitor all these vital
components. Everything is integrated and monitored as one system. The monitoring dashboard is ideal for
displaying on a large monitor in a network operations center, or in a background window, where a quick glance
tells you everything you need to know about the health of your environments. ENow’s single solution enables
organization to detect outages immediately, validate the end user experience, and increase adoption while
controlling costs.
11
Detect Outages Immediately
ENow’s solution is your own personal
outage detector that pertains solely
to your environment. Knowing
immediately when a problem
happens, where the fault lies, and why
the issue has occurred ensures that any
outages are detected and solved as
quickly as possible.
The graphic to the left depicts

ENow’s awarding-winning OneLook
dashboard.
ENow’s monitoring provides early warnings of outages in any of the components within the infrastructure,
including Office 365. It is, of course, preferable when administrators know that something has gone wrong and
have taken steps to rectify the issue before end users notice that they can’t get work done. Even if the problem
cannot be immediately addressed because it is within Office 365, a support call can be logged with Microsoft,
and a mitigation plan can be set in motion to advise end users what they can do to remain productive. For
instance, if Exchange Online is unavailable, you can tell users that clients such as Outlook Web App and mobile
devices won’t be able to do much work with email, and that they might be better off switching to Outlook. If
SharePoint Online is unavailable, users can be told to work with local copies of documents until online access
is restored. Being able to proactively coach and guide end users during outage situations is a much better
position to be in than having to scramble to understand what’s going on, and then rushing to react.
It’s important to note that many of the Office 365 outages mentioned previously were eventually reported in
the SHD. However, the key difference is the SHD is not tenant specific, meaning customers had no visibility into
how the outages impacted their organizations. The information presented is quite vague, as all it really tells
you is there is some sort of issue happening somewhere, and the the outage may or may not affect your users.
In comparison, ENow’s solution is tenant specific, and can instantly give admins visibility into what services
and which subset of their users are affected. For example, the Exchange Online outage on Jan 2, 2019 affected
the majority of Europe, but other areas remained undisturbed. In addition to Office 365 service outages, the
outage can also be part of the on-premises components you are responsible for. ENow monitors all major
components from the Network, Certificates, authentication, as well as your hybrid servers.
12
Validate End User Experience
The ultimate measure of a service is your end users’ ability to connect to the service and complete tasks
pertinent to their role. Unfortunately, Office 365 often leaves many admins in the dark, waiting for their end
users to bring a problem to light. When the issue does reach the help desk, there is often no easy way to
validate their complaint, which, in turn, increases MTTR.
The best way to measure the end-user experience is to perform synthetic transactions. Synthetic transactions
programmatically mimic end-user interactions with the service and typically focus on the most common
actions taken by a user. By continually performing the same operations that a “real” client, such as Outlook or
Microsoft Teams, would perform, these transactions give you an end-to-end view of the service, including
DNS and network operations, logon, as well as sending and receiving data. By establishing a baseline, which
includes latency and performance, you can quickly see over a historical time period if there is an issue that
needs attention. This could include issues with the service, hybrid components that you maintain or with
connectivity between the user’s location and the service.
Although synthetic transactions can provide a lot of information about a specific action, such as whether
the action was successful and how long it took to perform a certain task, the behavior of a transaction alone
does not necessarily tell you what component caused a specific problem or where an issue occurred. For
this, synthetic transactions must be complemented with the necessary application logic and additional tests
allowing the monitoring solution to correlate events. By unlocking this level of visibility, you will have detailed
insight into the cause and location of the problem.
ENow runs its suite of monitoring tests both from the workstation where it’s installed and from an independent
cloud provider, giving you a 360-degree perspective on connectivity and functionality issues. ENow’s Remote
Probes enable organizations to run monitoring tests from locations they specify. Remote Probes let you
monitor service quality and performance from the locations where your users work so you always have up-to-
date data about service quality in those locations. Monitoring the experience that individual end users have
when using the service is critical to identifying and localizing problems. By deploying Remote Probes where
your users are, you get complete visibility into performance and service quality at each individual location so
that you always know where problems are occurring and who is affected.
13
Increase Adoption & Optimize
License Spend
In an on premises world, Microsoft introduced new features every few years within major releases. Back then,
companies had the choice to deploy the software. If they did, configuration of several services (Exchange,
SharePoint, Lync) and training end users was required. Office 365 delivers new features nearly every month
that can improve end user collaboration and productivity. While the fast-paced cloud world reaps a plethora of
benefits, it also introduced a new set of challenges. How do you know if your user community is really adopting
and using these features? If they are not using these features, is your organization really taking advantage of
the licenses they purchased?
While Microsoft does offer an Adoption Pack, it has some limitations. For example, it is limited to the last 12
months or less, depending on the service. In addition, it requires elevated permissions to setup and customize
the reports. The user level details are only available for the last complete month, for users with an assigned
product license. Sharing reports presents a challenge for many companies, as both parties must have Power Bi
Pro or Premium. In essence, if one user has the free Power Bi service, they cannot share their dashboards.
ENow closes the gaps by properly monitoring adoption, enabling IT Pros to pinpoint underutilized licenses.
For example, one report can identify that a particular department, like marketing is not leveraging Microsoft
Teams. Upon investigation, IT Pros may find that the Marketing Team is: using a competing service that could
be eliminated, unware that Microsoft Teams was available to them, do not how to use it, or perhaps they don’t
need the application to meet their goals.
Out of the box, Office 365 gives very

limited information when it comes to
licensing. ENow’s solution empowers
organizations to obtain financial
accountability by driving hidden costs
out of their Office 365 investment.
Gaining insights into the usage
and attribution of those licenses is
important. It can help organizations
make better decisions on what licenses
to buy or which ones to swap for
another license plan.
14
Conclusion
Microsoft continues to invest huge amounts of money in growing the reach and expanding the size of
Office 365. They’re adding new services and capabilities, expanding into new regions, and ramping up their
marketing efforts to capture more (and larger!) customers. Along with all these activities, they are continuing to
slowly enhance their included monitoring tools. However, the nature of cloud services means that customers
will never get the complete visibility they need from Microsoft’s own tools; there are some components, such
as hybrid services, that must be monitored from the customer side. In addition, customer-centered monitoring
of service availability and quality is critical to detecting and resolving problems no matter their origin.
ENow is the market leader in Office 365 monitoring and reporting, combining synthetic transaction-based
monitoring with Microsoft’s own monitoring APIs to give you a clear, actionable picture of the state of your
cloud services. Its unique combination of reporting and monitoring functionality has led global companies
like Facebook, Experian, and The Wonderful Company to choose ENow. Lastly, ENow’s solution has been
recommended by the Gartner Group since 2015, as depicted in the graphic below.
Gartner
Published Research Note Title Research Gartner Analyst
Date Note ISD and Author
Tuesday,
Performance Strategy for Office
December 1, G00277137 Kyle Davis
365 (ENow)
2015
Daniel O’Connell,
SWOT: Microsoft Skype for
Friday, January Megan Marek
Business Online Worldwide G00292742
1, 2016 Fernandez, Bern
(ENow)
Elliot
Four Steps to Remedy
Monday,
SharePoint Online
February 1, G00293171 Kyle Davis
Performance for a Global
2016
Company (ENow)
Wednesday, Solution Path for SharePoint
G00303886 Craig Roth
May 11, 2016 (ENow)
Cameron Haight
Friday, June 10, Monitoring the Move to
G00281286 Vivek Bhalla, Sanjit
2016 Exchange Online
Gangeli
Thursday,
How to Manage Multitenant Adam Preset,
February 28, G00321578
Cloud Office Deployments Jeffrey Mann
2019
15

Addressing The Office 365 Monitoring Gap (7.2)

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Addressing The Office 365 Monitoring Gap (7.2)

Uploaded by

Copyright:

Available Formats

ADDRESSING THE

Over the past 18 years , Michael has specialized in a variety of products

Jaap is an independent consultant in the Netherlands focusing on Microsoft

Justin has over 22 years of experience in the Microsoft identity and

Microsoft’s Monitoring Native

Service Health Dashboard

Azure AD Connect Health

ENow’s Office 365 Platform Fills

The graphic to the left depicts

Out of the box, Office 365 gives very

You might also like