PagerDuty Blog

Register now for Register

Summit Now
ALL BEST
BLOGS
ENGINEERING

PRODUCT
INTEGRATIONS
PRACTICES
ANNOUNCEMENTS Contact Us Log in

Products Solutions Customers Pricing Integrations Resources GET STARTED

Home
-> Blog
-> Reliability -> End-to-End SMS Provider Testing, It's How We Ensure SMS Alerts are Delivered
START FREE TRIAL

End-to-End SMS Provider

Testing, It's How We Ensure SMS REQUEST A DEMO

Alerts are Delivered

by Ashwin Jiwane

All Categories
APRIL 28, 2014

Reliability is important to us. We even inject failure into our systems every Friday to prove it. But
when it comes to sending alerts, reliability goes beyonds writing flawless code. We rely on several
third-party carriers to deliver alerts to our customers. If an SMS isn’t delivered, you aren’t notified of Popular Integrations
an outage. We can’t stick our heads in the sand, telling you that it was because our providers were
down. That’s not good enough. We need a way to ensure that you will receive your alerts no matter
Application Performance
what. Management (APM)

Initially, we provided reliability with our providers with redundancy. We have several providers who
DATADOG
aid us in delivering our SMS alerts to our customers. Redundancy is great and it’s something that
we will continue to do. But we felt we can do even better.

Over the last 4 months, we’ve introduced End-to-End SMS Provider Testing to proactively identify Application Performance
any delays or outages our providers may be experiencing, even if their publicly facing status pages Management (APM)
are out-of-date. We often found that our providers would show that their systems were fully
available, but SMS messages would not be received, even going as far as sending us a delivery
NEW RELIC
confirmation. Using End-to-End Provider testing we can quickly switch the order of our primary,
secondary and tertiary providers to ensure that our customers alerts are not affected by
unexpected outages with our third-party providers. Before introducing this testing, we had no
choice other than to believe our providers and react accordingly. Infrastructure Monitoring

How We Conduct End-to-End PINGDOM

SMS Provider Testing Infrastructure Monitoring

We are continuously testing our SMS providers. For our short codes we send a test SMS every two-
minutes, while varying the frequency for our less-frequently used long codes. To conduct testing, PIVOTAL PLATFORM
we send alerts to ourselves all day using each of our providers. We can measure how long it takes
for the message to be received, if at all, to determine if a provider is available (and how well they are
performing). We then determine if we should demote or promote the usage of our providers.
Developer Tools

How We Automate End-to-End SMS ATLASSIAN JIRA

Provider Testing
(CLOUD)

Currently, we have 3 Android phones with different mobile carrier networks: AT&T, Verizon and T-
Mobile. (We will be adding Sprint soon)

Using our internally built android app for this testing system we send test SMS alerts from our
system to each of our Android phones in a round-robin rotation.

Using Datadog, we calculate the time taken for each SMS to reach the designated testing phone
and how long our testing app takes to reply back to us.

Based on measurements we determine if a provider is down or degraded, then take the

appropriate action.

We consider a provider to be degraded if their SMS delivery latency is more than 3 minutes or if an
SMS is not received for the last two attempts across each of the mobile carrier networks. If a
provider is degraded, our team is alerted and replaces the provider to proactively avoid our
customers missing any of their SMS alerts.
 The Future of End-to-End SMS Provider
Testing
Currently, the process to switch the priority level of our SMS providers is a manual process. We
receive an alert that a provider is degraded and we react accordingly. As we move forward we will
be automating by building a probabilistic model, moving away from receiving failure provider

alerts.

This process has allowed our team to get appropriate alerts from our testing systems so we don’t
have to set-up a variety of alerts on different resources to be aware of issues. This reduces the
overall number of pages our on-call engineer receives, allowing us to spend time solving issues
and improving processes without getting distracted by noise.

We will continue to focus on reliability here at PagerDuty. Not only do these tests and automation
practices keep our product available for you, we also are compiling many learnings from the state
of the systems we connect with. We have definitive knowledge into the performance of our mobile
providers without having to rely on external reports, allowing us to be proactive and deliver you an
awesome PagerDuty experience.

AT&T
Datadog
End-to-End Provider Testing
high availability
reliability

sms alerts
Sprint
T-Mobile
Verizon





You may also love these...

RELIABILITY PRODUCT, RELIABILITY

End to End (E2E) Testing Best PagerDuty: We Are Always On

Practices

PLATFORM SOLUTIONS RESOURCES COMPANY SUPPORT

DevOps Developers Resources Library About PagerDuty System Status

AiOps CollabOps Community Forum Locations Help & Support
Process Automation Customer Service Ops Guides Leadership Partner with PagerDuty
Business Operations Security Incident Customer Stories Newsroom Contact Us
Response
Developer Platform Integrations Guides Events Legal
Critical Event
Enterprise-Class Platform PagerDuty University PagerDuty.org Services
Management
Digital Operations Support Docs Investor Relations FAQs
Retail
Management
Blog Careers Accessibility
Financial Services
Demo
Healthcare

State & Local Government

Non-profit

All Use Cases

    © 2022 PagerDuty, Inc. All rights reserved.

🌐 Switch language

Privacy Policy | Website Terms of Use | California Privacy Notice | Modern Slavery Act | Cookies Settings | Download Assistive Technology