Office 365 Network Optimization

and Troubleshooting
Jeff Mealiffe
Principal Program Manager
Microsoft Office 365 Engineering

Microsoft has invested $15 billion in

infrastructure, building over 100 datacenters
and we are constantly evaluating new locations
Our high-performing network is one of the
top 3 in the world with public peering in 23
countries with 1,500 ISPs.
Our Datacenters support over 20 Million
businesses and over 200 Online Services.
Office 365 is sold in 131 markets, 43
languages, and 25 currencies.

Office 365
DC locations
Other Microsoft
DC locations

Microsoft has datacenter capacity around the worldand were growing

Quincy
Cheyenne

Dublin

Chicago
Boydton

Amsterdam

Shanghai
Hong Kong

Des Moines

San Antonio
Singapore

Brazil

35+ factors in site selection:

Proximity to customers
Energy, Fiber Infrastructure
Skilled workforce

Australia

1+ million servers
100+ datacenters
in 40+ countries

Japan

The green circles

represent Microsoft
Edge nodes live for the
Office 365 Portal.
There are many other Microsoft edge nodes
that are not yet compliant with Office 365.

EDGE

TCP
connect
SSL
connect
San Antonio DC

TTFB

Reusing existing
connections

Edge reuses connections between

to further improve performance

TTLB

Without Edge, entire

request over ISPs network

Content
Time
EDGE

With Edge, Microsofts network

brought closer to the user

Client RTT

Server RTT

App
Latency

Brisbane

Australia

Kuala Lumpur

Malaysia

Los Angeles

USA

Melbourne

Australia

Amsterdam

Netherlands

Miami

USA

Perth

Australia

Auckland

New Zealand

New York

USA

Sydney

Australia

Wellington

New Zealand

Palo Alto

USA

Vienna

Austria

Moscow

Russia

San Jose

USA

Luxembourg

Luxembourg

Singapore

Singapore

Seattle

USA

Sao Paulo

Brazil

Stockholm

Sweden

Montreal

Canada

Zurich

Switzerland

Toronto

Canada

Taipei

Taiwan

Prague

Czechoslovakia

London

UK

Paris

France

Ashburn

USA

Frankfurt

Germany

Atlanta

USA

Hong Kong

Hong Kong

Boston

USA

Dublin

Ireland

Chicago

USA

Milan

Italy

Dallas

USA

Turin

Italy

Denver

USA

Tokyo

Japan

Honolulu

USA

Seoul

Korea

Las Vegas

USA

Site data is published at

http://www.peeringdb.com/view
.php?asn=8075
Some cities have multiple
peering points
Peering locations may be on-net
or off-net
Peering may involve physical
connection and/or routing
advertisements
Data as of July 2014 is subject to
change

Outlook, Outlook Web Access, OneDrive for Business

Native clients on
tablets, PCs,
& desktops

Native clients
on mobile devices

Browser-based
Clients also cache
with HTML5

Browser-based
mobile clients

Office 365 offers a wide variety of options across devices for customers to access the service
OWA uses HTML 5 Offline Application Caching if enabled in Offline Settings

Server
workloads

Microsoft
network

Microsoft
edge nodes

Internet
peering and
routing

Content
delivery
network

Customer
Internet
connection

Rich client
applications

Exchange Online
http://aka.ms/tune
SharePoint Online
Similar to on-prem
HTTPS
views
of webpages,
uploads/downloads of content
Lync
Estimates
rely upon
on-prem baselines
Online
Office
Web Apps
or Office
Document
Cached modeediting
reduceswith
impact
and provides
for latency
tolerance (<325 mSec)
Perfmon,
IM is bursty,
latency
tolerant,
and
very
small
Netmon,
etc. tonew
baseline
Use
yourbut
pilot
to top,
predict
requirements
Bursty,
latency
tolerant
Voice
uses
RTAudio
Without
Definite
advantages
tono
Outlook
2013
Baseline
bandwidth
requirements
before
Office 365
baseline,
real
way
toSP1
estimate
50kbps
low, 80kbps
high,
autodetermination
Exchange Bandwidth Calculator
Netmon
Measure
bandwidth
for pilot users
Video depends
uponbaseline
resolution
to baseline
280kbps
4000kbps
high,
dependent
upon
resolution
Will
Extrapolate
for total
bandwidth
required
for your
workload
ramp low,
up
as
more
content
is loaded
into
SharePoint, MySites,
Desktop
depends
upon upgrades
desktop resolution
Assumes sharing
enough
time to order
if needed
OneDrive
for
Business
Peer-to-Peer versus Client-Server
Start with at least 20% head room
Lync Bandwidth Calculator
Evaluate options to conserve bandwidth

1.

1.

2.

2.

3.

3.

4.

4.

5.

5.
6.
7.
8.

6.

7.
8.
9.

Other configuration topics

Some customer scenarios have seen improved performance
Never a silver bullet and other measures should be investigated
Required to be disabled for debugging or support

IP Addresses not as quickly updated

PAC files, CONNECT, and are they helping or hurting

Agenda

NAT, Measuring Latency/Round Trip Time

TCP Window Scaling
GEO DNS issues
Proxy and Firewall port exhaustion
Packet Loss
Routing and Peering

TCP Idle time settings

Proxy Authentication
DNS performance
SACK and TCP MSS
Lync tests to Server
SharePoint customization performance

PSPing Demo

Putting it all together

Client

Proxy

Office 365 Datacentre

54.88ms

0.346ms

Internal RTT (ms)

External RTT (ms)

Total RTT to O365

54.88

346

400.88

Here we can see clearly, the poor RTT is outside the customers environment, on the ISP link to Office 365. If
this RTT is unexpected, the customer can engage their ISP to investigate.

TCP Window Scaling

TCP data packets
TCP ACK

TCP Window Scaling

enabled?

Maximum TCP receive

buffer (Bytes)

No

65535 (65k)

Yes

1073725440 (1gb)

Impact of TCP Window Scaling

Presuming a 1000 Mbps link here is the maximum throughput we can get with TCP window
scaling disabled and then with it enabled
Round Trip Time (ms)

Maximum Throughput
(Mbit/sec) without
scaling

300

1.71

447.36

200

2.56

655.32

100

5.12

1310.64

50

10.24

2684.16

25

20.48

5368.32

10

51.20

13420.80

102.40

26841.60

512.00

134208.00

Maximum Throughput
(Mbit/sec) with scaling

TCP Window Scaling

7692
12:28:03
14/03/2014
12:28:03.8450000
0.0000000
100.8450000
10.127.0.199
contoso47-48ipv4b.sharepointonline.com.akadns.net
TCP
TCP: [Bad CheckSum]Flags=......S.,
SrcPort=43511, DstPort=HTTPS(443), PayloadLen=0, Seq=3807440828, Ack=0, Win=65535 ( Negotiating scale factor 0x0 ) =
65535
7740
12:28:04
14/03/2014
12:28:04.1440000
0.2990000
101.1440000
contoso4748ipv4b.sharepointonline.com.akadns.net
10.127.0.199
TCP
TCP:Flags=...A..S., SrcPort=HTTPS(443),
DstPort=43511, PayloadLen=0, Seq=3293427307, Ack=3807440829, Win=4380 ( Negotiated scale factor 0x2 ) = 17520

7692
12:28:03
14/03/2014
12:28:03.8450000
0.0000000
100.8450000
1
0.127.0.199
contoso47-48ipv4b.sharepointonline.com.akadns.net
TCP
:
[Bad CheckSum]Flags=......S., SrcPort=43511, DstPort=HTTPS(443), PayloadLen=0, Seq=3807440828, Ack=0,
Win=65535 ( Negotiating scale factor 0x0 ) = 65535 {TCP:818, IPv4:122}
7740
12:28:04
14/03/2014
12:28:04.1440000
0.2990000
101.1440000
c
ontoso47-48ipv4b.sharepointonline.com.akadns.net
10.127.0.199
TCP
TCP:Flags=...A..S.,
SrcPort=HTTPS(443), DstPort=43511, PayloadLen=0, Seq=3293427307, Ack=3807440829, Win=4380 ( Negotiated
scale factor 0x2 ) = 17520
{TCP:818, IPv4:122}

Internet
egress point

Microsoft network

Customer network

Tracing route to OUTLOOK-APACNORTH.OFFICE365.COM [132.245.65.146]

over a maximum of 30 hops:
1

1 ms

1 ms

1 ms SkyRouter.Home [192.168.0.1]

11 ms

11 ms

11 ms ip-89-200-132-100.ov.easynet.net [89.200.132.100]

10 ms

10 ms

10 ms igbtmdistc7503.msft.net [195.66.236.140]

84 ms

84 ms

84 ms xe-0-3-2-0.ash-96cbe-1a.ntwk.msn.net [207.46.45.227]

96 ms

95 ms

95 ms ae2-0.atb-96cbe-1a.ntwk.msn.net [207.46.33.228]

9 140 ms 142 ms 140 ms 191.234.83.150

10 142 ms 138 ms 139 ms ae11-0.lax-96cbe-1b.ntwk.msn.net [207.46.47.11]
11 256 ms 256 ms 256 ms ae2-0.tya-96cbe-1a.ntwk.msn.net [207.46.46.149]
12 265 ms 265 ms 265 ms ae0-0.tya-96cbe-1b.ntwk.msn.net [204.152.140.181]
13 288 ms 290 ms 292 ms xe-7-0-1-0.sin-96cbe-1a.ntwk.msn.net [207.46.38.252]
14 290 ms 288 ms 287 ms xe-5-3-1-0.sin-96cbe-1b.ntwk.msn.net [207.46.41.39]
15 279 ms 279 ms 279 ms ae1-0.sg2-96cbe-1a.ntwk.msn.net [191.234.80.90]
18 280 ms 280 ms 279 ms 132.245.65.146

http://channel9.msdn.com/events/SharePoint-Conference/2014/SPC3993

http://technet.microsoft.com/en-us/library/office-365-system-requirements.aspx

http://trippams.online.lync.com
http://trippdb3.online.lync.com
http://tripphkn.online.lync.com
http://trippbl2.online.lync.com
http://trippsn2.online.lync.com
http://trippsg1.online.lync.com

What you can do