NTP Architecture, Protocol

and Algorithms

David L. Mills
University of Delaware
http://www.eecis.udel.edu/~mills
mailto:mills@udel.edu

Sir John Tenniel; Alice’s Adventures in Wonderland,Lewis Carroll

22-Jul-07 1
Process decomposition
Peer/Poll System
Server 1 Clock Discipline
1 Process
Process
Selection
Peer/Poll and Combining
Server 2 Loop Filter
2 Clustering Algorithm
Algorithms
Peer/Poll
Server 3
3

Remote VFO
Peer/Poll
Servers
Processes
Clock Adjust
Process
o Peer process runs when a packet is received.
o Poll process sends packets at intervals determined by the clock
discipline process and remote server.
o System process runs when a new peer process update is received.
o Clock discipline process runs at intervals determined by the measured
network phase jitter and clock oscillator (VFO) frequency wander.
o Clock adjust process runs at intervals of one second.
22-Jul-07 2
 NTP protocol header and timestamp formats
NTP Protocol Header Format (32 bits)
LI leap warning indicator
LI VN Mode Strat Poll Prec
VN version number (4)
Root Delay Strat stratum (0-15)
Root Dispersion Poll poll interval (log2)
Reference Identifier Prec precision (log2)

Reference Timestamp (64) NTP Timestamp Format (64 bits)

Seconds (32) Fraction (32)
Originate Timestamp (64)
Value is in seconds and fraction
Cryptosum Receive Timestamp (64) since 0h 1 January 1900

Transmit Timestamp (64) NTPv4 Extension Field

Field Length Field Type
Extension Field 1 (optional)
Extension Field
(padded to 32-bit boundary)
Extension Field 2… (optional) Last field padded to 64-bit boundary

Key/Algorithm Identifier
Authenticator NTP v3 and v4
(Optional) Message Hash (64 or 128) NTP v4 only
authentication only

Authenticator uses DES-CBC or MD5 cryptosum

of NTP header plus extension fields (NTPv4)
22-Jul-07 3
NTP packet header format

Packet header
Variables Description LI VN Mode Strat Poll Prec
Root Delay
leap leap indicator (LI)
version version number (VN) Root Dispersion
mode protocol mode Reference Identifier
stratum stratum
τ poll interval (log2 s) Reference Timestamp (64)
ρ clock reading precision (log2 s)
Δ root delay Originate Timestamp (64)
Ε root dispersion
refid reference ID Receive Timestamp (64)
reftime reference timestamp
T1 originate timestamp Transmit Timestamp (64)
T2 receive timestamp
T3 transmit timestamp MAC (optional 160)
T4 destination timestamp*
MAC MD5 message hash (optional)
* Strictly speaking, T4 is not a packet variable; it
is the value of the system clock upon arrival.

22-Jul-07 4
NTP date and timestamp formats and important dates
Year M D JDN NTP Date Era Timestamp
-4712 1 1 0 -208,657,814,400 -49 1,795,583,104 First day Julian Era
1 1 1 1,721,426 -59,926,608,000 -14 202,934,144 First day Common Era
1582 10 15 2,299,161 -10,010,304,000 -3 2,874,597,888 First day Gregorian Era
1900 1 1 2,415,021 0 0 0 First day NTP Era 0
1970 1 1 2,440,588 2,208,988,800 0 2,208,988,800 First day Unix Era
1972 1 1 2,441,318 2,272,060,800 0 2,272,060,800 First day UTC
2000 1 1 2,451,545 3,155,673,600 0 3,155,673,600 First day 21st century
2036 2 7 2,464,731 4,294,944,000 0 4,294,944,000 Last day NTP Era 0
2036 2 8 2,464,732 4,295,030,400 1 63,104 First day NTP Era 1
3000 1 1 2,816,788 34,712,668,800 8 352,930,432 4294967296

NTP Date (signed, twos-complement, 128-bit integer)

Seconds (64) Fraction (32 or 64)

Era Number NTP Timestamp (unsigned 64-bit integer)

Era (32) Seconds (32) Fraction (32)

22-Jul-07 5
Process decomposition
Peer/Poll System
Server 1 Clock Discipline
1 Process
Process
Selection
Peer/Poll and Combining
Server 2 Loop Filter
2 Clustering Algorithm
Algorithms
Peer/Poll
Server 3
3

Remote VFO
Peer/Poll
Servers
Processes
Clock Adjust
Process
o Peer process runs when a packet is received.
o Poll process sends packets at intervals determined by the clock
discipline process and remote server.
o System process runs when a new peer process update is received.
o Clock discipline process runs at intervals determined by the measured
network phase jitter and clock oscillator (VFO) frequency wander.
o Clock adjust process runs at intervals of one second.
22-Jul-07 6
 NTP one-step on-wire protocol
t2 t3 t6 t7
0 t1 = org t3 t5 = org
0 t2 = rec t4 t6 = rec Packet State Variables
Variables Name Description
t1 t3 = xmit t5 t7 = xmit
t2 = clock t6 = clock Peer B org originate timestamp
rec receive timestamp
org t1<>0? T1 = t1 t5<>T1? T5 = t5
State xmt transmit timestamp
rec T2 = t2 T2 T6 = t6 T6 Variables
xmt 0 T3 = clock t3==T3? T7 = clock Packet Header Variables
Name Description
t2 t3 t6 t7
tn originate timestamp
tn+1 receive timestamp
tn+2 transmit timestamp
tn+3 destination timestamp
t1 t4 t5 t8
t1 t4 t5 t8
0 t1 t3 = org t5 t7 <> T3? org Duplicate Test
0 t2 t4 = rec t6 Packet
Variables t5 == T5? xmt Bogus Test
t1 = xmit t3 t5 = xmit t7
t4 = clock t8 = clock Peer A
org 0 t3 <> 0? T3 = t3 t7 <> T3?
State
rec 0 T4 = t4 T4 T8 = t8 Variables
xmt T1 = clock t1 == T1? T5 = clock t5 == T5?

22-Jul-07 7
Timestamp interleaving
t2 t3 t6 t7 t10 t11

One Step

t1 t4 t5 t8 t9 t12

t2 t6 t3 t10 t7

Two Step

t4 t1 t8 t5 t12

o In the diagrams, transmit timestamps carry odd numbers, receive

timestamps carry even numbers.
• Receive timestamps are available immediately.
• In one-step mode, transmit timestamps are conveyed in the transmitted
packet.
• In two-step mode, transmit timestamps are conveyed in the following
transmitted packet.
• It takes two roundtrips to accumulate all four timestamps

22-Jul-07 8
 NTP two-step on-wire protocol
t2 t3 t6 t7 t10 t11

t1 t4 t5 t8 t9 t12

t1 t4 t5 t8 t9 t12
t01 t04 t3 =t5org t58 t7 =t5org t98
0 t2 t4 = rec t6 t8 = rec t10 Packet
0 0 t5 = xmit t7 (T3) t9 = xmit t11(T7) Variables
t4 = clock t8 = clock t12 = clock
org 0 t3 <> 0? T3 = t3 t7 <> T3? T7 = t7 t11 <> T7?
State
rec 0 T4 = t4 T4 T8 = t8 T8 T12 = t12
Variables
xmt 0 t1 == T1? T1 t5 == T1? T5 t9 == T5?

porg T1= clock T1 T5 = clock T5 T9 = clock T9 Extended

prec 0 T2 = t2 T2 T6 T6 T6 State
pxmt 0 T4 = t4 T4 T8 T8 T8 Variables

Round 1
Round 2
1
θ= [(T2 − T1 ) + (T3 − T4 )] 1
Round 3
2 θ= [(T6 − T5 ) + (T7 − T8 )]
δ = (T 4 − T1 ) − (T 3 − T 2 ) 2
δ = (T 8 − T 5 ) − (T 7 − T 6 )

22-Jul-07 9
IEEW 1588 (PTP) master-slave protocol

T2 T3
Slave
Delay_Req
Sync Delay_Resp(t4)
Follow_Up (T1)
Master T1 T4

o Ethernet NIC hardware strikes a timestamp after the preamble and

before the data separately for transmit and receive.
o In each round master sends Sync message at T1; slave receives at T2.
o In one-step variant T1 is inserted just before the data in the Sync
message; in two-step variant t1 is sent later in a Follow_Up message.
o Slave sends Delay_Req message at T3; master sends Delay_Resp
message with T4. Compute master offset θ and roundtrip delay δ:
1
θ = [(T2 − T1 ) + (T3 − T4 )] δ = (T 4 − T1 ) − (T 3 − T 2 )
2

22-Jul-07 10
Transition matrix

Packet Mode

Mode ACTIVE PASSIVE CLIENT SERVER BCAST

NO_PEER NEWPS FXMIT NEWMC NEWBC

ACTIVE PROC PROC

Mode
Association M

PASSIVE PROC ERROR

CLIENT PROC

SERVER

BCAST

BCLIENT ERROR PROC

The default (empty box) behavior is to discard the packet without comment.

22-Jul-07 11
Packet sanity tests

Test Comment Code Condition Routine

Packet Flashers
drop implementation error none T 3 = 0 or (T 1 = 0 and T 2 ≠ 0) or (T 1 ≠ 0 and T 2 = 0) receive
1 duplicate packet pkt_dupe T 3 = xmt receive
2 bogus packet pkt_bogus T 1 ≠ org receive
3 invalid timestamp pkt_proto mode ≠ BCST and T 1 = 0 and T 2 = 0 receive
4 access denied pkt_denied access restricted, untrusted key, etc. receive
5 authentication error pkt_auth MD5 message hash fails to match message digest receive
6 peer not synchronized pkt_unsync leap = 11 or stratum >= MAXSTRAT or T 3 < reftime packet
7 invalid distance pkt_dist Δ R < 0 or E R < 0 or Δ R / 2 + E R > MAXDISP packet
8 autokey keystream error pkt_autokey MD5 autokey hash fails to match previous key ID receive
9 autokey protocol error pkt_crypto key mismatch, certificate expired, etc. receive

Peer Flashers
10 peer stratum exceeded peer_stratum stratum > sys_stratum in non-symmetric mode accept
11 peer distance exceeded peer_dist distance greater than MAXDIST accept
12 peer synchronization loop peer_loop peer is synchronized to this host accept
13 peer unfit for synchronization peer_unfit unreachable, unsynchronized, noselect accept

22-Jul-07 12
Clock filter algorithm
T2 Server T3

θ0
T1 Client T4
θ = 1 [( T 2 − T1 ) + (T 3 − T 4 )]
2
δ = ( T4 − T1 ) − (T 3 − T 2 )

o The most accurate offset θ0 is measured at the lowest delay δ0 (apex of

the wedge scattergram).
o The correct time θ must lie within the wedge θ0 ± (δ − δ0)/2.
o The δ0 is estimated as the minimum of the last eight delay
measurements and (θ0 ,δ0) becomes the peer update.
o Each peer update can be used only once and must be more recent
than the previous update.

22-Jul-07 13
 Clock filter performance
4 4

3 3

2 2

Offset (ms)
Offset (ms)

1 1

0 0

−1 −1

−2 −2

−3 −3

−4 −4
0 5 10 15 20 0 5 10 15 20
Time (hr) Time (hr)

o Left figure shows raw time offsets measured for a typical path over a
24-hour period (mean error 724 μs, median error 192 μs)
o Right graph shows filtered time offsets over the same period (mean
error 192 μs, median error 112 μs).
o The mean error has been reduced by 11.5 dB; the median error by 18.3
dB. This is impressive performance.

22-Jul-07 14
Clock select principles
B
correctness interval = q - l £ q0 £ q + l
A
m = number of clocks
D C f = number of presumed falsetickers
A, B, C are truechimers
Correct DTS D is falseticker
Correct NTP

o The correctness interval for any candidate is the set of points in the
interval of length twice the synchronization distance centered at the
computed offset.
o The DTS interval contains points from the largest number of
correctness intervals, i.e., the intersection of correctness intervals.
o The NTP interval includes the DTS interval, but requires that the
computed offset for each candidate is contained in the interval.
o Formal correctness assertions require at least half the candidates be in
the NTP interval. If not, no candidate can be considered a truechimer.

22-Jul-07 15
system process: select algorithm
For each of m associations construct a correctness interval
[θ – rootdist, θ + rootdist]

Consider the lowpoint, midpoint and highpoint of these intervals. Sort these
values in a list from lowest to highest. Set the number of falsetickers f = 0.

Set the number of midpoints d = 0. Set c = 0. Scan from lowest endpoint to

highest. Add one to c for every lowpoint, subtract one for every highpoint,
add one to d for every midpoint. If c ≥ m − f, stop; set l = current lowpoint

Set c = 0. Scan from highest endpoint to lowest. Add one to c for every
highpoint, subtract one for every lowpoint, add one to d for every midpoint. If
c ≥ m − f, stop; set u = current highpoint.

If d ≤ f and l < u?
no yes
yes Success; the intersection
Add one to f. Is f < m / 2?
no interval is [l, u].
Failure; a majority clique
could not be found..

22-Jul-07 16
Cluster principles
peer jitter
select jitter

ϕR3 ϕR3
ϕS1 ϕS3
ϕR2 ϕR2
ϕR4 ϕR4

ϕR1

a b

o Candidate 1 is further from the others, so its select jitter ϕS1 is highest.

o (a) ϕmax = ϕS1 and ϕmin = ϕR2. Since ϕmax > ϕmin, the algorithm prunes
candidate 1 to reduce select jitter and continues.
o (b) ϕmax = ϕS3 and ϕmin = ϕR2. Since ϕmax < ϕmin, pruning additional
candidates will not reduce select jitter. So, the algorithm ends with ϕR2,
ϕR3 and ϕR4 as survivors.

22-Jul-07 17
system process: cluster algorithm

Let (θ, ϕR, Λ) represent a candidate with peer offset θ, jitter ϕ and a
weight factor Λ equal to stratum as the high order field and root
distance as the low order field.

Sort the candidates by increasing Λ. Let n be the number of

candidates and nmin ≤ n the minimum number of survivors.

For each candidate compute the selection jitter ϕS (RMS peer offset
differences between this and all other candidates).

Select ϕmax as the candidate with maximum ΛϕS.

Select ϕmin as the candidate with minimum ϕ.

ϕmax < ϕmin or n ≤ nmin or ϕmax is prefer peer? yes

no
Delete the outlyer candidate with ϕmax; reduce n by one.

Done. The remaining cluster survivors are the pick of the litter.

22-Jul-07 18
NTP dataflow analysis
Server 1 Peer 1
Δ, Ε θ, δ, ε, ϕ
Selection
Server 2 Peer 2 and System
Δ, Ε θ, δ, ε , ϕ Combining Θ, Δ, Ε, ϑ
Algorithms
Server 3 Peer 3
Δ, Ε θ, δ, ε , ϕ

o Each server provides delay Δ and dispersion Ε relative to the root of the
synchronization subtree.
o As each NTP message arrives, the peer process updates peer offset θ,
delay δ, dispersion ε and jitter ϕ.
o At system poll intervals, the clock selection and combining algorithms
updates system offset Θ, delay Δ, dispersion Ε and jitter ϑ.
o Dispersions ε and Ε increase with time at a rate depending on specified
frequency tolerance φ.

22-Jul-07 19
Error budget - notation
o Constants (peers A and B) o System variables
ρ maximum reading error Θ clock offset
φ maximum frequency error Δ root delay
w dispersion normalize: 0.5 Ε root dispersion
ϕs selection jitter
o Packet variables
ϕ jitter
ΔB peer root delay
τ interval since last update
ΕB peer root dispersion
m number of peers
o Sample variables
T1, T2, T3, T4 protocol timestamps o Peer variables
x clock offset θ clock offset
y roundtrip delay δ roundtrip delay
z dispersion ε dispersion
τ interval since last update ϕr filter jitter
n number of filter stages
τ interval since last update

22-Jul-07 20
Definitions

o Precision: elapsed time to read the system clock from userland.

o Resolution: significant bits of the timestamp fraction.
o Maximum error: maximum error due all causes (see error budget).
o Offset: estimated time offset relative to the server time.
o Jitter: exponential average of first-order time differences
o Frequency: estimated frequency offset relative to UTC.
o Wander: exponential average of first-order frequency differences.
o Dispersion: maximum error due oscillator frequency tolerance.
o Root delay: accumulated roundtrip delay via primary server.
o Root dispersion: accumulated total dispersion from primary server.
o Estimated error: RMS accumulation from all causes (see error budget).

22-Jul-07 21
Time values and computations
Packet Variables Peer Variables System Variables

x= 1
2 [(T2 − T1 ) + (T3 − T4 )] θ = x0 Θ = combine(θ j )

y = (T4 − T1 ) − (T3 −ΔT2 )= Δ δ = y0 Σ ρB = ρ Δ = ΔR + δ

B

zi +1 = zi + Φ μ ε = ∑ zi +i 1 Σ Ε = Ε R + ε + Φμ + ϕ + θ
i 2

z0 = ρ R + ρ + Φ(T4 − T1 ) ϕ= 1

n
∑(x 0 − xi ) 2 ϕS = 1
∑ (θ − θ ) j
2
i m j

Client
ρR Σ

ΕR = Ε + Φμ ΔR = Δ

Server ϑ= ϕ 2 + ϕ S2

o Packet variables are computed directly from the packet header.

o Peer variables are groomed by the clock filter.
o System variables are groomed from the available peers.
22-Jul-07 22
Clock discipline algorithm
θr+
NTP Phase Vd Vs
θc− Detector
Clock Filter

VFO Loop Filter

x
Vc Clock Phase/Freq
Adjust y Prediction

o Vd is a function of the phase difference between NTP and the VFO.

o Vs depends on the stage chosen on the clock filter shift register.
o x and y are the phase update and frequency update, respectively,
computed by the prediction functions.
o Clock adjust process runs once per second to compute Vc, which
controls the frequency of the local clock oscillator.
o VFO phase is compared to NTP phase to close the feedback loop.

22-Jul-07 23
NTP clock discipline with PPS steering
θr+
NTP Phase Vd Vs
θo− Detector
Clock Filter

Vc
VFO Loop Filter

y Frequency
PPS
Estimator

o NTP daemon disciplines variable frequency oscillator (VFO) phase Vc

relative to accurate and reliable network sources.
o Kernel disciplines VFO frequency y to pulse-per-second (PPS) signal.
o Clock accuracy continues to be disciplined even if NTP daemon or
sources fail.
o In general, the accuracy is only slightly degraded relative to a local
reference source.

22-Jul-07 24
Measured PPS time error for Alpha 433

Standard error 51.3 ns

22-Jul-07 25
Further information

o NTP home page http://www.ntp.org

• Current NTP Version 3 and 4 software and documentation
• FAQ and links to other sources and interesting places
o David L. Mills home page http://www.eecis.udel.edu/~mills
• Papers, reports and memoranda in PostScript and PDF formats
• Briefings in HTML, PostScript, PowerPoint and PDF formats
• Collaboration resources hardware, software and documentation
• Songs, photo galleries and after-dinner speech scripts
o Udel FTP server: ftp://ftp.udel.edu/pub/ntp
• Current NTP Version software, documentation and support
• Collaboration resources and junkbox
o Related projects http://www.eecis.udel.edu/~mills/status.htm
• Current research project descriptions and briefings

22-Jul-07 26