See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/265729617

Analysis of Windows Authentication Protocols: NTLM and Kerberos

Conference Paper · March 2014

DOI: 10.13140/2.1.2087.5528

CITATIONS READS

0 2,766

3 authors:

Randhir Bhandari Nagesh Kumar

AP Goyal Shimla University Jaypee University of Information Technology
5 PUBLICATIONS   3 CITATIONS    8 PUBLICATIONS   163 CITATIONS   

SEE PROFILE SEE PROFILE

Sachin Sharma
Shoolini University
4 PUBLICATIONS   3 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Laravel Frame Work View project

Forest fire detection using WSN: opportunistic routing perspective View project

All content following this page was uploaded by Nagesh Kumar on 18 September 2014.

The user has requested enhancement of the downloaded file.

 Analysis of Windows Authentication Protocols: NTLM and Kerberos

Randhir Bhandari1,a, Nagesh Kumar 2,b, Sachin Sharma 1,c

1
Computer Science Department
Shoolini University, Solan, (H.P), India

2
Computer Science and Engineering Department
GHEC, Solan, (H.P), India
(arandhir.12csmtc10@shooliniuniversity.com, bengg.nagesh2@gmail.com, csachin@shooliniuniversity.com)

Abstract— In today’s environment where data travels a lot on network and hence cannot be send in
plain text hence there is a need of protocols. Authentication Protocols are one of the same which can
provide the authentication, confidentiality & integrity. On Windows as platform in this paper we
analyze two basic protocols known as NTLM (Network LAN Manager) & Kerberos Authentication
Protocol (developed by Massachusetts Institute of Technology (MIT)).
The Kerberos provides authentication by encrypting essential information. It is widely
adopted by organizations and comes in different versions latest available versions are version 4 &
version 5. In one of over previous paper we have generalized the ticket exchange process of version
5. In this paper we have compared Kerberos version 4 with version 5 and also NTLMv1 and
NTLMv2 on the basis of different parameters.
Index Terms- NTLM, Kerberos, cryptography, encryption, decryption, ticket.

I. INTRODUCTION
In beginning of computer era the security of data mostly depend on the user or system and the
authenticity of the user depend on the single password set by the user only. When the Windows based
environment started the passwords were stored on the client machines only. To enhance the
authentication measures, Microsoft introduced LAN Manager or LM (Microsoft NTLM 2013) as an
authentication protocol. The LM was used in Windows 95, NT 3.5 and earlier versions. The LM
password length was restricted to 14 characters only and every seven characters are encrypted
separately (Microsoft NTLM, 2013). The LM could not distinguish between the upper and lower case
characters. Later to overcome these limitations LM was enhanced and the new protocol NTLM which
came into existence in 1993 (NT LAN Manager 2013) and was a challenge – response type protocol.
This protocol was used on different networks that included Windows operating systems. Also this
protocol can operate on stand-alone systems. The NTLM protocol was used as default authentication
protocol (NT LAN Manager 2013) in the Windows NT 4.0 and is still incorporated with new versions
(Windows 7, 8) for the compatibility with older versions(Win9X,NT4.0,SP4).
Microsoft does not recommend NTLM as the recent cryptographic techniques like AES or SHA
256 are not supported in this protocol. NTLM works on Cyclic Redundancy Check (CRC) or Message
digest algorithm to provide integrity of data and it incorporates RC4 cryptographic method for
encryption. NTLM comes with Version 1 and 2. NTLM v1 sends 8 byte random number and secret
key (password) to communicate with server, but as it is prone to a dictionary attack so NTLM v2
came in existence. It includes MD4 hashed 8 byte client challenge in addition to NTLM v1 to immune
itself to dictionary attack. It does not allow a server to pass the client credential to the server on the
same network. Before we further explain here is brief background of Kerberos.
The Kerberos in Greek Mythology means a three headed dog which stands at the gates of house of
Hades. Here the three heads denotes the AAA known as Authentication, Authorization and
Accounting. These three are the basic requirements for designing any secure environment.
The Kerberos protocol is designed & developed to provide authentication services. The MIT designed
Kerberos for providing authentication across unsecure networks by (Kerberos Papers and
Documentation 2004) using the private key cryptography. The authentication in Kerberos is done
through a trusted third party who is denoted as an authentication server. Till now MIT developed five
versions of Kerberos. Version 1 through 3 was used internally by MIT where as version 4 is accepted
beyond MIT. Models for administration and use of computer services differ from site to site and some
environments require support that is not present in Version 4. Version 5 of the Kerberos protocol
incorporates new features suggested by experience with Version 4, making it useful in more
situations.
"Kerberos is an authentication protocol for trusted hosts on untrusted networks". Kerberos is
not beneficial if the host is not trustworthy. Otherwise, the intruder can use host as a key to get
authentic. Intruder can impersonate by obtaining IP address for that server.

II. THE NTLM WORKING MODEL

NTLM family protocols are embedded in applications the application caller sends parameters to
the NTLM which then reply with an authentication method that the caller uses into its own messages
which are to be transmitted.

NEGOTIATE_MSG

Client Server
Machine CHALLENGE_MSG Machine

AUTHENTICATE_MSG

Figure 1: The NTLM Family Protocols Working Model

 1. NEGOTIATE_MSG (Client to server): To begin the communication client sends the message
(Request) to the server to access the service. In this message the client specifies its supported
NTLM options to the server. The client shows its capabilities to the server whether it can use
NTLM family protocols or not. This message can contain different fields like domain names,
work station name version structure and payload (variable).
2. CHALLENGE_MSG (Server to Client): This message is send by the server to the client to
challenge the client for proving its identity. This message is generated by server in response to
the client’s negotiate message. This message can contain the fields like message type, target
name, server challenge (8 bytes), target information, version structure and payload.
3. AUTHENTICATE _MSG (Client to Server): This message is sent from client to server in
response to the challenge given by the server. This contains fields like message type, challenge
response (8 bytes), domain name, user name, work station, encrypted random session key,
version structure, message integrity and payload.
NTLM v1
It works on the basic model of NTLM family explained above. The server and the client share a
secret key. When the communication starts, the client requests the server through a negotiate message.
The server sends an 8 byte random number as challenge (CH) to the client in response to its negotiate
message (NEG_MSG). After receiving the 8 bytes random number the client compute the value using
challenge and secret key (one of the two password hashes). This computed value returned by the
client to the server as 24 bytes result (Response1, Response2). The server now verifies the client’s
identity using its response (client_response). The server checks whether the value commuted by the
client is correct or not. If the value is correct then the client is valid user and hence authenticated.
In NTLM v1 client uses both hashes (NT-hash and LM-hash) to compute the value and send both
results to the server in 24 bytes packet. The hashes produce 16 bytes quantities. To make these
quantities 21 bytes, five bytes of zeros are added at the end of 16 bytes quantities. The 21 bytes are
then divided into three seven bytes (56 bits) values (key1, key2, key3). These 56 bits values used as
secret keys for encryption by DES. By using these 64 bytes, challenge is encrypted and sent to the
server.
=8 � ℎ
� / � / � = −ℎ ℎ−5 � −
= � , / � , / � ,
� / � / � = −ℎ ℎ−5 � −
= � , / � , / � ,
� _ = /
NTLM v2
The working of NTLM v2 is slightly different from NTLM v1. It sends two 16 bytes
responses (client_response (LMv2 and NTv2)) to an 8 bytes server challenge (SCH (random
number)). Instead of DES it uses HMAC-MD5 algorithm to compute the value on the client machine.
The first response (LMv2) contains HMAC-MD5 hash of server challenge (SCH), a client challenge
(CCH (random number)), client’s password hashed by HMAC-MD5 (v2-hash) and other client related
information (identity information). The second response (NTv2) sent by NTLM v2 from client
machine uses client challenge of variable length (CCH*). The client challenge (CCH*) contains a) the
current time, b) 8 bytes random value (CCH), c) domain name and d) standard format values
(MSG_format_values).
=8 � ℎ
=8 � � ℎ
∗= _ _ , _� , , � _
−ℎ ℎ= � − 5 −ℎ ℎ, � _ , � _
= � − 5 −ℎ ℎ, ,
= � − 5 −ℎ ℎ, , ∗
� _ = / / / ∗

III. THE KERBEROS TICKET EXCHANGE MODEL

Kerberos came in existence to provide identity authentication between client and server. To fulfill
this purpose, the message exchange is done between client, authentication server and application
server. Initially, client presents its identity to the server using ticket, which contains principal,
authenticator, service principal. Issuing of ticket is catered by Key Distribution Center (KDC). The
KDC Stores the secret keys of clients and servers in the Database. These keys serve the client and
server for the authenticity of the tickets they received. A ticket is limited in its lifetime, which decides
its expiration.
The Kerberos ticket exchange model is shown in the figure 2. It consists of KDC which
contains:
1. Authentication Server (AS): It serves as authentication part of the Kerberos environment.

2. Ticket granting Server (TGS): It serves as service ticket provider for the Kerberos
environment.

3. Database (DB): It serves as storage of secret keys of clients and servers for the Kerberos
environment.
 Key Distribution Centre
AS_REQ
Authentication
Client
server
(Secret Key)
AS_REP
(TGT) DATA
TGS_REQ (TGT) BASE
Ticket (Client’s
Client
Granting secret
Server key)
TGS_REP
(Service ticket)

AP_REQ (Service ticket)

Application
Client Server
AP_REP (Optional)

Figure 2: Kerberos Ticket exchange model (Bhandari and Sharma 2013)

In Kerberos model client communicates with Authentication Server (AS_REQ, AS_REP) and
Ticket Granting Server (TGS_REQ, TGS_REP) to get authenticated plus service ticket ( �, )
respectively. In the end using the �, , application server (AP_REQ, AP_REP) gives access to
required service.

IV. KERBEROS WORKING

Client to KDC (AS_REQ): The client sends its credentials to get authenticated by authentication
server (AS), the part of Key Distribution Center (KDC) environment.
KDC to Client (AS_REP): AS replies to the request of the client with TGT and session key. TGT is
encrypted using the TGS secret key and session key encrypted using user’s secret key.
Client to KDC (TGS_REQ): Client sends request to TGS for service ticket. It contains TGT from the
previous message and authenticator encrypted with the help of session key.
KDC to Client (TGS_REP): The TGS replies to the request of the client. It contains the service ticket
encrypted using secret key of the service and session key generated by TGS and encrypts same using
session key generated in previous message by AS.
Client to Application Server (AP_REQ): In this client sends a request to access the service it
requires. The request contains the service ticket, service name and the authenticator generated by
client encrypted by service session key generated by TGS.
Application Server to Client (AP_REP): It’s an optional reply to the client to prove authenticity of
application server. It is generated when mutual authentication is required.
 TICKET GENERATION PROCESS

TGT
1. � → : , ,
2. →{ �, } ∶ { , � � , �, }
3. → � :{ �, , } � , { �, }
4. � ℎ � �� = { , � } �,

5. � → : {�� } �, ,{ �, }
(In version 4, message 3 is { �, , ,{ �, } } � } ) (Kohl et.al. 1994)

KDC (2)

(3) (1)

(5) Authentication
Client (4) Server

Figure 3: Process of getting TGT (Bhandari and Sharma 2013)

TICKET GRANTING TICKET

Initial Ticket Exchange

Step1: The client sends the request to the KDC, containing credentials for identifying itself to the
authentication server. It presents name of the client, service and a non-repeating identifier.

� → : , ,
Step 2: In this KDC generates a ticket containing random encryption key called session key ( �, ).

→{ �, } ∶ { , � � , �, }
Step 3: KDC uses the shared secret key (which is shared between the KDC and the Application
Server) to encrypt session key ( �, ) and the lifetime of TGT.

→ � :{ �, , } � , { �, }
Step 4: In this client generates the authenticator (�� ) which is encrypted using session key ( �, ) and
contains timestamp and user (C) and present TGT and the authenticator (�� ) to the server.

� ℎ � �� = { , � } �,
Step 5: Server extracts the credentials of client and session key ( �, ) by decrypting ticket using
secret key ( ) shared between KDC and server.

� → : {�� } �, ,{ �, }
Step 6: The Authentication Server uses session key ( �, ) to decrypt the authenticator, which verifies
the timestamp which proves that the client possesses the session key ( �, ).
Co-located
KDC (2)
--------------
TGS (6)
(5)
(3) (1)

(7)
Application
Client (4) Server
(8)
Figure 4: Process of getting ticket from TGS (Bhandari and Sharma 2013)

TGS
1. � → : , ,
2. = �, � = , , , � � , �, �

3. → � :{ �, � , } � , { �, � } �

4. � ℎ � = �� = { , � } �, �

5. � → : �� , { �, � } � , ,
6. �, = � ���� = , , , �� , � � , �,

7. → � :{ �, , } �, � ,{ �, }
8. � → : { �� } �, ,{ �, }
(In version 4, message 3 is �, � , ,{ �, � } � } �,

and message 5 is { �, , ,{ �, } } �, � ) (Kohl et.al. 1994)

V. TICKET GRANTING SERVER

The service ticket generation

In the initial ticket exchange the client obtain the ticket to start communication with the TGS.
TGS is a special server which decreases the risk of exposure of the client’s secret key K c. TGS also
make the KDC environment transparent to the client (user). As soon as the client received the TGT
to communicate with the TGS it deletes its private (secret) key.
We can differentiate the TGS logically from KDC. But TGS take the help of the KDC database and
run on the same machine on which the KDC runs.
Step 1: [Client to KDC (TGS)]. The client starts communicating with TGS by providing the KDC
Host, its name (identity or Principal, C), timestamp (n) and the TGS address (TGSs).
� → : , ,
Step 2: to simplify the operation the TGT is defined as below. The TGT contains the username
(principal, C), service name (S), timestamp (n), lifetime and the session key of the client and TGS
( �, � ).
= �, � = , , , � � , �, �

Step 3: [KDC (TGS) to Client]. The KDC (TGS) replies to the client the session key ( �, � ) and
timestamp (n) encrypted with the secret key of the client ( � ). It also includes the TGT encrypted
with the secret key of TGS.
→ � :{ �, � , } � , { �, � } �

Step 4: Now the client has to include the authenticator who can verify the identity of the client to
TGS. The authenticator provides the information of the client i.e. the username, timestamp. It is
encrypted with the session key of the client and the TGS.
� ℎ � = �� = { , � } �, �

Step 5: [Client to TGS]. The client sends the authenticator (�� ), TGT, name of service (S) and
timestamp to the TGS.
� → : �� , { �, � } � , ,
Step 6: Now as the TGS received the request from the client that what service it needs. The TGS has
to verify the identity of the client with the help of the TGT, Authenticator, and the database of the
KDC. TGS decrypts the authenticator and the TGT and check whether the client is the genuine user
who can use the specified service or not. As the client get authenticated, the TGS generates the
service ticket ( �, or (T e vice ). The ticket contains the username, service name, timestamp, IPlist,
session key ( �, ), lifetime of the ticket. The client cannot decrypt this ticket because this is
encrypted with the secret key which is shared between the KDC environment and the application
server.
�, = � ���� = , , , �� , � � , �,

Step 7: [TGS to Client]. As the ticket has been generated by TGS it sends this ticket to the client
encrypted with the secret key shared between the application server and the TGS. It also sends the
information of the client’s session key with TGS.
→ � :{ �, , } �, � ,{ �, }
Step 8: [Client to Application Server]. The client now starts communicating with the application
server. The client sends the authenticator and the ticket to the application server. The client and the
server now verify each other’s identity to themselves.
 � → : { �� } �, ,{ �, }
Step 9: Now the communication has been started between the client and the application server. To
start exchanging messages the client and the server share a common session key ( �, )) to encrypt
these messages which are sent over the network.

VI. COMPARISON OF PROTOCOLS

The following Table presents the comparison of above discussed authentication protocols.
Table 1: Comparison of Windows Authentication Protocols
Trusted Microsoft
Cryptographic Security Message Ease to Client/Server
Protocols Third Supported
Technique Level Type Use Challenge
Party Platforms
Win
95,98,ME,
NTLM Symmetric Key Random Domain NT 4.0,
Less Easy Server Only
v1 Cryptography number Controller 2000,XP,
2003/R2,
Vista
Win

More MD4
95,98,ME,
NTLM Symmetric Key Than hash & Domain NT 4.0, Both Client &
Easy
v2 Cryptography NTLM Random Controller 2000,XP, server
v1 number
2003/R2,
Vista
Win2K,
Symmetric Key More Domain
Encrypted
Cryptography than Controller XP,
Kerberos Ticket Both Client &
and NTLM Complex & Key Windows
v4 Using server
Asymmetric v1 and Distribution 2003/R2,
DES
Cryptography v2 Center
Vista
Win2K,
Symmetric Key Domain
More Encrypted
Cryptography Controller XP,
Kerberos than Ticket Both Client &
and Complex & Key Windows
v5 Kerberos Using server
Asymmetric Distribution 2003/R2,
v4 MD5
Cryptography Center
Vista,7,8
 VII. CONCLUSION

In this paper the working of two Windows authentication protocols has been presented and in
last protocols had been compared on the basis of different parameters. The Kerberos v5 proves to be
better than other protocols due to its enhanced security measures and techniques. There are still some
flaws in it. Work is going on to rectify them. In our next publication we will present our test analysis
on these protocols.

VIII. REFERENCES

Neuman, B. C. and T’so, T. Y. 1994. “Kerberos: An Authentication Service for Computer

Networks”, IEEE Communications, 32(9):33-38.
Kohl, J. T.; Neuman, B. C. and T’so, T. Y. 1994. “The Evolution of the Kerberos Authentication
System”. In Distributed Open Systems, IEEE Computer Society Press, pages 78-94.
Yu, T.; Hartman, S. and Raeburn, K. 2004. "The Perils of Unauthenticated Encryption: Kerberos
Version 4”. In Proceedings of the Network and Distributed System Security Symposium. The Internet
Society.
Bhandari, R. and Sharma, S. 2013. “Kerberos: Simplified Ticketing”, IJARCSSE, 3(11), pp. 647-650.

Kerberos Papers and Documentation. 2004 <http://web.mit.edu/kerberos/papers.html>.[20

November 2013]

Kerberos Protocol Tutorial, 2007. <http://kerberos.org/software/tutorial.html>.[25 November 2013]

NT LAN Manager, 2013. <en.wikipedia.org/wiki/NT_LAN_Manager>.[30 November 2013]

Microsoft NTLM, 2013. <http://msdn.microsoft.com/en-us/library>.[3 December 2013]

Authentication in Windows Server 2008 R2 and Windows 7, 2010.

<http://windowsitpro.com/security>. [8 December 2013]

View publication stats