Professional Documents
Culture Documents
Gitprint Snowch HSM Guide Blob Master Book - MD
Gitprint Snowch HSM Guide Blob Master Book - MD
Guide
An introduction to HSMs.
Authors
Chris Snow
Colin Cook
Nick Bitounis
Juris Lacis
Zeeshan Ahmad
Other contributors ...
Book License
CC0 1.0 Universal
Preface
About this book
A primary role of a HSM is the secure management of digital keys. This book introduces digital key
management concepts and reinforces those concepts with exercises that the reader can perform on an open
source Thales HSM Simulator.
The reader should have a basic understanding of symmetric and asymmetric cryptography. Appendix A,
Introductory Books in Cryptography provides a list of introductory books in cryptography for those wishing to
learn the basics or just wanting to refresh their knowledge in the field of cryptography.
This book is developed by the community - please send me your contributions.
The source of this book is available here (https://github.com/snowch/hsm-guide/blob/master/book.md)
You can view the print version of the book here (https://gitprint.com/snowch/hsm-guide/blob/master/book.md)
Audience
Undergraduate and graduate students should find that this book supplements their studies in the theoretical
concepts of cryptography with practical applications.
Software Engineers and Architects designing and building security solutions using the Thales brand of HSM
will learn concepts and patterns of key management that can be applied to their designs
Book Organisation
Chapter 1, Introduction describes the role fulfilled by a HSM. The Thales series of HSM's are then introduced
1 of 22
with a short history of their evolution. Finally, an open source Thales Simulator project is presented, with
hands on exercises for the reader to install the Simulator and then connect to it from Java, C#, and ruby
clients.
Chapter 2, Theory covers the cryptography basics, different types of cryptographic functions and algorithms.
That chapter includes the basic information of encryption and message authentication functions used in
cryptography. TODO finish about Chapter 2
Chapter 3, HSM Local Master Keys (LMKs) covers in detail the concept of the LMKs. The knowledge gained in
this chapter is fundamental to your understanding of the Thales HSM. Almost all other chapters depend on
you understanding the material covered in this chapter. This chapter concludes with exercise with the Thales
Simulator to help instill the concepts of LMKs.
Chapter 4, Key Concepts describes general key concepts that you need to know in addition to the material
covered in the previous chapter about LMKs. The knowledge in this chapter is of vital importance when
interacting with the Thales HSM. Exercises are provided with the Thales Simulator to put the concepts learnt
in this chapter into practise.
Chapter 5, Secure Key Exchange two sites that are secured with HSMs need to have a set of keys that are
shared between the HSMs. This chapter describes how keys are created and shared. Exercises are given to
setup two demo sites with the Thales Simulator and generate and share keys between the demo sites.
Chapter 6, Dynamic Key Exchange
TODO describe other chapters
Introduction
Overview
A primary role of a HSM is the secure management of digital keys. This document describes digital key
management concepts, and also describes some key management patterns for solving specific problems.
Other acronyms for Hardware Security Modules include:
HSM : Hardware Security Module / Host Security Module
TRSM : Tamper Resistant Security Module
SCD : Secure Cryptographic Devices
TODO: Why digital key management? Life without a HSM? TODO: Thales Simulator project overview.
Console Commands
2 of 22
Host Commands
Host commands fall into the categories:
TODO
Theory
TODO introduction to this section
Cryptographic functions
Cryptographic functions fall into two main classes called encryption and message authentication functions.
Encryption functions are used to transmit data securely. Encryption functions are two way functions and have
both encryption and decryption algorithms. The sender encrypts the message to be sent with the encryption
algorithm and the cryptographic key. After receiving the encrypted data the recipient uses a decryption
algorithm and cryptographic key to decrypt the encrypted data received:
Authentication functions are used to ensure that the message was not modified during transmission. The
message authentication function result is not reversable. The sender calculates a check sum of the message
and sends it together with plaintext or encrypted message. The sender, using the same message
authentication algorithm can verify the check summ received together with the message. If the check sum is
valid, the receiver can be sure that the message was not modified during transmission:
Encryption functions
Encryption functions are divided into two classes, symmetric and asymmetric.
Symmetic encryption algorithms uses the same key for both encryption of plaintext data and decryption of
ciphertext. Usually, the decryption algorithm is completely reversed encryption algorithm. The symmetric
encryption can use stream ciphers or block ciphers. This document covers only block cipher encryption.
Block cipher algorithms operate with fixed length blocks of bits on input, if the plain text message can not be
split into blocks of the required length before encryption, it has to be padded with additional bytes to meet the
required length.
Some of the most commonly used symmetric algorithms are:
Data Encryption Standard (DES / 3DES)
3 of 22
Asymmetric, also known as public-key algorithms use different keys (key pairs) for data encryption and
decryption, these are called the private key and public key. Both the private and public keys are mathematicaly
linked. The asymmetric encryption and decryption algorithms are also different. Before communication the
receiver generates the pair of keys and keeps private key secret. The public key is distributed with other parties
involved in data exchange. The public key used to encrypt data can't be used to decrypt it. Data can only be
decrypted with the private key.
Some of the most commonly used asymmetric algorithms are:
Rivest Shamir Adleman (RSA)
Diffie-Hellman
Digital Signature Algorithm (DSA)
ElGamal
Elyptic Curve Cipher (ECC)
TODO - brief paragraph to say when to use symmetric and when to use assymetric?
MAC functions also called keyed hash functions are cryptographic based message authentication functions to
produce the short output check sum over the message used to ensure, that the message was not modified
during transmission. In contrast to hash functions, MACs are generated using cryptographic algorithms and
secret keys. The MAC can be securely applied to plain text messages without message encryption because it
uses cryptographic keys to produce MAC.
The most commonly used MAC algorithms are defined in ISO-9797 standard.
TODO - can we list some of the algorithms?
TODO - brief paragraph to say when to use hash and when to use mac?
4 of 22
Padding
TODO - Introduction to paddings
5 of 22
message meets length requirements. Method One padding can not be used for message encryption because it
is impossible to differentiate padding bytes from the message content.
6 of 22
i n t padIdx = 0;
f o r ( i n t i = paddedMsg. length - 1; i > 0; i- - ) {
i f ( paddedMsg[ i] = = ( b y t e ) 0x80) {
padIdx = i;
}
}
b y t e [ ] message = n e w b y t e [ paddedMsg. length - padIdx] ;
System. arraycopy( paddedMsg, 0, message, 0, message. length) ;
r e t u r n message;
}
When the ciphertext is decrypted, the padding structure must be validated. In a case of structure mismatch the
error should be returned. The conditions of valid padding are:
1st byte must be 0x00;
2nd byte must be 0x02;
Encrypted message must have a byte with value 0x00 separating random byte octet from message;
Random byte octet must be at least 8 non-zero bytes long.
The example of PKCS#1 v1.5 padding on Java:
/*
* The example below adds PKCS#1 padding of length to make message the same
* size as pubic key.
*/
p u b l i c s t a t i c b y t e [ ] p k c s 1 P a d ( b y t e [ ] message, i n t keyLen) {
java. util. Random r = n e w java. util. Random( ) ;
i n t padLen = keyLen - message. length - 3;
b y t e [ ] rand = n e w b y t e [ padLen] ;
b y t e [ ] paddedMsg = n e w b y t e [ message. length + padLen + 3] ;
paddedMsg[ 0] = ( b y t e ) 0x00;
paddedMsg[ 1] = ( b y t e ) 0x02;
paddedMsg[ padLen + 2] = ( b y t e ) 0x00;
i n t n = 0;
f o r ( i n t i = 0; i < padLen; i+ + ) {
n = r. nextInt( 256) ;
i f ( n = = 0) {
i- - ;
c o n t i n u e ;
}
rand[ i] = ( b y t e ) n;
}
System. arraycopy( rand, 0, paddedMsg, 2, rand. length) ;
7 of 22
Exercises
Asymmetric cipher paddings
1) To verify the PKCS#1 v1.5 padding the following Java code can be used:
i m p o r t java.security.KeyPair;
i m p o r t java.security.KeyPairGenerator;
i m p o r t java.security.PrivateKey;
i m p o r t java.security.PublicKey;
i m p o r t javax.crypto.Cipher;
p u b l i c c l a s s R S A P a d d i n g T e s t {
p r i v a t e s t a t i c String b y t e 2 h e x ( b y t e bs[ ] ) {
// refer to Appendix A for that method code
}
/*
* Method generates the private and public RSA key pair
*/
p r i v a t e s t a t i c KeyPair g e n e r a t e K e y P a i r ( i n t keyLen) t h r o w s Exception {
KeyPairGenerator kpg = KeyPairGenerator. getInstance( "RSA") ;
kpg. initialize( keyLen) ;
KeyPair kp = kpg. genKeyPair( ) ;
r e t u r n kp;
}
/*
* Method encrypts passed message with RSA public key applying
8 of 22
9 of 22
2. Unzip the downloaded file and execute the file ThalesWinSimulatorSetup.msi, accepting the default
options.
Starting the Thales Simulator
1. Navigate to the folder where you installed the Simulator (E.g. C:\Program Files (x86)\NTG\Thales
Simulator\)
2. Execute ThalesWinSimulator.exe (if your are running Windows 7, right click the file and select Run As
Administrator)
3. Click the Start Simulator button:
4. In the Application Events window, the simulator will inform you that it could not find a file containing
the LMK keys so it will create a new set of keys for you. The Simulator will always create the same keys.
10 of 22
11 of 22
ND the response code. The response from HSM always is command code with incremented second letter
00 error code, 00 means that no errors occured during command processing
7B44AC1DDEE2A94B Local Master Key (see corresponding chapter) check value
0007-E000 means the HSM firmware revision number
Each command has its own response specification, see "Host command reference manual" for more details.
LMK Overview
Local Master Keys are a sets of 40 DES keys. They are stored securely in the HSM making it very difficult for an
attacker to gain access to them. LMKs are the only keys that are stored in the HSM.
LMKs are not used for encrypting data, but are instead used to encrypt and decrypt other keys as these enter or
leave the HSM. LMKs are used to ensure that even if the data traffic between the HSM and an application is
recorded, the clear values of any exchanged keys are not compromised.
LMKs come in pairs and the Thales HSM contains several LMK pairs. Different LMK pairs are used to
encrypt/decrypt different types of security keys. LMK pairs are identified by two numbers, for example LMK
pair 04-05, LMK pair 14-15, etc. See the diagram below.
Each LMK pair is assigned a code. LMK pair 04-05 is assigned code 00, while LMK pair 14-15 is assigned code
02. The full list of HSM key pairs are listed in Table 2.1, LMK Key Pairs, below. Note that HSM key pairs do
not start at 00-01, instead the numbering starts at 04-05, and runs non-contiguously to 38-39.
Key Pair
Code
04-05
00
06-07
01
14-15
02
16-17
03
18-19
04
20-21
05
22-23
06
24-25
07
26-27
08
28-26
09
30-31
10
12 of 22
Key Pair
Code
32-33
11
34-35
12
36-37
13
38-39
14
Each HSM has a unique set of LMK pairs that can be either randomly generated or loaded from smart cards.
HSM users jealously guard the LMKs because the integrity of the key management security scheme depends
upon them.
LMK Variants
Back when the HSM had only a handful of LMK pairs, more than the type of keys that had to be encrypted, a
way had to be found to ensure that different key types can be used but also provide a way to identify parity
errors with these key types. Variants are an easy way to pseudo-multiply your LMK pairs. (TODO validate this)
Keys are encrypted under LMK pairs using either the clear value of the LMK or a variant of the LMK. An LMK
variant is created by performing a XOR operation with a value on the LMK key. For example, variant 1 of an
LMK is created by XORing the LMK with the value 000000000000000000000000000000A6. The Thales HSM
supports 10 variants for each LMK pair, with variant 0 being the clear LMK itself. The full list of variant
calculation functions can be seen in the Table below.
Variant number
000000000000000000000000000000A6
0000000000000000000000000000005A
0000000000000000000000000000006A
000000000000000000000000000000DE
0000000000000000000000000000002B
00000000000000000000000000000050
00000000000000000000000000000074
0000000000000000000000000000009C
000000000000000000000000000000FA
Exercises
TODO - what exercises could be performed on the Simulator around LMK Concepts? Could we walk through
the default keys with the HSM simulator and how to change those keys?
13 of 22
Key Concepts
Key Types
Each different key supported by the Thales HSM has a unique code called the key type code. The key type is a
three-digit number and is made up from the code of the LMK pair and the LMK variant. Therefore, keys
encrypted under LMK pair 04-05 using a variant 1 will have a key type equal to 00 + 1 = 001. It is important
to understand key types since several Thales commands expect key type codes as parameters. The full list of
key types can be seen in the table below.
LMK Key Pair
LMK Code
Variant
Key Type
04-05
00
000
ZMK
06-07
01
001
ZPK
14-15
02
002
14-15
02
402
CVK, CSCK
16-17
03
003
TAK
22-23
06
006
WWK
26-27
08
008
ZAK
28-29
09
009
BDK1
28-29
09
109
MK-AC
28-29
09
209
MK-SMI
28-29
09
309
MK-SMC
28-29
09
409
MK-DAC
28-29
09
509
MK-DN
28-29
09
609
BDK2
28-29
09
709
MK-CVC3
28-29
09
809
BDK3
30-31
0A
00A
ZEK
32-33
0B
00B
DEK
32-33
0B
30B
TEK
In the latest payShield9000 and HSM8000 firmwares Thales has introduced second, PCI-HSM standard
compliant, key type table with the changes around 002 key type - PVK, TMK, TPK. The changes moves TMK
and TPK to different LMK pair and Variants leaving PVK the only key of 002 type:
LMK Key Pair
LMK Code
Variant
14 of 22
Key Type
LMK Code
Variant
Key Type
14-15
02
002
PVK
36-37
0D
70D
TPK
36-37
0D
80D
TMK
Key Formats
The Thales HSM uses two major formats when processing security keys. These are Variant and ANSI.
The variant concept is similar to the one used to form LMK variants: a key is XORed with a value to form the
key variant. Double-length keys are XORed with the value:
00000000000000A6 000000000000005A
and triple-length keys are XORed with the value:
00000000000000A6 000000000000005A 000000000000006A
TODO: how does the variant concept differ to the LMK variant concept?
The ANSI format is much simpler: a key is used as-is without performing any additional operations on it.
Key Scheme
Depending on their length and key format, keys are designated by a key scheme that helps to quickly identify
the nature of a key. Key schemes are the following:
Key Scheme
Description
15 of 22
Exercises
TODO - what exercises could be performed on the Simulator around LMK Concepts?
16 of 22
Exercises
Creating a Zone Master Key (ZMK)
In this exercise, we create a Zone Master Key (ZMK) using console commands.
The ZMK is distributed manually as components. To create the ZMK, we first create three ZMK components.
Generate the ZMK Components
Generate three ZMK components using the console command Generate Component (GC). Repeat the
command three times as shown below:
GC # User input
Key length [1,2,3]: 2 # User input
Key Type: 000 # User input
Key Scheme: U # User input
Clear Component: 79CD 2380 9B4F C1C4 7F9E FB2A DF2A 674A
Encrypted Component: U 1BA5 185A FCF1 5A1B 274B E1E0 03B4 7C2A
Key check value: 7A5B C7
GC # User input
Key length [1,2,3]: 2 # User input
Key Type: 000 # User input
Key Scheme: U # User input
Clear Component: 0157 B3DF 6116 3402 372C 54FD 62F2 1C91
Encrypted Component: U FCE4 7AF7 FFF8 9F40 2407 A35A F063 D3E1
Key check value: 1E79 CB
GC # User input
Key length [1,2,3]: 2 # User input
Key Type: 000 # User input
Key Scheme: U # User input
Clear Component: 7AEA B5A4 1A9E 9B68 EF80 494C 0819 4ADA
Encrypted Component: U EE8D 4F9E C8B2 ADF4 9CD2 F0D2 7F5C 95C5
Key check value: 277A 5F
Description
We are generating a Double-length key, so we choose a Key Length of 2. See Key Schemes for a list of the key
schemes.
We are generating ZMK keys, so we use the Key Type 000. See Key Types for a list of the key types.
We are generating a Double-length variant key, which has a Key Scheme of U. See Key Schemes for a list of the
key schemes.
TODO: refer to a section that describes how to decide which scheme and type of key is most suitable for your
particular encryption requirements? E.g. why have we chosen Double-length variant key and not Single-length
ANSI keys for this example?
Generate the ZMK from the components
This step uses the FK command to generate the ZMK from the three ZMK components previously generated:
TODO: describe the various command options and how to choose which option values to use.
FK # User input
Key length [1,2,3]: 2 # User input
Key Type: 000 # User input
Key Scheme: U # User input
17 of 22
18 of 22
Key Translation
In our previous example, the value C9A62E96ADFB52A7 815BE8D7E730B24E represents the randomly created
ZPK encrypted under the previously created ZMK. But imagine that one of the parties that have exchanged this
ZPK needs to transmit it to another party with which they share a different ZMK which well call ZMK2 . To
properly transmit the ZPK to the other zone that is secured with ZMK2, the ZPK has to be:
Decrypted under ZMK.
Encrypted under ZMK2.
This process is called key translation. Key translation does not happen with specific key types as the ZMK that
was used in the previous example but is a more general process - for example it is possible to translate a key
from encryption under the ZMK to encryption under an LMK.
Key translation always takes place within the HSM to avoid exposing the clear value of the key being
translated.
Exercises
Creating a Zone PIN Key (ZPK)
In this exercise, we create a Zone PIN Key (ZPK) using console commands.
When prompted for the ZMK, use the encrypted ZMK value from the section called Creating a Zone Master
Key (ZMK).
KG
Key length [1,2,3]: 2
Key Type: 001
Key Scheme (LMK): U
Key Scheme (ZMK) [ENTER FOR NONE]: X
Enter encrypted ZMK [ENTER FOR NONE]: U 104C 4216 A751 FEEE FF55 698B 26C5 7789
Enter ZMK check value [ENTER TO SKIP CV TEST]:
Key under LMK: U 8586 51EC 83AF CA66 8175 804F 5B7D CD6B
Key encrypted for transmission: X BAA5 18AA D10D 28A2 D32A 5688 317F 44EB
Key check value: 6543 F4
PIN translation
MACing
CVV/CVV2/iCVV
Appendix A - Commonly used code in examples
p r i v a t e s t a t i c String b y t e 2 h e x ( b y t e bs[ ] ) {
i n t i;
String s = n e w String( ) ;
String hex_digits = "0123456789ABCDEF";
b y t e c;
i f ( bs = = n u l l | | bs. length = = 0) {
r e t u r n s;
}
f o r ( i = 0; i < bs. length; + + i) {
c = bs[ i] ;
s + = hex_digits. charAt( ( c > > 4) & 0xf) ;
s + = hex_digits. charAt( c & 0xf) ;
}
r e t u r n s;
}
#!/usr/bin/env perl
u s e IO::Socket::INET;
m y $sock = n e w IO::Socket::INET(PeerAddr= > "localhost:9998") o r die;
$sock- > send(pack "H*","0006303030304e43");
$sock- > recv($data, 1024); p r i n t $data;
The command should output a response similar to the following:
! 0000ND007B44AC1DDEE2A94B0007- E000
The response from HSM can be broken down as follows:
! actualy is 0021, it is software header returned by HSM, actual response length
0000 is HSM response header which is set the same as for received command
ND the response code. The response from HSM always is command code with incremented second letter
00 error code, 00 means that no errors occured during command processing
7B44AC1DDEE2A94B Local Master Key (see corresponding chapter) check value
0007-E000 means the HSM firmware revision number
Each command has its own response specification, see "Host command reference manual" for more details.
Connecting with a Java Client
In this session, we connect to the HSM over TCP/IP using Java. When we connect using Java, we can send
Host Commands to the HSM.
In the code example, below, we send the command Perform Diagnostics (NC), and print the response to
System.out.
For a full treatment of Host Programming the Thales HSM, refer to the Thales documentation Host
Programmers Manual. For a full list of Host Commands, refer to the Thales documentation Host Command
Reference Manual
i m p o r t java.io.BufferedOutputStream;
i m p o r t java.io.InputStream;
i m p o r t java.io.OutputStream;
i m p o r t java.net.Socket;
i m p o r t java.nio.ByteBuffer;
p u b l i c c l a s s M a i n {
p u b l i c s t a t i c v o i d m a i n ( String[ ] args) t h r o w s Exception {
Socket socket = n e w Socket( "localhost", 9998) ;
String command = "0006303030304e43";
// the following line converts the hex command string to a byte array
b y t e [ ] bytes = ByteBuffer. allocate( 8) . putLong( Long. parseLong( command, 16) ) . array( ) ;
OutputStream out = socket. getOutputStream( ) ;
BufferedOutputStream bufferedOut = n e w BufferedOutputStream( out, 1024) ;
bufferedOut. write( bytes) ;
bufferedOut. flush( ) ;
InputStream in = socket. getInputStream( ) ;
i n t result;
w h i l e ( ( result = in. read( ) ) ! = - 1) {
System. out. print( ( c h a r ) result) ;
21 of 22
}
socket. close( ) ;
}
}
See the description from Connecting with a perl client (/book.md#connecting-with-a-perl-client) for the format of the
command: 0006303030304e43.
Connecting with a Unix client
If you have netcat (nc) installed, you can run a command by using echo convert a hex string to binary and send
it to the Thales simulator using netcat.
Here we send the Thales command NC which asks the Thales Simulator to Perform Diagnostics and return the
result:
$ echo - ne '\x00\x06\x30\x30\x30\x30\x4e\x43' | nc localhost 9998
! 0000ND007B44AC1DDEE2A94B0007- E000
See the description from Connecting with a perl client (/book.md#connecting-with-a-perl-client) for the format of the
command: 0006303030304e43.
Connecting with a PL/SQL client
If you want to call HSM using PL/SQL you can use the UTL_TCP package as in example procedure below:
C R E A T E O R R E P L A C E P R O C E D U R E call_hsm I S
v_to_hsm VARCHAR2(255);
n_length NUMBER(2);
v_from_hsm VARCHAR2(255);
rw_from_hsm RAW(100);
c utl_tcp.c o n n e c t i o n ;
B E G I N
c := utl_tcp.open_connection('192.168.229.1', '9998', N U L L , N U L L , N U L L , N U L L , N U L L , N U
L L , 1);
v_to_hsm := '0000' | | 'NC';
v_to_hsm := CHR(0) | | CHR(L E N G T H (v_to_hsm)) | | v_to_hsm;
v_from_hsm := utl_tcp.write_text(c , v_to_hsm);
n_length := utl_tcp.available(c , 1);
n_length := utl_tcp.read_raw(c , rw_from_hsm, 100);
S E L E C T SUBSTR(C A S T (rw_from_hsm A S VARCHAR2(255)), 5)
I N T O v_from_hsm
F R O M dual;
v_from_hsm := utl_i18n.raw_to_char(v_from_hsm);
dbms_output.put_line(v_from_hsm);
utl_tcp.close_connection(c );
E N D ;
22 of 22