Architected for Performance

NVM ExpressTM Management Interface

August 11, 2015

John Carroll, Storage Architect, Intel

Peter Onufryk, Storage Director, Product Development, PMC-Sierra
Austin Bolen, Storage Technologist, Dell
Agenda
• NVMe Management Interface Overview
– Definition
– Comparison to NVM Express Specification interface
– Benefits over in-band management
– To standardize or not to standardize
• NVMe-MI Usage
– A real world example – Automated Remote Health Monitoring

• NVMe-MI Architecture
– NVM Subsystem, Port, Management Endpoint, Command Slot
• Overview of Features/Functionality
– NVMe Management Commands
– NVMe Admin Commands
– PCIe Commands
– Control Primitives
– VPD
• Standardization Status

2
NVMe Management Interface

What is the NVMe Management Interface?

 A programming interface that allows out-of-band management of an NVMe Field

Replaceable Unit (FRU) or an embedded NVMe NVM Subsystem

3
Field Replaceable Unit (FRU)

FRU definition (Wikipedia):

 A circuit board, part or assembly that can be
quickly and easily removed from a computer
or other piece of electronic equipment, and
replaced by the user or a technician without
having to send the entire product or system
to a repair facility.

4
Management Fundamentals
What is meant by “management”?

Four pillars of systems management: Management operational times:

• Inventory • Deployment (No OS)
• Configuration • Pre-OS (e.g. UEFI/BIOS)
• Monitoring • Runtime
• Change Management • Auxiliary Power
• Decommissioning

5
In-Band vs Out-of-Band Management
Host Processor
Host Operating System
NVMe Driver
Management Controller
• NVMe driver communicates
(BMC)
BMC Operating
Operating System
System to NVMe controllers over
BMC
PCIe per NVMe Spec
NVMe-MI Driver

• MC runs on its own OS on it

PCIe Root PCIe Root PCIe Root own processor independent
SMBus/I2C
Port Port Port from host OS and driver
PCIe PCIe VDM SMBus/I2C • Two OOB paths: PCIe VDM
Bus PCIe Bus and SMBus
PCIe Port SMBus/I2C
• PCIe VDMs are completely
NVM Subsystem separate from in-band PCIe
PCIe SSD traffic though they share the
same physical connection

6
In-band vs Out-of-Band Management Cont.
In-Band Management Application
• Many host OSes to support (Windows, Linux, VMWare, etc.)
• Several different flavors/distros of each
• New revisions of OS and NVMe driver released over time
• Developing and maintaining a management application for every OS variant is resource/cost
prohibitive
• Management features vary per OS

Out-of-Band Management Application

• Develop management application in one operating environment
• Works the same across any host OS the user installs
• Works across no OS cases (pre-boot, deployment)

7
Why Standardize NVMe Storage Device Management?

Reduces Cost and Broadens Adoption

 Allows OEMs to source storage devices from multiple suppliers
 Eliminates need for NVMe storage device suppliers to develop custom OEM specific
management features

Consistent Feature Set

 All storage devices that implement management implement a common baseline feature set
 Optional features are implemented in a consistent manner

Industry Ecosystem
 Compliance tests / program
 Development tools

8
A Real World Example – Automated Remote Health Monitoring
The Problem:
• Datacenter with hundreds of servers
• Each servers consists of dozens of Field Replaceable Units
• Some number of FRUs fail weekly (or even daily)
• Manually discovering and resolving issues due to failed FRUs is prohibitively time
consuming and expensive

The Solution:
• Each server has a BMC to manage all FRUs
• Each BMC is connected to a network accessible via a remote management console
• BMC detects NVMe FRU failures using NVMe-MI and reports failures to a remote
administrator

9
Remote Health Monitoring – Management Infrastructure
Server
Server Management
Controller

Server Management
Controller
Network
Server Management Controller

Server Management Controller

Management
Power Memory Host Controller
NVMe
NICs
Supplies DIMMs Processors SSDs
Power Memory Host NVMe
NICs
Supplies DIMMs Processors SSDs
Power Memory Host NVMe
NICs
Supplies DIMMs Processors SSDs
NICs
Power Memory Host NVMe Remote Mangement
Supplies DIMMs Processors SSDs
Power Memory Host NVMe
Console
NICs
Supplies DIMMs Processors SSDs

10
Remote Health Monitoring – Set up Alerts

Enable e-mail alerts for NVMe health events 11

Remote Health Monitoring – Detect Error Using NVMe-MI

• Management Controller issues NVM

Server Subsystem Health Status Poll
Server Management
Controller
command to NVMe drive
Server Management
Controller

Server Management Controller

Network • NVMe drive responds indicating a
Server Management Controller
Critical Warning bit is set
Management
Controller
NICs
Power
Supplies
Memory
DIMMs
Host
Processors
NVMe
SSDs • Management Controller then issues a
Power Memory Host NVMe
NICs
Supplies DIMMs Processors SSDs Controller Health Status Poll
Power Memory Host NVMe
NICs
Supplies
Power
DIMMs Processors SSDs command to the drive
Memory Host NVMe
NICs
Supplies DIMMs Processors SSDs
NICs
Power
Supplies
Memory
DIMMs
Host
Processors
NVMe
SSDs • NVMe drive responds indicating a
Reliability Degraded error occurred
Remote Mangement
Console
• Management Controller sends email
notification

12
Remote Health Monitoring – Receive E-Mail Alert

E-mail alert sent if a drive health event occurs

13
Remote Health Monitoring – Check Event Log

Check event log on remote system 14

Remote Health Monitoring – Drives Overview

Review drive table to see status of all drives in the system 15

Remote Health Monitoring – Drive Detail

Expand a drive’s view to see its details 16

Management Controller GUI – Export Log Files

Export logs from the drive 17

Remote Health Monitoring – Check Log File

Check the log for failure details 18

Remote Health Monitoring – Blink Drive LED

Admin blinks the indicator LED for the failed drive

Local Datacenter Technician finds and replaces faulty drive
19
Agenda
• NVMe Management Interface Overview
– Definition
– Comparison to NVM Express Specification interface
– Benefits over in-band management
– To standardize or not to standardize
• NVMe-MI Usage
– A real world example – Automated Remote Health Monitoring

• NVMe-MI Architecture
– NVM Subsystem, Port, Management Endpoint, Command Slot
• Overview of Features/Functionality
– NVMe Management Commands
– NVMe Admin Commands
– PCIe Commands
– Control Primitives
– VPD
• Standardization Status

20
NVMe Architecture (review)

NVM Subsystem - one or more controllers, one or more namespaces, one or

more PCI Express ports, a non-volatile memory storage medium, and an
interface between the controller(s) and non-volatile memory storage medium
PCIe Port PCIe Port x PCIe Port y

NVM Subsystem NVM Subsystem

PCI Function 0
NVMe Controller
PCI Function 0
NVMe Controller
NVMe Controller
One Controller/Port PCI Function 0 Two Controllers/Ports NSID 1 NSID 2 NSID 3 NSID 2

NS NS
NSID 1 NSID 2 A C

NS NS
A B NS
B

21
NVMe Field Replaceable Units with NVMe-MI
PCIe PCIe PCIe
Port Port 0 Port 1

NVMe Controller Management NVMe Controller Management NVMe Controller NVMe Controller Management
PCI Function 0 Endpoint PCI Function 0 Endpoint PCI Function 1 PCI Function 0 Endpoint
Management
SMBus/I2C
Controller Controller Controller Controller Endpoint
Management Interface Management Interface Management Interface Management Interface

NVM Subsystem NVM Subsystem

An NVMe FRU consists of one and only one NVM Subsystem with
 One or more PCIe ports
 An optional SMBus/I2C interface
 One or more Management Endpoints

22
VPD – Vital Product Data
Vital Product Data typically available in a serial EEPROM

PCIe PCIe
NVMe-MI defined standard VPD contents including: Port 0 Port 1

• Device Form factor

SMBus/I2C
• Initial and peak power usage by power rail NVM Subsystem

Serial

• RefClk/SRIS capability PCIe SSD

EEPROM

• and more …

NVMe-MI makes VPD contents accessible out-of-band

23
NVMe-MI Defines the Protocol for Managing NVMe

Leverage existing PCIe and SMBus

Management Applications (e.g., Remote Console)
Application
Management Controller Layer
(BMC or Host Processor)
MCTP defines the transport layer
Protocol
NVMe Management Interface
• Refer Management
to http://dmtf.org/ for Management
Layer
Applications (e.g., Applications (e.g.,
more info on MCTP Remote Console) Management Component Transport Protocol (MCTP) Remote Console)
Transport
MCTP over MCTP over Layer
SMBus/I2C Binding PCIe Binding
NVMe-MI is the protocol for
Physical
applications to information SMBus/I2C PCIe VDM
Layer

24
Types of MCTP Messages
NVMe-MI Message Other MCTP
Messages
Request Message Response (e.g., MCTP
Message control)
Command Message
NVMe-MI PCIe Control Success
Command Command Primitive
NVMe Admin
Command Error

25
Command Slots

• Each Management Endpoint has two Command Slots to service Command Messages
• Each Command Slot follows this state machine

Response Message
Transmitted Idle Start of Message
or
Abort
Abort or
Error

Transmit Receive
More
Processing Abort
Required
Response
Required
Command Message
or Process
Received
Resume

26
Command Slots

• Each Management Endpoint has two Command Slots to service Command Messages
• Each Command Slot follows this state machine

Response Message
Transmitted Idle Start of Message
or
Abort
Abort or
Error

Transmit Receive
More
Processing Abort
Required
Response
Required
Command Message
or Process
Received
Resume

27
Management Interface Command Set
Command O/M
• Discover Device Capabilities
Configuration Set Mandatory
Configuration Get Mandatory
• Monitor Health Status Controller Health Status Poll Mandatory
NVM Subsystem Health Status Poll Mandatory
Read NVMe-MI Data Structure Mandatory
• Modify Configuration
Reset Mandatory
VPD Read Mandatory
VPD Write Mandatory
Vendor Specific Optional

28
NVMe Admin Commands
Command O/M
Get Features Mandatory
• NVMe-MI defines mechanism to Get Log Page Mandatory
send existing NVMe Admin
Identify Mandatory
Commands out-of-band
Firmware Activate/Commit Optional
Firmware Image Download Optional
• Admin Commands target a Format NVM Optional
controller in the NVM subsystem Namespace Management Optional
Security Send Optional
Security Receive Optional
Set Features Optional
Vendor Specific Optional

29
PCIe Commands

• PCIe Commands provide optional Command O/M

functionality to read and modify
PCIe Configuration Read Optional
PCIe memory
PCIe Configuration Write Optional
PCIe Memory Read Optional
PCIe Memory Write Optional
PCIe I/O Read Optional
PCIe I/O Write Optional

30
Control Primitives

• Control Primitives enable a Management

Control Primitive O/M
Controller to detect and recover from
errors Pause Mandatory
Resume Mandatory
Abort Mandatory
• Control Primitives fit into a single packet
and do not require message assembly Get State Mandatory
Replay Mandatory

31
Summary

NVMe-MI standardizes out of band management to discover and configure NVMe

devices

NVMe-MI 1.0 specification under member review – will be published on NVMe site after
ratification

Join NVMe to shape the future of NVMe-MI

32
Architected for Performance