Add NVMe support to the smartmontools

[Alex Samorukov]

About

It would be great to add NVMe support to the smartctl. This task is not very trivial - because smart in NVMe is implemented differently then in SCSI/SAS or SATA. This ticket will be used to track changes in smartmontools and to collect important related links.

NVMe standard and vendor specific log pages

• Official specifications: ​http://nvmexpress.org/specifications/
• ​Intel NVMe documentation with Intel-specific log pages (CAh, C5h, DDh, C1h/C2h).
• ​OCZ Z-Drive specification, with vendor-specific smart log page (E0h)

Hardware and emulation

• ​QEMU fork which can emulates NVMe device with most of the features supported, including SMART log pages, namespaces, etc. Tested with Linux and FreeBSD, full list of the options could be found in the ​source code.

Linux NVMe support

• NVMe kernel header ​linux/nvme.h
• ​nvme-cli - NVM-Express user space tooling for Linux
• ​nvme-user - another opensource userspace utility

FreeBSD NVMe support

• NVMe kernel header ​dev/nvme/nvme.h
• ​nvmecontrol utility, ​opensource.

Windows NVMe support

• ​OpenFabrics NVMe driver for Windows. It provides a NVME_PASS_THROUGH I/O-control which is reportedly also supported by the NVMe Windows drivers from Intel and OCZ.
• ​IOCTL_STORAGE_PROTOCOL_COMMAND - NVMe pass-through I/O-control supported by the generic NVMe driver for Windows (Stornvme.sys). Requires Windows 10.
• ​Naraeon NVMe Tools windows ​open source NVMe SMART / Health Information Log Viewer on Windows
  • ​Samsung 950 Pro’s NVMe Commands
  • ​Intel 750 Series’ NVMe Commands
• Intel® ​Solid-State Drive Data Center Tool

OS X NVMe support

• Kind of official support added from 10.10.3, closes source, not documented, only apple-branded NVMe devices are supported. Implements NVMeSMARTClient library, however API is not documented.
• ​MacVidCards NVMe driver - third party closed-source driver, coming with closed-source nvme-cli management utility, which is based on open-source one.

Solaris NVMe support

• ​nvmeadm - NVM Express controller command line interface, interface is unknown
• ​Illumos NVMe driver, ​no management interface/tool yet implemented

[Alex Samorukov]

​https://github.com/linux-nvme/nvme-cli - open source tool to get NVME information on Linux (including SMART data).
​https://github.com/xnox/nvme-user - another NVMe Linux utility
​http://nvmexpress.org/specifications/ - specifications, including SCSI translation interface
​http://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/ssd-dc-p3608-spec.pdf - Intel specification with some Intel-only pages documented

[Christian Franke]

Adding NVMe support makes sense.

Logical structure of NVMe commands and returned SMART related information is different both to ATA and SCSI. A new (-d nvme) pass_through interface and related printing routines are required. Unlike SAT, the NVMe SCSI translation interface is not very useful as it does not (yet?) provide a NVMe pass-through command.

[Christian Franke]

​https://svn.openfabrics.org/svnrepo/nvmewin/ - OpenFabrics NVMe driver for Windows.
It provides a NVME_PASS_THROUGH I/O-control which is reportedly also supported by the NVMe Windows drivers from Intel and OCZ.

[Alex Samorukov]

In the FreeBSD NVMe SMART data could be obtained using ​nvmecontrol utility.

[Alex Samorukov]

Windows opensource NVMe tool set: ​http://naraeon.net/en/latest-naraeon-ssd-tools/

[Alex Samorukov]

QEMU with nvme support: ​https://github.com/OpenChannelSSD/qemu-nvme/. It could be used for the development purpose.

[Christian Franke]

​IOCTL_STORAGE_PROTOCOL_COMMAND - NVMe pass-through I/O-control supported by the generic NVMe driver for Windows (Stornvme.sys). Requires Windows 10.

[Alex Samorukov]

Recent (10.10.3+) OSX/Darwin also supports NVME and there is undocumented NVMeSMARTClient library. No userland client [yet] found. Also only apple-branded devices are supported by official driver.

Non-apple NVMe devices are supported by 3rd party driver, see ​http://www.macvidcards.com/nvme-faq.html. This driver has some management tool, but no API description found.

[Christian Franke]

Work is in progress.

[Christian Franke]

r4238, r4239 added basic NVMe support for smartctl on Linux:

smartctl [-d nvme[,NSID]] -i -H -A /dev/nvmeX[nY]

[Christian Franke]

r4240 added NVMe Error Information log support:

smartctl -l error[,NUM] /dev/nvmeX[nY]

[Christian Franke]

r4242 added hex dump support for NVMe logs:

smartctl -l nvmelog,PAGE,SIZE /dev/nvmeX[nY]

Useful for testing.

[Christian Franke]

r4249 added basic NVMe support to smartd:

/dev/nvmeN [-d nvme[,NSID]] -H -l error -W DIFF[,INFO[,CRIT]]

[Alex Samorukov]

r4257:4258 added initial FreeBSD support

[Christian Franke]

r4262 added NVMe DEVICESCAN support on Linux. It needs to be explicitly enabled with -d nvme unless --with-nvme-devicescan is used during build.

With r4265, it is possible to add NVMe to existing device scanning:

DEVICESCAN -d ata -d scsi -d sat -d nvme -a

[Christian Franke]

r4278:4281 added initial NVMe support for Windows.

Support for DEVICESCAN ... -d nvme or smartctl -d nvme --scan is included.

Devices are accessed via NVME_PASS_THROUGH I/O-control as specified by ​OpenFabrics NMVe driver. This should work (at least) with NVMe drivers from Intel, OCZ and Samsung. It does not work with the ​Microsoft NVMe driver.

Pseudo device names are similar to Linux:

smartctl [-d nvme[,NSID]] -x /dev/nvmeX[nY]

Alternatively /dev/nvmesX[nY] could be used to directly address NVMe device behind Windows device name \\.\ScsiX:.

[Alex Samorukov]

I created initial version of the NVMe support page for the smartmontools wiki - https://www.smartmontools.org/wiki/NVMe_Support . Feel free to edit.

[Christian Franke]

r4311 added smartctl -c which prints info about various capabilites (optional commands, firmware slots, power states, LBA sizes).

[Christian Franke]

Sample smartctl -x output:

smartctl 6.5 2016-04-27 r4312 [x86_64-w64-mingw32-win10] (daily-20160427)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 950 PRO 256GB
Serial Number:                      ...
Firmware Version:                   1B0QBXX7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Controller ID:                      1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          256,060,514,304 [256 GB]
Namespace 1 Utilization:            117,410,267,136 [117 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Thu Apr 28 19:32:07 2016 CEST
Firmware Updates (0x06):            3 Slots
Optional Admin Commands (0x0007):   Security Format Frmw_DL
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         32 Pages

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.50W       -        -    0  0  0  0        5       5
 1 +     5.80W       -        -    1  1  1  1       30      30
 2 +     3.60W       -        -    2  2  2  2      100     100
 3 -   0.0700W       -        -    3  3  3  3      500    5000
 4 -   0.0050W       -        -    4  4  4  4     2000   22000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        40 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    1,769,281 [905 GB]
Data Units Written:                 1,384,224 [708 GB]
Host Read Commands:                 24,646,213
Host Write Commands:                19,105,374
Controller Busy Time:               38
Power Cycles:                       32
Power On Hours:                     129
Unsafe Shutdowns:                   6
Media and Data Integrity Errors:    0
Error Information Log Entries:      44

Error Information (NVMe Log 0x01, max 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0         44     0  0x002a  0x4016  0x000            0   255     -
  1         43     0  0x0029  0x4016  0x000            0   255     -
  2         42     0  0x0049  0x4016  0x000            0   255     -
  3         41     0  0x0048  0x4016  0x000            0   255     -
  4         40     0  0x001f  0x4004  0x000            0     0     -
  5         39     0  0x001e  0x4004  0x000            0     0     -
  6         38     0  0x001f  0x4004  0x000            0     0     -
  7         37     0  0x001e  0x4004  0x000            0     0     -
  8         36     0  0x001f  0x4004  0x000            0     0     -
  9         35     0  0x001e  0x4004  0x000            0     0     -
 10         34     0  0x001d  0x4004  0x000            0     0     -
 11         33     0  0x001c  0x4004  0x000            0     0     -
 12         32     0  0x001d  0x4004  0x000            0     0     -
 13         31     0  0x001c  0x4004  0x000            0     0     -
 14         30     0  0x001d  0x4004  0x000            0     0     -
 15         29     0  0x001c  0x4004  0x000            0     0     -
... (28 entries not shown)

[Christian Franke]

Basic NVMe support is now available for Linux, FreeBSD and Windows. See above comments for details.

For NVMe related bug reports and enhancement requests, please create new tickets (e.g. #687, #688, #689, #690, #691).