Access smart data from devices behind Megaraid software stack (ESRT2)

[AndyLinux]

For a long time Intel platforms have supported ESRT2 which basically runs the LSI megaraid stack as a kernel module talking thru the chipset which has to be placed into ESRT2 mode.

This has the advantages that the platform operates as though it had a real RAID card and anything talking to that (e.g. the LSI/Avago/Broadcom tools MegaCLI, storcli, CmdTool264 etc.) just work unchanged.

Unfortunately smartctl does not appear to work.

I'm using an Intel S2600WF system with the latest ESRT2 drivers on Linux, using a 4.9 kernel in a custom distribution. I've tried with the RHEL6 smartmontools package (6.6.2) and don't notice any obvious changes between this version and smartmontools 7 Megaraid support in the change history.

The first problem is that the the megasr kernel module creates its device node at /dev/megaraid_swr_ioctl_node not the normal /dev/megaraid_sas_ioctl_node - this is worked around using a symlink or a secondary device node with the same major/minor numbers.

Once this is done, the smartctl fails in a fairly blunt fashion:

# smartctl -a -d sat+megaraid,0 -T permissive /dev/bus/0
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.9.125-2.11.0.12.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

Read Device Identity failed: megasas_cmd result: 0.0 = 25/255

=== START OF INFORMATION SECTION ===
Device Model:     [No Information Found]
Serial Number:    [No Information Found]
Firmware Version: [No Information Found]
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   [No Information Found]
Local Time is:    Thu Jul 11 15:21:01 2019 UTC
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 82-83 don't show if SMART supported.
SMART support is: Ambiguous - ATA IDENTIFY DEVICE words 85-87 don't show if SMART is enabled.
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

Adding the -r ioctl,3 options simple shows a dump of a frame full of 0's.

The smartctl scan options show this info:

# smartctl --scan -r ioctl,3
glob(3) found no matches for pattern /dev/hd[a-t]
sda -> /sys/class/scsi_host/host0/proc_name: "megasr"
glob(3) found no matches for pattern /dev/sd[a-c][a-z]
glob(3) found no matches for pattern /dev/nvme[0-9]
glob(3) found no matches for pattern /dev/nvme[1-9][0-9]
/dev/sda -d scsi # /dev/sda, SCSI device
# smartctl --scan-open -r ioctl,3
glob(3) found no matches for pattern /dev/hd[a-t]
sda -> /sys/class/scsi_host/host0/proc_name: "megasr"
glob(3) found no matches for pattern /dev/sd[a-c][a-z]
glob(3) found no matches for pattern /dev/nvme[0-9]
glob(3) found no matches for pattern /dev/nvme[1-9][0-9]
>>>> do_scsi_cmnd_io: sg_io_ver=3
 [inquiry: 12 00 00 00 24 00 ]
  scsi_status=0x0, sg_transport_status=0x0, sg_driver_status=0x0
  sg_info=0x0  sg_duration=0 milliseconds  resid=0
  Incoming data, len=36:
 00     00 00 05 02 fa 00 00 32  49 6e 74 65 6c 20 20 20
 10     4d 65 67 61 53 52 20 20  00 00 00 00 00 00 00 00
 20     31 2e 30 20
/dev/sda -d scsi # /dev/sda, SCSI device

I suspect that the problem relates to the fact that unlike normal Megaraid controllers, the drives do not show as being in an enclosure, with the software raid the enclosure is N/A:

# CmdTool264 pdlist a0 | grep -i -e slot -e id
Enclosure Device ID: N/A
Slot Number: 0
Device Id: 0
Media Type: Solid State Device
Enclosure Device ID: N/A
Slot Number: 1
Device Id: 1
Media Type: Solid State Device

If I boot the system using a Live CD and don't load the megasr module, the AHCI module will find the devices and report the SMART data quite happily.

I won't be able to provide debug access but it's not a production system so can run whatever debug commands are required to provide more info as needed.

[Christian Franke]

Is the source code of the megasr kernel module available?
Is there any evidence that the module implements ​MEGASAS_IOC_FIRMWARE?

[AndyLinux]

Replying to Christian Franke:

Is the source code of the megasr kernel module available?

Only partially, it is a mostly closed source module except for the inevitable shim layer. It can be found at:
​https://downloadcenter.intel.com/download/28304/Intel-Embedded-Server-RAID-Technology-2-ESRT2-RAID-driver-for-Linux-?v=t

Is there any evidence that the module implements ​MEGASAS_IOC_FIRMWARE?

I have no idea; looking in the above code I see a token LSRAID_IOC_FIRMWARE used, defined as:

#define LSRAID_IOC_FIRMWARE	_IOWR('R', 1, struct megasas_iocpacket)

Which I guess is similar but different to that in the kernel code:

#define MEGASAS_IOC_FIRMWARE	_IOWR('M', 1, struct megasas_iocpacket)

If this will show up in an strace of the LSI tools being used, then I can provide that, if that helps? Let me know what CmdTool264 etc. command to use and I can grab a trace.

[AndyLinux]

I had a quick look at an strace comparison between a 'CmdTool pdinfo' command on a real RAID system and a megasr software RAID system.

There are 2 basic differences:

1. It notices the difference in /proc/devices between "250 megaraid_sas_ioctl" and "250 megaraid_swr_ioctl" as an indication that it's talking to a software RAID controller, and uses that to determine the correct /dev/ node to open, as noted above
2. It uses subtly different ioctl's - hardware RAID shows a sequence of ioctl(0x3, 0xc1944d01, 0x7f6d9f452000) whereas software RAID shows ioctl(0x3, 0xc1945201, 0x7f3485c52000) - where of course the 3rd octet is 4d or 52 - 'M' or 'R' from the above #define's

If you can give me some ideas on how to see the contents of the ioctl memory, I can forward that if it helps.

Looking at the structures referenced in the #define values above, the structures appear identical.

[Alex Samorukov]

You can get ioctl content with a number of ways.

1. If it is non-staic binary - just use LD_PRELOAD to override ioctl
2. If it is static - use any kernel-level debugger (stap?) to do the same

However, without documentation that would be a lot of work.

[Christian Franke]

Patch for os_linux.cpp to do a minimal test of LSRAID_IOC_FIRMWARE

[Christian Franke]

Replying to AndyLinux:

... hardware RAID shows a sequence of ioctl(0x3, 0xc1944d01, 0x7f6d9f452000) whereas software RAID shows ioctl(0x3, 0xc1945201, 0x7f3485c52000) ...

Result of _IOWR() expansion:

#define MEGASAS_IOC_FIRMWARE 0xc1944d01UL
#define LSRAID_IOC_FIRMWARE  0xc1945201UL

These values actually match the strace result. The struct size and buffer addresse are identical in both calls. This suggests that both I/O-controls may only differ in ioctl code and device node.

If possible, please apply os_linux-lsraid_ioc-test.patch​ to smartmontools source, build smartctl and try it on a test(!) machine. Patch should work for 6.6 and 7.0 tarballs and current SVN. Test could be run from build directory, there is no need to run make install. The patch does neither include creation of missing /dev/megaraid_swr_ioctl_node nor device scan support.

[AndyLinux]

Thanks for the prompt patch.

I tried this with 6.6, fixed a minor typo, tweaked the patch to replace all instances of MEGASAS_IOC_FIRMWARE with LSIRAID_IOC_FIRMWARE, also all instances of megaraid_sas_ioctl_node with megaraid_swr_ioctl_node; it's close but not quite functioning yet:

# smartctl -a -d megaraid,0 -T permissive -r ioctl,4 /dev/bus/0
smartctl 6.6 2019-06-19 r4923 [x86_64-linux-4.9.125-2.11.0.12.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

 [inquiry: 12 00 00 00 24 00 ]
Got MegaRAID inquiry.. ATA     Micron_5200_MTFDU004
 [ata pass-through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 ]
sat_device::ata_pass_through: scsi_pass_through() failed, errno=5 [megasas_cmd result: 0.0 = 0/35]
 [inquiry: 12 01 00 00 fc 00 ]
 [inquiry: 12 00 00 00 24 00 ]
 [read capacity(10): 25 00 00 00 00 00 00 00 00 00 ]
User Capacity:        960,197,124,096 bytes [960 GB]
Logical block size:   512 bytes
 [mode sense(6): 1a 00 84 00 40 00 ]
 [mode sense(6): 1a 00 1c 00 40 00 ]
Vital Product Data (VPD) INQUIRY failed [3]
Vital Product Data (VPD) INQUIRY failed [3]
Device type:          CD/DVD
 [mode sense(6): 1a 00 19 00 40 00 ]
Local Time is:        Mon Jul 15 11:04:57 2019 UTC
 [test unit ready: 00 00 00 00 00 00 ]
SMART support is:     Unavailable - device lacks SMART capability.
 [Input/output error]

=== START OF READ SMART DATA SECTION ===
 [log sense: 4d 00 40 00 00 00 00 00 04 00 ]
 [log sense: 4d 00 40 00 00 00 00 00 06 00 ]
 [log sense: 4d 00 6f 00 00 00 00 00 04 00 ]
 [log sense: 4d 00 6f 00 00 00 00 00 0c 00 ]
 [request sense: 03 00 00 00 12 00 ]
 [log sense: 4d 00 4d 00 00 00 00 00 04 00 ]
 [log sense: 4d 00 4d 00 00 00 00 00 10 00 ]

 [log sense: 4d 00 4d 00 00 00 00 00 04 00 ]
 [log sense: 4d 00 4d 00 00 00 00 00 10 00 ]
Current Drive Temperature:     29 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

 [mode sense(6): 1a 00 0a 00 40 00 ]
Device does not support Self Test logging

If I use sat+megaraid,0 then I see this output - I've removed the identify packet dumps as they are all 0's.

# smartctl -a -d sat+megaraid,0 -T permissive -r ioctl,4 /dev/bus/0
smartctl 6.6 2019-06-19 r4923 [x86_64-linux-4.9.125-2.11.0.12.x86_64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org


REPORT-IOCTL: Device=/dev/bus/0 Command=IDENTIFY DEVICE
 Input:   FR=...., SC=0x01, LL=...., LM=...., LH=...., DEV=...., CMD=0xec IN
 [ata pass-through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00 ]
sat_device::ata_pass_through: scsi_pass_through() failed, errno=5 [megasas_cmd result: 0.0 = 0/35]
REPORT-IOCTL: Device=/dev/bus/0 Command=IDENTIFY DEVICE returned -1 errno=5 [megasas_cmd result: 0.0 = 0/35]

===== [IDENTIFY DEVICE] DATA START (BASE-16) =====
...
===== [IDENTIFY DEVICE] DATA END (512 Bytes) =====


REPORT-IOCTL: Device=/dev/bus/0 Command=IDENTIFY PACKET DEVICE
 Input:   FR=...., SC=0x01, LL=...., LM=...., LH=...., DEV=...., CMD=0xa1 IN
 [ata pass-through(16): 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 a1 00 ]
sat_device::ata_pass_through: scsi_pass_through() failed, errno=5 [megasas_cmd result: 0.0 = 0/35]
REPORT-IOCTL: Device=/dev/bus/0 Command=IDENTIFY PACKET DEVICE returned -1 errno=5 [megasas_cmd result: 0.0 = 0/35]

===== [IDENTIFY PACKET DEVICE] DATA START (BASE-16) =====
...
===== [IDENTIFY PACKET DEVICE] DATA END (512 Bytes) =====

Read Device Identity failed: megasas_cmd result: 0.0 = 0/35
(followed by empty information section)

It looks like the CmdTool2 binary is dynamically linked against glibc so I will see if I can use gdb to capture some ioctl content, unless there's anything else I can try?

[AndyLinux]

I managed to use gdb to capture the ioctl parameters when running CmdTool264 pdinfo physdrv [:0] a0 (which should return the data about the same drive as identified above)

For each time ioctl is called, it dumps the ioctl megasas_ioc_packet and 128 bytes pointed to by the first sgl entry in that ioc_packet, and then dumps them again after the syscall returns.

Hope this is of use, let me know if there's more data that can be gathered.

[AndyLinux]

gdb capture of ioctl calls for "CmdTool264 pdinfo physdrv [:0] a0"

[Christian Franke]

Most cmd types (at offset 0x14) are MFI_CMD_DCMD (0x05), only two commands are actual pass-through types (MFI_CMD_PD_SCSI_IO, 0x04).

The first pass-through command is a SAT ATA PASS-THROUGH-12 (0xa1) with a CDB for ATA IDENTIFY DEVICE (0xec). A quick look shows two fields which differ from smartctl:

1. megasas_pthru_frame.sense_buf_phys_addr_lo/hi is set to some sense buffer address.
2. megasas_pthru_frame.context is set to some magic value.

825	0x7ffff5e52010:	0x00	0x00	0x00	0x00	[0x04]	0x00	0xff	0x00  <== cmd
826	0x7ffff5e52018:	0x00	0x00	0x0c	0x01	[0x00	0xa2	0xe6	0xf5] <== context
827	0x7ffff5e52020:	0xff	0x7f	0x00	0x00	0x14	0x00	0x1e	0x00
828	0x7ffff5e52028:	0x00	0x02	0x00	0x00	[0x9c	0x70	0xe6	0xf5] <== sense_buf_phys_addr_lo
829	0x7ffff5e52030:	[0xff	0x7f	0x00	0x00]	[0xa1	0x08	0x0b	0x00] <== "_hi, cdb[0..3]
830	0x7ffff5e52038:	[0x00	0x00	0x00	0x00	0x00	0xec	0x00	0x00] <== cdb[4..11]

The second pass-through command is SCSI LOG SENSE (0x4d). The context is set to a slightly different value:

948	0x7ffff5e52018:	0x00	0x00	0x0a	0x01	[0x00	0xa4	0xe6	0xf5] <== context

So a nonzero context may be required.

BTW: Output data is missing for the pass-through commands because gdb interpreted megasas_pthru_frame.cdb[8..15] as the buffer address:

830	0x7ffff5e52038:	0x00	0x00	0x00	0x00	[0x00	0xec	0x00	0x00
831	0x7ffff5e52040:	0x00	0xb1	0x0d	0x54]	0xbc	0x70	0xe6	0xf5
...
843	0x540db1000000ec00:	runmegacli.gdb:24: Error in sourced command file:
844	Cannot access memory at address 0x540db1000000ec00

[AndyLinux]

If you look at the 4 bytes following the context value (pad_0), combined with the context value it forms a memory pointer in a similar range to the memory being decoded. I assumed that it was some opaque handle back to the requesting program but have added some dump of that too for the passthrough commands.

Picking the wrong address is simply me not having a sufficiently detailed understanding of the command structures, I am now slightly more enlightened.

Attached is an updated run, which now dumps the sgl and sense_buf for megasas_pthru_frame correctly, also the dereferenced "context" address.

[AndyLinux]

Updated gdb output showing dump of context and passthru frames

[AndyLinux]

gdb dump of ioctl's from "CmdTool264 pdinfo physdrv [252:0] a0" on a hardware RAID system with NL-SAS (==SATA) drives.

[AndyLinux]

I added a similar ioctl capture from the same command running on a different system, which has a proper hardware megaraid controller and NL-SAS drives that work as expected with smartctl, for comparison purposes, in case that is of use.

In this instance, the RAID "device ID" (which would be provided as part of smartctl "-d sat+megaraid,XX" is 36.

[AndyLinux]

I used similar gdb techniques to track the ioctl's for smartctl -a -d megaraid,0 -T permissive /dev/bus/0 and have attached the output from that.

I notice that there are some complaints from the kernel module about unsupported operations; I have inserted these into the corresponding ioctl output in case they are of significance.

I notice that smartctl is using 32 bit sgl pointers whereas CmdTool264 is using 64 bit. Could that be significant? Would I be naive in asking how the kernel module would know the uppermost part of the address (since this changes between stack and heap based sgl entries)?

[AndyLinux]

gdb trace of smartctl ioctl calls

[AndyLinux]

I also tried using Intel's isdct3 SSD management program. This also supports drives behind megasr and although it doesn't find any Intel SSDs, it does also use the same ioctl's to probe for drives.

I have uploaded an ioctl trace of this being run in case it's of use. ioctl memory blocks are truncated to 256 bytes since some of the buffer sizes are huge, I can increase this limit if it is of use.

[AndyLinux]

ioctl trace from running Intel isdct3

[AndyLinux]

I found that the Intel / Broadcom storcli tool shows a bit more retrieved SMART data, so captured the ioctl's from that too.

[AndyLinux]

ioctl trace from running storcli show /c0 /s0 all

[AndyLinux]

ioctl trace from running storcli show /c0 /s0 smart