JSON output for SCSI contains bogus string "Pending defect count:"

[mircea.voicu]

Hello,

We have 2 servers and on one of them smartctl is not able to identify SATA disks when using JSON output.
NOTE:
s14 - working server
s16 - problem server
Both servers have identical configuration.

For example on s14 we have:

[root@s14 ~]# sudo -n smartctl -a /dev/bus/1 -d megaraid,0 -j | head
{
  "json_format_version": [
    1,
    0
  ],
  "smartctl": {
    "version": [
      7,
      2
    ],

[root@s14 ~]# sudo -n smartctl --scan -d sat
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/bus/0 -d megaraid,0 # /dev/bus/0 [megaraid_disk_00], SCSI device
/dev/bus/0 -d megaraid,1 # /dev/bus/0 [megaraid_disk_01], SCSI device
/dev/bus/0 -d megaraid,2 # /dev/bus/0 [megaraid_disk_02], SCSI device
/dev/bus/0 -d megaraid,3 # /dev/bus/0 [megaraid_disk_03], SCSI device

...
2024/07/29 11:12:23.404053 [Smart] command sudo -n smartctl -a /dev/bus/0 -d megaraid,3 -j  smartctl raw response: {
  "json_format_version": [
...
    "flags": {
      "value": 0,
      "remainder_scan_enabled": false
    },
    "power_up_scan_resume_minutes": 0
  }
}
....

[root@s14 ~]# smartctl --version
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.14.0-162.22.2.el9_1.x86_64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

smartctl comes with ABSOLUTELY NO WARRANTY. This is free
software, and you are welcome to redistribute it under
the terms of the GNU General Public License; either
version 2, or (at your option) any later version.
See http://www.gnu.org for further details.

smartmontools release 7.2 dated 2020-12-30 at 16:48:30 UTC
smartmontools SVN rev 5155 dated 2020-12-30 at 16:49:18
smartmontools build host: x86_64-redhat-linux-gnu
smartmontools build with: C++17, GCC 11.3.1 20221121 (Red Hat 11.3.1-4)
smartmontools configure arguments: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-selinux' '--with-libcap-ng=yes' '--with-libsystemd' '--with-systemdsystemunitdir=/usr/lib/systemd/system' '--sysconfdir=/etc/smartmontools/' '--with-systemdenvfile=/etc/sysconfig/smartmontools' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CXX=g++' 'CXXFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 ' 'CC=gcc' 'CFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

And on s16 we have this:

[root@s16 ~]# sudo -n smartctl -a /dev/bus/1 -d megaraid,0 -j | head
  Pending defect count:{
  "json_format_version": [
    1,
    0
  ],
  "smartctl": {
    "version": [
      7,
      2
    ],

[root@s16 ~]# sudo -n smartctl --scan -d sat
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/bus/1 -d megaraid,0 # /dev/bus/1 [megaraid_disk_00], SCSI device
/dev/bus/1 -d megaraid,1 # /dev/bus/1 [megaraid_disk_01], SCSI device
/dev/bus/1 -d megaraid,2 # /dev/bus/1 [megaraid_disk_02], SCSI device
/dev/bus/1 -d megaraid,3 # /dev/bus/1 [megaraid_disk_03], SCSI device

...
2024/07/29 11:12:30.605948 [Smart] command sudo -n smartctl -a /dev/bus/1 -d megaraid,0 -j  smartctl raw response:   Pending defect count:{
  "json_format_version": [
...
    }
  },
  "pending_defects": {
    "count": 0
  }
}
2024/07/29 11:12:30.605972 [Smart] failed to unmarshal megaraid device with name /dev/bus/1 -d megaraid,0, invalid character 'P' looking for beginning of value
...

[root@s16 ~]# smartctl --version
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.14.0-427.20.1.el9_4.x86_64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

smartctl comes with ABSOLUTELY NO WARRANTY. This is free
software, and you are welcome to redistribute it under
the terms of the GNU General Public License; either
version 2, or (at your option) any later version.
See http://www.gnu.org for further details.

smartmontools release 7.2 dated 2020-12-30 at 16:48:30 UTC
smartmontools SVN rev 5155 dated 2020-12-30 at 16:49:18
smartmontools build host: x86_64-redhat-linux-gnu
smartmontools build with: C++17, GCC 11.4.1 20231218 (Red Hat 11.4.1-3)
smartmontools configure arguments: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--program-prefix=' '--disable-dependency-tracking' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--localstatedir=/var' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--with-selinux' '--with-libcap-ng=yes' '--with-libsystemd' '--with-systemdsystemunitdir=/usr/lib/systemd/system' '--sysconfdir=/etc/smartmontools/' '--with-systemdenvfile=/etc/sysconfig/smartmontools' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'CXX=g++' 'CXXFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'LDFLAGS=-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 ' 'CC=gcc' 'CFLAGS=-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64 -march=x86-64-v2 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' 'PKG_CONFIG_PATH=:/usr/lib64/pkgconfig:/usr/share/pkgconfig'

The problem seems to be here:

s14:

...
2024/07/29 11:12:23.404053 [Smart] command sudo -n smartctl -a /dev/bus/0 -d megaraid,3 -j  smartctl raw response: {
  "json_format_version": [
...
    "flags": {
      "value": 0,
      "remainder_scan_enabled": false
    },
    "power_up_scan_resume_minutes": 0
  }
}
...

versus

s16:

...
2024/07/29 11:12:30.605948 [Smart] command sudo -n smartctl -a /dev/bus/1 -d megaraid,0 -j  smartctl raw response:   Pending defect count:{
  "json_format_version": [
...
    }
  },
  "pending_defects": {
    "count": 0
  }
}
2024/07/29 11:12:30.605972 [Smart] failed to unmarshal megaraid device with name /dev/bus/1 -d megaraid,0, invalid character 'P' looking for beginning of value
...

Somehow on s16 server JSON output is not ok.
How can I fix this?

Thank you

[Christian Franke]

Somehow on s16 server JSON output is not ok.

Then please provide the full questionable JSON output as an attachment to this ticket.

[mircea.voicu]

s14_JSON_output

[mircea.voicu]

s16_JSON_output

[mircea.voicu]

s14_JSON_increased_verbosity

[mircea.voicu]

s16_JSON_increased_verbosity

[Christian Franke]

s14_JSON.output:

  "device": {
    "name": "/dev/bus/0",
    "info_name": "/dev/bus/0 [megaraid_disk_00] [SAT]",
    "type": "sat+megaraid,0",
    "protocol": "ATA"
  },
  "model_name": "ST16000NM000J-2TW103",

The ST16000NM000J is a Seagate Exos X18 SATA disk.

s16_JSON.output:

  "device": {
    "name": "/dev/bus/1",
    "info_name": "/dev/bus/1 [megaraid_disk_00]",
    "type": "megaraid,0",
    "protocol": "SCSI"
  },
  "vendor": "SEAGATE",
  "product": "ST16000NM004J",
  "model_name": "SEAGATE ST16000NM004J",

The ST16000NM004J is a Seagate Exos X18 SAS disk. It is correctly identified by smartctl.

ATA/SATA and SCSI/SAS diagnostics differ considerably from and so do smartctl outputs.

Both servers have identical configuration.

This is obviously not the case :-)

[Christian Franke]

The ST16000NM004J is not a SATA disk but a SAS disk.

[mircea.voicu]

Hello,

I understand, but my problem is that when I run this command the JSON output includes invalid characters, as shown below - "[Smart] failed to unmarshal megaraid device with name /dev/bus/1 -d megaraid,0, invalid character 'P' looking for beginning of value"
Because of this, it is not possible to make discovery on megaraid disks.
Please see attachement "S16_JSON_increased_verbosity.txt" for more details.

[root@s16 ~]# zabbix_agent2 -t 'smart.disk.discovery' -v -v -v
...
2024/07/29 11:12:30.605948 [Smart] command sudo -n smartctl -a /dev/bus/1 -d megaraid,0 -j  smartctl raw response:   Pending defect count:{
  "json_format_version": [
...
    }
  },
  "pending_defects": {
    "count": 0
  }
}
2024/07/29 11:12:30.605972 [Smart] failed to unmarshal megaraid device with name /dev/bus/1 -d megaraid,0, invalid character 'P' looking for beginning of value
...

[Christian Franke]

Reopen ticket and adjust summary.

[Christian Franke]

Already fixed since smartmontools 7.3, see r5283.