Using NVMe Command Line Tools to Check NVMe Flash Health
In this blog post, I’ll look at the types of NVMe flash health information you can get from using the NVMe command line tools.
Checking SATA-based drive health is easy. Whether it’s an SSD or older spinning drive, you can use the smartctl command to get a wealth of information about the device’s performance and health. As an example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
root@blinky:/var/lib/mysql# smartctl -A /dev/sda smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-62-generic] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0 5 Reallocated_Sector_Ct 0x0032 100 100 010 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 41 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 2 171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 173 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 1 174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 194 Temperature_Celsius 0x0022 065 059 000 Old_age Always - 35 (Min/Max 21/41) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0 202 Unknown_SSD_Attribute 0x0030 100 100 001 Old_age Offline - 0 206 Unknown_SSD_Attribute 0x000e 100 100 000 Old_age Always - 0 246 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 145599393 247 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 4550280 248 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 582524 180 Unused_Rsvd_Blk_Cnt_Tot 0x0033 000 000 000 Pre-fail Always - 1260 210 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 |
While smartctl might not know all vendor-specific smart values, typically you can Google the drive model along with “smart attributes” and find documents like this to get more details.
If you move to newer generation NVMe-based flash storage, smartctl won’t work anymore – at least it doesn’t work for the packages available for Ubuntu 16.04 (what I’m running). It looks like support for NVMe in Smartmontools is coming, and it would be great to get a single tool that supports both SATA and NVMe flash storage.
In the meantime, you can use the nvme tool available from the nvme-cli package. It provides some basic information for NVMe devices.
To get information about the NVMe devices installed:
|
1 2 3 4 5 6 7 |
root@alex:~# nvme list Node SN Model Version Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- -------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 S3EVNCAHB01861F Samsung SSD 960 PRO 1TB 1.2 1 689.63 GB / 1.02 TB 512 B + 0 B 1B6QCXP7 |
To get SMART information:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
root@alex:~# nvme smart-log /dev/nvme0 Smart Log for NVME device:nvme0 namespace-id:ffffffff critical_warning : 0 temperature : 34 C available_spare : 100% available_spare_threshold : 10% percentage_used : 0% data_units_read : 3,465,389 data_units_written : 9,014,689 host_read_commands : 89,719,366 host_write_commands : 134,671,295 controller_busy_time : 310 power_cycles : 11 power_on_hours : 21 unsafe_shutdowns : 8 media_errors : 0 num_err_log_entries : 1 Warning Temperature Time : 0 Critical Composite Temperature Time : 0 Temperature Sensor 1 : 34 C Temperature Sensor 2 : 47 C Temperature Sensor 3 : 0 C Temperature Sensor 4 : 0 C Temperature Sensor 5 : 0 C Temperature Sensor 6 : 0 C |
To get additional SMART information (not all devices support it):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
root@ts140i:/home/pz/workloads/1m# nvme smart-log-add /dev/nvme0 Additional Smart Log for NVME device:nvme0 namespace-id:ffffffff key normalized raw program_fail_count : 100% 0 erase_fail_count : 100% 0 wear_leveling : 62% min: 1114, max: 1161, avg: 1134 end_to_end_error_detection_count: 100% 0 crc_error_count : 100% 0 timed_workload_media_wear : 100% 37.941% timed_workload_host_reads : 100% 51% timed_workload_timer : 100% 446008 min thermal_throttle_status : 100% 0%, cnt: 0 retry_buffer_overflow_count : 100% 0 pll_lock_loss_count : 100% 0 nand_bytes_written : 100% sectors: 16185227 host_bytes_written : 100% sectors: 6405605 |
Some of this information is self-explanatory, and some of it isn’t. After looking at the NVMe specification document, here is my read on some of the data:
Available Spare. Contains a normalized percentage (0 to 100%) of the remaining spare capacity that is available.
Available Spare Threshold. When the Available Spare capacity falls below the threshold indicated in this field, an asynchronous event completion can occur. The value is indicated as a normalized percentage (0 to 100%).
(Note: I’m not quite sure what the practical meaning of “asynchronous event completion” is, but it looks like something to avoid!)
Percentage Used. Contains a vendor specific estimate of the percentage of the NVM subsystem life used, based on actual usage and the manufacturer’s prediction of NVM life.
(Note: the number can be more than 100% if you’re using storage for longer than its planned life.)
Data Units Read/Data Units Written. This is the number of 512-byte data units that are read/written, but it is measured in an unusual way. The first value corresponds to 1000 of the 512-byte units. So you can multiply this value by 512000 to get value in bytes. It does not include meta-data accesses.
Host Read/Write Commands. The number of commands of the appropriate type issued. Using this value, as well as one below, you can compute the average IO size for “physical” reads and writes.
Controller Busy Time. Time in minutes that the controller was busy servicing commands. This can be used to gauge long-term storage load trends.
Unsafe Shutdowns. The number of times a power loss happened without a shutdown notification being sent. Depending on the NVMe device you’re using, an unsafe shutdown might corrupt user data.
Warning Temperature Time/Critical Temperature Time. The time in minutes a device operated above a warning or critical temperature. It should be zeroes.
Wear_Leveling. This shows how much of the rated cell life was used, as well as the min/max/avg write count for different cells. In this case, it looks like the cells are rated for 1800 writes and about 1100 on average were used
Timed Workload Media Wear. The media wear by the current “workload.” This device allows you to measure some statistics from the time you reset them (called the “workload”) in addition to showing the device lifetime values.
Timed Workload Host Reads. The percentage of IO operations that were reads (since the workload timer was reset).
Thermal Throttle Status. This shows if the device is throttled due to overheating, and when there were throttling events in the past.
Nand Bytes Written. The bytes written to NAND cells. For this device, the measured unit seems to be in 32MB values. It might be different for other devices.
Host Bytes Written. The bytes written to the NVMe storage from the system. This unit also is in 32MB values. The scale of these values is not very important, as they are the most helpful for finding the write amplification of your workload. This ratio is measured in writes to NAND and writes to HOST. For this example, the Write Amplification Factor (WAF) is 16185227 / 6405605 = 2.53
As you can see, the NVMe command line tools provide a lot of good information for understanding the health and performance of NVMe devices. You don’t need to use vendor specific tools (like isdct).
Hi Peter!
Sadly, you cannot rely on the values reported by smartctl giving any good indication of the SSD’s wearout.
German magazine c’t had an article in its issue 1/2017 where they had done a long-term test of SSDs, continuously writing data to them and checking the smartctl values – the results were very disappointing.
And even worse, when a SSD was worn out, it would not only fail to write anything but also deny read access.
All this was about SATA SSDs, but I wouldn’t be surprised if it were no better for NVMe devices.
So: Beware!
Regards,
Jörg
Hi Jorg,
What article are you referring too ? The article I read on this found what SSDs tend to operate well beyond their file time which SSD would operate
http://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead
Hi Peter!
Sorry about the late reply …
The article I refer to appeared in the 1/2017 issue of c’t, a German IT paper. It seems the full text is not available online, just the start; this is the URL:
https://www.heise.de/ct/ausgabe/2017-1-Flash-Speicher-im-Langzeittest-3573503.html
To get the full article, you have to buy it (online, from that link), and it is German only.
You are right about the SSDs operating much longer than specified by the manufacturer, c’t also reported that. So my “disappointing” does not refer to the overall quality, it refers to the lack of clear diagnostics that would give a meaningful warning sign to the admin. I’m sorry my comment didn’t make that obvious, my fault.
I followed your link, that text sounds more positive on the helpfulness of SMART data than c’t is. I suspect that is an area where manufacturers differ (or: where quality differs). Or might it have changed from that 2015 report to the c’t 2017 one?
c’t had tried these SSDs of 240 – 256 GB: Crucial BX 200, Samsung 750 Evo and 850 Pro, SanDisk Extreme Pro, SanDisk Ultra II, and Toshiba OCZ TR. It might be significant that in both reports, the Samsung Pro could stand the largest amount of data written.
Also, both reports agree that SSDs fail completely, do not fall back into a read-only mode – and IMO, that is a really sad fact.
Regards,
Jörg
Hi,
Oh yes. I would not count on SSDs to become read only after they wear out. I think the idea is when you’re operating SSD beyond its operational parameters you’re using it on your own risk.
Hi Peter,
Thank you for the great article, I have question about the write amplification pls.
My Intel nvme reports a bigger value on the host_bytes_written attribute compared to the nand_bytes_written which doesn’t make sense, even when i try to write the disk, the host_bytes_written attribute seems to increase more than the nand_bytes_written! do you have any explanation to that ? given the fact that the nand_bytes_written should be bigger since it includes also the data written by the host and the overhead of the garbage collector/Wear leveler !