The strange I / O behavior of ZFS

Support
1

If you see it, I just read the file from the ZFS storage pool and copy it to the / tmp folder, but why do you have written operations? I didn’t write any files to this Seagate HDD.
And I use the Manjaro default ZFS configuration and join the SSD to enable L2ARC and set up to 16G memory using ARC.
ARC should have already hit the file I read, but why is there a strange write behavior?
This makes it impossible for me to sleep my HDD, resulting in additional hard disk life consumption and energy waste.

mount
kate on /home/test/d type zfs (rw,xattr,noacl)

sudo hdparm -y /dev/disk/by-id/ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087

/dev/disk/by-id/ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087:
 issuing standby command

sudo smartctl -i -n standby /dev/disk/by-id/ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087
doas (test@test-HomePC) password: 
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.15-1-MANJARO] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

Device is in STANDBY mode, exit(2)

cat ./loopcp.sh
#!/bin/bash
while true
do
cp /home/test/d/manjaro-kde-21.1.2-210904-linux513.iso /tmp
done

screen -dmS loopcp ./loopcp.sh 

lsof ~/d
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
      Output information may be incomplete.
COMMAND    PID   USER   FD   TYPE DEVICE   SIZE/OFF NODE NAME
cp      106407 test    3r   REG   0,75 3456624640  128 /home/test/d/manjaro-kde-21.1.2-210904-linux513.iso


sudo zpool iostat -v kate 1
                                                     capacity     operations     bandwidth 
pool                                               alloc   free   read  write   read  write
-------------------------------------------------  -----  -----  -----  -----  -----  -----
kate                                               3.22G   925G      0     12    689  8.74M
  ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087        3.22G   925G      0     12    689  8.74M
cache                                                  -      -      -      -      -      -
  ata-INTEL_SSDSCKHB340G4M_118000041_BTWM7AK02UHC  3.17G   314G      0     64    712  7.92M
-------------------------------------------------  -----  -----  -----  -----  -----  -----
                                                     capacity     operations     bandwidth 
pool                                               alloc   free   read  write   read  write
-------------------------------------------------  -----  -----  -----  -----  -----  -----
kate                                               3.22G   925G      0     48      0   300K
  ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087        3.22G   925G      0     48      0   300K
cache                                                  -      -      -      -      -      -
  ata-INTEL_SSDSCKHB340G4M_118000041_BTWM7AK02UHC  3.17G   314G      0      1      0  7.99K
-------------------------------------------------  -----  -----  -----  -----  -----  -----


sudo smartctl -i -n standby /dev/disk/by-id/ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.16-1-MANJARO] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Samsung SpinPoint M8 (AF)
Device Model:     ST1000LM024 HN-M101MBB
Serial Number:    S31LJ9DFB19087
LU WWN Device Id: 5 0004cf 20eb028c2
Firmware Version: 2BA30001
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      2.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Sep 14 20:28:37 2021 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Power mode is:    ACTIVE or IDLE

cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=17179869184
options zfs zfs_arc_min=8589934592
2

  1. zfs was not meant to be used on machines that are being put to sleep. It’s a server-oriented type of filesystem.

  2. Your HDD will wear out faster by putting it to sleep and waking it up all the time. The best usage for a HDD is to keep it spinning at a constant velocity.

1 Like
3

Since I have very few writes to the file server, I don’t understand why the file system writes to HDDS in the event of an ARC hit. I don’t write at all.
So, this is writing to the log?

4

What lives on that pool? Logs? Configs?

The L2ARC will be written to in order to fill it with cache, regardless.

I don’t see any separately configured SLOG device, so your ZiL lives solely on the HDD. (Not relevant for reads, however.)

Somewhat off topic: You’re not using any type of redundancy, which puts everything on your pool at risk for permanent data loss.

If you want to a see closer to a “real-time” view of I/O bandwidth, revise your original command like so:

zpool iostat -v -y kate 1

Using “-y” will drop the cumulative bandwidth.

Use CTRL+C to abort after you’ve seen enough.

5

In fact, this was just testing ZFS on my PC, and did not actually deploy ZFS to the storage server.
This storage pool is only used to store files, and I am only testing reading files from zfs.
Then I will test the zpool iostat -v -y kate 1 command.

6

Even if it adds ZIL, it is still a strange write operation, and there is nothing improvement.

sudo zpool iostat -v -y kate 1

                                                           capacity     operations     bandwidth 
pool                                                     alloc   free   read  write   read  write
-------------------------------------------------------  -----  -----  -----  -----  -----  -----
kate                                                     3.22G   925G      0     39      0   284K
  ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087-part1        3.22G   925G      0     39      0   284K
logs                                                         -      -      -      -      -      -
  nvme-INTEL_SSDPE21D280GA_PHM274900219280AGN-part1          0   260G      0      0      0      0
cache                                                        -      -      -      -      -      -
  ata-INTEL_SSDSCKHB340G4M_118000041_BTWM7AK02UHC-part1  3.22G   313G      0      1      0  7.99K
-------------------------------------------------------  -----  -----  -----  -----  -----  -----
                                                           capacity     operations     bandwidth 
pool                                                     alloc   free   read  write   read  write
-------------------------------------------------------  -----  -----  -----  -----  -----  -----
kate                                                     3.22G   925G      0      3      0  16.0K
  ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087-part1        3.22G   925G      0      3      0  16.0K
logs                                                         -      -      -      -      -      -
  nvme-INTEL_SSDPE21D280GA_PHM274900219280AGN-part1          0   260G      0      0      0      0
cache                                                        -      -      -      -      -      -
  ata-INTEL_SSDSCKHB340G4M_118000041_BTWM7AK02UHC-part1  3.22G   313G      0      1      0  7.99K
-------------------------------------------------------  -----  -----  -----  -----  -----  -----
                                                           capacity     operations     bandwidth 
pool                                                     alloc   free   read  write   read  write
-------------------------------------------------------  -----  -----  -----  -----  -----  -----
kate                                                     3.22G   925G      0      0      0      0
  ata-ST1000LM024_HN-M101MBB_S31LJ9DFB19087-part1        3.22G   925G      0      0      0      0
logs                                                         -      -      -      -      -      -
  nvme-INTEL_SSDPE21D280GA_PHM274900219280AGN-part1          0   260G      0      0      0      0
cache                                                        -      -      -      -      -      -
  ata-INTEL_SSDSCKHB340G4M_118000041_BTWM7AK02UHC-part1  3.22G   313G      0      1      0  7.99K
-------------------------------------------------------  -----  -----  -----  -----  -----  -----
7