Various benchmark for storage and Kubernetes.

=========================
FIO Benchmark Summary
For: test_device
CPU Idleness Profiling: enabled
Size: 50G
Mode: full
=========================
IOPS (Read/Write)
        Random:            98368 / 89200
  CPU Idleness:                      68%

Bandwidth in KiB/sec (Read/Write)
    Sequential:          552052 / 521330
  CPU Idleness:                      99%

Latency in ns (Read/Write)
        Random:            97222 / 44548
  CPU Idleness:                      72%

================================
FIO Benchmark Comparsion Summary
For: 850-pro-raw vs 850-pro-ext4
SIZE: 50G
QUICK MODE: DISABLED
================================
                             850-pro-raw   vs             850-pro-ext4    :              Change
IOPS (Read/Write)
        Random:          96,735 / 89,565   vs          98,135 / 88,990    :      1.45% / -0.64%
  CPU Idleness:                      68%   vs                      66%    :                 -2%

Bandwidth in KiB/sec (Read/Write)
    Sequential:        551,825 / 522,936   vs        552,337 / 520,364    :      0.09% / -0.49%
  CPU Idleness:                      98%   vs                      98%    :                  0%

Latency in ns (Read/Write)
        Random:          82,899 / 44,437   vs         114,331 / 45,104    :      37.92% / 1.50%
  CPU Idleness:                      72%   vs                      73%    :                  1%

For official benchmarking:

1. SIZE environmental variable: the size should be at least 25 times the read/write bandwidth to avoid the caching impacting the result.

2. If you're testing a distributed storage solution like Longhorn, always test against the local storage first to know what's the baseline.

   • You can install a storage provider for local storage like Local Path Provisioner for this test if you're testing with Kubernetes.

3. CPU_IDLE_PROF environmental variable: the CPU idleness profiling measures the CPU idleness, but it introduces extra overhead and reduces the storage performance. By default, the flag is disabled.

4. MODE environmental variable: selects which benchmark workloads to run. Defaults to full. Supported values:

   • full — run all IOPS, bandwidth, and latency workloads (default, recommended for official benchmarking).
   • quick — a short-running debug mode; not suitable for reporting numbers.
   • random-read-iops / random-write-iops — run only the corresponding random small-block IOPS workload.
   • sequential-read-bandwidth / sequential-write-bandwidth — run only the corresponding sequential large-block bandwidth workload.
   • random-read-latency / random-write-latency — run only the corresponding random single-queue latency workload.

   Note: the legacy QUICK_MODE variable is deprecated; use MODE="quick" instead.

• IOPS: IO operations per second. Higher is better.
  • It's a measurement of how many IO operations can the device handle in a second, mainly concerning the smaller IO chunks, e.g. 4k.
• Bandwidth: Also called Throughput. Higher is better.
  • It's a measurement of how many data can the device read/write in a second. It's mainly concering the bigger IO chunks, e.g. 128k.
• Latency: The total time each request spent in the IO path. Lower is better.
  • It's a measurement of how efficient is the storage system to handle each request.
  • The data path overhead of a storage system can be expressed as the latency it added on top of the native storage system (SSD/NVMe).
• CPU Idleness: How idle is the CPU on the node that's running the test. Higher is better.
  • It's a measurement of the CPU load/overhead that the storage device has generated.
  • Notice this is idleness, so if the value is higher, it means the CPUs on that node have more free cycles.
  • Unforunately at this moment, this measurement cannot reflect the load on the whole cluster for distributed storage systems. But it's still a worthy reference regarding the storage client's CPU load when benchmarking (depends on how distributed storage was architected).
• For comparison benchmark, the Change column indicates what's the percentage differences when comparing the second volume to the first volume.
  • For IOPS, Bandwidth, CPU Idleness, positive percentage is better.
  • For Latency, negative percentage is better.
  • For CPU Idleness, instead of showing the percentage of the change, we are showing the difference.

Each benchmark category uses a fixed fio workload profile. The parameters below are the ones that most directly define what each number means; see the corresponding fio/*-include.fio for the full definition.

Common settings shared across all workloads (from fio/common-include.fio): ioengine=libaio, direct=1, time_based=1, ramp_time=60s, runtime=60s, group_reporting=1.

The three parameters above can be overridden per category via environment variables. This is useful for exploring how a storage system behaves under different queue depths or block sizes, but numbers produced with non-default values are not comparable with numbers produced by the kbench defaults — treat them as your own custom benchmark, not as a kbench score.

Notes:

• Only lines that already exist in the include file are overridden; unset env vars leave the file untouched.
• For meaningful latency measurements, keep LAT_IODEPTH=1 and LAT_NUMJOBS=1. Anything larger measures queuing latency, not per-op latency.
• The Kubernetes manifests (deploy/fio.yaml, deploy/fio-cmp.yaml) already set all nine variables to their default values, so all knobs are visible in one place.

For a distributed storage system, you always need to test the local storage first as a baseline.

Something is wrong when:

1. You're getting lower read latency than local storage.
   • You might get higher read IOPS/bandwidth than local storage due to the storage engines can aggregate performance from different nodes/disks. But you shouldn't be able to get lower latency on reading compare to the local storage.
   • If that happens, it's most likely due to there is a cache. Increase the SIZE to avoid that.
2. You're getting better write IOPS/bandwidth/latency than local storage.
   • It's almost impossible to get better write performance compare to the local storage for a distributed storage solution, unless you have a persistent caching device in front of the local storage.
   • If you are getting this result, it's likely the storage solution is not crash-consistent, so it doesn't commit the data into the disk before respond, which means in the case of an incident, you might lose data.
3. You're getting low CPU Idleness for Latency benchmark , e.g. <40%.
   • For Latency, the CPU Idleness should be at least 40% to guarantee the test won't be impacted by CPU starving.
   • If this happens, adding more CPUs to the node, or move to a beefer machine.

By default:

1. The benchmark will use the default storage class.
   • You can specify the storage class with the YAML locally.
2. Filesystem mode will be used.
   • You can switch to the block mode with the YAML locally.
3. The test requires a 33G PVC temporarily.
   • You can change the test size with the YAML locally.
   • As mentioned above, for formal benchmark, the size should be at least 25 times the read/write bandwidth to avoid the caching impacting the result.

Step to deploy:

1. One line to start benchmarking your default storage class:

   kubectl apply -f https://raw.githubusercontent.com/longhorn/kbench/main/deploy/fio.yaml
   

2. Observe the Result:

   kubectl logs -l kbench=fio -f
   

3. Cleanup:

   kubectl delete -f https://raw.githubusercontent.com/longhorn/kbench/main/deploy/fio.yaml
   

Note: a single benchmark for FIO will take about 6 minutes to finish.

See ./deploy/fio.yaml for available options.

1. Get a local copy of fio-cmp.yaml

   wget https://raw.githubusercontent.com/longhorn/kbench/main/deploy/fio-cmp.yaml
   

2. Set the storage class for each volume you want to compare.
   • By default, it's local-path vs longhorn.
   • You can install a storage provider for local storage like Local Path Provisioner for this test if you're testing with Kubernetes.
3. Update the FIRST_VOL_NAME and SECOND_VOL_NAME fields in the yaml to the name you want to call them
   • By default, it's also Local-Path vs Longhorn.
4. Update the SIZE of PVCs and the benchmark configuration.
   • By default, the size for comparison benchmark is 30G.
   • As mentioned above, for formal benchmark, the size should be at least 25 times the read/write bandwidth to avoid the caching impacting the result.
5. Deploy using

   kubectl apply -f fio-cmp.yaml
   

6. Observe the result:

   kubectl logs -l kbench=fio -f
   

7. Cleanup:

   kubectl delete -f fio-cmp.yaml
   

Note: a comparison benchmark for FIO will take about 12 minutes to finish.

See ./deploy/fio-cmp.yaml for available options.

docker run -v /volume longhornio/kbench:latest /volume/test.img

e.g.

docker run -e "SIZE=100M" -v /volume longhornio/kbench:latest /volume/test.img

Notice in this case, fio is required locally.

./fio/run.sh <test_file> <output_prefix>

e.g.

./fio/run.sh /dev/sdaxxx ~/fio-results/Samsung_850_PRO_512GB/raw-bloc

Intermediate result will be saved into <output_prefix>-iops.json, <output_prefix>-bandwidth.json and <output_prefix>-latency.json. The output will be printed out as well as saved into <output_prefix>.summary.

Copyright (c) 2021 Sheng Yang

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.