Important difference in timing of I/O writes in SLE 12

SUSE Linux Enterprise Server 12

 SLE 12 introduced a number of I/O improvements that enhance thread fairness, predictability of I/O behavior and minimize I/O stalls.  The bulk of these changes are transparent to the user but there are exceptions.
This TID discusses a change in the default vm.dirty_ratio sysctl.

The system is required to limit what percentage of the systems memory contains file-backed data that needs writing to disk. This guarantees that the system can always allocate the necessary data structures to complete I/O. The maximum amount of memory that may be dirty and requires writing at any given time is controlled by vm.dirty_ratio and the defaults are:

SLE-11-SP3:    vm.dirty_ratio = 40
SLE-12:        vm.dirty_ratio = 20

The primary advantage of using the lower ratio in SLE 12 is that page reclamation and allocation in low memory situations completes faster as there is a higher probability that old clean pages will be quickly found and discarded.  The secondary advantage is that if all data on the system must be synchronized then the time to complete the operation on SLE-12 will be lower than SLE-11-SP3 by default. Most workloads will not notice this change as data is synchronized with fsync() by the application or data is not dirtied quickly enough to hit the limits.

There are exceptions and if your application is affected by this, it will manifest as an unexpected stall during writes. To prove it is affected by dirty data rate limiting then monitor /proc/PID_OF_APPLICATION/stack and it will be observed that the application spends significant time in balance_dirty_pages_ratelimited.  If this is observed and it is a problem  then increase the value of vm.dirty_ratio to 40 to restore the SLE-11-SP3 behaviour.

It is important to note that the overall I/O throughput is the same regardless of the setting. The only difference is the timing of when the I/O is queued.
This is an example of using dd to asynchronously write 30% of memory to disk which would happen to be affected by the change in vm.dirty_ratio.

$ MEMTOTAL_MBYTES=`free -m | grep Mem: | awk '{print $2}'`
$ sysctl vm.dirty_ratio=40
$ dd if=/dev/zero of=zerofile ibs=1048576 count=$((MEMTOTAL_MBYTES*30/100))
2507145216 bytes (2.5 GB) copied, 8.00153 s, 313 MB/s
$ sysctl vm.dirty_ratio=20
dd if=/dev/zero of=zerofile ibs=1048576 count=$((MEMTOTAL_MBYTES*30/100))
2507145216 bytes (2.5 GB) copied, 10.1593 s, 247 MB/s

Note that the parameter affects the time it takes for the command to complete as well as the apparent write speed of the device. With dirty_ratio=40, more of the data is cached and written to disk in the background by the kernel. It is very important to note that the speed of I/O is identical in both cases. To demonstrate, this is the result when dd synchronizes the data before exiting.

$ sysctl vm.dirty_ratio=40
$ dd if=/dev/zero of=zerofile ibs=1048576 count=$((MEMTOTAL_MBYTES*30/100)) conv=fdatasync
2507145216 bytes (2.5 GB) copied, 21.0663 s, 119 MB/s
$ sysctl vm.dirty_ratio=20
$ dd if=/dev/zero of=zerofile ibs=1048576 count=$((MEMTOTAL_MBYTES*30/100)) conv=fdatasync
2507145216 bytes (2.5 GB) copied, 21.7286 s, 115 MB/s

Note that dirty_ratio had almost no impact here and is within the natural variability of a command. Hence, dirty_ratio does not directly impact I/O performance but it may affect the apparent performance of a workload that writes data asynchronously without synchronising.