Post

Linux -- IO

Posted Updated
By Xiong Ding 2 min read

iostat is a simple tool to monitor disk activity and latency. I ran it inside an AWS EC2 instance, and output is shown below.

1
2
3
4
5
6
7
8
9
10
11
12

> iostat -d -x -m 1
Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1          6.00      0.05     0.00   0.00    3.17     8.00 2561.00    125.05    40.00   1.54    5.81    50.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.89 100.00

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1         59.00      0.48     0.00   0.00    2.05     8.27 2608.00    124.62    42.00   1.58    6.59    48.93    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   17.32 100.00

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1         14.00      0.12     0.00   0.00    2.64     9.14 2566.00    124.72    48.00   1.84    6.09    49.77    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   15.67 100.00

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0n1         33.00      0.26     0.00   0.00    3.24     8.00 1925.00    124.94    47.00   2.38    9.57    66.46    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   18.54 100.00

The above command prints io stats every second. Some important columns:

  • IPOS = r/s + w/s which is in the range 2000 - 2500.
  • throughput = rMB/s + wMB/s ~= 125MB/s
  • aqu-sz: average queue length.
  • %util: bandwidth utilization for the device. 100% means device saturation. Usually, if aqu-sz is large, then %util is 100%.

You can find corresponding monitoring graphs in the corresponding EC2 dashboard.

io_uring

Submission queue + completion queue

TODO: read more on this topic

select, poll, epoll

Stack overflow has some posts that discuess why epoll is faster than select/poll. For example this one. The key difference in my opinion is that epoll registers callback to all file descriptor it has interest in, so any event arrives, it will get triggered. This is a typical event driven design. While select/poll iterates over all file descriptors each time it is called.

TODO: read the source code

os, linux
linux io
This post is licensed under CC BY 4.0 by the author.