fio 离线下载包 fio-3.29

This document describes use and implementation of a new fio histogram-based latency

percentile measurement tool.

History

Why the old tool wasn’t good enough

Design and implementation

Histogram log parsing

History

As a result of the need to measure I/O latency percentiles for a cluster with hundreds or even

thousands of storage users, Karl Cronberg and I worked on the fiologparser_hist.py tool 2 years

ago [1]. First he added the capability into fio to emit periodic latency histograms, which it

already had been maintaining in memory, to a log file. Once he wrote the tool to postprocess

this data, we then had a new ability to calculate latency percentiles as a function of time for a

large distributed-storage cluster, so that we could understand how cluster events, such as node

failures, impact response time of real applications. The results were pretty shocking - max

2) upstream release did.

Why the old tool wasn’t good enough

I got frustrated with fiologparser_hist.py, the tool for calculating fio latency percentiles as a

function of time, because of:

- its dependency on python pandas module, which pulls in a LOT of packages

- its slowness (can take as long as an hour to process a set of histogram logs)

- hard to understand what it's doing

- hard to prove that it is generating the right results

I'm not aware of any other tool that does the same thing. So I rewrote it as

fio-histo-log-pctiles.py. Differences between this implementation and the old one are:

to be giving very similar answers. It is also giving similar answers to fiologparser_hist.py.

To see the syntax, just run “python fio-histo-log-pctiles.py --help”, it is fairly self-explanatory with

the exception of the --time-quantum parameter, which is explained below.

Design and implementation

In summary, all input histogram logs are normalized to a fixed set of aligned time intervals (assumes good time

synchronization across hosts), so that histograms can then be added directly to obtain cluster-wide histogram,

and then latency percentiles are computed from that by summation.

● direction - 0 if the histogram is for reads, 1 if for writes.

direction : (read/write) is only useful at present for finding the end time of the histogram time interval (present

in the next histogram record). The tool merges read and write histograms together at present. However, we

need to keep this field because someday the tool might support separate read and write perf latency

percentiles (yes they can be really different).

time_ms : Separate histogram records are emitted for reads and for writes on the same time interval. And they

can be emitted in any order (read,write) or (write,read). Consequently, if we want to find the subsequent

histogram record for that I/O direction (read or write) in order to get the end_time for the current record, we may

have to read as many as 3 records farther to get it. For example, here’s an excerpt from a real histogram log:

have to be identical.

When we encounter the last histogram record (for an I/O direction) in the log, we can no longer get the end

time from the next histogram record, since there isn’t one, so we instead estimate the end time as the test end

time in millisec.

bs - this field is really not used at all. There are 2 cases: either we are using a fixed I/O transfer size or a

variable one. If the I/O transfer size is fixed, then it is in the command or job file used to run fio and is not

needed in the result. If the I/O transfer size is variable, which fio supports, then the bs field is meaningless

since a variety of I/O transfer sizes would have been used in a single histogram interval. So either way we can