Cloud-oriented Life

Cloud Native Technology Improves Lives

Posted on Edited on In , , Word count in article: 2.6k Reading time ≈ 2 mins.

postmark-rails

Postmark: Application Email Service with Exceptional Delivery - https://postmarkapp.com/ allows you to send your emails with high delivery rates. It also includes detailed statistics. In addition, Postmark can parse incoming emails which are forwarded back to your application.

The Postmark Rails Gem is a drop-in plug-in for ActionMailer to send emails via [Postmark - https://postmarkapp.com/)(https://postmarkapp.com/). The gem has been created for fast implementation and fully supports all of Postmark’s features - https://postmarkapp.com/why.

Read more »

Posted on Edited on In , , Word count in article: 1k Reading time ≈ 1 mins.

TruffleRuby

TruffleRuby is the GraalVM high-performance implementation of the Ruby programming language.

GraalVM is a high-performance multilingual runtime. It is designed to accelerate the execution of applications written in Java and other JVM languages while also providing runtimes for JavaScript, Ruby, Python, and a number of other popular languages. GraalVM’s polyglot capabilities make it possible to mix multiple programming languages in a single application while eliminating any foreign language call costs.

Read more »

Posted on Edited on In , Word count in article: 2.3k Reading time ≈ 2 mins.

Mirrors

Mirror sites or mirrors are replicas of other websites or any network node. The concept of mirroring applies to network services accessible through any protocol, such as HTTP or FTP. Such sites have different URLs than the original site, but host identical or near-identical content.

The purpose of mirrors is to reduce network traffic, improve access speed, ensure availability of the original site for technical or political reasons or provide a real-time backup of the original site.

This article describe a few useful mirrors in China Mainland.

Read more »

Posted on Edited on In , Word count in article: 1.6k Reading time ≈ 1 mins.

ctop

Top-like interface for container metrics

ctop provides a concise and condensed overview of real-time metrics for multiple containers:

ctop

as well as a single container view for inspecting a specific container.

ctop comes with built-in support for Docker and runC; connectors for other container and cluster systems are planned for future releases.

Read more »

Posted on Edited on In , Word count in article: 17k Reading time ≈ 15 mins.

FIO

Fio was originally written to save me the hassle of writing special test case programs when I wanted to test a specific workload, either for performance reasons or to find/reproduce a bug. The process of writing such a test app can be tiresome, especially if you have to do it often. Hence I needed a tool that would be able to simulate a given I/O workload without resorting to writing a tailored test case again and again.

A test work load is difficult to define, though. There can be any number of processes or threads involved, and they can each be using their own way of generating I/O. You could have someone dirtying large amounts of memory in an memory mapped file, or maybe several threads issuing reads using asynchronous I/O. fio needed to be flexible enough to simulate both of these cases, and many more.

Fio spawns a number of threads or processes doing a particular type of I/O action as specified by the user. fio takes a number of global parameters, each inherited by the thread unless otherwise parameters given to them overriding that setting is given. The typical use of fio is to write a job file matching the I/O load one wants to simulate.

Installation

1
2
3
4
5
# CentOS
$ yum install -y fio

# Debian, Ubuntu
$ apt install -y fio

Usages

Running fio

Running fio is normally the easiest part - you just give it the job file (or job files) as parameters:

1
$ fio [options] [jobfile] ...

and it will start doing what the jobfile tells it to do. You can give more than one job file on the command line, fio will serialize the running of those files. Internally that is the same as using the stonewall parameter described in the parameter section.

If the job file contains only one job, you may as well just give the parameters on the command line. The command line parameters are identical to the job parameters, with a few extra that control global parameters. For example, for the job file parameter iodepth=2, the mirror command line option would be --iodepth 2 or --iodepth=2. You can also use the command line for giving more than one job entry. For each --name option that fio sees, it will start a new job with that name. Command line entries following a --name entry will apply to that job, until there are no more entries or a new --name entry is seen. This is similar to the job file options, where each option applies to the current job until a new [] job entry is seen.

fio does not need to run as root, except if the files or devices specified in the job section requires that. Some other options may also be restricted, such as memory locking, I/O scheduler switching, and decreasing the nice value.

If jobfile is specified as -, the job file will be read from standard input.

See 1.9. How fio works | 1. fio - Flexible I/O tester rev. 3.27 — fio 3.27 documentation - https://fio.readthedocs.io/en/latest/fio_doc.html#how-fio-works to learn more.

Interpreting the output

Some important metrics within the output.

  • bw

    Bandwidth statistics based on samples. Same names as the xlat stats, but also includes the number of samples taken (samples) and an approximate percentage of total aggregate bandwidth this thread received in its group (per). This last value is only really useful if the threads in this group are on the same disk, since they are then competing for disk access.

  • iops

    IOPS statistics based on samples. Same names as bw.

See 1.14. Interpreting the output | 1. fio - Flexible I/O tester rev. 3.27 — fio 3.27 documentation - https://fio.readthedocs.io/en/latest/fio_doc.html#interpreting-the-output to learn more.

Test disk device

Create or edit fio.conf.disk configuration file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# fio.conf.disk

[global]
ioengine=libaio
iodepth=128
time_based
direct=1
thread=1
group_reporting
randrepeat=0
norandommap
numjobs=32
timeout=6000
runtime=120

[randread-4k]
rw=randread
bs=4k
filename=/dev/vdb
rwmixread=100
stonewall

[randwrite-4k]
rw=randwrite
bs=4k
filename=/dev/vdb
stonewall

[read-512k]
rw=read
bs=512k
filename=/dev/vdb
stonewall

[write-512k]
rw=write
bs=512k
filename=/dev/vdb
stonewall

Run fio fio.conf.disk to test disk performance.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
$ fio fio.conf.disk
randread-4k: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
randwrite-4k: (g=1): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
read-512k: (g=2): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=128
...
write-512k: (g=3): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=128
...
fio-3.19
Starting 128 threads
Jobs: 32 (f=32): [_(96),W(32)][4.0%][w=222MiB/s][w=443 IOPS][eta 03h:12m:00s]
randread-4k: (groupid=0, jobs=32): err= 0: pid=3426725: Wed Nov 17 20:56:13 2021
  read: IOPS=11.9k, BW=46.5MiB/s (48.8MB/s)(5581MiB/120025msec)
    slat (nsec): min=1283, max=228640k, avg=2685615.46, stdev=6012190.22
    clat (msec): min=10, max=1288, avg=341.29, stdev=118.00
     lat (msec): min=10, max=1289, avg=343.98, stdev=118.80
    clat percentiles (msec):
     |  1.00th=[   70],  5.00th=[  180], 10.00th=[  211], 20.00th=[  251],
     | 30.00th=[  279], 40.00th=[  300], 50.00th=[  330], 60.00th=[  359],
     | 70.00th=[  388], 80.00th=[  430], 90.00th=[  489], 95.00th=[  550],
     | 99.00th=[  684], 99.50th=[  751], 99.90th=[  919], 99.95th=[ 1003],
     | 99.99th=[ 1133]
   bw (  KiB/s): min=14374, max=118348, per=99.78%, avg=47509.90, stdev=473.74, samples=7648
   iops        : min= 3584, max=29578, avg=11876.11, stdev=118.43, samples=7648
  lat (msec)   : 20=0.01%, 50=0.34%, 100=1.16%, 250=18.68%, 500=70.66%
  lat (msec)   : 750=8.67%, 1000=0.43%, 2000=0.05%
  cpu          : usr=0.06%, sys=0.25%, ctx=1264741, majf=0, minf=4128
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=1428809,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
randwrite-4k: (groupid=1, jobs=32): err= 0: pid=3433113: Wed Nov 17 20:56:13 2021
  write: IOPS=11.8k, BW=46.2MiB/s (48.4MB/s)(5546MiB/120027msec); 0 zone resets
    slat (nsec): min=1190, max=253603k, avg=2702464.17, stdev=6005163.86
    clat (msec): min=10, max=1312, avg=342.81, stdev=117.60
     lat (msec): min=10, max=1322, avg=345.52, stdev=118.39
    clat percentiles (msec):
     |  1.00th=[  110],  5.00th=[  180], 10.00th=[  211], 20.00th=[  251],
     | 30.00th=[  279], 40.00th=[  300], 50.00th=[  330], 60.00th=[  359],
     | 70.00th=[  388], 80.00th=[  430], 90.00th=[  493], 95.00th=[  550],
     | 99.00th=[  684], 99.50th=[  743], 99.90th=[  961], 99.95th=[ 1062],
     | 99.99th=[ 1167]
   bw (  KiB/s): min=14798, max=106798, per=99.86%, avg=47246.76, stdev=464.02, samples=7648
   iops        : min= 3694, max=26697, avg=11809.98, stdev=116.01, samples=7648
  lat (msec)   : 20=0.01%, 50=0.15%, 100=0.68%, 250=19.28%, 500=70.51%
  lat (msec)   : 750=8.90%, 1000=0.40%, 2000=0.08%
  cpu          : usr=0.08%, sys=0.26%, ctx=1269905, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,1419651,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
read-512k: (groupid=2, jobs=32): err= 0: pid=3439498: Wed Nov 17 20:56:13 2021
  read: IOPS=443, BW=222MiB/s (233MB/s)(26.1GiB/120314msec)
    slat (usec): min=4, max=1521.5k, avg=71910.72, stdev=86011.65
    clat (msec): min=228, max=15650, avg=8887.05, stdev=1962.66
     lat (msec): min=267, max=15959, avg=8958.96, stdev=1967.34
    clat percentiles (msec):
     |  1.00th=[ 1435],  5.00th=[ 5873], 10.00th=[ 7013], 20.00th=[ 7819],
     | 30.00th=[ 8221], 40.00th=[ 8658], 50.00th=[ 8926], 60.00th=[ 9329],
     | 70.00th=[ 9731], 80.00th=[10268], 90.00th=[10939], 95.00th=[11745],
     | 99.00th=[13355], 99.50th=[13892], 99.90th=[14697], 99.95th=[15100],
     | 99.99th=[15368]
   bw (  KiB/s): min=32703, max=648006, per=99.86%, avg=226951.41, stdev=3447.82, samples=7109
   iops        : min=   45, max= 1262, avg=440.60, stdev= 6.74, samples=7109
  lat (msec)   : 250=0.02%, 500=0.23%, 750=0.18%, 1000=0.21%, 2000=0.82%
  lat (msec)   : >=2000=98.53%
  cpu          : usr=0.01%, sys=0.06%, ctx=110802, majf=0, minf=519723
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.5%, 16=1.0%, 32=1.9%, >=64=96.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=53407,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
write-512k: (groupid=3, jobs=32): err= 0: pid=3445908: Wed Nov 17 20:56:13 2021
  write: IOPS=443, BW=222MiB/s (232MB/s)(26.0GiB/120361msec); 0 zone resets
    slat (usec): min=16, max=1017.5k, avg=72029.58, stdev=71276.32
    clat (msec): min=34, max=13151, avg=8858.97, stdev=1733.50
     lat (msec): min=34, max=13301, avg=8931.01, stdev=1737.43
    clat percentiles (msec):
     |  1.00th=[ 1045],  5.00th=[ 5738], 10.00th=[ 7617], 20.00th=[ 8221],
     | 30.00th=[ 8557], 40.00th=[ 8792], 50.00th=[ 9060], 60.00th=[ 9329],
     | 70.00th=[ 9597], 80.00th=[ 9866], 90.00th=[10402], 95.00th=[10805],
     | 99.00th=[11745], 99.50th=[11879], 99.90th=[12684], 99.95th=[12953],
     | 99.99th=[13087]
   bw (  KiB/s): min=36806, max=606274, per=100.00%, avg=227128.25, stdev=2963.38, samples=7087
   iops        : min=   60, max= 1177, avg=438.87, stdev= 5.80, samples=7087
  lat (msec)   : 50=0.04%, 100=0.08%, 250=0.11%, 500=0.28%, 750=0.23%
  lat (msec)   : 1000=0.22%, 2000=0.82%, >=2000=98.22%
  cpu          : usr=0.06%, sys=0.05%, ctx=118979, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.5%, 16=1.0%, 32=1.9%, >=64=96.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,53323,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=46.5MiB/s (48.8MB/s), 46.5MiB/s-46.5MiB/s (48.8MB/s-48.8MB/s), io=5581MiB (5852MB), run=120025-120025msec

Run status group 1 (all jobs):
  WRITE: bw=46.2MiB/s (48.4MB/s), 46.2MiB/s-46.2MiB/s (48.4MB/s-48.4MB/s), io=5546MiB (5815MB), run=120027-120027msec

Run status group 2 (all jobs):
   READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=26.1GiB (28.0GB), run=120314-120314msec

Run status group 3 (all jobs):
  WRITE: bw=222MiB/s (232MB/s), 222MiB/s-222MiB/s (232MB/s-232MB/s), io=26.0GiB (27.0GB), run=120361-120361msec

Disk stats (read/write):
  vdb: ios=1554368/1554036, merge=2825/305, ticks=60894783/60980848, in_queue=121875631, util=99.56%

Test disk file

Create or edit fio.conf.file configuration file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# fio.conf.file

[global]
ioengine=libaio
iodepth=128
time_based
direct=1
thread=1
group_reporting
randrepeat=0
norandommap
numjobs=32
timeout=6000
runtime=120
size=5G

[randread-4k]
rw=randread
bs=4k
filename=/root/test.img
rwmixread=100
stonewall

[randwrite-4k]
rw=randwrite
bs=4k
filename=/root/test.img
stonewall

[read-512k]
rw=read
bs=512k
filename=/root/test.img
stonewall

[write-512k]
rw=write
bs=512k
filename=/root/test.img
stonewall

Run fio fio.conf.file to test disk performace.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
$ fio fio.conf.file
randread-4k: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
randwrite-4k: (g=1): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
...
read-512k: (g=2): rw=read, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=128
...
write-512k: (g=3): rw=write, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=128
...
fio-3.19
Starting 128 threads
randread-4k: Laying out IO file (1 file / 5120MiB)
Jobs: 30 (f=30): [_(96),W(12),_(1),W(6),_(1),W(12)][37.3%][w=360MiB/s][w=720 IOPS][eta 13m:30s]
randread-4k: (groupid=0, jobs=32): err= 0: pid=1535608: Thu Nov 18 09:11:15 2021
  read: IOPS=11.9k, BW=46.4MiB/s (48.6MB/s)(5564MiB/120025msec)
    slat (nsec): min=1534, max=168704k, avg=2694411.30, stdev=4645972.12
    clat (msec): min=10, max=1069, avg=342.33, stdev=74.36
     lat (msec): min=17, max=1118, avg=345.03, stdev=74.85
    clat percentiles (msec):
     |  1.00th=[  107],  5.00th=[  241], 10.00th=[  262], 20.00th=[  288],
     | 30.00th=[  309], 40.00th=[  321], 50.00th=[  342], 60.00th=[  359],
     | 70.00th=[  380], 80.00th=[  401], 90.00th=[  430], 95.00th=[  460],
     | 99.00th=[  531], 99.50th=[  567], 99.90th=[  718], 99.95th=[  776],
     | 99.99th=[  902]
   bw (  KiB/s): min=24114, max=117969, per=99.85%, avg=47395.97, stdev=305.48, samples=7648
   iops        : min= 6026, max=29481, avg=11848.53, stdev=76.36, samples=7648
  lat (msec)   : 20=0.01%, 50=0.04%, 100=0.89%, 250=5.86%, 500=91.40%
  lat (msec)   : 750=1.73%, 1000=0.07%, 2000=0.01%
  cpu          : usr=0.06%, sys=0.27%, ctx=1296227, majf=0, minf=4128
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=1424347,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
randwrite-4k: (groupid=1, jobs=32): err= 0: pid=1541968: Thu Nov 18 09:11:15 2021
  write: IOPS=11.8k, BW=46.0MiB/s (48.3MB/s)(5527MiB/120027msec); 0 zone resets
    slat (nsec): min=1603, max=169727k, avg=2712193.76, stdev=4974963.08
    clat (usec): min=501, max=921119, avg=344609.13, stdev=67128.86
     lat (usec): min=504, max=929592, avg=347321.59, stdev=67565.75
    clat percentiles (msec):
     |  1.00th=[  178],  5.00th=[  249], 10.00th=[  271], 20.00th=[  292],
     | 30.00th=[  309], 40.00th=[  330], 50.00th=[  342], 60.00th=[  359],
     | 70.00th=[  380], 80.00th=[  401], 90.00th=[  430], 95.00th=[  451],
     | 99.00th=[  510], 99.50th=[  542], 99.90th=[  676], 99.95th=[  751],
     | 99.99th=[  852]
   bw (  KiB/s): min=25024, max=86996, per=99.89%, avg=47100.88, stdev=261.59, samples=7648
   iops        : min= 6255, max=21748, avg=11775.01, stdev=65.40, samples=7648
  lat (usec)   : 750=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.09%
  lat (msec)   : 100=0.23%, 250=4.98%, 500=93.49%, 750=1.14%, 1000=0.05%
  cpu          : usr=0.07%, sys=0.28%, ctx=1285046, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,1414991,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
read-512k: (groupid=2, jobs=32): err= 0: pid=1548302: Thu Nov 18 09:11:15 2021
  read: IOPS=443, BW=222MiB/s (233MB/s)(26.1GiB/120317msec)
    slat (usec): min=9, max=930894, avg=71954.82, stdev=63916.45
    clat (msec): min=121, max=13298, avg=8889.65, stdev=1711.42
     lat (msec): min=259, max=13401, avg=8961.61, stdev=1714.61
    clat percentiles (msec):
     |  1.00th=[ 1401],  5.00th=[ 6141], 10.00th=[ 7483], 20.00th=[ 8154],
     | 30.00th=[ 8490], 40.00th=[ 8792], 50.00th=[ 9060], 60.00th=[ 9329],
     | 70.00th=[ 9597], 80.00th=[10000], 90.00th=[10537], 95.00th=[11073],
     | 99.00th=[11745], 99.50th=[12013], 99.90th=[12818], 99.95th=[12953],
     | 99.99th=[13221]
   bw (  KiB/s): min=35829, max=499679, per=99.12%, avg=225160.35, stdev=2695.71, samples=7169
   iops        : min=   67, max=  975, avg=438.92, stdev= 5.27, samples=7169
  lat (msec)   : 250=0.03%, 500=0.21%, 750=0.22%, 1000=0.19%, 2000=0.84%
  lat (msec)   : >=2000=98.51%
  cpu          : usr=0.01%, sys=0.07%, ctx=121043, majf=0, minf=524320
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.5%, 16=1.0%, 32=1.9%, >=64=96.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=53381,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128
write-512k: (groupid=3, jobs=32): err= 0: pid=1550227: Thu Nov 18 09:11:15 2021
  write: IOPS=442, BW=221MiB/s (232MB/s)(26.0GiB/120384msec); 0 zone resets
    slat (usec): min=16, max=1135.3k, avg=72030.60, stdev=59737.86
    clat (msec): min=79, max=14388, avg=8853.39, stdev=1881.03
     lat (msec): min=79, max=14496, avg=8925.42, stdev=1887.27
    clat percentiles (msec):
     |  1.00th=[ 1003],  5.00th=[ 5537], 10.00th=[ 7416], 20.00th=[ 8020],
     | 30.00th=[ 8423], 40.00th=[ 8792], 50.00th=[ 9060], 60.00th=[ 9329],
     | 70.00th=[ 9597], 80.00th=[10000], 90.00th=[10671], 95.00th=[11208],
     | 99.00th=[13221], 99.50th=[13624], 99.90th=[14160], 99.95th=[14160],
     | 99.99th=[14295]
   bw (  KiB/s): min=40950, max=624837, per=99.90%, avg=226555.80, stdev=2902.55, samples=7107
   iops        : min=   78, max= 1219, avg=441.72, stdev= 5.67, samples=7107
  lat (msec)   : 100=0.05%, 250=0.16%, 500=0.33%, 750=0.25%, 1000=0.22%
  lat (msec)   : 2000=0.84%, >=2000=98.16%
  cpu          : usr=0.06%, sys=0.05%, ctx=122810, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.5%, 16=1.0%, 32=1.9%, >=64=96.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,53322,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=46.4MiB/s (48.6MB/s), 46.4MiB/s-46.4MiB/s (48.6MB/s-48.6MB/s), io=5564MiB (5834MB), run=120025-120025msec

Run status group 1 (all jobs):
  WRITE: bw=46.0MiB/s (48.3MB/s), 46.0MiB/s-46.0MiB/s (48.3MB/s-48.3MB/s), io=5527MiB (5796MB), run=120027-120027msec

Run status group 2 (all jobs):
   READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=26.1GiB (27.0GB), run=120317-120317msec

Run status group 3 (all jobs):
  WRITE: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=26.0GiB (27.0GB), run=120384-120384msec

Disk stats (read/write):
  vda: ios=1560424/1561420, merge=620/2869, ticks=60567966/61121545, in_queue=121689511, util=99.22%

FAQs

you need to specify size=

1
2
write-512k: you need to specify size=
fio: pid=0, err=22/file:filesetup.c:1057, func=total_file_size, error=Invalid argument

Specify size in the configuration file to fix that issue.

1
2
3
[global]
# ...
size=5G

References

[1] axboe/fio: Flexible I/O Tester - https://github.com/axboe/fio

[2] Welcome to FIO’s documentation! — fio 3.27 documentation - https://fio.readthedocs.io/en/latest/index.html

Posted on Edited on In , Word count in article: 3.1k Reading time ≈ 3 mins.

rsync

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Read more »

Posted on Edited on In , Word count in article: 3.8k Reading time ≈ 3 mins.

sshpass

The sshpass utility is designed to run SSH using the keyboard-interactive password authentication mode, but in a non-interactive way.

SSH uses direct TTY access to ensure that the password is indeed issued by an interactive keyboard user. sshpass runs SSH in a dedicated TTY, fooling SSH into thinking it is getting the password from an interactive user.

Installation

1
2
3
4
5
6
7
8
9
10
# CentOS, 
$ suod yum install -y sshpass

# macOS
$ brew install hudochenkov/sshpass/sshpass

Or
# $ brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb

# $ brew install http://git.io/sshpass.rb

Usages

Synopsis

1
sshpass [-ffilename|-dnum|-ppassword|-e] [options] command arguments

Options

1
2
3
4
5
6
7
8
9
10
11
-  -ppassword
      The password is given on the command line. 

-   -ffilename
      The password is the first line of the file filename. 

-   -dnumber
      number is a file descriptor inherited by sshpass from the runner. The password is read from the open file descriptor. 

-   -e
      The password is taken from the environment variable "SSHPASS".

SSH

Use sshpass with ssh:

Use the -p option

1
$ sshpass -p "<Your Password>" ssh [email protected]

Use the -e option

1
2
3
$ export SSHPASS="<Your Password>"

$ sshpass -e ssh -Y [email protected]

Use the -f option

1
2
3
4
5
$ echo "<Your Password>" > ./pass_file

$ chmod 0400 ./pass_file

$ sshpass -f ./pass_file ssh [email protected]

Rsync

Use sshpass with rsync:

Uses the -e option, which passes the password to the environment variable SSHPASS

1
$ SSHPASS='<Your Password>' rsync --rsh="sshpass -e ssh -l username" /custom/ host.example.com:/opt/custom/

We can use the -f switch like this:

1
$ rsync --rsh="sshpass -f pass_file ssh -l username" /custom/ host.example.com:/opt/custom/

Scp

Use sshpass with scp:

1
$ scp -r /var/www/html/example.com --rsh="sshpass -f pass_file ssh -l user" host.example.com:/var/www/html

GPG

You can also use sshpass with a GPG-encrypted file. When the -f switch is used, the reference file is in plaintext. Let’s see how we can encrypt a file with GPG and use it.

First, create a file as follows:

1
$ echo '<Your Password>' > .sshpasswd

Next, encrypt the file using the gpg command:

1
$ gpg -c .sshpasswd

Remove the file which contains the plaintext:

1
$ rm .sshpasswd

Finally, use it as follows:

1
$ gpg -d -q .sshpassword.gpg > pass_file; sshpass -f pass_file ssh [email protected]

FAQs

We won’t add sshpass because it makes it too easy for novice SSH users to ruin SSH’s security on macOS with brew

1
2
3
4
5
6
$ brew install sshpass
...
Warning: No available formula or cask with the name "sshpass". 
...
We won't add sshpass because it makes it too easy for novice SSH users to
ruin SSH's security.

Install from the unofficial package.

1
2
3
4
5
6
7
# macOS
$ brew install hudochenkov/sshpass/sshpass

Or
# $ brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb

# $ brew install http://git.io/sshpass.rb

Non-checksummed download of sshpass formula file from an arbitrary URL is unsupported!

1
2
$ brew install https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb
`brew extract` or `brew create` and `brew tap-new` to create a formula file in a tap on GitHub instead.: Invalid usage: Non-checksummed download of sshpass formula file from an arbitrary URL is unsupported!  (UsageError)

Try to download and brew install it.

1
2
3
$ wget https://raw.githubusercontent.com/kadwanev/bigboybrew/master/Library/Formula/sshpass.rb

$ brew install sshpass.rb

References

[1] sshpass(1) - Linux man page - https://linux.die.net/man/1/sshpass

[2] SSH password automation in Linux with sshpass | Enable Sysadmin - https://www.redhat.com/sysadmin/ssh-automation-sshpass

[3] Avoid typing ssh passwords with sshpass on macOS | Bits and Dragons - https://bitsanddragons.wordpress.com/2021/05/27/avoid-typing-ssh-passwords-with-sshpass-on-macos/

[4] How to install sshpass on mac? | Newbedev - https://newbedev.com/how-to-install-sshpass-on-mac

[5] Ansible: To use the ‘ssh’ connection type with passwords, you must install the sshpass program - bidhankhatri.com.np - https://bidhankhatri.com.np/automation/ansible-ssh-connection-type-with-passwords-error/

Posted on Edited on In , Word count in article: 2.2k Reading time ≈ 2 mins.

OpenSSL

OpenSSL is a robust, commercial-grade, and full-featured toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols. It is also a general-purpose cryptography library. For more information about the team and community around the project, or to start making your own contributions, start with the community page. To get the latest news, download the source, and so on, please see the sidebar or the buttons at the top of every page.

In this article, we’ll learn how to create a self-signed certificate with OpenSSL.

Read more »
0%