写在前面

我突然又明白，死亡是聪明的兄长，我们可以放心地把自己托付给他，他会知道在我们有所准备的适当时刻前来。我也突然懂得，原来痛苦、失望和悲愁不是为了惹恼我们，使我们气馁或者无地自容；它们的存在，是为了使我们心智成熟，臻于完善。—赫尔曼·黑塞《彼得·卡门青》

零、系统调优原理概述：

通过调整系统设置来提高计算机资源的利用率、数据吞吐量或用戶体验的过程，消除或减轻系统中的排队现象。

分析系统性能的方法：

执行系统变更的步骤：

一、如何检测系统的性能瓶颈

1、检测系统当前运行负载：vmstat、mpstat、iostat、top 等

装包，一般不需要

yum install -y sysstat

sysstat包含监测系统性能及效率的一组工具，这些工具对于我们收集系统性能数据，比如 CPU 使用率、硬盘和网络吞吐数据，这些数据的收集和分析，有利于我们判断系统是否正常运行，是提高系统运行效率、安全运行服务器的得力助手。

vmstat 内存子系统监控

┌──[root@liruilongs.github.io]-[~] └─$ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 3228884 2076 594152 0 0 10 6 102 152 0 0 100 0 0 ┌──[root@liruilongs.github.io]-[~] └─$ vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 3228852 2076 594184 0 0 10 6 102 152 0 0 100 0 0 # 平均值 0 0 0 3228852 2076 594184 0 0 0 0 197 289 0 1 99 0 0 0 0 0 3228852 2076 594184 0 0 0 0 192 290 0 0 100 0 0 1 0 0 3228852 2076 594184 0 0 0 0 180 285 0 0 100 0 0 0 0 0 3228852 2076 594184 0 0 0 0 174 276 1 0 99 0 0 0 0 0 3228852 2076 594184 0 0 0 0 173 267 0 0 100 0 0 0 0 0 3228852 2076 594184 0 0 0 0 176 268 0 0 100 0 0 0 0 0 3228852 2076 594184 0 0 0 0 173 274 0 0 100 0 0 ^C ┌──[root@liruilongs.github.io]-[~] └─$

mpstat CUP子系统监控

┌──[root@liruilongs.github.io]-[~] └─$ mpstat 1 Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/23/2021 _x86_64_ (2 CPU) 01:41:22 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 01:41:23 AM all 0.52 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 99.48 01:41:24 AM all 0.00 0.00 0.51 0.00 0.00 0.00 0.00 0.00 0.00 99.49 01:41:25 AM all 0.00 0.00 0.00 0.00 0.00 0.52 0.00 0.00 0.00 99.48 01:41:26 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 01:41:27 AM all 0.00 0.00 0.52 0.00 0.00 0.00 0.00 0.00 0.00 99.48 01:41:28 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 01:41:29 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 ^C Average: all 0.07 0.00 0.15 0.00 0.00 0.07 0.00 0.00 0.00 99.71 ┌──[root@liruilongs.github.io]-[~] └─$ mpstat 1 3 Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/23/2021 _x86_64_ (2 CPU) 01:42:06 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle 01:42:07 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 01:42:08 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 01:42:09 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 ┌──[root@liruilongs.github.io]-[~] └─$

iostat IO子系统监控

┌──[root@liruilongs.github.io]-[~] └─$ iostat 1 3 Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/23/2021 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.21 0.00 0.24 0.05 0.00 99.50 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 0.56 19.86 12.15 387999 237243 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.52 0.00 99.48 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 1.00 8.00 0.00 8 0 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 0.00 0.00 100.00 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn sda 0.00 0.00 0.00 0 0 ┌──[root@liruilongs.github.io]-[~] └─$

top 综合情况监控

top - 01:46:43 up 5:29, 1 user, load average: 0.01, 0.02, 0.05 Tasks: 151 total, 1 running, 150 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu1 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 4030172 total, 3227596 free, 205988 used, 596588 buff/cache KiB Swap: 10485756 total, 10485756 free, 0 used. 3531292 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND 844 etcd 20 0 10.308g 22996 11100 S 2.7 0.6 11:10.71 etcd 572 root 20 0 220312 5208 3508 S 0.3 0.1 0:06.03 rsyslogd 592 root 20 0 298868 6260 4916 S 0.3 0.2 1:32.71 vmtoolsd 1 root 20 0 125408 3836 2440 S 0.0 0.1 0:09.03 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:00.61 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root rt 0 0 0 0 S 0.0 0.0 0:00.27 migration/0 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root 20 0 0 0 0 S 0.0 0.0 0:05.72 rcu_sched 10 root rt 0 0 0 0 S 0.0 0.0 0:01.25 watchdog/0 11 root rt 0 0 0 0 S 0.0 0.0 0:01.15 watchdog/1 12 root rt 0 0 0 0 S 0.0 0.0 0:00.17 migration/1 13 root 20 0 0 0 0 S 0.0 0.0 0:00.54 ksoftirqd/1 15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H 17 root 20 0 0 0 0 S 0.0 0.0 0:00.05 kdevtmpfs 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 19 root 20 0 0 0 0 S 0.0 0.0 0:00.03 khungtaskd 20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback 21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd 22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 23 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd

• load average: 0.01 不能超过 0.7
• 需要除以 cpu核数 ；数字1

2、检测系统历史运行负载：sar

┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system] └─$ pwd /usr/lib/systemd/system ┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system] └─$ cat sysstat.service # /usr/lib/systemd/system/sysstat.service # (C) 2012 Peter Schiffer (pschiffe <at> redhat.com) # # sysstat-10.1.5 systemd unit file: # Insert a dummy record in current daily data file. # This indicates that the counters have restarted from 0. [Unit] Description=Resets System Activity Logs [Service] Type=oneshot RemainAfterExit=yes User=root ExecStart=/usr/lib64/sa/sa1 --boot [Install] WantedBy=multi-user.target ┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system] └─$ systemctl status sysstat.service ● sysstat.service - Resets System Activity Logs Loaded: loaded (/usr/lib/systemd/system/sysstat.service; enabled; vendor preset: enabled) Active: active (exited) since Wed 2021-10-13 01:53:41 CST; 1 weeks 3 days ago Main PID: 584 (code=exited, status=0/SUCCESS) CGroup: /system.slice/sysstat.service Oct 13 01:53:41 liruilongs.github.io systemd[1]: Starting Resets System Activity Logs... Oct 13 01:53:41 liruilongs.github.io systemd[1]: Started Resets System Activity Logs.

红帽8没有，需要自己开启，红帽7有

┌──[root@liruilongs.github.io]-[/] └─$ cd /var/log/sa ┌──[root@liruilongs.github.io]-[/var/log/sa] └─$ ls sa01 sa03 sa04 sa05 sa06 sa08 sa09 sa13 sa14 sa23 sa24 sa30 sar03

CPU 监控信息

┌──[root@liruilongs.github.io]-[/var/log/sa] └─$ sar -f sa24 -p # CPU 监控信息 Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/24/2021 _x86_64_ (2 CPU) 08:14:46 AM LINUX RESTART 08:20:03 AM CPU %user %nice %system %iowait %steal %idle 09:40:01 AM all 1.44 0.00 1.72 0.21 0.00 96.64 09:50:01 AM all 0.80 0.00 1.51 0.01 0.00 97.69 10:00:01 AM all 0.81 0.00 1.56 0.00 0.00 97.63 10:10:01 AM all 0.74 0.00 1.40 0.02 0.00 97.83 10:20:01 AM all 0.65 0.00 1.26 0.00 0.00 98.08 10:30:01 AM all 0.78 0.02 1.50 0.04 0.00 97.67 10:40:01 AM all 3.24 0.00 5.65 0.01 0.00 91.11 Average: all 1.18 0.00 2.10 0.03 0.00 96.69

负载信息

┌──[root@liruilongs.github.io]-[/var/log/sa] └─$ sar -f sa24 -q # 负载信息 Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/24/2021 _x86_64_ (2 CPU) 08:14:46 AM LINUX RESTART 08:20:03 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked 09:40:01 AM 0 215 0.06 0.40 0.42 0 09:50:01 AM 0 215 0.41 0.21 0.28 0 10:00:01 AM 1 215 0.27 0.13 0.20 0 10:10:01 AM 1 216 0.03 0.08 0.14 0 10:20:01 AM 0 216 0.00 0.04 0.10 0 10:30:01 AM 1 217 0.00 0.03 0.06 0 10:40:01 AM 0 215 0.07 0.16 0.14 0 Average: 0 216 0.12 0.15 0.19 0

内存信息

┌──[root@liruilongs.github.io]-[/var/log/sa] └─$ sar -f sa24 -r # 内存信息 Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/24/2021 _x86_64_ (2 CPU) 08:14:46 AM LINUX RESTART 08:20:03 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 09:40:01 AM 3472808 557364 13.83 2076 266132 832840 5.74 157400 244716 8 09:50:01 AM 3470960 559212 13.88 2076 266516 832840 5.74 158004 244652 4 10:00:01 AM 3466736 563436 13.98 2076 270432 832972 5.74 158272 248436 0 10:10:01 AM 3465552 564620 14.01 2076 270856 833480 5.74 159624 247772 0 10:20:01 AM 3465732 564440 14.01 2076 271172 833584 5.74 159720 248028 0 10:30:01 AM 3457092 573080 14.22 2076 272904 833772 5.74 161872 247732 0 10:40:01 AM 3458440 571732 14.19 2076 271864 833032 5.74 160824 247256 0 Average: 3465331 564841 14.02 2076 269982 833217 5.74 159388 246942 2

网卡信息

┌──[root@liruilongs.github.io]-[/var/log/sa] └─$ sar -f sa24 -n DEV # 网卡信息 Linux 3.10.0-693.el7.x86_64 (liruilongs.github.io) 10/24/2021 _x86_64_ (2 CPU) 08:14:46 AM LINUX RESTART 08:20:03 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 09:40:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00 09:40:01 AM ens32 0.34 0.37 0.04 0.33 0.00 0.00 0.00 09:40:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00 09:40:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 09:50:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00 09:50:01 AM ens32 1.19 0.78 0.10 0.36 0.00 0.00 0.00 09:50:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00 09:50:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:00:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:00:01 AM ens32 0.20 0.13 0.02 0.01 0.00 0.00 0.00 10:00:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00 10:00:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:10:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:10:01 AM ens32 0.49 0.50 0.04 0.54 0.00 0.00 0.00 10:10:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00 10:10:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:20:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:20:01 AM ens32 0.17 0.11 0.01 0.01 0.00 0.00 0.00 10:20:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00 10:20:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:30:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:30:01 AM ens32 0.04 0.03 0.00 0.00 0.00 0.00 0.00 10:30:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00 10:30:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:40:01 AM br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10:40:01 AM ens32 0.08 0.06 0.01 0.00 0.00 0.00 0.00 10:40:01 AM lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00 10:40:01 AM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s Average: br-4b3da203747c 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: ens32 0.36 0.28 0.03 0.17 0.00 0.00 0.00 Average: lo 0.13 0.13 0.01 0.01 0.00 0.00 0.00 Average: docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ┌──[root@liruilongs.github.io]-[/var/log/sa] └─$

3、通过 Performance Co-Pilot(PCP) 收集性能数据

安装相关包

┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system] └─$ yum -y install pcp ┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system] └─$ yum install pcp-gui -y

启动相关服务

┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system] └─$ systemctl enable --now pmcd # 启动监控服务 Created symlink from /etc/systemd/system/multi-user.target.wants/pmcd.service to /usr/lib/systemd/system/pmcd.service. ┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system] └─$ systemctl enable --now pmlogger # 启动存数据的数据库 Created symlink from /etc/systemd/system/multi-user.target.wants/pmlogger.service to /usr/lib/systemd/system/pmlogger.service. ┌──[root@liruilongs.github.io]-[/usr/lib/systemd/system]

通过pmchart，pminfo，pmval读取系统当前及历史运行数据

查看所有的监控内容

┌──[root@liruilongs.github.io]-[~] └─$ pminfo jbd2.njournals jbd2.transaction.count jbd2.transaction.requested ...... ┌──[root@liruilongs.github.io]-[~] └─$ pminfo | wc -l 2052

查看所有的监控参数详细信息

┌──[root@liruilongs.github.io]-[~] └─$ pminfo -dt proc.nprocs #参数详细信息 proc.nprocs [instantaneous number of processes] Data Type: 32-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff Semantics: instant Units: none ┌──[root@liruilongs.github.io]-[~] └─$ pminfo proc.nprocs #进程监控 proc.nprocs

监控进程信息 pmval proc.nprocs

┌──[root@liruilongs.github.io]-[~] └─$ pmval proc.nprocs #进程监控 metric: proc.nprocs host: liruilongs.github.io semantics: instantaneous value units: none samples: all 158 158 158 158 158 158 ^C

监控剩余内存的量 pmval -t 2 mem.freemem

┌──[root@liruilongs.github.io]-[~] └─$ pmval -t 2 mem.freemem # 剩余内存信息 metric: mem.freemem host: liruilongs.github.io semantics: instantaneous value units: Kbyte samples: all 3471612 3471636 3471636 3471636 3471636 ^C ┌──[root@liruilongs.github.io]-[~] └─$

查看监控日志

┌──[root@liruilongs.github.io]-[~] └─$ pminfo | grep mem.freemem mem.freemem ┌──[root@liruilongs.github.io]-[~] └─$ cd /var/log/pcp ┌──[root@liruilongs.github.io]-[/var/log/pcp] └─$ ls NOTICES pmcd pmie pmlogger pmproxy ┌──[root@liruilongs.github.io]-[/var/log/pcp] └─$ cd pmlogger/ ┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger] └─$ ls liruilongs.github.io pmlogger_check.log.prev pmlogger_daily-K.log.prev pmlogger_daily.stamp ┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger] └─$ cd liruilongs.github.io/ ┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger/liruilongs.github.io] └─$ ls 20211023.02.05.0.xz 20211023.02.30-00.0.xz 20211023.02.30.0.xz 20211024.08.16.0 Latest 20211023.02.05.index 20211023.02.30-00.index 20211023.02.30.index 20211024.08.16.index pmlogger.log 20211023.02.05.meta.xz 20211023.02.30-00.meta.xz 20211023.02.30.meta.xz 20211024.08.16.meta pmlogger.log.prior ┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger/liruilongs.github.io] └─$ pmval -a 20211024.08.16.0 mem.freemem # 查看监控日志 每秒 metric: mem.freemem archive: 20211024.08.16.0 host: liruilongs.github.io start: Sun Oct 24 08:17:06 2021 end: Sun Oct 24 10:02:06 2021 semantics: instantaneous value units: Kbyte samples: 6301 interval: 1.00 sec 08:17:06.095 No values available 08:17:07.095 No values available 08:17:08.095 3651184 08:17:09.095 3651184 08:17:10.095 3651184 08:17:11.095 3651184 08:17:12.095 3651184 08:17:13.095 3651184 08:17:14.095 3651184 08:17:15.095 3651184 08:17:16.095 3651184 08:17:17.095 3651184 08:17:18.095 3651184 08:17:19.095 3651184 08:17:20.095 3651184 08:17:21.095 3651184 08:17:22.095 3651184 08:17:23.095 3651184 08:17:24.095 3651184 08:17:25.095 3651184 08:17:26.095 3651184 ......

加时间粒度

┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger/liruilongs.github.io] └─$ pmval -a 20211024.08.16.0 -t 500 mem.freemem # 加时间粒度 metric: mem.freemem archive: 20211024.08.16.0 host: liruilongs.github.io start: Sun Oct 24 08:17:06 2021 end: Sun Oct 24 10:08:06 2021 semantics: instantaneous value units: Kbyte samples: 14 interval: 500.00 sec 08:17:06.095 No values available 08:25:26.095 3488128 08:33:46.095 3488128 08:42:06.095 3488128 08:50:26.095 3488128 08:58:46.095 3488128 09:07:06.095 3488128 09:15:26.095 3488128 09:23:46.095 3488128 09:32:06.095 3488128 09:40:26.095 3472784 09:48:46.095 3472024 09:57:06.095 3467052 10:05:26.095 3466276 ┌──[root@liruilongs.github.io]-[/var/log/pcp/pmlogger/liruilongs.github.io] └─$

二、如何进行内核参数调优

/proc & /sys

/proc 存放内存中数据，每次重启就没了。

┌──[root@liruilongs.github.io]-[/] └─$ cd proc/ ┌──[root@liruilongs.github.io]-[/proc] └─$ ls 1 16836 241 269 284 3 313 362 45 647 853 crypto keys pagetypeinfo tty 10 16842 242 270 285 300 314 363 469 649 856 devices key-users partitions uptime 1041 16843 250 271 286 301 315 364 47 66 857 diskstats kmsg sched_debug version 1043 16850 251 272 287 302 316 365 5 6671 861 dma kpagecount schedstat vmallocinfo 1045 16862 253 273 288 303 317 366 506 7 864 driver kpageflags scsi vmstat 1050 17 254 274 289 304 318 367 509 7603 8922 execdomains loadavg self zoneinfo 1051 18 255 275 290 305 319 368 570 7604 9 fb locks slabinfo 1054 19 256 276 291 306 32 369 581 7606 98 filesystems mdstat softirqs 1055 2 262 277 292 307 320 370 583 8 acpi fs meminfo stat 1056 20 263 278 293 308 321 371 585 8095 buddyinfo interrupts misc swaps 11 21 264 279 294 309 322 372 594 8158 bus iomem modules sys 12 22 265 280 295 31 33 373 595 8159 cgroups ioports mounts sysrq-trigger 13 23 266 281 296 310 34 42 596 8160 cmdline irq mpt sysvipc 15 24 267 282 298 311 348 44 627 8161 consoles kallsyms mtrr timer_list 16006 240 268 283 299 312 361 441 642 8162 cpuinfo kcore net timer_stats ┌──[root@liruilongs.github.io]-[/proc] └─$ pwd /proc ┌──[root@liruilongs.github.io]-[/proc] └─$ cd sys ┌──[root@liruilongs.github.io]-[/proc/sys] └─$ ls abi crypto debug dev fs kernel net user vm

内存调优目录

┌──[root@liruilongs.github.io]-[/proc/sys] └─$ cd vm # 内存调优目录 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ ls admin_reserve_kbytes hugepages_treat_as_movable mmap_min_addr overcommit_memory block_dump hugetlb_shm_group mmap_rnd_bits overcommit_ratio compact_memory laptop_mode mmap_rnd_compat_bits page-cluster dirty_background_bytes legacy_va_layout nr_hugepages panic_on_oom dirty_background_ratio lowmem_reserve_ratio nr_hugepages_mempolicy percpu_pagelist_fraction dirty_bytes max_map_count nr_overcommit_hugepages stat_interval dirty_expire_centisecs memory_failure_early_kill nr_pdflush_threads swappiness dirty_ratio memory_failure_recovery numa_zonelist_order user_reserve_kbytes dirty_writeback_centisecs min_free_kbytes oom_dump_tasks vfs_cache_pressure drop_caches min_slab_ratio oom_kill_allocating_task zone_reclaim_mode extfrag_threshold min_unmapped_ratio overcommit_kbytes ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$

查看内核参数：cat & sysctl -a

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ sysctl -a # 查看所有调优参数 abi.vsyscall32 = 1 crypto.fips_enabled = 0 debug.exception-trace = 1 debug.kprobes-optimization = 1 debug.panic_on_rcu_stall = 0 dev.hpet.max-user-freq = 64 dev.mac_hid.mouse_button2_keycode = 97 dev.mac_hid.mouse_button3_keycode = 100 dev.mac_hid.mouse_button_emulation = 0 dev.parport.default.spintime = 500 dev.parport.default.timeslice = 200 dev.raid.speed_limit_max = 200000 dev.raid.speed_limit_min = 1000 dev.scsi.logging_level = 0 fs.aio-max-nr = 65536 fs.aio-nr = 0 ..............

根据变量找对应文件参数

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ sysctl -a | grep net.ipv4.ip_forward net.ipv4.ip_forward = 1 net.ipv4.ip_forward_use_pmtu = 0 sysctl: reading key "net.ipv6.conf.all.stable_secret" sysctl: reading key "net.ipv6.conf.br-4b3da203747c.stable_secret" sysctl: reading key "net.ipv6.conf.default.stable_secret" sysctl: reading key "net.ipv6.conf.docker0.stable_secret" sysctl: reading key "net.ipv6.conf.ens32.stable_secret" sysctl: reading key "net.ipv6.conf.lo.stable_secret" ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ cd ../net/ipv ipv4/ ipv6/ ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ cd ../net/ipv4/ ┌──[root@liruilongs.github.io]-[/proc/sys/net/ipv4] └─$ cat ip_forward 1

调整内核参数：echo & sysctl

设置调优参数：临时调整

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ cat swappiness 30 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ echo 40 > swappiness ## 临时调整 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ cat swappiness 40

设置调优参数：永久调整

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ cat /etc/sysc sysconfig/ sysctl.conf sysctl.d/ ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ cat /etc/sysctl.conf # sysctl settings are defined through files in # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/. # # Vendors settings live in /usr/lib/sysctl.d/. # To override a whole file, create a new file with the same in # /etc/sysctl.d/ and put new settings there. To override # only specific settings, add a file with a lexically later # name in /etc/sysctl.d/ and put new settings there. # # For more information, see sysctl.conf(5) and sysctl.d(5). net.ipv4.ip_forward = 1 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ echo "vm.swappiness = 20" >> /etc/sysctl.conf ## 永久调整 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ cat /etc/sysctl.conf # sysctl settings are defined through files in # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/. # # Vendors settings live in /usr/lib/sysctl.d/. # To override a whole file, create a new file with the same in # /etc/sysctl.d/ and put new settings there. To override # only specific settings, add a file with a lexically later # name in /etc/sysctl.d/ and put new settings there. # # For more information, see sysctl.conf(5) and sysctl.d(5). net.ipv4.ip_forward = 1 vm.swappiness = 20 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ sysctl -p net.ipv4.ip_forward = 1 vm.swappiness = 20 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$

内核调优之清理缓存

如何使用帮助调整内核参数

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ man -K drop_caches --Man-- next: proc(5) [ view (return) | skip (Ctrl-D) | quit (Ctrl-C) ]

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ cat drop_caches #缓存处理 0 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ man -K drop_caches No manual entry for drop_caches ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ man -K drop_caches No manual entry for drop_caches ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ man -K ip_forward ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ free -m total used free shared buff/cache available Mem: 3935 212 3357 16 366 3440 Swap: 10239 0 10239 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ echo 3 > /proc/sys/vm/drop_caches ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ free -m total used free shared buff/cache available Mem: 3935 200 3575 16 159 3504 Swap: 10239 0 10239 ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$

调整内核模块参数：lsmod、modinfo、modprobe

加载的内核模块列表 lsmod

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ lsmod # 大小 调用次数 Module Size Used by binfmt_misc 17468 1 xt_conntrack 12760 2 ipt_MASQUERADE 12678 2 nf_nat_masquerade_ipv4 13412 1 ipt_MASQUERADE nf_conntrack_netlink 40449 0 nfnetlink 14696 2 nf_conntrack_netlink xt_addrtype 12676 2 iptable_filter 12810 1 iptable_nat 12875 1 nf_conntrack_ipv4 15053 3 nf_defrag_ipv4 12729 1 nf_conntrack_ipv4 。。。。。 。。。。。。 crct10dif_common 12595 3 crct10dif_pclmul,crct10dif_generic,crc_t10dif crc32c_intel 22079 1 ahci 34042 0 drm 370825 2 ttm,drm_kms_helper libahci 31992 1 ahci mptspi 22542 2 ata_piix 35038 0 scsi_transport_spi 30732 1 mptspi mptscsih 40150 1 mptspi libata 238896 5 ahci,pata_acpi,libahci,ata_generic,ata_piix e1000 137500 0 serio_raw 13413 0 mptbase 105960 2 mptspi,mptscsih i2c_core 40756 3 drm,i2c_piix4,drm_kms_helper ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$

查看内核模块参数 modinfo

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ modinfo kvm # 查看内核模块参数 filename: /lib/modules/3.10.0-693.el7.x86_64/kernel/arch/x86/kvm/kvm.ko.xz license: GPL author: Qumranet rhelversion: 7.4 srcversion: FA3AAB0FB1DD5C7B9D69811 depends: irqbypass intree: Y vermagic: 3.10.0-693.el7.x86_64 SMP mod_unload modversions signer: CentOS Linux kernel signing key sig_key: DA:18:7D:CA:7D:BE:53:AB:05:BD:13:BD:0C:4E:21:F4:22:B6:A4:9C sig_hashalgo: sha256 parm: ignore_msrs:bool parm: min_timer_period_us:uint parm: kvmclock_periodic_sync:bool parm: tsc_tolerance_ppm:uint parm: lapic_timer_advance_ns:uint parm: vector_hashing:bool parm: halt_poll_ns:uint parm: halt_poll_ns_grow:uint parm: halt_poll_ns_shrink:uint ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$

内核模块调优之嵌套虚拟化设置

┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ modinfo kvm filename: /lib/modules/3.10.0-693.el7.x86_64/kernel/arch/x86/kvm/kvm.ko.xz license: GPL author: Qumranet rhelversion: 7.4 srcversion: FA3AAB0FB1DD5C7B9D69811 depends: irqbypass intree: Y vermagic: 3.10.0-693.el7.x86_64 SMP mod_unload modversions signer: CentOS Linux kernel signing key sig_key: DA:18:7D:CA:7D:BE:53:AB:05:BD:13:BD:0C:4E:21:F4:22:B6:A4:9C sig_hashalgo: sha256 parm: ignore_msrs:bool parm: min_timer_period_us:uint parm: kvmclock_periodic_sync:bool parm: tsc_tolerance_ppm:uint parm: lapic_timer_advance_ns:uint parm: vector_hashing:bool parm: halt_poll_ns:uint parm: halt_poll_ns_grow:uint parm: halt_poll_ns_shrink:uint ┌──[root@liruilongs.github.io]-[/proc/sys/vm] └─$ modinfo kvm | grep ignore_msrs # 设置允许嵌套虚拟化 parm: ignore_msrs:bool ┌──[root@liruilongs.github.io]-[/etc/modprobe.d] └─$ echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf ##从起机器

如何限制服务的资源占用

资源消耗分为用户和服务，消耗：CPU，内存和 I/O

经典的限制通过pam_limit模块配合/etc/security/limits.conf实现（用户）

用户登录会加载pam_limit模块，pam_limit模块读取配置文件 /etc/security/limits.conf限制用户资源的占用

┌──[root@liruilongs.github.io]-[/etc/modprobe.d] └─$ cat /etc/security/limits.conf # /etc/security/limits.conf # #This file sets the resource limits for the users logged in via PAM. #It does not affect resource limits of the system services. # #Also note that configuration files in /etc/security/limits.d directory, #which are read in alphabetical order, override the settings in this #file in case the domain is the same or more specific. #That means for example that setting a limit for wildcard domain here #can be overriden with a wildcard setting in a config file in the #subdirectory, but a user specific setting here can be overriden only #with a user specific setting in the subdirectory. # #Each line describes a limit for a user in the form: # #<domain> <type> <item> <value> # #Where: #<domain> can be: # - a user name # - a group name, with @group syntax # - the wildcard *, for default entry # - the wildcard %, can be also used with %group syntax, # for maxlogin limit # #<type> can have the two values: # - "soft" for enforcing the soft limits # - "hard" for enforcing hard limits # #<item> can be one of the following: # - core - limits the core file size (KB) 核心文件大小 # - data - max data size (KB) # - fsize - maximum filesize (KB) # - memlock - max locked-in-memory address space (KB) # - nofile - max number of open file descriptors # 访问文件数量 # - rss - max resident set size (KB) # - stack - max stack size (KB) # - cpu - max CPU time (MIN) # cup时间 # - nproc - max number of processes # - as - address space limit (KB) # - maxlogins - max number of logins for this user #最多的登录数 # - maxsyslogins - max number of logins on the system # - priority - the priority to run user process with # - locks - max number of file locks the user can hold # - sigpending - max number of pending signals # - msgqueue - max memory used by POSIX message queues (bytes) # - nice - max nice priority allowed to raise to values: [-20, 19] # - rtprio - max realtime priority # #<domain> <type> <item> <value> # #* soft core 0 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #@student - maxlogins 4 # End of file

限制服务的资源之限制用户通过ssh的登录数

┌──[root@liruilongs.github.io]-[/etc/modprobe.d] └─$vim /etc/security/limits.conf student hard maxlogins 4

也可以通过 cgroup 实现资源的限制(服务)

┌──[root@liruilongs.github.io]-[/etc/modprobe.d] └─$ md5sum /dev/urandom & #启动一个进程 cpu100% [1] 38162 ┌──[root@liruilongs.github.io]-[/etc/modprobe.d] └─$ top top - 13:37:42 up 4:10, 1 user, load average: 1.03, 0.80, 0.43 Tasks: 158 total, 2 running, 156 sleeping, 0 stopped, 0 zombie %Cpu(s): 2.0 us, 53.9 sy, 0.0 ni, 43.3 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st KiB Mem : 4030172 total, 3637840 free, 207252 used, 185080 buff/cache KiB Swap: 10485756 total, 10485756 free, 0 used. 3575580 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND 861 etcd 20 0 10.254g 22572 11092 S 9.7 0.6 22:07.49 etcd 9 root 20 0 0 0 0 S 1.7 0.0 0:28.00 rcu_sched 595 root 20 0 298868 6260 4916 S 1.6 0.2 2:27.63 vmtoolsd 371 root 20 0 0 0 0 S 0.4 0.0 1:23.11 xfsaild/sda1 1 root 20 0 51688 3960 2624 S 0.4 0.1 0:31.08 systemd 36765 root 20 0 0 0 0 S 0.2 0.0 0:03.13 kworker/1:0 583 root 20 0 21616 1284 972 S 0.2 0.0 0:09.59 irqbalance 857 root 20 0 573468 16760 6020 S 0.2 0.4 0:14.24 tuned 13 root 20 0 0 0 0 S 0.2 0.0 0:06.16 ksoftirqd/1 864 root 20 0 1015116 42652 13740 S 0.1 1.1 0:22.33 containerd 853 root 20 0 228220 5128 3240 S 0.1 0.1 0:09.13 httpd 594 root 20 0 224504 12424 3348 S 0.1 0.3 0:08.61 rsyslogd 649 chrony 20 0 117704 1772 1312 S 0.1 0.0 0:03.47 chronyd 7604 root 20 0 151984 5352 4088 R 0.1 0.1 0:05.89 sshd 38166 root 20 0 161908 2264 1556 R 0.1 0.1 0:00.10 top 1055 root 20 0 81744 3132 2168 S 0.0 0.1 0:10.09 pmdalinux 8095 root 20 0 0 0 0 S 0.0 0.0 0:01.54 kworker/u256:0 11 root rt 0 0 0 0 S 0.0 0.0 0:02.17 watchdog/1 1051 root 20 0 81628 2700 1912 S 0.0 0.1 0:03.07 pmdaproc 2 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:01.56 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root rt 0 0 0 0 S 0.0 0.0 0:01.08 migration/0 [1] Terminated md5sum /dev/urandom ┌──[root@liruilongs.github.io]-[/etc/modprobe.d] └─$

现在我们把这个写成一个服务，然后通过cgroup做简单限制。

┌──[root@liruilongs.github.io]-[~] └─$ vim /etc/systemd/system/md5sum.service ┌──[root@liruilongs.github.io]-[~] └─$ cat /etc/systemd/system/md5sum.service [Unit] Description=MD5 Demo [Service] CPUQuota=40% ExecStart=/usr/bin/md5sum /dev/urandom [Install] WantedBy=multi-user.target ┌──[root@liruilongs.github.io]-[~] └─$ ┌──[root@liruilongs.github.io]-[~] └─$ systemctl daemon-reload ┌──[root@liruilongs.github.io]-[~] └─$ systemctl start md5sum.service ┌──[root@liruilongs.github.io]-[~] └─$ top top - 19:12:14 up 9:44, 1 user, load average: 0.29, 0.20, 0.11 Tasks: 159 total, 2 running, 157 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.5 us, 20.3 sy, 0.0 ni, 78.8 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st KiB Mem : 4030172 total, 3606960 free, 209832 used, 213380 buff/cache KiB Swap: 10485756 total, 10485756 free, 0 used. 3558764 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME COMMAND 59598 root 20 0 107920 612 516 R 39.7 0.0 0:02.54 md5sum ##CPU限制为40% 861 etcd 20 0 10.254g 22572 11092 S 6.3 0.6 51:24.88 etcd 59601 root 20 0 161908 2268 1560 R 1.0 0.1 0:00.08 top 595 root 20 0 298868 6260 4916 S 0.7 0.2 5:35.80 vmtoolsd 1 root 20 0 51688 4000 2648 S 0.3 0.1 1:15.99 systemd 9 root 20 0 0 0 0 S 0.3 0.0 0:50.90 rcu_sched 371 root 20 0 0 0 0 S 0.3 0.0 3:02.41 xfsaild/sda1 6671 root 20 0 1309148 66444 25620 S 0.3 1.6 0:48.35 dockerd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.13 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:04.25 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root rt 0 0 0 0 S 0.0 0.0 0:02.39 migration/0 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 10 root rt 0 0 0 0 S 0.0 0.0 0:03.13 watchdog/0 11 root rt 0 0 0 0 S 0.0 0.0 0:04.80 watchdog/1 12 root rt 0 0 0 0 S 0.0 0.0 0:02.72 migration/1 13 root 20 0 0 0 0 S 0.0 0.0 0:11.18 ksoftirqd/1 15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H 17 root 20 0 0 0 0 S 0.0 0.0 0:00.11 kdevtmpfs 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 19 root 20 0 0 0 0 S 0.0 0.0 0:00.08 khungtaskd 20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback 21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd ┌──[root@liruilongs.github.io]-[~] └─$

这个具体小伙伴可以看看这篇博客: https://blog.csdn.net/sanhewuyang/article/details/120735766

关于cgroup参数设置可以使用帮助文档

┌──[root@liruilongs.github.io]-[~] └─$ man -k systemd ┌──[root@liruilongs.github.io]-[~] └─$ man systemd.resource-control ┌──[root@liruilongs.github.io]-[~] └─$

为 systemd unit 实现资源限制

这个老师没讲，以后研究下，应该也和cgroup有关

自定义tuned调优配置集：

tuned 实现系统傻瓜化和集成化调优,操作系统预装多个调优场景配置集,这是一道RHCAS的考试题

查看调优策略

┌──[root@liruilongs.github.io]-[~] └─$ tuned-adm list Available profiles: - balanced - General non-specialized tuned profile - desktop - Optimize for the desktop use-case - latency-performance - Optimize for deterministic performance at the cost of increased power consumption - network-latency - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance - network-throughput - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G networks - powersave - Optimize for low power consumption - throughput-performance - Broadly applicable tuning that provides excellent performance across a variety of common server workloads - virtual-guest - Optimize for running inside a virtual guest - virtual-host - Optimize for running KVM guests Current active profile: virtual-guest ┌──[root@liruilongs.github.io]-[~] └─$ tuned-adm recommend # 查看推荐的策略 virtual-guest ┌──[root@liruilongs.github.io]-[~] └─$ tuned-adm profile virtual-guest

调优策略位置

┌──[root@liruilongs.github.io]-[~] └─$ cd /usr/lib/tuned/;ls ## 调优参数 balanced latency-performance powersave virtual-guest desktop network-latency recommend.conf virtual-host functions network-throughput throughput-performance ┌──[root@liruilongs.github.io]-[/usr/lib/tuned] └─$ cd virtual-guest/ ┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest] └─$ ls tuned.conf ┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest] └─$ cat tuned.conf # # tuned configuration # [main] summary=Optimize for running inside a virtual guest include=throughput-performance [sysctl] # If a workload mostly uses anonymous memory and it hits this limit, the entire # working set is buffered for I/O, and any more write buffering would require # swapping, so its time to throttle writes until I/O can catch up. Workloads # that mostly use file mappings may be able to use even higher values. # # The generator of dirty data starts writeback at this percentage (system default # is 20%) vm.dirty_ratio = 30 # Filesystem I/O is usually much more efficient than swapping, so try to keep # swapping low. Its usually safe to go even lower than this on systems with # server-grade storage. vm.swappiness = 30 ┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest] └─$

在/etc/tuned目录中编辑自定义tuned profile,配置集中的条目可以相互引用，并通过各种plugin自定义
获取 yum仓库的tuned profile

┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest] └─$ yum list | grep tuned tuned.noarch 2.8.0-5.el7 @anaconda tuned.noarch 2.11.0-11.el7_9 updates tuned-gtk.noarch 2.11.0-11.el7_9 updates tuned-profiles-atomic.noarch 2.11.0-11.el7_9 updates tuned-profiles-compat.noarch 2.11.0-11.el7_9 updates tuned-profiles-cpu-partitioning.noarch 2.11.0-11.el7_9 updates tuned-profiles-mssql.noarch 2.11.0-11.el7_9 updates tuned-profiles-oracle.noarch 2.11.0-11.el7_9 updates tuned-utils.noarch 2.11.0-11.el7_9 updates tuned-utils-systemtap.noarch 2.11.0-11.el7_9 updates ┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest] └─$ yum -y install tuned-profiles-oracle.noarch .... ┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest] └─$ tuned-adm list Available profiles: - balanced - General non-specialized tuned profile - desktop - Optimize for the desktop use-case - hpc-compute - Optimize for HPC compute workloads - latency-performance - Optimize for deterministic performance at the cost of increased power consumption - network-latency - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance - network-throughput - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G networks - oracle - Optimize for Oracle RDBMS - powersave - Optimize for low power consumption - throughput-performance - Broadly applicable tuning that provides excellent performance across a variety of common server workloads - virtual-guest - Optimize for running inside a virtual guest - virtual-host - Optimize for running KVM guests Current active profile: virtual-guest ┌──[root@liruilongs.github.io]-[/usr/lib/tuned/virtual-guest] └─$

通过SystemTap配置内核模块进行系统底层分析：

嗯，这部分有些复杂，有些包需要订阅，先记录下，以后有机会学习