福建项目：泉州移动4G功能端丢包

ID	Creation Date	Assignee	Status
OMPUB-723	2022-12-02T20:18:48.000+0800	彭宣正	已关闭

192.168.13.1-4 4台功能端出现丢包现象原因：服务器CPU爆满但是近期无策略变化，目前观察分时段出现丢包现象liuxueli commented on 2022-12-02T20:48:23.773+0800:

查看现场，CPU使用率较高 ** {code:java} %Cpu0 : 83.8 us, 2.4 sy, 0.0 ni, 13.5 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu1 : 96.3 us, 3.0 sy, 0.0 ni, 0.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu2 : 95.0 us, 2.7 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu3 : 96.4 us, 1.7 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu4 : 93.4 us, 4.6 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu5 : 96.0 us, 2.3 sy, 0.0 ni, 1.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu6 : 89.7 us, 8.3 sy, 0.0 ni, 1.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu7 : 93.3 us, 3.0 sy, 0.0 ni, 3.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu8 : 92.0 us, 3.7 sy, 0.0 ni, 4.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu9 : 95.7 us, 2.7 sy, 0.0 ni, 1.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu10 : 95.0 us, 3.0 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu11 : 94.0 us, 3.3 sy, 0.0 ni, 2.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu12 : 94.4 us, 2.3 sy, 0.0 ni, 3.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu13 : 77.6 us, 3.0 sy, 0.0 ni, 19.1 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu14 : 81.3 us, 4.0 sy, 0.0 ni, 14.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu15 : 82.8 us, 2.0 sy, 0.0 ni, 14.8 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu16 : 94.4 us, 2.0 sy, 0.0 ni, 3.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu17 : 95.4 us, 1.7 sy, 0.0 ni, 3.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu18 : 94.0 us, 1.7 sy, 0.0 ni, 4.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu19 : 88.7 us, 2.3 sy, 0.0 ni, 9.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu20 : 86.3 us, 3.0 sy, 0.0 ni, 10.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu21 : 93.4 us, 1.7 sy, 0.0 ni, 5.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu22 : 91.2 us, 2.0 sy, 0.0 ni, 6.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu23 : 83.6 us, 3.0 sy, 0.0 ni, 13.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu24 : 88.1 us, 2.3 sy, 0.0 ni, 9.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu25 : 79.3 us, 4.8 sy, 0.0 ni, 16.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu26 : 84.1 us, 4.1 sy, 0.0 ni, 11.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu27 : 79.5 us, 4.1 sy, 0.0 ni, 16.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu28 : 86.9 us, 2.3 sy, 0.0 ni, 10.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu29 : 76.5 us, 1.3 sy, 0.0 ni, 22.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu30 : 77.7 us, 1.7 sy, 0.0 ni, 20.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu31 : 76.7 us, 1.4 sy, 0.0 ni, 22.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu32 : 90.0 us, 3.7 sy, 0.0 ni, 6.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu33 : 97.0 us, 1.7 sy, 0.0 ni, 1.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu34 : 94.7 us, 1.7 sy, 0.0 ni, 3.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu35 : 93.4 us, 4.0 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu36 : 94.7 us, 2.7 sy, 0.0 ni, 2.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu37 : 94.4 us, 2.3 sy, 0.0 ni, 3.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu38 : 94.4 us, 1.7 sy, 0.0 ni, 4.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu39 : 91.3 us, 4.7 sy, 0.0 ni, 4.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu40 : 93.7 us, 4.0 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st %Cpu41 : 96.4 us, 2.0 sy, 0.0 ni, 1.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu42 : 95.7 us, 2.0 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu43 : 94.7 us, 2.3 sy, 0.0 ni, 2.6 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st {code}
使用perf top -C cpuid(11/12)查看，multi_getsock和multi_runsingle函数占用CPU较多，如下图: ** !image-2022-12-02-20-45-15-083.png! ** !image-2022-12-02-20-46-03-148.png!

pengxuanzheng commented on 2022-12-05T10:45:57.154+0800:

hos 修改配置 hos_curl_easy_handle_per_multi_handle = 20，降低curl_multi_perform调用频率

pengxuanzheng commented on 2022-12-06T11:33:40.186+0800:

hos修改配置后，服务器现状：

!image-2022-12-06-11-36-25-098.png!

查看8,9,10,11 核，hos CPU使用率均有所下降

!image-2022-12-06-11-36-02-699.png!

观察一段时间发现，hos CPU使用率在2%-9%之间，可以加大hos_curl_easy_handle_per_multi_handle 至 50，继续降低curl_multi_perform调用频率

pengxuanzheng commented on 2022-12-06T14:28:18.014+0800:

改完之后，功能端不再丢包

pengxuanzheng commented on 2022-12-06T14:43:54.475+0800:

hos_curl_easy_handle_per_multi_handle 的作用是每个curl multi handle 收到多少个 curl easy handle 之后才会开始curl_multi_perform, 也就是一次curl_multi_perform 携带的put请求个数，默认值是1（配合此前的交互式上传），后续会修改默认值为20

Attachments

Attachment: image-2022-12-02-20-45-15-083.png

Attachment: image-2022-12-02-20-46-03-148.png

Attachment: image-2022-12-06-11-36-02-699.png

Attachment: image-2022-12-06-11-36-25-098.png

6.2 KiB Raw Permalink Blame History Unescape Escape

福建项目：泉州移动4G功能端丢包

Attachments

6.2 KiB

Raw Permalink Blame History