132 lines
5.6 KiB
Markdown
132 lines
5.6 KiB
Markdown
# 福建项目：泉州移动4G功能端丢包
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-723 | 2022-12-02T20:18:48.000+0800 | 彭宣正 | 已关闭 |
|
||
|
||
|
||
---
|
||
|
||
192.168.13.1-4 4台功能端出现丢包现象
|
||
原因：服务器CPU爆满
|
||
但是近期无策略变化，目前观察分时段出现丢包现象**liuxueli** commented on *2022-12-02T20:48:23.773+0800*:
|
||
|
||
* 查看现场，CPU使用率较高
|
||
**
|
||
{code:java}
|
||
%Cpu0 : 83.8 us, 2.4 sy, 0.0 ni, 13.5 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
|
||
%Cpu1 : 96.3 us, 3.0 sy, 0.0 ni, 0.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu2 : 95.0 us, 2.7 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu3 : 96.4 us, 1.7 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu4 : 93.4 us, 4.6 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu5 : 96.0 us, 2.3 sy, 0.0 ni, 1.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu6 : 89.7 us, 8.3 sy, 0.0 ni, 1.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
|
||
%Cpu7 : 93.3 us, 3.0 sy, 0.0 ni, 3.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu8 : 92.0 us, 3.7 sy, 0.0 ni, 4.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu9 : 95.7 us, 2.7 sy, 0.0 ni, 1.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu10 : 95.0 us, 3.0 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu11 : 94.0 us, 3.3 sy, 0.0 ni, 2.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu12 : 94.4 us, 2.3 sy, 0.0 ni, 3.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu13 : 77.6 us, 3.0 sy, 0.0 ni, 19.1 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
|
||
%Cpu14 : 81.3 us, 4.0 sy, 0.0 ni, 14.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu15 : 82.8 us, 2.0 sy, 0.0 ni, 14.8 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
|
||
%Cpu16 : 94.4 us, 2.0 sy, 0.0 ni, 3.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu17 : 95.4 us, 1.7 sy, 0.0 ni, 3.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu18 : 94.0 us, 1.7 sy, 0.0 ni, 4.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu19 : 88.7 us, 2.3 sy, 0.0 ni, 9.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu20 : 86.3 us, 3.0 sy, 0.0 ni, 10.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu21 : 93.4 us, 1.7 sy, 0.0 ni, 5.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu22 : 91.2 us, 2.0 sy, 0.0 ni, 6.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu23 : 83.6 us, 3.0 sy, 0.0 ni, 13.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu24 : 88.1 us, 2.3 sy, 0.0 ni, 9.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu25 : 79.3 us, 4.8 sy, 0.0 ni, 16.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu26 : 84.1 us, 4.1 sy, 0.0 ni, 11.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu27 : 79.5 us, 4.1 sy, 0.0 ni, 16.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu28 : 86.9 us, 2.3 sy, 0.0 ni, 10.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu29 : 76.5 us, 1.3 sy, 0.0 ni, 22.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu30 : 77.7 us, 1.7 sy, 0.0 ni, 20.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu31 : 76.7 us, 1.4 sy, 0.0 ni, 22.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu32 : 90.0 us, 3.7 sy, 0.0 ni, 6.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu33 : 97.0 us, 1.7 sy, 0.0 ni, 1.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu34 : 94.7 us, 1.7 sy, 0.0 ni, 3.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu35 : 93.4 us, 4.0 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
|
||
%Cpu36 : 94.7 us, 2.7 sy, 0.0 ni, 2.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu37 : 94.4 us, 2.3 sy, 0.0 ni, 3.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu38 : 94.4 us, 1.7 sy, 0.0 ni, 4.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu39 : 91.3 us, 4.7 sy, 0.0 ni, 4.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu40 : 93.7 us, 4.0 sy, 0.0 ni, 2.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
|
||
%Cpu41 : 96.4 us, 2.0 sy, 0.0 ni, 1.3 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
|
||
%Cpu42 : 95.7 us, 2.0 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
|
||
%Cpu43 : 94.7 us, 2.3 sy, 0.0 ni, 2.6 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
|
||
{code}
|
||
|
||
* 使用perf top -C cpuid(11/12)查看，multi_getsock和multi_runsingle函数占用CPU较多，如下图:
|
||
** !image-2022-12-02-20-45-15-083.png!
|
||
** !image-2022-12-02-20-46-03-148.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**pengxuanzheng** commented on *2022-12-05T10:45:57.154+0800*:
|
||
|
||
hos 修改配置
|
||
hos_curl_easy_handle_per_multi_handle = 20， 降低curl_multi_perform调用频率
|
||
|
||
|
||
|
||
---
|
||
|
||
**pengxuanzheng** commented on *2022-12-06T11:33:40.186+0800*:
|
||
|
||
hos修改配置后，服务器现状：
|
||
|
||
!image-2022-12-06-11-36-25-098.png!
|
||
|
||
查看8,9,10,11 核，hos CPU使用率均有所下降
|
||
|
||
!image-2022-12-06-11-36-02-699.png!
|
||
|
||
观察一段时间发现，hos CPU使用率在2%-9%之间，可以加大hos_curl_easy_handle_per_multi_handle 至 50，继续降低curl_multi_perform调用频率
|
||
|
||
|
||
|
||
---
|
||
|
||
**pengxuanzheng** commented on *2022-12-06T14:28:18.014+0800*:
|
||
|
||
改完之后，功能端不再丢包
|
||
|
||
|
||
|
||
---
|
||
|
||
**pengxuanzheng** commented on *2022-12-06T14:43:54.475+0800*:
|
||
|
||
hos_curl_easy_handle_per_multi_handle 的作用是每个curl multi handle 收到多少个 curl easy handle 之后才会开始curl_multi_perform, 也就是一次curl_multi_perform 携带的put请求个数， 默认值是1（配合此前的交互式上传），后续会修改默认值为20
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|
||
**33344/image-2022-12-02-20-45-15-083.png**
|
||
|
||
---
|
||
|
||
**33345/image-2022-12-02-20-46-03-148.png**
|
||
|
||
---
|
||
|
||
**33397/image-2022-12-06-11-36-02-699.png**
|
||
|
||
---
|
||
|
||
**33396/image-2022-12-06-11-36-25-098.png**
|
||
|
||
---
|
||
|