136 lines
4.7 KiB
Markdown
136 lines
4.7 KiB
Markdown
# P-POC现场流量高于50Gbps时10秒左右会出现一次丢包
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-834 | 2023-03-06T03:59:55.000+0800 | 彭宣正 | 已关闭 |
|
||
|
||
|
||
---
|
||
|
||
* POC现场10秒左右会出现一次丢包
|
||
** 当时流量情况参见[^POC-monit-stream.txt]
|
||
** 使用perf top -C cpuid查看CPU使用情况，tsg_conn_sketch消耗CPU较多
|
||
*** !image-2023-03-06-03-59-27-105.png!
|
||
*** !image-2023-03-06-03-59-53-901.png!**pengxuanzheng** commented on *2023-03-07T20:44:48.234+0800*:
|
||
|
||
udp_entry CPU占用率高的原因有2:
|
||
* udp_entry 看到的pkttype是PKT_TYPE_TCPREORDER 状态，一直在加1
|
||
** !image-2023-03-07-20-43-35-085.png!
|
||
* tsg_conn_get_trans_layer_label函数占用率高
|
||
|
||
|
||
解决方案：
|
||
* udp_entry 不再调用tsg_conn_update_translayer_label函数（本身udp 不用更新translayer label）
|
||
* 拆分 tsg_conn_get_trans_layer_label函数
|
||
|
||
|
||
|
||
---
|
||
|
||
**pengxuanzheng** commented on *2023-03-07T20:53:38.006+0800*:
|
||
|
||
更新后结果：
|
||
* 没有再丢包
|
||
** !image-2023-03-07-20-50-52-064.png!
|
||
* CPU占用率降低
|
||
** !image-2023-03-07-20-51-45-548.png!
|
||
* perf 结果 tsg_conn_sketch不再占用过高CPU
|
||
** !image-2023-03-07-20-53-03-633.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuyang** commented on *2023-03-08T14:25:21.783+0800*:
|
||
|
||
补充：[~luqiuwen]关闭重复流量处理插件
|
||
|
||
|
||
|
||
---
|
||
|
||
**luqiuwen** commented on *2023-03-08T14:32:58.759+0800*:
|
||
|
||
关闭connsketch插件的udp入口后，发现全部处理器的sys周期性增高，每次增高持续时间约1秒，大约10秒钟增高一次。怀疑和重复流量识别机制的bloom filter轮换机制有关。关闭sapp的重复流量识别功能，恢复正常，cpu利用率保持稳定（每个核心约50%），sys使用率较低（1%左右）。
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2023-03-09T16:45:02.001+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/d05c305b952f126a315ec48e9101e31bf29d7121] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(OMPUB-827,OMPUB-834): update tsg_conn_sketch-3.3.15 to tsg_conn_sketch-3.3.16.1{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2023-03-09T16:46:08.497+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/merge_requests/1157] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(OMPUB-827,OMPUB-834): update tsg_conn_sketch-3.3.15 to tsg_conn_sketch-3.3.16.1{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2023-03-16T13:59:17.963+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tango/tsg_conn_record/-/commit/7c84a198b48305faf7b8d02f4e9df963551dd5dc] of [TSG Appliance / tsg_conn_record|https://git.mesalab.cn/tango/tsg_conn_record] on branch [feature-tsg-14070|https://git.mesalab.cn/tango/tsg_conn_record/-/tree/feature-tsg-14070]:{quote}🐞 fix(OMPUB-834): 删除tsg_conn_get_trans_layer_label函数, udp 不再调用tsg_conn_update_translayer_label{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2023-03-16T17:42:52.320+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a commit|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/commit/4c24cf51e85c009a5fe1e65617cce4f71d8d3c68] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(TSG-14070,OMPUB-827,OMPUB-834): update tsg_conn_sketch-3.4.0 to tsg_conn_sketch-3.4.2{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2023-03-16T17:43:36.544+0800*:
|
||
|
||
[彭宣正|https://git.mesalab.cn/pxz] mentioned this issue in [a merge request|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/merge_requests/1177] of [TSG / tsg-os-buildimage|https://git.mesalab.cn/tsg/tsg-os-buildimage] on branch [update-tsg_conn_sketch|https://git.mesalab.cn/tsg/tsg-os-buildimage/-/tree/update-tsg_conn_sketch]:{quote}🐞 fix(TSG-14070,OMPUB-827,OMPUB-834): update tsg_conn_sketch-3.4.0 to tsg_conn_sketch-3.4.2{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|
||
**35666/fs2_sysinfo.log**
|
||
|
||
---
|
||
|
||
**35665/image-2023-03-06-03-59-27-105.png**
|
||
|
||
---
|
||
|
||
**35664/image-2023-03-06-03-59-53-901.png**
|
||
|
||
---
|
||
|
||
**35811/image-2023-03-07-20-43-35-085.png**
|
||
|
||
---
|
||
|
||
**35812/image-2023-03-07-20-50-52-064.png**
|
||
|
||
---
|
||
|
||
**35813/image-2023-03-07-20-51-45-548.png**
|
||
|
||
---
|
||
|
||
**35814/image-2023-03-07-20-53-03-633.png**
|
||
|
||
---
|
||
|
||
**35667/POC-monit-stream.txt**
|
||
|
||
---
|
||
|