geedge-jira/md/OMPUB-484.md

# 【XJ-NPM现场】功能端sapp间断丢包

| ID | Creation Date | Assignee | Status |
|----|----------------|----------|--------|
| OMPUB-484 | 2022-05-16T10:39:32.000+0800 | 贾依蒙 | 已关闭 |


---

在nezha监控中发现rx_drops_total sapp 间断性丢包。如图1所示。 !图1.png|thumbnail!**yangwei** commented on *2022-05-16T11:39:08.844+0800*:

* 据4月25号交流结果，当时报{*}省口移动机房{*}有20台左右功能端存在丢包较多的情况，NEZHA监控信息如下：

!image-2022-05-16-11-23-25-277.png|width=600,height=262!

monit_stream查看信息如下，初步判断为{*}单网卡流量过载(ens1f0网卡上Rx+Drop >1.2Mpps，远大于ens2f0上的152K pps){*}

!image-2022-05-16-11-25-54-223.png|width=645,height=210!
 * 初步交流，确认当时没有新增策略，排查因为策略扫描命中率，导致丢包的原因，进一步查看NEZHA上较长时间范围内，sapp丢包曲线如下：
 * !image-2022-05-16-11-28-07-255.png|width=787,height=362!
 * 定位到一台正在丢包现场的功能端，执行monit_stream -IHs --per-stream，按线程查看sapp处理情况如下：
 * !image-2022-05-16-11-28-57-505.png|width=667,height=294!
 * 进一步判断原因为2号和12号包处理线程流量过载，两个线程drop约1.7Mpps包，比其他包处理线程正常处理的包(609K pps)高约3倍
 * 通过在丢包功能端对应工作线程捕包，发现异常流量基本来自相同二元组，疑似DoS：
 * !image-2022-05-16-11-37-13-225.png|width=901,height=183!
 * 小结：4月25号丢包的原因，基本判定为二元组DoS攻击->单线程分流不匀->功能端工作线程过载，建议改为四元组分流，可以缓解这种简单的DoS攻击，对于DDoS仍然存在问题



---

**yangwei** commented on *2022-05-16T11:39:52.170+0800*:

请[~jiayimeng] 补充IDC机房环境出现丢包的详细情况



---

**jiayimeng** commented on *2022-05-16T17:00:10.073+0800*:

新疆联通IDC环境有10台左右的功能端出现间断丢包现象，查看monit_device结果显示不丢包，查看monit_stream -lHs --per-stream，单线程存在分流不均导致的丢包 ，因丢包现象不持续，按线程抓包未能准确在丢包时抓取。

!image-2022-05-16-16-56-18-321.png|thumbnail!

查看sysinfo.log，功能端记录的链路质量显示丢包率百分之3多，

!screenshot-1.png|thumbnail!

根据省口处理经验，修改mrzcpd分流模式为内层四元组分流，目前修改了0.14，待观察。



---

**jiayimeng** commented on *2022-05-17T16:17:43.464+0800*:

172.16.0.14修改分流模式后，未出现丢包，17日中午修改全部TSG-X分流模式为内层四元组。

当前24小时内丢包情况如下。

!image-2022-05-17-16-16-42-319.png|thumbnail!

待明日观察今晚流量高峰期丢包情况。



---

**jiayimeng** commented on *2022-05-18T10:41:50.778+0800*:

今日功能端丢包情况，几乎无丢包。

!image-2022-05-18-10-41-34-224.png|thumbnail!



---



# Attachments

Attachment: image-2022-05-16-11-23-25-277.png
![image-2022-05-16-11-23-25-277.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/27993/image-2022-05-16-11-23-25-277.png)



Attachment: image-2022-05-16-11-25-54-223.png
![image-2022-05-16-11-25-54-223.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/27994/image-2022-05-16-11-25-54-223.png)



Attachment: image-2022-05-16-11-28-07-255.png
![image-2022-05-16-11-28-07-255.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/27996/image-2022-05-16-11-28-07-255.png)



Attachment: image-2022-05-16-11-28-57-505.png
![image-2022-05-16-11-28-57-505.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/27997/image-2022-05-16-11-28-57-505.png)



Attachment: image-2022-05-16-11-29-36-228.png
![image-2022-05-16-11-29-36-228.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/27998/image-2022-05-16-11-29-36-228.png)



Attachment: image-2022-05-16-11-37-13-225.png
![image-2022-05-16-11-37-13-225.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/27999/image-2022-05-16-11-37-13-225.png)



Attachment: image-2022-05-16-16-56-18-321.png
![image-2022-05-16-16-56-18-321.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/28011/image-2022-05-16-16-56-18-321.png)



Attachment: image-2022-05-17-16-16-42-319.png
![image-2022-05-17-16-16-42-319.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/28042/image-2022-05-17-16-16-42-319.png)



Attachment: image-2022-05-18-10-41-34-224.png
![image-2022-05-18-10-41-34-224.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/28073/image-2022-05-18-10-41-34-224.png)



Attachment: screenshot-1.png
![screenshot-1.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/28010/screenshot-1.png)



Attachment: 图1.png
![图1.png](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/27988/图1.png)