174 lines
9.6 KiB
Markdown
174 lines
9.6 KiB
Markdown
|
# 福建项目：分流不均导致sapp丢包
|
|||
|
|
|
|||
|
|
| ID | Creation Date | Assignee | Status |
|
|||
|
|
|----|----------------|----------|--------|
|
|||
|
|
| OMPUB-750 | 2022-12-20T11:01:14.000+0800 | 刘学利 | 已关闭 |
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
涉及IP：联通康居城域网192.168.28.1-3 联通橘园洲城域网192.168.27.1-3
|
|||
|
|
|
|||
|
|
!image-2022-12-20-11-00-43-292.png|thumbnail!
|
|||
|
|
!image-2022-12-20-11-00-50-796.png|thumbnail!
|
|||
|
|
!image-2022-12-20-11-00-56-385.png|thumbnail! **liuxueli** commented on *2022-12-20T11:21:04.479+0800*:
|
|||
|
|
|
|||
|
|
* 使用monit_stream -lHs --per-stream命令发现分流不均导致一直丢包
|
|||
|
|
** !image-2022-12-20-11-15-48-255.png!
|
|||
|
|
** 在丢包线程随机捕1000个数据包分析
|
|||
|
|
*** 捕包命令: /opt/tsg/framework/bin/tcpdump_mesa -P 9345 -k 7 -g -c 10000 -s0 -w 10000.pcap
|
|||
|
|
*** 发现固定二元组流量较大
|
|||
|
|
**** !image-2022-12-20-11-17-48-307.png!
|
|||
|
|
**** !image-2022-12-20-11-18-06-105.png!
|
|||
|
|
** 调整驱动的分流模式，由二元组改成四元组，观察10分钟未发现丢包现象。
|
|||
|
|
*** 调整驱动分流模式，由distmode=4改成distmode=2，驱动分流模式包含5种方式：
|
|||
|
|
**** 0：软件二元组哈希分流
|
|||
|
|
**** 1：软件四元组哈希分流
|
|||
|
|
**** 2：软件二元组哈希分流(解析到隧道内层，即内层二元组分流)
|
|||
|
|
**** 3：软件四元组哈希分流(解析到隧道内层，即内层四元组分流)
|
|||
|
|
**** 4：使用硬件计算结果分流
|
|||
|
|
** 调整分流模式后，使用monit_stream -lHs --per-stream的结果，流量已均分到各个线程，如下：
|
|||
|
|
***
|
|||
|
|
{code:java}
|
|||
|
|
Time: Tue Dec 20 11:13:35 2022, App: sapp4, Device: ens7f0
|
|||
|
|
-------------- ---------- ---------- ---------- ---------- ---------- ----------
|
|||
|
|
RxPkts RxBits RxDrops TxPkts TxBits TxDrops
|
|||
|
|
-------------- ---------- ---------- ---------- ---------- ---------- ----------
|
|||
|
|
RX[0]TX[0] 87.42K 515.32M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[1]TX[1] 94.78K 702.03M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[2]TX[2] 88.29K 591.35M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[3]TX[3] 86.72K 598.86M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[4]TX[4] 73.98K 429.44M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[5]TX[5] 66.98K 381.60M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[6]TX[6] 177.50K 1.59G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[7]TX[7] 76.48K 477.76M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[8]TX[8] 260.42K 2.57G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[9]TX[9] 77.54K 455.05M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[10]TX[10] 92.61K 627.67M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[11]TX[11] 83.90K 539.58M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[12]TX[12] 77.92K 526.27M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[13]TX[13] 79.68K 518.44M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[14]TX[14] 71.52K 381.38M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[15]TX[15] 84.90K 553.59M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[16]TX[16] 180.67K 1.62G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[17]TX[17] 83.17K 539.52M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[18]TX[18] 205.41K 1.84G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[19]TX[19] 77.73K 495.12M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[20]TX[20] 73.57K 448.88M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[21]TX[21] 82.27K 527.89M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[22]TX[22] 86.59K 551.34M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[23]TX[23] 83.68K 583.11M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[24]TX[24] 83.14K 533.06M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[25]TX[25] 65.70K 414.19M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[26]TX[26] 78.50K 510.63M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[27]TX[27] 70.14K 424.56M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[28]TX[28] 77.12K 509.90M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[29]TX[29] 96.06K 648.83M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[30]TX[30] 158.18K 1.43G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[31]TX[31] 104.00K 749.84M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[32]TX[32] 70.08K 430.50M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[33]TX[33] 83.39K 551.89M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[34]TX[34] 92.93K 547.16M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[35]TX[35] 173.06K 1.54G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[36]TX[36] 162.02K 1.45G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[37]TX[37] 282.40K 2.77G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[38]TX[38] 86.82K 561.90M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[39]TX[39] 73.38K 460.07M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[40]TX[40] 89.98K 591.51M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[41]TX[41] 165.47K 1.49G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[42]TX[42] 78.62K 524.17M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[43]TX[43] 169.57K 1.53G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[44]TX[44] 85.06K 572.66M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[45]TX[45] 81.38K 491.24M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[46]TX[46] 173.54K 1.54G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[47]TX[47] 79.01K 539.73M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[48]TX[48] 97.57K 689.31M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[49]TX[49] 66.59K 387.96M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[50]TX[50] 64.54K 375.59M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[51]TX[51] 79.39K 503.77M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[52]TX[52] 72.10K 439.01M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[53]TX[53] 73.50K 422.83M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[54]TX[54] 73.15K 468.97M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[55]TX[55] 170.69K 1.55G 0.00 0.00 0.00 0.00
|
|||
|
|
RX[56]TX[56] 77.73K 498.15M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[57]TX[57] 81.54K 527.32M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[58]TX[58] 83.74K 569.12M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[59]TX[59] 88.03K 591.66M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[60]TX[60] 64.19K 362.00M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[61]TX[61] 92.93K 606.12M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[62]TX[62] 79.94K 514.02M 0.00 0.00 0.00 0.00
|
|||
|
|
RX[63]TX[63] 71.94K 396.58M 0.00 0.00 0.00 0.00
|
|||
|
|
Total 6.47M 47.78G 0.00 0.00 0.00 0.00 {code}
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhangzhihan** commented on *2022-12-20T11:24:11.922+0800*:
|
|||
|
|
|
|||
|
|
调整为按四元组分流后，丢包现象缓解
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**liuxueli** commented on *2022-12-20T14:37:49.070+0800*:
|
|||
|
|
|
|||
|
|
* 使用perf top分析：发现丢包时http_deleteEmptyRow函数消耗CPU较高，如下：
|
|||
|
|
** 丢包时刻
|
|||
|
|
*** !image-2022-12-20-14-36-21-332.png!
|
|||
|
|
** perf top -C 31结果
|
|||
|
|
*** !image-2022-12-20-14-36-43-853.png!
|
|||
|
|
* http_deleteEmptyRow函数消耗CPU较高参见: TSG-12072
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**liuxueli** commented on *2022-12-20T17:21:04.759+0800*:
|
|||
|
|
|
|||
|
|
* 登录192.168.27.3，使用perf top查看丢包线程，CPU使用均不高，捕包发现丢包线程出现IPv6 分片数据包，参见：TSG-13139
|
|||
|
|
** 捕包命令：/opt/tsg/framework/bin/tcpdump_mesa -P 9345 -k 26 -c 10000 -s0 -w 10000.pcap -v
|
|||
|
|
** {color:#de350b}10000个数据包有6706个分片包{color}
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Attachments
|
|||
|
|
|
|||
|
|
**33807/image-2022-12-20-11-00-43-292.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33806/image-2022-12-20-11-00-50-796.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33805/image-2022-12-20-11-00-56-385.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33808/image-2022-12-20-11-15-48-255.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33809/image-2022-12-20-11-17-48-307.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33810/image-2022-12-20-11-18-06-105.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33820/image-2022-12-20-14-36-21-332.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33821/image-2022-12-20-14-36-43-853.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|