113 lines
3.3 KiB
Markdown
113 lines
3.3 KiB
Markdown
# 【E21现场】Bole-IGW NPB04 频繁出现单核CPU占用100%
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-1081 | 2023-12-11T23:13:41.000+0800 | 刘文坛 | 已关闭 |
|
||
|
||
|
||
---
|
||
|
||
*现象*
|
||
* 包处理核短时间内使用率100%，触发Overload Protection，一定概率导致自检不通过
|
||
* 使用 top -d 0.5提高cpu占用显示频率，占用100%的时间多维持在1-2秒
|
||
* CPU占用100%的线程无特定规律，不定时在各个核之前变化，存在同时多个（2-3个）核占用100%的情况
|
||
|
||
*现场信息*
|
||
* perf top -C [core-id]，该线程memcpy和memset调用排名靠前，同时maat相关调用显著高于其他线程
|
||
* 生成火焰图，当包处理占用CPU 70%（100%占满持续时间较短，70%为生成火焰图过程的平均值），maat占用约24%，显著高于其他线程（正常约占4%）
|
||
|
||
*推测*
|
||
* 部分流量触发maat命中，导致CPU占用和内存拷贝显著增加
|
||
* 现场maat版本 v4.0.42
|
||
|
||
*[^master.maat.status]无异常*
|
||
|
||
[^cpu20.perf.svg]!image-2023-12-11-23-13-05-395.png|width=1390,height=839!
|
||
|
||
!image-2023-12-11-23-13-27-176.png|width=1487,height=985!
|
||
|
||
!image-2023-12-11-23-15-29-293.png!**yangwei** commented on *2023-12-12T16:43:43.429+0800*:
|
||
|
||
* 对应时段产生了死锁检测的日志，显示Firewall调用maat_scan_string时耗时长
|
||
|
||
!image-2023-12-12-16-42-59-624.png|width=1492,height=554!
|
||
* 由于backtrace日志中maat的堆栈未显示，安装debuginfo进一步排查
|
||
** addr2line -e /opt/tsg/framework/lib/libmaat4.so.4 0x5c2cc6查找对应的堆栈现场，显示？？，无法定位到具体的代码行
|
||
** gdb /opt/tsg/framework/lib/libmaat4.so.4
|
||
|
||
*
|
||
**
|
||
*** list *0x5c2cc6
|
||
*** 显示对应的堆栈在执行hs_scan_stream，此时输入的扫描内容为HTTP Host
|
||
|
||
与[~liuwentan] 确认后，当前maat4对于字符串扫描，无论是否完整扫描，底层都直接调用{*}Hyperscan的流式扫描接口{*}，需要确认Hyperscan单核流式扫描的benchmark，以便进一步的分析流式扫描性能不足，是否是本次CPU占用100%的原因
|
||
|
||
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuwentan** commented on *2023-12-12T17:30:16.832+0800*:
|
||
|
||
*Hyperscan流式扫描*
|
||
# 每次扫描只命中1条规则，将单核 cpu 跑满的吞吐在{*}77MB{*}左右浮动
|
||
# 每次扫描命中100条规则，将单核 cpu 跑满的吞吐在5{*}MB{*}左右浮动
|
||
|
||
|
||
|
||
---
|
||
|
||
**gitlab** commented on *2023-12-21T10:24:59.437+0800*:
|
||
|
||
[liuwentan|https://git.mesalab.cn/liuwentan] mentioned this issue in [a commit|https://git.mesalab.cn/MESA_framework/maat/-/commit/759f625cb178ada2751a9980062c4c9045a83675] of [MESA Framework / Maat|https://git.mesalab.cn/MESA_framework/maat] on branch [dev-23.07|https://git.mesalab.cn/MESA_framework/maat/-/tree/dev-23.07]:{quote}[PATCH]add bloom filter to optimize performance => OMPUB-1081{quote}
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuwentan** commented on *2023-12-21T11:15:13.339+0800*:
|
||
|
||
maat v4.0.49 增加 bloom filter 进行 pattern_id 的去重，优化性能
|
||
|
||
|
||
|
||
---
|
||
|
||
**yangwei** commented on *2024-03-25T13:50:39.909+0800*:
|
||
|
||
升级后问题暂未复现，关闭本issue
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|
||
**47681/cpu20.perf.svg**
|
||
|
||
---
|
||
|
||
**47680/image-2023-12-11-23-13-05-395.png**
|
||
|
||
---
|
||
|
||
**47679/image-2023-12-11-23-13-27-176.png**
|
||
|
||
---
|
||
|
||
**47682/image-2023-12-11-23-15-29-293.png**
|
||
|
||
---
|
||
|
||
**47690/image-2023-12-12-16-42-59-624.png**
|
||
|
||
---
|
||
|
||
**47683/master.maat.status**
|
||
|
||
---
|
||
|