138 lines
3.6 KiB
Markdown
138 lines
3.6 KiB
Markdown
|
# 【E21现场】升级23.07后，Bole-IGW出现因记录HTTP会话日志申请大量内存，导致连续重启
|
|||
|
|
|
|||
|
|
| ID | Creation Date | Assignee | Status |
|
|||
|
|
|----|----------------|----------|--------|
|
|||
|
|
| OMPUB-1054 | 2023-11-05T12:10:53.000+0800 | 杨威 | 已关闭 |
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Bole-IGW NPB03从2023.11.3 12点(UTC+3)左右开始持续重启，现象是watchdog timeout触发重启，原因是部分线程大量申请内存导致缺页，如果不开watchdog的话，则会在启动后迅速触发oom**liuyang** commented on *2023-11-05T12:11:43.168+0800*:
|
|||
|
|
|
|||
|
|
暂时定位到原因是流量中有异常的HTTP URL，触发会话插件占用内存异常，暂时将会话插件对于HTTP只记录host，具体是修改对应NPB中session_record.inf
|
|||
|
|
[HTTP]
|
|||
|
|
#FUNC_FLAG=ALL
|
|||
|
|
FUNC_FLAG=HTTP_HOST
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2023-11-06T14:24:02.557+0800*:
|
|||
|
|
|
|||
|
|
故障开始时间：UTC+3 2023-11-03 11:40
|
|||
|
|
|
|||
|
|
故障排查时段：UTC+3 2023-11-03 15:11-17:48
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
重启现象：运行一段时间后，单包处理延迟高，触发watchdog timeout
|
|||
|
|
|
|||
|
|
!anydesk00001.png|thumbnail!
|
|||
|
|
|
|||
|
|
重启现场：session_record调用sendlog发送日志，调用realloc时间长导致超时
|
|||
|
|
|
|||
|
|
!anydesk00000.png|thumbnail!
|
|||
|
|
|
|||
|
|
定位过程：
|
|||
|
|
* 关闭watchdog，前台gdb运行，观察到运行数分钟后，部分CPU核 sys调用上涨，同时内存快速增长，触发OOM
|
|||
|
|
|
|||
|
|
!anydesk00002.png|thumbnail!
|
|||
|
|
* perf top -C ’sys调用上涨的核‘，观察到内存分配相关函数调用高，对应的火焰图显示sys调用上涨原因为触发缺页中断
|
|||
|
|
|
|||
|
|
!anydesk00003.png|thumbnail!
|
|||
|
|
|
|||
|
|
!image-2023-11-06-14-14-36-831.png|width=468,height=336!
|
|||
|
|
* bt查看对应sys调用高的核，调用栈主要在HTTP日志处理
|
|||
|
|
|
|||
|
|
!image-2023-11-06-14-16-02-180.png|width=636,height=358!
|
|||
|
|
* 尝试关闭session_record中的HTTP数据处理入口，内存上涨的现象未复现
|
|||
|
|
* 尝试session_record仅处理HTTP URL相关数据，内存上涨现象复现
|
|||
|
|
* 继续尝试，session_record仅处理HTTP Host，内存上涨现象未复现，同时检查重启现象的范围，全网对应时段仅有Bol-IGW NPB03一台，暂时定位原因为流量中的异常HTTP URL导致session_record拼接日志内存使用异常，触发watchdog timeout
|
|||
|
|
|
|||
|
|
故障处理
|
|||
|
|
* UTC+3 2023-11-03 17:48暂时将Bole-IGW NPB 03中session_record处理HTTP的入口改为仅处理Host，待观察后续情况
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2023-11-06T14:32:12.243+0800*:
|
|||
|
|
|
|||
|
|
UTC+3 2023-11-03 21:02 Bole-IGW NPB03出现重启，重启现象依然为watchdog timeout，对应时段PSI监控CPU waiting较高
|
|||
|
|
|
|||
|
|
!image-2023-11-06-14-29-10-155.png|width=521,height=1098!
|
|||
|
|
|
|||
|
|
同时自17:48修改session_record参数后，内存使用未见异常，但是Application Drop计数明显上涨，推测原因仍然与异常流量有关
|
|||
|
|
|
|||
|
|
!image-2023-11-06-14-30-31-368.png|width=507,height=535!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2023-11-27T09:49:58.473+0800*:
|
|||
|
|
|
|||
|
|
11-24 *Bole-IGW* {*}NOB04{*}重现，UTC+3 14:29开启频繁重启，现象为重启后，触发watchdog timeout
|
|||
|
|
|
|||
|
|
执行11-03 NBP03相同的处理方式，恢复正常
|
|||
|
|
|
|||
|
|
!image-2023-11-27-09-47-55-825.png|width=802,height=435!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**liuyang** commented on *2024-08-31T18:34:04.849+0800*:
|
|||
|
|
|
|||
|
|
系统已升级至TSG24.02，关闭此bug。升级后系统再出现类似问题，重新创建bug
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Attachments
|
|||
|
|
|
|||
|
|
**46696/anydesk00000.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**46695/anydesk00001.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**46697/anydesk00002.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**46698/anydesk00003.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**46700/image-2023-11-06-14-14-36-831.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**46701/image-2023-11-06-14-16-02-180.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**46705/image-2023-11-06-14-29-10-155.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**46706/image-2023-11-06-14-30-31-368.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**47582/image-2023-11-27-09-47-55-825.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**46699/perf+(3).svg**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|