199 lines
5.3 KiB
Markdown
199 lines
5.3 KiB
Markdown
|
# 【WMS-UTR】MSH01和PCAP06存在周期性内存占用上涨导致重启
|
|||
|
|
|
|||
|
|
| ID | Creation Date | Assignee | Status |
|
|||
|
|
|----|----------------|----------|--------|
|
|||
|
|
| OMPUB-1225 | 2024-04-11T23:53:16.000+0800 | 段冬梅 | 已解决 |
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
开始时段：当地时间4月10日03:25前后
|
|||
|
|
|
|||
|
|
MSH01周期性内存占用达到80%后，大概率重启（marsio+Firewall）
|
|||
|
|
|
|||
|
|
!image-2024-04-11-23-49-06-869.png|width=659,height=345!
|
|||
|
|
|
|||
|
|
PCAP06同样出现周期性内存增长
|
|||
|
|
|
|||
|
|
!image-2024-04-11-23-50-22-777.png|width=585,height=224!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
近期hotfix操作如下：
|
|||
|
|
|
|||
|
|
1、firewall更新，增加异步日志缓存队列大小
|
|||
|
|
|
|||
|
|
2、jemalloc参数更新，降低内存脏页回收频率，避免出现madvise竞争
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
需要回溯hotfix之前的内存占用情况，请[~caoshanfeng] 帮忙导出过去1周两台设备的资产监控方便对比和排查。
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-04-12T21:18:04.925+0800*:
|
|||
|
|
|
|||
|
|
4.12观察功能端运行状态，在未进行更新的前提下，内存周期性上涨的现象未复现。
|
|||
|
|
|
|||
|
|
同时进一步观察msh01的历史监控，4.9凌晨1:00开始出现内存持续上涨，持续到凌晨5:00，此时流量回落至20Gbps左右，内存上涨趋势与流量规模无相关性，推测与https://jira.geedge.net/browse/OMPUB-1221相关，待进一步观察。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-04-15T09:15:10.561+0800*:
|
|||
|
|
|
|||
|
|
* 持续观察流量较大的MSH01，PCAP06h和TWA01三台设备，自当地时间4月12日8:00之后，未再出现内存持续上涨的现象。
|
|||
|
|
|
|||
|
|
!image-2024-04-15-09-09-45-903.png|width=421,height=250!!image-2024-04-15-09-10-04-061.png|width=427,height=260!!image-2024-04-15-09-11-51-188.png|width=490,height=265!
|
|||
|
|
* 4月13日至4月14日两天，全网未再出现功能端重启或者丢包告警
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-04-24T16:02:00.049+0800*:
|
|||
|
|
|
|||
|
|
4月22日-4月23日复现，持续时间为4.22 21:50，至4.23 11:21，正在与SD排查是否为对应时段多次推送全量相关
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-04-24T16:29:08.895+0800*:
|
|||
|
|
|
|||
|
|
切换pcap03至SD的测试Redis，SD 4次重启同步程序（发起4次全量同步），该设备内存都出现持续上涨的趋势，同步程序停止后，内存回落，现象与前段时间内存持续上涨一直，待进一步排查触发的原因。
|
|||
|
|
|
|||
|
|
!image-2024-04-24-16-27-40-357.png|width=821,height=453!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-04-24T19:40:23.542+0800*:
|
|||
|
|
|
|||
|
|
分析现场的日志，和观察内存使用，发现监控中内存持续上涨分成两个面板，一个是整机内存，一个是vsys1的内存，实际top观察，sapp的内存占用量持续稳定，但是redis-server的内存持续上涨，最终可能造成oom
|
|||
|
|
|
|||
|
|
!image-2024-04-24-19-39-02-962.png|width=1213,height=393!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhengchao** commented on *2024-04-29T16:31:19.987+0800*:
|
|||
|
|
|
|||
|
|
Redis内存增长的原因是MAAT_UPDATE_STATUS未清理？[~duandongmei]
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**duandongmei** commented on *2024-04-29T17:54:42.072+0800*:
|
|||
|
|
|
|||
|
|
Redis内存增长的问题是SD MAAT_UPDATE_STATUS清理功能存在BUG导致，26号已解决此问题，SD Redis内存已下降。
|
|||
|
|
|
|||
|
|
!image-2024-04-29-17-54-08-948.png|width=1556,height=686!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhengchao** commented on *2024-04-30T12:24:23.728+0800*:
|
|||
|
|
|
|||
|
|
1. 这张图看不到26日Redis内存下降；2. OS的Redis内存下降了吗?
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**duandongmei** commented on *2024-04-30T13:04:08.631+0800*:
|
|||
|
|
|
|||
|
|
嗯，我把Redis内存指标放上；TSG-OS的Redis内存在28号进行hotfix之后下降了，具体的hotfix内存我不太了解。
|
|||
|
|
|
|||
|
|
!image-2024-04-30-13-03-10-717.png!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2024-04-30T16:30:29.350+0800*:
|
|||
|
|
|
|||
|
|
附图是用于测试的tsgx-pcap03设备的监控信息，蓝色为OS整体的内存占用情况（包含OS上的Redis），绿色为sapp进程的内存占用。
|
|||
|
|
|
|||
|
|
从绿色监控线条可知自4月23日14:00之后，sapp的内存使用趋于稳定，占用整体小于60GB.
|
|||
|
|
|
|||
|
|
但是蓝色的整机占用显示，{*}在4月23日14:00之后，OS的内存占用仍呈现较大的波动，直到4月28日8:00之后趋于稳定，与28日在os上进行hotfix的时段相符。{*}
|
|||
|
|
|
|||
|
|
由于未对Redis的内存做单独的监控，通过蓝色和绿色的监控值之间的差别，可以推算中间内存的波动应该是由于Redis同步造成。
|
|||
|
|
|
|||
|
|
!image-2024-04-30-16-20-52-287.png|width=421,height=308!!image-2024-04-30-16-21-06-766.png|width=414,height=296!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**duandongmei** commented on *2024-05-07T10:17:33.908+0800*:
|
|||
|
|
|
|||
|
|
SD Redis内存：26号之后SD Redis内存占用保持在3.5G。
|
|||
|
|
|
|||
|
|
TSG-OS Redis内存：28号hotfix后，未出现内存周期性上涨问题。
|
|||
|
|
|
|||
|
|
28号后，TSG-OS Redis与SD Redis内存基本保持在3.5G左右，关闭该问题。
|
|||
|
|
|
|||
|
|
!image-2024-05-07-10-15-23-958.png|width=636,height=323!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Attachments
|
|||
|
|
|
|||
|
|
**54969/image-2024-04-11-23-49-06-869.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**54968/image-2024-04-11-23-50-22-777.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**55177/image-2024-04-15-09-09-45-903.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**55178/image-2024-04-15-09-10-04-061.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**55179/image-2024-04-15-09-11-51-188.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**56164/image-2024-04-24-16-27-40-357.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**56218/image-2024-04-24-19-39-02-962.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**56475/image-2024-04-29-17-54-08-948.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**56553/image-2024-04-30-13-03-10-717.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**56574/image-2024-04-30-16-20-52-287.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**56575/image-2024-04-30-16-21-06-766.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**56769/image-2024-05-07-10-15-23-958.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|