187 lines
4.0 KiB
Markdown
187 lines
4.0 KiB
Markdown
# 11月30日18:20左右，192.168.44.3虚拟机死机
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-1072 | 2023-12-04T11:19:56.000+0800 | 雷军 | 已解决 |
|
||
|
||
|
||
---
|
||
|
||
人为重启虚拟机后恢复
|
||
麻烦帮忙排查原因**leijun** commented on *2023-12-04T17:06:47.358+0800*:
|
||
|
||
11月30日18:20左右 192.168.44.3虚拟机Xshell无法连接，通过 Proxmox VE 对44.3虚拟机先进行 STOP，然后START启动
|
||
|
||
以下是16:00点前的内存观察
|
||
|
||
!image-2023-12-04-17-01-07-825.png|width=694,height=263!
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
---
|
||
|
||
**xubotao** commented on *2023-12-05T09:32:07.423+0800*:
|
||
|
||
经过排查当天的系统日志、安全日志、系统引导日志、核心日志及服务日志没有找到造成卡死的原因
|
||
|
||
|
||
|
||
---
|
||
|
||
**xubotao** commented on *2023-12-05T11:42:26.298+0800*:
|
||
|
||
经过以下几点排查发现可能引起44.3卡死或宕机的情况发生
|
||
|
||
1.141宿主机内存资源使用过高，减去buff/cache占用的资源，基本已经跑满，这台宿主机下运行的虚拟机所有分配的内存已经超过物理机最大内存，虚拟机测试状态下资源上下浮动，可能会导致虚拟机宕机现象
|
||
|
||
2.44.3虚拟机I/O利用率基本写满，导致磁盘I/O负载过高，可能会引起延迟或宕机
|
||
|
||
3.44.3虚拟机网络连接数增长，网络节点负载升高，可能会导致性能下降、延迟或者宕机
|
||
|
||
|
||
|
||
141宿主机内存资源及分配情况
|
||
|
||
!image-2023-12-05-11-39-11-323.png!
|
||
|
||
!image-2023-12-05-11-42-17-560.png!
|
||
|
||
44.3 I/O利用率
|
||
|
||
!image-2023-12-05-11-39-32-258.png!
|
||
|
||
|
||
|
||
44.3网络链接数
|
||
|
||
!image-2023-12-05-11-40-26-255.png!
|
||
|
||
|
||
|
||
44.3网络节点负载
|
||
|
||
!image-2023-12-05-11-40-44-908.png!
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
---
|
||
|
||
**niuxiang** commented on *2023-12-15T16:52:42.246+0800*:
|
||
|
||
2023年12月15日15：48出现宕机现象，如图。从PVE宿主机，查看整台设备内存使用率接近100%，通过进程使用的内存排序情况如图。主要占用内存的虚拟机是44.3和44.17，各占了25%。 !image-2023-12-15-16-51-09-078.png!
|
||
|
||
!image-2023-12-15-16-51-22-664.png!
|
||
|VID|VNAME|在PVE宿主机中查看虚拟机进程内存占用率|
|
||
|102|zhangwei-192.168.44.3-long|25.5|
|
||
|118|dongxiaoyan-192.168.44.17-long|25.4|
|
||
|104|zhangwei-192.168.44.5-long|12.7|
|
||
|261|doufenghu-192.168.44.14-long|9.5|
|
||
|134|duandongmei-192.168.44.136|6.3|
|
||
|133|yangwei-dpi|5.6|
|
||
|106|dongxiaoyan-192.168.40.6-win-long|3.2|
|
||
|103|zhangwei-192.168.44.4-long|3.1|
|
||
|109|dongxiaoyan-192.168.44.9-long|2.8|
|
||
|149|luwenpeng-192.168.44.128-long|2.6|
|
||
|153|duandongmei-192.168.44.137|2.4|
|
||
|
||
!image-2023-12-15-16-52-40-908.png!
|
||
|
||
|
||
|
||
---
|
||
|
||
**liuyang** commented on *2023-12-18T09:44:07.168+0800*:
|
||
|
||
* 宿主机内存512G，查看44.3内存使用了25%，即128G
|
||
* 192.168.44.3虚拟机申请内存128G，通过NZ监控查看死机前44.3内存占用40%左右，即不到60G
|
||
!image-2023-12-18-09-46-58-427.png|thumbnail!
|
||
[~niuxiang]上述两处查看44.3使用内存不一致，是否可以通过系统日志或者其他信息，帮忙再次确认44.3内存使用
|
||
|
||
|
||
|
||
---
|
||
|
||
**leijun** commented on *2024-01-03T19:10:52.548+0800*:
|
||
|
||
2024/01/03 18:50左右，192.168.44.3虚拟机死机
|
||
* 通过NZ监控查看死机前44.3内存占用了36%，CPU使用了51%左右
|
||
|
||
!image-2024-01-03-18-55-19-559.png|width=625,height=321!
|
||
|
||
44.3虚拟机控制台信息
|
||
|
||
!image-2024-01-03-18-58-42-358.png|width=608,height=456!
|
||
|
||
|
||
|
||
---
|
||
|
||
**leijun** commented on *2024-07-08T09:36:55.104+0800*:
|
||
|
||
44.3虚拟机迁移新物理机后，没有再出现此现象
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
## Attachments
|
||
|
||
**47616/image-2023-12-04-17-01-07-825.png**
|
||
|
||
---
|
||
|
||
**47626/image-2023-12-05-11-39-11-323.png**
|
||
|
||
---
|
||
|
||
**47627/image-2023-12-05-11-39-32-258.png**
|
||
|
||
---
|
||
|
||
**47628/image-2023-12-05-11-40-26-255.png**
|
||
|
||
---
|
||
|
||
**47629/image-2023-12-05-11-40-44-908.png**
|
||
|
||
---
|
||
|
||
**47630/image-2023-12-05-11-42-17-560.png**
|
||
|
||
---
|
||
|
||
**47704/image-2023-12-15-16-51-09-078.png**
|
||
|
||
---
|
||
|
||
**47705/image-2023-12-15-16-51-22-664.png**
|
||
|
||
---
|
||
|
||
**47706/image-2023-12-15-16-52-40-908.png**
|
||
|
||
---
|
||
|
||
**47711/image-2023-12-18-09-46-58-427.png**
|
||
|
||
---
|
||
|
||
**49380/image-2024-01-03-18-55-19-559.png**
|
||
|
||
---
|
||
|
||
**49382/image-2024-01-03-18-58-42-358.png**
|
||
|
||
---
|
||
|