183 lines
4.6 KiB
Markdown
183 lines
4.6 KiB
Markdown
|
# 信息港生产环境OLAP磁盘被写满
|
|||
|
|
|
|||
|
|
| ID | Creation Date | Assignee | Status |
|
|||
|
|
|----|----------------|----------|--------|
|
|||
|
|
| OMPUB-730 | 2022-12-06T16:13:51.000+0800 | 张洪庆 | 已解决 |
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
对应OLAP IP地址：192.168.44.30
|
|||
|
|
密码 PUB
|
|||
|
|
经排查发现/home/docker/overlay2目录下写满文件导致
|
|||
|
|
!screenshot-1.png|thumbnail! **zhanghongqing** commented on *2022-12-08T11:47:47.687+0800*:
|
|||
|
|
|
|||
|
|
使用命令查看arangodb相关docker目录
|
|||
|
|
1. 查询命令
|
|||
|
|
{code:java}
|
|||
|
|
docker ps -q | xargs docker inspect --format '{{.State.Pid}}, {{.Id}}, {{.Name}}, {{.GraphDriver.Data.WorkDir}}' {code}
|
|||
|
|
|
|||
|
|
|
|||
|
|
!image-2022-12-08-11-42-17-892.png|width=924,height=272!
|
|||
|
|
|
|||
|
|
2. 进入到/home/docker/overlay2/*/merged,可以看到有很多core.文件。core文件其实就是内存的映像,当程序崩溃时,存储内存的相应信息,主用用于对程序进行调试。
|
|||
|
|
|
|||
|
|
3. 在docker-compose.yml文件中禁用core 。
|
|||
|
|
|
|||
|
|
!image-2022-12-08-11-44-43-041.png!
|
|||
|
|
|
|||
|
|
4. 重新创建容器并进入容器查看，core已经被禁用
|
|||
|
|
|
|||
|
|
!image-2022-12-08-11-45-20-708.png|width=393,height=242!
|
|||
|
|
|
|||
|
|
5. 将持续观察一段时间
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**doufenghu** commented on *2022-12-08T12:40:13.246+0800*:
|
|||
|
|
|
|||
|
|
* 分析下Arango DB 具体的异常信息
|
|||
|
|
* Docker 下生成core文件，其它组件或服务是否也存在该现象？处置方案是限制core文件大小，还是禁用？
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhanghongqing** commented on *2022-12-20T16:23:12.272+0800*:
|
|||
|
|
|
|||
|
|
arangodb 报错跟踪记录：
|
|||
|
|
|
|||
|
|
!image-2022-12-20-15-14-29-290.png!!image-2022-12-20-15-22-22-899.png!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhanghongqing** commented on *2023-01-11T16:08:06.013+0800*:
|
|||
|
|
|
|||
|
|
arango问题跟踪记录：
|
|||
|
|
# 历史日志中发现 vm.max_map_count 进程虚拟内存映射过小提示，Linux默认65530 。
|
|||
|
|
|
|||
|
|
!image-2023-01-11-15-22-01-898.png|width=908,height=257!
|
|||
|
|
|
|||
|
|
2. 打开之前对core日志的禁用，修改/etc/sysctl.conf 中 vm.max_map_count = 2048000，sysctl -p 生效，arangodb 重启docker restart arangodb
|
|||
|
|
|
|||
|
|
相关issue： [https://github.com/arangodb/arangodb/issues/5641]
|
|||
|
|
|
|||
|
|
!image-2023-01-11-15-57-54-134.png!
|
|||
|
|
|
|||
|
|
3. {-}观察一段时间发现 arangodb docker目录下未出现core.pid日志，未发生磁盘满载问题{-}。
|
|||
|
|
|
|||
|
|
{*}总结{*}：{-}原因可能由arango 服务器上 vm.max_map_count配置过小，此服务器性能不足，导致arangodb程序崩溃，并产生大量core日志导致磁盘满载。增大服务器 vm.max_map_count可以解决。{-}
|
|||
|
|
|
|||
|
|
其它：继续跟踪观察调小arango单机配置如：缓存，线程数是否也能解决程序崩溃问题。
|
|||
|
|
|
|||
|
|
arango日志中关于transparent_hugepage 透明巨页的推荐提示将后续跟踪测试。
|
|||
|
|
|
|||
|
|
*2023.02.20*
|
|||
|
|
|
|||
|
|
现象：vm.max_map_count配置过了一段时间后又产生core.pid日志，但未影响arango正常使用。目前可使用禁用core日志的方式解决。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhanghongqing** commented on *2023-03-06T11:42:45.509+0800*:
|
|||
|
|
|
|||
|
|
*2023.03.06*
|
|||
|
|
|
|||
|
|
** 现象：修改服务器参数“透明大页面”，“内核参数”，运行一段时间发现仍然生成新的core.pid日志
|
|||
|
|
|
|||
|
|
!image-2023-03-06-11-37-49-222.png|width=614,height=141!
|
|||
|
|
|
|||
|
|
修改命令
|
|||
|
|
Page Sizes :
|
|||
|
|
|
|||
|
|
sudo bash -c "echo madvise >/sys/kernel/mm/transparent_hugepage/enabled"
|
|||
|
|
sudo bash -c "echo madvise >/sys/kernel/mm/transparent_hugepage/defrag"
|
|||
|
|
|
|||
|
|
Overcommit MemoryPermalink :
|
|||
|
|
|
|||
|
|
sudo bash -c "echo 0 >/proc/sys/vm/overcommit_memory"
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhanghongqing** commented on *2023-03-20T11:32:41.009+0800*:
|
|||
|
|
|
|||
|
|
*2023.03.20*
|
|||
|
|
|
|||
|
|
{*}现象：{*}在arnago的docker-compose.yml文件中设置core大小为10485760（10M）运行一段时间后，docker目录下产生多个10M大小core.pid 文件
|
|||
|
|
|
|||
|
|
!image-2023-03-20-11-31-14-302.png!!image-2023-03-20-11-30-35-563.png!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**zhanghongqing** commented on *2023-07-10T09:46:08.497+0800*:
|
|||
|
|
|
|||
|
|
关闭arangodb core文件产生后，观察一段时间后未发生磁盘满的问题。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Attachments
|
|||
|
|
|
|||
|
|
**33528/image-2022-12-08-11-42-17-892.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33529/image-2022-12-08-11-44-43-041.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33530/image-2022-12-08-11-45-20-708.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33824/image-2022-12-20-15-14-29-290.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33825/image-2022-12-20-15-22-22-899.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**34348/image-2023-01-11-15-22-01-898.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**34349/image-2023-01-11-15-28-07-711.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**34352/image-2023-01-11-15-57-54-134.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**35672/image-2023-03-06-11-37-49-222.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36380/image-2023-03-20-11-30-35-563.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36381/image-2023-03-20-11-31-14-302.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**33409/screenshot-1.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|