45 lines
1.3 KiB
Markdown
45 lines
1.3 KiB
Markdown
# 【M22】Clickhouse数据节点出现High Disk IO > 90%告警
|
||
|
||
| ID | Creation Date | Assignee | Status |
|
||
|----|----------------|----------|--------|
|
||
| OMPUB-1335 | 2024-06-20T17:50:36.000+0800 | 王成成 | 已关闭 |
|
||
|
||
|
||
---
|
||
|
||
*具体现象：*
|
||
* clickhouse集群数据节点在6月19日9:50-11:00期间io使用率增高，达到100%。
|
||
* 对应时间段Merged rate和CPU Usage增高。
|
||
* 对应的时间段内出现CM-API Average Request Latency。**wangchengcheng** commented on *2024-07-01T18:49:50.604+0800*:
|
||
|
||
排查日志发现，在6月19日09:50-11:00时间段内多次触发clickhouse字段发现功能，导致clickhouse集群io使用率超过90%。
|
||
|
||
解决方案：将字段发现功能时间片分割粒度由86400s改成3600s。调整后，同样条件的查询磁盘io资源使用率降低至50%-60%左右，后续持续观察此问题。
|
||
|
||
|
||
|
||
---
|
||
|
||
|
||
|
||
# Attachments
|
||
|
||
Attachment: ck监控.png
|
||
|
||
|
||
|
||
|
||
|
||
Attachment: cn-sql样例.txt
|
||
|
||
[cn-sql样例.txt](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/59230/cn-sql样例.txt)
|
||
|
||
|
||
|
||
Attachment: qgw-slowQuery.log
|
||
|
||
[qgw-slowQuery.log](https://gfwleak.exec.li/admin/geedge-jira/raw/branch/master/attachment/59232/qgw-slowQuery.log)
|
||
|
||
|
||
|