167 lines
3.3 KiB
Markdown
167 lines
3.3 KiB
Markdown
|
# 【E21现场】多站点频繁出现tsg_9140_service_sapp_running告警
|
|||
|
|
|
|||
|
|
| ID | Creation Date | Assignee | Status |
|
|||
|
|
|----|----------------|----------|--------|
|
|||
|
|
| OMPUB-858 | 2023-03-22T15:57:04.000+0800 | 刘学利 | 已关闭 |
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
查询时间为2023-03-07 – 2023-03-21；
|
|||
|
|
|
|||
|
|
查询过去两周的告警，tsg_9140_service_sapp_running告警共出现81次，集中在BOL-IGW（24次）和BOL-PE（31次）；
|
|||
|
|
|
|||
|
|
现挑选BOL-IGW和BOL-PE这两站点频繁告警的NPB：BOL-IGW-10.225.11.4（8次）和 BOL-PE-10.229.11.1（8次），导出其core文件、Alerts以及过去7天的Assets放在Core.zip压缩包内。
|
|||
|
|
|
|||
|
|
详情见附件。
|
|||
|
|
|
|||
|
|
**liuxueli** commented on *2023-03-22T20:10:58.241+0800*:
|
|||
|
|
|
|||
|
|
* [~chengsiyuan] [~liuju] 暂时关闭BOL-IGW-10.225.11.4和BOL-PE-10.229.11.1上的SIP插件，观察是否还存在重启的现象
|
|||
|
|
** 修改配置文件
|
|||
|
|
*** /opt/tsg/sapp/tsgconf/main.conf
|
|||
|
|
|
|||
|
|
*
|
|||
|
|
**
|
|||
|
|
***
|
|||
|
|
****
|
|||
|
|
{code:java}
|
|||
|
|
[SYSTEM]
|
|||
|
|
...
|
|||
|
|
IDENTIFY_PROTO_NAME="HTTP;SSL;DNS;FTP;BGP;MAIL;STREAMING_MEDIA;QUIC;SSH;Stratum;"
|
|||
|
|
{code}
|
|||
|
|
|
|||
|
|
|
|||
|
|
*** /opt/tsg/sapp/plug/conflist.inf
|
|||
|
|
****
|
|||
|
|
{code:java}
|
|||
|
|
[platform]
|
|||
|
|
...
|
|||
|
|
|
|||
|
|
[protocol]
|
|||
|
|
...
|
|||
|
|
#./plug/protocol/sip/sip.inf
|
|||
|
|
...
|
|||
|
|
|
|||
|
|
[business]
|
|||
|
|
...
|
|||
|
|
#./plug/business/fw_voip_plug/fw_voip_plug.inf
|
|||
|
|
...
|
|||
|
|
{code}
|
|||
|
|
|
|||
|
|
*
|
|||
|
|
**
|
|||
|
|
*** /opt/tsg/sapp/plug/business/tsg_conn_sketch/tsg_conn_sketch.inf
|
|||
|
|
|
|||
|
|
*
|
|||
|
|
**
|
|||
|
|
***
|
|||
|
|
****
|
|||
|
|
{code:java}
|
|||
|
|
#[SIP]
|
|||
|
|
#FUNC_FLAG=ALL
|
|||
|
|
#FUNC_NAME=tsg_record_sip_entry {code}
|
|||
|
|
|
|||
|
|
*
|
|||
|
|
** 修改配置文件后，重启sapp
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**chengsiyuan** commented on *2023-03-22T22:13:57.605+0800*:
|
|||
|
|
|
|||
|
|
2023-03-07 – 2023-03-21期间，出现告警的其他NPB core文件已导出，详情见附件core_other.zip
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**liuxueli** commented on *2023-03-23T11:13:48.765+0800*:
|
|||
|
|
|
|||
|
|
* 对重启的core现场进行分类：
|
|||
|
|
** SIP解析层出现47次：参见: TSG-14390
|
|||
|
|
** HTTP解析层出现12次: 参见: TSG-12926
|
|||
|
|
*** 福建环境在v22.11版本已修复
|
|||
|
|
** 怀疑内存被写越界造成的重启
|
|||
|
|
*** sapp出现5次重启: 参见：TSG-14396
|
|||
|
|
*** LRU淘汰的TCP链接，CLOSE状态回调出现2次，参见: TSG-14397
|
|||
|
|
*** [^E21.restart.txt]
|
|||
|
|
**** ^rapidjson: 3次^
|
|||
|
|
**** ^AppSketch：1次^
|
|||
|
|
**** ^无栈信息: 5次^
|
|||
|
|
**** ^FieldStat2: 1次^
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**chengsiyuan** commented on *2023-03-23T20:44:11.779+0800*:
|
|||
|
|
|
|||
|
|
已经按提供的操作，暂时关闭BOL-IGW-10.225.11.4和BOL-PE-10.229.11.1上的SIP插件；
|
|||
|
|
|
|||
|
|
更新时间：BOL-IGW-10.225.11.4（2023-03-22 15:52左右）和BOL-PE-10.229.11.1（2023-03-22 16:28左右）；
|
|||
|
|
|
|||
|
|
截止到目前为止，未出现重启现象
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**chengsiyuan** commented on *2023-03-24T15:36:02.160+0800*:
|
|||
|
|
|
|||
|
|
截止到目前（2023-03-24）为止，未出现重启现象
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**chengsiyuan** commented on *2023-03-27T15:21:00.059+0800*:
|
|||
|
|
|
|||
|
|
截止到目前（2023-03-27）为止
|
|||
|
|
|
|||
|
|
BOL-PE-10.229.11.1 未出现重启现象
|
|||
|
|
|
|||
|
|
BOL-IGW-10.225.11.4 于 2023-03-26 重启过一次，并产生了core文件，已导出对应的core文件，详情见附件core_10.225.11.4_Mar26
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**liuxueli** commented on *2023-03-27T16:41:15.279+0800*:
|
|||
|
|
|
|||
|
|
* 分析现场core文件，怀疑是内存被写越界。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2023-06-12T09:03:31.493+0800*:
|
|||
|
|
|
|||
|
|
升级后未复现，关闭issue，后续有类似问题，新开issue围绕当时版本讨论
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Attachments
|
|||
|
|
|
|||
|
|
**36650/core_10.225.11.4_Mar26**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36518/core_other.zip**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36496/Core.zip**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36537/E21.restart.txt**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|