169 lines
5.1 KiB
Markdown
169 lines
5.1 KiB
Markdown
|
# 【XJ-CUCC】192.227sapp频繁重启
|
|||
|
|
|
|||
|
|
| ID | Creation Date | Assignee | Status |
|
|||
|
|
|----|----------------|----------|--------|
|
|||
|
|
| OMPUB-888 | 2023-04-06T12:10:24.000+0800 | 刘学利 | 已关闭 |
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
h2. 192.227sapp服务频繁重启，经初步排查发现问题如下
|
|||
|
|
h3. 1.maat.conf配置文件:
|
|||
|
|
|
|||
|
|
配置文件路径：/home/mesasoft/sapp_run/tsgconf/maat.conf
|
|||
|
|
|
|||
|
|
问题：EFFECITIVE_RANGE_FILE配置项文件不存在
|
|||
|
|
|
|||
|
|
配置文件截图：
|
|||
|
|
|
|||
|
|
!image-20230406114930460.png|width=292,height=142!
|
|||
|
|
h3. 2.master日志报错：
|
|||
|
|
|
|||
|
|
问题：日志中报错的文件不存在
|
|||
|
|
|
|||
|
|
__tsglog_tsg_master.2023-04-06日志截图
|
|||
|
|
|
|||
|
|
!企业微信截图_16807528427825.png|width=493,height=55!
|
|||
|
|
h3. 3.app_sketch服务连接超时：
|
|||
|
|
|
|||
|
|
__tsglog_app_sketch_local_app_sketch_local.2023-04-06日志截图
|
|||
|
|
|
|||
|
|
!企业微信截图_16807530107311.png|width=438,height=142!
|
|||
|
|
|
|||
|
|
!企业微信截图_16807530545301.png|width=406,height=132!
|
|||
|
|
|
|||
|
|
{color:#333333}runtimelog.2023-04-06日志截图:{color}
|
|||
|
|
|
|||
|
|
{color:#333333}!企业微信截图_16807531161478.png|width=414,height=136!{color}
|
|||
|
|
|
|||
|
|
{color:#333333}!企业微信截图_16807531403302.png|width=416,height=133!{color}**yangwei** commented on *2023-04-10T09:31:24.364+0800*:
|
|||
|
|
|
|||
|
|
* 现象分析
|
|||
|
|
** 现象1和2中报错的文件不存在，原因是运营商省口的系统不是使用os安装，不影响正常运行
|
|||
|
|
** 现象3的日志反馈出两个问题
|
|||
|
|
*** 1、app sketch扫描tcp/udp首包负载耗时过长，截图显示单包（长度1000+字节）扫描耗时短时间内多次出现超过1秒的情况，部分耗时超过10秒
|
|||
|
|
*** 2、sip、tsg_master、ssl业务扫描，在报首包扫描耗时长的同一时段（日志截图中的10:38前后），也出现扫描耗时超过1秒的告警
|
|||
|
|
* 原因
|
|||
|
|
** 初步怀疑为app sketch中，配置tcp/udp首包负载特征导致单包扫描耗时长
|
|||
|
|
* 处理 [~jiayimeng]
|
|||
|
|
** 帮忙检查一下省口系统中，配置有tcp/udp首包负载特征（tcp.payload or udp.payload）的app有哪些？配置的都是一些什么负载特征，以及是否有必要保留？
|
|||
|
|
** 鉴于192.227重启较为频繁，推测触发扫描耗时长的流量在这个节点出现比较频繁，尝试在这个节点上对报超时（TIMEOUT）的服务端IP+端口进行捕包
|
|||
|
|
** 如果app特征在省口和IDC机房一致，则其他节点出现重启的原因可能与192.227相同，检查一下https://jira.geedge.net/browse/OMPUB-887和https://jira.geedge.net/browse/OMPUB-890出现重启时，对应的功能端日志，是否报与192.227类似的TIMEOUT告警
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2023-04-10T11:24:39.732+0800*:
|
|||
|
|
|
|||
|
|
省口系统中，配置有tcp/udp首包负载特征（tcp.payload or udp.payload）的自定义app有两个，钉钉和微信，配置的负载特征均不长，钉钉配置了0001000200076465，微信配置负载特征如下 [^weixin-signature.txt] ，钉钉与微信的负载经过测试，且为145个名单中的APP，需要保留。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2023-04-10T11:28:09.356+0800*:
|
|||
|
|
|
|||
|
|
省口通过maat_redis_tool拉取现网配置 APP_SIG_SESSION_ATTRIBUTE_STRING表中共配置了1667条负载特征，IDC通过maat_redis_tool拉取现网配置 APP_SIG_SESSION_ATTRIBUTE_STRING表中共配置了396条负载特征；除微信和钉钉外，其余负载特征应该都是app sketch db中的特征。
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2023-04-10T11:29:06.368+0800*:
|
|||
|
|
|
|||
|
|
IDC环境TSG中无自定义APP
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2023-04-10T12:09:31.598+0800*:
|
|||
|
|
|
|||
|
|
除227外其他（省口和IDC）重启的机器，__tsglog_app_sketch_local_app_sketch_local日志中无TIMEOUT报错
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**sunjiajia** commented on *2023-04-10T12:51:06.543+0800*:
|
|||
|
|
|
|||
|
|
省口04.05-04.06重启机器__tsglog_app_sketch_local_app_sketch_local日志文件不存在，查看了04.09重启机器_tsglog_app_sketch_local_app_sketch_local日志文件，以192.179为例，如下图所示：
|
|||
|
|
!image-2023-04-10-12-50-54-905.png!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**sunjiajia** commented on *2023-04-10T12:55:19.439+0800*:
|
|||
|
|
|
|||
|
|
IDC环境重启机器（以172.16.0.5为例）查看了runtimelog、__tsglog_app_proto_identify_app_proto_identify、__tsglog_maat_tsg_maat_log、 __tsglog_tsg_conn_sketch_tsg_conn_sketch_log日志情况；如下图所示：
|
|||
|
|
!image-2023-04-10-12-52-14-687.png|width=515,height=226!
|
|||
|
|
|
|||
|
|
!image-2023-04-10-12-54-13-840.png|width=608,height=202!
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**yangwei** commented on *2023-04-17T08:54:35.228+0800*:
|
|||
|
|
|
|||
|
|
4月10日下午检查192.227问题，发现操作卡顿（排除远程网络连接原因），尝试重启服务器后，失联，已联系联通集成进行处理，暂无处理完成的信息
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**jiayimeng** commented on *2023-04-25T15:50:39.261+0800*:
|
|||
|
|
|
|||
|
|
集成近日未在新疆，待五一后处理
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
## Attachments
|
|||
|
|
|
|||
|
|
**36835/image-20230406114930460.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**37011/image-2023-04-10-12-50-54-905.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**37012/image-2023-04-10-12-52-14-687.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**37013/image-2023-04-10-12-54-13-840.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**37010/weixin-signature.txt**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36834/企业微信截图_16807528427825.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36833/企业微信截图_16807530107311.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36832/企业微信截图_16807530545301.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36831/企业微信截图_16807531161478.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
**36830/企业微信截图_16807531403302.png**
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|